GPT-2

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw text TokenizerByte-level BPE vocab · mergesfixed assets Token IDs[B, T]Token embedding

\mathrm{WTE}

50,257 × 768 E_t[B, T, 768]Position IDs

0,\ldots,T-1

Position embedding

\mathrm{WPE}

1,024 × 768 +Dropout

p_{\mathrm{embd}}

x₀[B, T, 768]

12 independent
blocks Open →

x₁₂[B, T, 768]Final LayerNorm

\mu,\;\sigma^2

gain · bias Tied projection

h_{\mathrm{final}}WTE^{\mathsf T}

0 new parameters Logits[B, T, 50257]

Position embedding table

learnedtf.gather(wpe, positions_for(X, past_length))

One learned row per absolute position, 0 through 1,023.

{\htmlData{light=wpe}{E_{\mathrm{pos}}}}=W_{\mathrm{position}}[\mathrm{position\_ids}]

in: [B batch, T positions]
out: $E_{\mathrm{pos}}$ [B batch, T positions, 768 d]
parameters: 786,432

Parameters

786,432 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
table	$1{,}024 \times 768$	786,432
Per-instance total		786,432

Explanation

The table has one row per position, so a sequence, cached positions included, holds at most 1,024 tokens (model.py 152).

Codemodel.py 152, 157 · modeling_gpt2.py 493, 576

model.py 152

        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],

model.py 157

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

modeling_gpt2.py 493

        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)

modeling_gpt2.py 576

        position_embeds = self.wpe(position_ids)

Sources2

Direct evidence

wpe is created alongside wte and gathered by position index.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt

        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],

… lines 153–156 omitted …

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Direct evidence

The constructor sets the position-table row count from max_position_embeddings; the lookup selects rows by position ID. The model configuration supplies the numeric count.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 492–577

Read code excerpt

        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)

… lines 494–575 omitted …

        position_embeds = self.wpe(position_ids)

GPT-2

GPT-2 model overview