GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Token embedding

×

learnedtf.gather(wte, X)

Look up a learned feature vector for each token ID.

Et=Wtoken[input_ids]{\htmlData{light=token-embedding wte}{E_t}}=W_{\mathrm{token}}[\mathrm{input\_ids}]
in
[B batch, T positions]
out
EtE_t [B batch, T positions, 768 d]
parameters
38,597,376
Parameters

38,597,376 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Token embedding50,257×76850{,}257 \times 76838,597,376
Per-instance total38,597,376

The vocabulary projection reuses this table’s transpose; its parameters are counted here.

Derived from the configured table dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 152–157 · modeling_gpt2.py 492–577
model.py 152
        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],
model.py 154
        wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd],
model.py 157
        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))
modeling_gpt2.py 492–493
        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)
modeling_gpt2.py 569
            inputs_embeds = self.wte(input_ids)
modeling_gpt2.py 576–577
        position_embeds = self.wpe(position_ids)
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)
Sources2

Direct evidence

The original TensorFlow graph creates both tables and adds their gathered rows.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt
        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],

… lines 153–153 omitted …

        wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd],

… lines 155–156 omitted …

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Direct evidence

Transformers names the tables wte and wpe, matching the released checkpoint keys.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 492–577

Read code excerpt
        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)

… lines 494–568 omitted …

            inputs_embeds = self.wte(input_ids)

… lines 570–575 omitted …

        position_embeds = self.wpe(position_ids)
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)