GPT-2

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw text TokenizerByte-level BPE vocab · mergesfixed assets Token IDs[B, T]Token embedding

\mathrm{WTE}

50,257 × 768 E_t[B, T, 768]Position IDs

0,\ldots,T-1

Position embedding

\mathrm{WPE}

1,024 × 768 +Dropout

p_{\mathrm{embd}}

x₀[B, T, 768]

12 independent
blocks Open →

x₁₂[B, T, 768]Final LayerNorm

\mu,\;\sigma^2

gain · bias Tied projection

h_{\mathrm{final}}WTE^{\mathsf T}

0 new parameters Logits[B, T, 50257]

Embedding addition

no parameterstf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Add token and position vectors element-wise before training-only embedding dropout initializes the residual stream.

{\htmlData{light=embed-add}{E}}=W_{\mathrm{token}}[\mathrm{input\_ids}]+W_{\mathrm{position}}[\mathrm{position\_ids}]

in: $E_{t}$ [B batch, T positions, 768 d] $E_{\mathrm{pos}}$ [B batch, T positions, 768 d]
out: $E$ [B batch, T positions, 768 d]

During training the maintained code applies dropout to $E$ before the first block (modeling_gpt2.py 604): $x_0=\operatorname{Dropout}(E;p_{\mathrm{embd}})$ . At evaluation, and in the released code, $x_0=E$ .

Codemodeling_gpt2.py 577–608 · model.py 157

modeling_gpt2.py 577

        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

modeling_gpt2.py 604

        hidden_states = self.drop(hidden_states)

modeling_gpt2.py 608

        for i, block in enumerate(self.h):

model.py 157

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Sources2

Direct evidence

For the base GPT-2 text path, embedding dropout follows token-plus-position addition and precedes the block loop; optional token-type embeddings are outside the base text path.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 576–608

Read code excerpt

        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

… lines 578–603 omitted …

        hidden_states = self.drop(hidden_states)

… lines 605–607 omitted …

        for i, block in enumerate(self.h):

Direct evidence

The original graph gathers the two tables and adds them into the initial stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

GPT-2

GPT-2 model overview