GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Transformer block

Open inside →×

learnedblock(h, 'h%d' % layer, past=past, hparams=hparams)

A pre-normalized residual update: read the stream, compute attention, add; read again, compute the feed-forward, add.

xℓ+1=Block⁡ℓ(xℓ)x_{\ell+1}=\operatorname{Block}_{\ell}(x_{\ell})
in
xnx_{n} [B batch, T positions, 768 d]
out
xn+1x_{n+1} [B batch, T positions, 768 d]
parameters
7,087,872 · × 12 blocks = 85,054,464
Parameters

7,087,872 parameters per block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Pre-attention LayerNorm768+768768 + 7681,536
Masked multi-head self-attention2,362,368 across its components2,362,368
Pre-MLP LayerNorm768+768768 + 7681,536
Position-wise feed-forward network4,722,432 across its components4,722,432
Per-instance total7,087,872

Across 12 independent blocks: 7,087,872×12=85,054,4647{,}087{,}872 \times 12 = 85{,}054{,}464 parameters.

Each of the 12 block addresses owns independent weights. The implementation names the first and last addresses h.0 and h.11.

Codemodel.py 126–130 · modeling_gpt2.py 272–307
model.py 126–130
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present
modeling_gpt2.py 272–280
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(
            hidden_states,
            past_key_values=past_key_values,
            attention_mask=attention_mask,
            use_cache=use_cache,
            **kwargs,
        )
modeling_gpt2.py 282
        hidden_states = attn_output + residual
modeling_gpt2.py 303–307
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)
        # residual connection
        hidden_states = residual + feed_forward_hidden_states
Sources2

Direct evidence

The released block performs both pre-normalized residual writes and returns the updated stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Direct evidence

The maintained implementation preserves the same two residual branches.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–307

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(
            hidden_states,
            past_key_values=past_key_values,
            attention_mask=attention_mask,
            use_cache=use_cache,
            **kwargs,
        )

… lines 281–281 omitted …

        hidden_states = attn_output + residual

… lines 283–302 omitted …

        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)
        # residual connection
        hidden_states = residual + feed_forward_hidden_states