GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

Attention residual merge

×

no parametersx + a

Add the mechanism’s update to the residual stream element by element.

xn′=xn+Dresid(Δattn){\htmlData{light=x-attn attn-add}{x'_{n}}}={\htmlData{light=x-in}{x_{n}}}+D_{\mathrm{resid}}({\htmlData{light=attention}{\Delta_{\mathrm{attn}}}})
in
xnx_{n} [B batch, T positions, 768 d]Dresid(Δattn)D_{\mathrm{resid}}(\Delta_{\mathrm{attn}}) [B batch, T positions, 768 d]
out
xn′x'_{n} [B batch, T positions, 768 d]

DresidD_{\mathrm{resid}} is dropout during training and the identity at evaluation. The maintained code applies it inside the attention module (modeling_gpt2.py 224); the released code adds aa as it is.

Codemodel.py 126–127 · modeling_gpt2.py 272–274, 282
model.py 126–127
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
modeling_gpt2.py 272–274
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(
modeling_gpt2.py 282
        hidden_states = attn_output + residual
Sources2

Direct evidence

First branch of the original GPT-2 block.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a

Direct evidence

The PyTorch block saves residual before normalization and adds it after attention.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–282

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(

… lines 275–281 omitted …

        hidden_states = attn_output + residual

Incoming transform evidence

The maintained training path applies residual dropout after c_proj and before the residual write; evaluation makes dropout the identity.

Attention residual dropout · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 222–226

Read code excerpt
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights