GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

Post-attention residual state

×

tensorx · 127

The residual features after the attention update, feeding the MLP branch and its bypass.

shape
xn′x'_{n} [B batch, T positions, 768 d]
Sources

Contextual evidence from Attention residual merge

First branch of the original GPT-2 block.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a

Contextual evidence from Attention residual merge

The PyTorch block saves residual before normalization and adds it after attention.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–282

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(

… lines 275–281 omitted …

        hidden_states = attn_output + residual

Contextual evidence from Pre-MLP LayerNorm

The second normalization appears directly inside the MLP call.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–128

Read code excerpt
def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x

… lines 39–127 omitted …

        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)

Contextual evidence from Pre-MLP LayerNorm

A second nn.LayerNorm instance precedes the MLP.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 254–305

Read code excerpt
        self.ln_2 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)

… lines 255–302 omitted …

        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)

Contextual evidence from MLP residual merge

Second branch and block return in the reference graph.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Contextual evidence from MLP residual merge

The same residual contract in the maintained PyTorch implementation.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 303–309

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)

… lines 306–306 omitted …

        hidden_states = residual + feed_forward_hidden_states