GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

Position-wise feed-forward network

Open inside →×

learnedmlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)

Transforms each position’s features independently to produce an update for the residual stream.

Δmlp=MLP⁡(Z2){\htmlData{light=mlp}{\Delta_{\mathrm{mlp}}}}=\operatorname{MLP}({\htmlData{light=ln2}{Z_2}})
in
Z2Z_2 [B batch, T positions, 768 d]
out
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
parameters
4,722,432 · × 12 blocks = 56,669,184
Parameters

4,722,432 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Feature expansion768×3,072+3,072768 \times 3{,}072 + 3{,}0722,362,368
Feature projection3,072×768+7683{,}072 \times 768 + 7682,360,064
Per-instance total4,722,432

Across 12 independent blocks: 4,722,432×12=56,669,1844{,}722{,}432 \times 12 = 56{,}669{,}184 parameters.

Codemodel.py 115–120, 128 · modeling_gpt2.py 232–260
model.py 115–120
def mlp(x, scope, n_state, *, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        h = gelu(conv1d(x, 'c_fc', n_state))
        h2 = conv1d(h, 'c_proj', nx)
        return h2
model.py 128
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
modeling_gpt2.py 232–243
        embed_dim = config.hidden_size
        self.c_fc = Conv1D(intermediate_size, embed_dim)
        self.c_proj = Conv1D(embed_dim, intermediate_size)
        self.act = ACT2FN[config.activation_function]
        self.dropout = nn.Dropout(config.resid_pdrop)

    def forward(self, hidden_states: tuple[torch.FloatTensor] | None) -> torch.FloatTensor:
        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states
modeling_gpt2.py 249–250
        hidden_size = config.hidden_size
        inner_dim = config.n_inner if config.n_inner is not None else 4 * hidden_size
modeling_gpt2.py 260
        self.mlp = GPT2MLP(inner_dim, config)
Sources2

Direct evidence

Original two projections with a four-times expansion.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 115–128

Read code excerpt
def mlp(x, scope, n_state, *, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        h = gelu(conv1d(x, 'c_fc', n_state))
        h2 = conv1d(h, 'c_proj', nx)
        return h2

… lines 121–127 omitted …

        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)

Direct evidence

Transformers resolves the activation from the checkpoint configuration.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 232–260

Read code excerpt
        embed_dim = config.hidden_size
        self.c_fc = Conv1D(intermediate_size, embed_dim)
        self.c_proj = Conv1D(embed_dim, intermediate_size)
        self.act = ACT2FN[config.activation_function]
        self.dropout = nn.Dropout(config.resid_pdrop)

    def forward(self, hidden_states: tuple[torch.FloatTensor] | None) -> torch.FloatTensor:
        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

… lines 244–248 omitted …

        hidden_size = config.hidden_size
        inner_dim = config.n_inner if config.n_inner is not None else 4 * hidden_size

… lines 251–259 omitted …

        self.mlp = GPT2MLP(inner_dim, config)