GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Position-wise feed-forward network

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Transformer block › MLP

Feature expansion

×

learnedconv1d(x, 'c_fc', n_state)

Expands the feature axis from 768 to 3,072 at each sequence position.

h=xW1+b1{\htmlData{light=pre-activation c_fc}{h}}={\htmlData{light=mlp-input}{x}}W_1+b_1
in
xx [B batch, T positions, 768 d]
out
hh [B batch, T positions, 3072 d_ff]
parameters
2,362,368 · × 12 blocks = 28,348,416
Parameters

2,362,368 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×3,072768 \times 3{,}0722,359,296
Output bias3,0723{,}0723,072
Per-instance total2,362,368

The reference module holds the weight input by output, [in, out].

Across 12 independent blocks: 2,362,368×12=28,348,4162{,}362{,}368 \times 12 = 28{,}348{,}416 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 50–128 · modeling_gpt2.py 232–260
model.py 50–56
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c
model.py 115–118
def mlp(x, scope, n_state, *, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        h = gelu(conv1d(x, 'c_fc', n_state))
model.py 128
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
modeling_gpt2.py 232–243
        embed_dim = config.hidden_size
        self.c_fc = Conv1D(intermediate_size, embed_dim)
        self.c_proj = Conv1D(embed_dim, intermediate_size)
        self.act = ACT2FN[config.activation_function]
        self.dropout = nn.Dropout(config.resid_pdrop)

    def forward(self, hidden_states: tuple[torch.FloatTensor] | None) -> torch.FloatTensor:
        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states
modeling_gpt2.py 249–250
        hidden_size = config.hidden_size
        inner_dim = config.n_inner if config.n_inner is not None else 4 * hidden_size
modeling_gpt2.py 260
        self.mlp = GPT2MLP(inner_dim, config)
Sources3

Direct evidence

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Direct evidence

The MLP's c_fc projection opens the four-times-wider feature workspace before GELU.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 115–128

Read code excerpt
def mlp(x, scope, n_state, *, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        h = gelu(conv1d(x, 'c_fc', n_state))

… lines 119–127 omitted …

        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)

Direct evidence

The maintained MLP defines c_fc at the configured intermediate size and applies it before activation.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 232–260

Read code excerpt
        embed_dim = config.hidden_size
        self.c_fc = Conv1D(intermediate_size, embed_dim)
        self.c_proj = Conv1D(embed_dim, intermediate_size)
        self.act = ACT2FN[config.activation_function]
        self.dropout = nn.Dropout(config.resid_pdrop)

    def forward(self, hidden_states: tuple[torch.FloatTensor] | None) -> torch.FloatTensor:
        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

… lines 244–248 omitted …

        hidden_size = config.hidden_size
        inner_dim = config.n_inner if config.n_inner is not None else 4 * hidden_size

… lines 251–259 omitted …

        self.mlp = GPT2MLP(inner_dim, config)