GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Merge and project

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Full sequenceTrainingPrefillDecode

Transformer block › Attention

Attention output projection

×

learnedconv1d(a, 'c_proj', n_state)

Applies a learned projection to the merged 768-feature field, producing the attention update.

Δattn=HWO+bO{\htmlData{light=attn-update attn-proj}{\Delta_{\mathrm{attn}}}}={\htmlData{light=h merge-heads}{H}}W_O+b_O
in
HH [B batch, T positions, 768 d]
out
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
parameters
590,592 · × 12 blocks = 7,087,104
Parameters

590,592 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×768768 \times 768589,824
Output bias768768768
Per-instance total590,592

The reference module holds the weight input by output, [in, out].

Across 12 independent blocks: 590,592×12=7,087,104590{,}592 \times 12 = 7{,}087{,}104 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 50–56, 109–112 · modeling_gpt2.py 222–226
model.py 50–56
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c
model.py 109–112
        a = multihead_attn(q, k, v)
        a = merge_heads(a)
        a = conv1d(a, 'c_proj', n_state)
        return a, present
modeling_gpt2.py 222–226
        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights
Sources3

Direct evidence

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Direct evidence

The attention output merges its head axis and passes through c_proj before returning to the residual stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 109–112

Read code excerpt
        a = multihead_attn(q, k, v)
        a = merge_heads(a)
        a = conv1d(a, 'c_proj', n_state)
        return a, present

Direct evidence

The maintained attention path merges heads, applies c_proj, and returns the residual update.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 222–226

Read code excerpt
        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights