GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Merge and project

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Full sequenceTrainingPrefillDecode
in
HH [B batch, 12 heads, 1, 64 d_h]
out
Δattn\Delta_{\mathrm{attn}} [B batch, 1, 768 d]
parameters
590,592 · × 12 blocks = 7,087,104
mode
decode · P = 1,000 cached positions · L = P + 1 = 1,001 keys read
Parameters

590,592 parameters per block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Merge attention heads—0
Attention output projection768×768+768768 \times 768 + 768590,592
Per-instance total590,592

Across 12 independent blocks: 590,592×12=7,087,104590{,}592 \times 12 = 7{,}087{,}104 parameters.

The group references these owners; it adds no weights of its own.

Explanation

Merging moves the head axis beside the feature axis and concatenates 12 × 64 = 768 features into HH. The output projection then combines information across heads at every position: Δattn=HWO+bO\Delta_{\mathrm{attn}}=HW_O+b_O. Only the projection owns parameters.

Intermediate states
Merged attention values
Codemodel.py 50–112 · modeling_gpt2.py 222–226
model.py 79
    def merge_heads(x):
model.py 81
        return merge_states(tf.transpose(x, [0, 2, 1, 3]))
model.py 50–56
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c
model.py 109–112
        a = multihead_attn(q, k, v)
        a = merge_heads(a)
        a = conv1d(a, 'c_proj', n_state)
        return a, present
modeling_gpt2.py 222–226
        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights
Sources4

Evidence for the operations this group contains. The group itself is a presentation and has no implementation of its own.

Contextual evidence from Merge attention heads

The released merge_heads function reverses the head permutation and merges the final two axes.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 79–81

Read code excerpt
    def merge_heads(x):

… lines 80–80 omitted …

        return merge_states(tf.transpose(x, [0, 2, 1, 3]))

Contextual evidence from Attention output projection

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Contextual evidence from Attention output projection

The attention output merges its head axis and passes through c_proj before returning to the residual stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 109–112

Read code excerpt
        a = multihead_attn(q, k, v)
        a = merge_heads(a)
        a = conv1d(a, 'c_proj', n_state)
        return a, present

Contextual evidence from Attention output projection

The maintained attention path merges heads, applies c_proj, and returns the residual update.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 222–226

Read code excerpt
        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights