GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Position-wise feed-forward network

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Transformer block › MLP

MLP residual update

×

tensorh2

The second projection returns to 768 features. During training the MLP module's residual dropout acts on it before the block's addition.

shape
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
Sources

Contextual evidence from MLP residual merge

Second branch and block return in the reference graph.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Contextual evidence from MLP residual merge

The same residual contract in the maintained PyTorch implementation.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 303–309

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)

… lines 306–306 omitted …

        hidden_states = residual + feed_forward_hidden_states

Contextual evidence from Feature projection

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Contextual evidence from Feature projection

The c_proj contraction maps the activated wide state back to the block width.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 115–120

Read code excerpt
        h = gelu(conv1d(x, 'c_fc', n_state))
        h2 = conv1d(h, 'c_proj', nx)
        return h2

Contextual evidence from Feature projection

After activation, c_proj contracts the intermediate state and returns the residual-width update.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 233–243

Read code excerpt
        self.c_fc = Conv1D(intermediate_size, embed_dim)
        self.c_proj = Conv1D(embed_dim, intermediate_size)
        self.act = ACT2FN[config.activation_function]
        self.dropout = nn.Dropout(config.resid_pdrop)

… lines 237–238 omitted …

        hidden_states = self.c_fc(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states