Transformer block › MLP
Activated MLP features
×tensorh
Element-wise tanh-GELU transforms h without changing its 3,072-feature width.
- shape
- [B batch, T positions, 3072 d_ff]
The code names this tensor h (model.py 118); here is the expansion before GELU.
Sources
Contextual evidence from Approximate GELU
The released graph defines the tanh approximation used by GPT-2.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 25–26
Read code excerpt
def gelu(x):
return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3))))Contextual evidence from Feature projection
The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.
linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56
Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
with tf.variable_scope(scope):
*start, nx = shape_list(x)
w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
return cContextual evidence from Feature projection
The c_proj contraction maps the activated wide state back to the block width.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 115–120
Read code excerpt
h = gelu(conv1d(x, 'c_fc', n_state))
h2 = conv1d(h, 'c_proj', nx)
return h2Contextual evidence from Feature projection
After activation, c_proj contracts the intermediate state and returns the residual-width update.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 233–243
Read code excerpt
self.c_fc = Conv1D(intermediate_size, embed_dim)
self.c_proj = Conv1D(embed_dim, intermediate_size)
self.act = ACT2FN[config.activation_function]
self.dropout = nn.Dropout(config.resid_pdrop)… lines 237–238 omitted …
hidden_states = self.c_fc(hidden_states)
hidden_states = self.act(hidden_states)
hidden_states = self.c_proj(hidden_states)
hidden_states = self.dropout(hidden_states)
return hidden_states