GPT-2 Small · Final LayerNorm

Final LayerNorm

learnednorm(h, 'ln_f')

Center and scale each position’s features, then apply learned gain and bias.

\begin{aligned}\mu_d(x)&=\frac{1}{d}\sum_{j=1}^d x_j\\\sigma_d^2(x)&=\frac{1}{d}\sum_{j=1}^d(x_j-\mu_d(x))^2\\y&=\gamma_{f}\odot\frac{x-\mu_d(x)}{\sqrt{\sigma_d^2(x)+\varepsilon}}+\beta_{f}\\\varepsilon&=10^{-5}\end{aligned}

in: $x_{12}$ [B batch, T positions, 768 d]
out: $h_{\mathrm{final}}$ [B batch, T positions, 768 d]
parameters: 1,536

Parameters

1,536 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Learned gain	$768$	768
Learned bias	$768$	768
Per-instance total		1,536

Each normalization instance owns its learned gain and bias vectors.

Derived from the configured feature width; checkpoint tensors have not been independently counted.

Explanation

Eight invented features of one token, before and after centering and division by √(variance + 10⁻⁵). The learned gain and bias are not applied in this example. Illustrative values.

Codemodel.py 28–38, 163–167 · modeling_gpt2.py 496–497

model.py 28–38

def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x

model.py 163–167

        for layer, past in enumerate(pasts):
            h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)
            presents.append(present)
        results['present'] = tf.stack(presents, axis=1)
        h = norm(h, 'ln_f')

modeling_gpt2.py 496–497

        self.h = nn.ModuleList([GPT2Block(config, layer_idx=i) for i in range(config.num_hidden_layers)])
        self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)

Sources2

Direct evidence

The final norm follows the layer loop.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–167

Read code excerpt

def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x

… lines 39–162 omitted …

        for layer, past in enumerate(pasts):
            h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)
            presents.append(present)
        results['present'] = tf.stack(presents, axis=1)
        h = norm(h, 'ln_f')

Direct evidence

Transformers registers ln_f once on the base model.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 492–497

Read code excerpt

        self.h = nn.ModuleList([GPT2Block(config, layer_idx=i) for i in range(config.num_hidden_layers)])
        self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)

GPT-2

GPT-2 model overview