GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

Pre-attention LayerNorm

×

learnednorm(x, 'ln_1')

Center and scale each position’s features, then apply learned gain and bias.

μd(x)=1d∑j=1dxjσd2(x)=1d∑j=1d(xj−μd(x))2y=γ1⊙x−μd(x)σd2(x)+ε+β1ε=10−5\begin{aligned}\mu_d(x)&=\frac{1}{d}\sum_{j=1}^d x_j\\\sigma_d^2(x)&=\frac{1}{d}\sum_{j=1}^d(x_j-\mu_d(x))^2\\y&=\gamma_{1}\odot\frac{x-\mu_d(x)}{\sqrt{\sigma_d^2(x)+\varepsilon}}+\beta_{1}\\\varepsilon&=10^{-5}\end{aligned}
in
xnx_n [B batch, T positions, 768 d]
out
Z1Z_1 [B batch, T positions, 768 d]
parameters
1,536 · × 12 blocks = 18,432
Parameters

1,536 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Learned gain768768768
Learned bias768768768
Per-instance total1,536

Each normalization instance owns its learned gain and bias vectors.

Across 12 independent blocks: 1,536×12=18,4321{,}536 \times 12 = 18{,}432 parameters.

Derived from the configured feature width; checkpoint tensors have not been independently counted.

Explanation
one token · 8 illustrative featuresbeforeaftermean 1.50 → 0
Eight invented features of one token, before and after centering and division by √(variance + 10⁻⁵). The learned gain and bias are not applied in this example. Illustrative values.
Codemodel.py 28–38 · modeling_gpt2.py 252, 272–273
model.py 28–38
def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x
modeling_gpt2.py 252
        self.ln_1 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
modeling_gpt2.py 272–273
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
Sources2

Direct evidence

Original LayerNorm implementation and ε = 10⁻⁵.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–38

Read code excerpt
def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x

Direct evidence

PyTorch delegates the operation to nn.LayerNorm with the checkpoint epsilon.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 252–273

Read code excerpt
        self.ln_1 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)

… lines 253–271 omitted …

        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)