Llama 3.1 8B · Transformer block

Transformer block

learnedlayer(h, start_pos, freqs_cis, mask)

Combine sequence context and position-wise feature transformations through two residual updates.

x'_n=x_n+\Delta_{\mathrm{attn}}

x_{n+1}=x'_n+\Delta_{\mathrm{mlp}}

All 24 steps

x_{\mathrm{attn}}=\operatorname{RMSNorm}(x_n)

\text{Grouped-query attention:}

{\htmlClass{vl-stream-q}{Q}}=xW_Q

{\htmlClass{vl-stream-k}{K}}=xW_K

{\htmlClass{vl-stream-v}{V}}=xW_V

\begin{bmatrix}z_{2j}^{\prime}\\z_{2j+1}^{\prime}\end{bmatrix}=\begin{bmatrix}\cos(t\omega'_j)&-\sin(t\omega'_j)\\\sin(t\omega'_j)&\cos(t\omega'_j)\end{bmatrix}\begin{bmatrix}z_{2j}\\z_{2j+1}\end{bmatrix}

\begin{bmatrix}z_{2j}^{\prime}\\z_{2j+1}^{\prime}\end{bmatrix}=\begin{bmatrix}\cos(t\omega'_j)&-\sin(t\omega'_j)\\\sin(t\omega'_j)&\cos(t\omega'_j)\end{bmatrix}\begin{bmatrix}z_{2j}\\z_{2j+1}\end{bmatrix}

{\htmlClass{vl-stream-k}{K_{\mathrm G}}}[h]={\htmlClass{vl-stream-k}{K_{\mathrm r}}}\big[\lfloor h/4\rfloor\big]

{\htmlClass{vl-stream-v}{V_{\mathrm G}}}[h]={\htmlClass{vl-stream-v}{V}}\big[\lfloor h/4\rfloor\big]

S={\htmlClass{vl-stream-q}{Q_{\mathrm r}}}{\htmlClass{vl-stream-k}{K_{\mathrm G}^{\mathsf T}}}

S_{\mathrm s}=S/\sqrt{128}

S_{\mathrm m}=S_{\mathrm s}+{\htmlClass{vl-stream-context}{M}}

A[b,i,t,s]=\frac{\exp(S_{\mathrm m}[b,i,t,s])}{\sum_{u=0}^{T-1}\exp(S_{\mathrm m}[b,i,t,u])}

H=AV_{\mathrm G}

H=\operatorname{reshape}(H)

\Delta_{\mathrm{attn}}=HW_O

x'_n=x_n+\Delta_{\mathrm{attn}}

x_{\mathrm{mlp}}=\operatorname{RMSNorm}(x'_n)

\text{SwiGLU MLP:}

g = xW_{\mathrm{gate}}

u = xW_{\mathrm{up}}

a=g\,\sigma(g)

h = a \odot u

\Delta_{\mathrm{mlp}} = hW_{\mathrm{down}}

x_{n+1}=x'_n+\Delta_{\mathrm{mlp}}

in: $x_0$ [B batch, T positions, 4096 d] $R$ [T positions, 64 pairs] $M$ [T queries, T keys]
out: $x_{32}$ [B batch, T positions, 4096 d]
parameters: 218,112,000 · × 32 blocks = 6,979,584,000

Meta’s code serves inference: the model’s forward runs under torch.inference_mode() (model.py 277), and no dropout is applied to the attention weights or to either sublayer’s update (179–190, 236–248).

Parameters

218,112,000 parameters per block, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
RMSNorm 1	$4{,}096$	4,096
Grouped-query attention	41,943,040 across its components	41,943,040
RMSNorm 2	$4{,}096$	4,096
SwiGLU MLP	176,160,768 across its components	176,160,768
Per-instance total		218,112,000

Across 32 independent blocks: $218{,}112{,}000 \times 32 = 6{,}979{,}584{,}000$ parameters.

Derived from the pinned catalogue dimensions and implementation. Checkpoint tensors have not been independently counted.

Codemodel.py 35–300 · sku_list.py 19, 231–248

model.py 262–266

        self.layers = torch.nn.ModuleList()
        for layer_id in range(params.n_layers):
            self.layers.append(TransformerBlock(layer_id, params))

        self.norm = RMSNorm(params.dim, eps=params.norm_eps)

model.py 298–300

        for layer in self.layers:
            h = layer(h, start_pos, freqs_cis, mask)
        h = self.norm(h)

model.py 236–248

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out

model.py 35–46

class RMSNorm(torch.nn.Module):
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def _norm(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

    def forward(self, x):
        output = self._norm(x.float()).type_as(x)
        return output * self.weight

model.py 236–237

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

model.py 266

        self.norm = RMSNorm(params.dim, eps=params.norm_eps)

model.py 93–120

        self.n_kv_heads = args.n_heads if args.n_kv_heads is None else args.n_kv_heads
        model_parallel_size = fs_init.get_model_parallel_world_size()
        self.n_local_heads = args.n_heads // model_parallel_size
        self.n_local_kv_heads = self.n_kv_heads // model_parallel_size
        self.n_rep = self.n_local_heads // self.n_local_kv_heads
        self.head_dim = args.dim // args.n_heads

        self.wq = ColumnParallelLinear(
            args.dim,
            args.n_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )
        self.wk = ColumnParallelLinear(
            args.dim,
            self.n_kv_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )
        self.wv = ColumnParallelLinear(
            args.dim,
            self.n_kv_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )

model.py 153–160

        bsz, seqlen, _ = x.shape
        xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)

        xq = xq.view(bsz, seqlen, self.n_local_heads, self.head_dim)
        xk = xk.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
        xv = xv.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)

        xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)

model.py 121–127

        self.wo = RowParallelLinear(
            args.n_heads * self.head_dim,
            args.dim,
            bias=False,
            input_is_parallel=True,
            init_method=lambda x: x,
        )

model.py 184–190

        scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(self.head_dim)
        if mask is not None:
            scores = scores + mask  # (bs, n_local_heads, seqlen, cache_len + seqlen)
        scores = F.softmax(scores.float(), dim=-1).type_as(xq)
        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
        return self.wo(output)

model.py 193–206

class FeedForward(nn.Module):
    def __init__(
        self,
        dim: int,
        hidden_dim: int,
        multiple_of: int,
        ffn_dim_multiplier: Optional[float],
    ):
        super().__init__()
        hidden_dim = int(2 * hidden_dim / 3)
        # custom dim factor multiplier
        if ffn_dim_multiplier is not None:
            hidden_dim = int(ffn_dim_multiplier * hidden_dim)
        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)

model.py 229–234

        self.feed_forward = FeedForward(
            dim=args.dim,
            hidden_dim=4 * args.dim,
            multiple_of=args.multiple_of,
            ffn_dim_multiplier=args.ffn_dim_multiplier,
        )

model.py 208–219

        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

sku_list.py 19

LLAMA3_VOCAB_SIZE = 128256

sku_list.py 231–248

        Model(
            core_model_id=CoreModelId.llama3_1_8b,
            description="Llama 3.1 8b model",
            huggingface_repo="meta-llama/Llama-3.1-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": True,
            },
            pth_file_count=1,
        ),

Sources8

Direct evidence

The model constructs independent blocks and passes the same sequence-specific RoPE coefficients and mask through the block loop.

model-stack · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 262–300

Read code excerpt

        self.layers = torch.nn.ModuleList()
        for layer_id in range(params.n_layers):
            self.layers.append(TransformerBlock(layer_id, params))

        self.norm = RMSNorm(params.dim, eps=params.norm_eps)

… lines 267–297 omitted …

        for layer in self.layers:
            h = layer(h, start_pos, freqs_cis, mask)
        h = self.norm(h)

Direct evidence

Meta's Llama 3.1 8B base catalogue entry supplies dimensions, head counts, the vocabulary (LLAMA3_VOCAB_SIZE, line 19), MLP rounding, epsilon, rope_theta and use_scaled_rope: True.

config · meta-llama/llama-models / models/sku_list.py
Pinned revision 0e0b8c519242 ↗ · lines 19–248

Read code excerpt

LLAMA3_VOCAB_SIZE = 128256

… lines 20–230 omitted …

        Model(
            core_model_id=CoreModelId.llama3_1_8b,
            description="Llama 3.1 8b model",
            huggingface_repo="meta-llama/Llama-3.1-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": True,
            },
            pth_file_count=1,
        ),

Direct evidence

Two independently normalized mechanism branches are added to their corresponding residual inputs.

block-residuals · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 236–248

Read code excerpt

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out

Direct evidence

RMSNorm rescales the final feature axis and applies one learned gain vector; block and final callers supply configured epsilon.

rms · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 35–266

Read code excerpt

class RMSNorm(torch.nn.Module):
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def _norm(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

    def forward(self, x):
        output = self._norm(x.float()).type_as(x)
        return output * self.weight

… lines 47–235 omitted …

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

… lines 238–265 omitted …

        self.norm = RMSNorm(params.dim, eps=params.norm_eps)

Direct evidence

Separate bias-free Q/K/V projections use configured query and KV head widths, then reshape into heads.

qkv · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 93–160

Read code excerpt

        self.n_kv_heads = args.n_heads if args.n_kv_heads is None else args.n_kv_heads
        model_parallel_size = fs_init.get_model_parallel_world_size()
        self.n_local_heads = args.n_heads // model_parallel_size
        self.n_local_kv_heads = self.n_kv_heads // model_parallel_size
        self.n_rep = self.n_local_heads // self.n_local_kv_heads
        self.head_dim = args.dim // args.n_heads

        self.wq = ColumnParallelLinear(
            args.dim,
            args.n_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )
        self.wk = ColumnParallelLinear(
            args.dim,
            self.n_kv_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )
        self.wv = ColumnParallelLinear(
            args.dim,
            self.n_kv_heads * self.head_dim,
            bias=False,
            gather_output=False,
            init_method=lambda x: x,
        )

… lines 121–152 omitted …

        bsz, seqlen, _ = x.shape
        xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)

        xq = xq.view(bsz, seqlen, self.n_local_heads, self.head_dim)
        xk = xk.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
        xv = xv.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)

        xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)

Direct evidence

Normalized attention weights multiply values; head outputs are merged and passed through a learned bias-free output projection.

retrieval · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 121–190

Read code excerpt

        self.wo = RowParallelLinear(
            args.n_heads * self.head_dim,
            args.dim,
            bias=False,
            input_is_parallel=True,
            init_method=lambda x: x,
        )

… lines 128–183 omitted …

        scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(self.head_dim)
        if mask is not None:
            scores = scores + mask  # (bs, n_local_heads, seqlen, cache_len + seqlen)
        scores = F.softmax(scores.float(), dim=-1).type_as(xq)
        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
        return self.wo(output)

Direct evidence

The block passes four times model width; FeedForward truncates, applies the configured multiplier and rounds to the configured multiple.

mlp-width · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 193–234

Read code excerpt

class FeedForward(nn.Module):
    def __init__(
        self,
        dim: int,
        hidden_dim: int,
        multiple_of: int,
        ffn_dim_multiplier: Optional[float],
    ):
        super().__init__()
        hidden_dim = int(2 * hidden_dim / 3)
        # custom dim factor multiplier
        if ffn_dim_multiplier is not None:
            hidden_dim = int(ffn_dim_multiplier * hidden_dim)
        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)

… lines 207–228 omitted …

        self.feed_forward = FeedForward(
            dim=args.dim,
            hidden_dim=4 * args.dim,
            multiple_of=args.multiple_of,
            ffn_dim_multiplier=args.ffn_dim_multiplier,
        )

Direct evidence

The MLP owns three bias-free projections. Its forward expression applies SiLU to w1(x), multiplies w3(x) element-wise and projects through w2.

mlp · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 208–219

Read code excerpt

        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))