Llama 3

All models, with previews →

Meta · Apr 2024Dense decoder

Grouped-query attention

Reference code meta-llama/llama3 @ a0940f9, configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

Full sequencePrefillDecode

32 independent transformer blocks › Attention

Raw similarities

×

tensortorch.matmul(xq, keys.transpose(2, 3))

Query–key dot products before scaling and causal masking.

shape
SS [B batch, 32 q heads, 1 queries, 1001 keys]
Codemodel.py 179–184
model.py 179–184
        xq = xq.transpose(1, 2)  # (bs, n_local_heads, seqlen, head_dim)
        keys = keys.transpose(1, 2)  # (bs, n_local_heads, cache_len + seqlen, head_dim)
        values = values.transpose(
            1, 2
        )  # (bs, n_local_heads, cache_len + seqlen, head_dim)
        scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(self.head_dim)
Sources1

Contextual evidence

Queries multiply transposed keys over their common head-feature axis.

attn-dot · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 179–184

Read code excerpt
        xq = xq.transpose(1, 2)  # (bs, n_local_heads, seqlen, head_dim)
        keys = keys.transpose(1, 2)  # (bs, n_local_heads, cache_len + seqlen, head_dim)
        values = values.transpose(
            1, 2
        )  # (bs, n_local_heads, cache_len + seqlen, head_dim)
        scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(self.head_dim)