Construct causal mask
×no parameterstorch.triu(mask, diagonal=1)
Identify the key positions permitted for every query position.
- in
- [T positions]
- out
- [T queries, T keys]
Built only when T > 1 (model.py 284–285). With cached positions it is [T, start_pos + T], its first start_pos columns zero (294–296); the reference generate loop calls with T = 1 after its first step, so there the mask is None (generation.py 179–203).
Codemodel.py 284–300
mask = None
if seqlen > 1:
mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device)
mask = torch.triu(mask, diagonal=1)
# When performing key-value caching, we compute the attention scores
# only for the new sequence. Thus, the matrix of scores is of size
# (seqlen, cache_len + seqlen), and the only masked entries are (i, j) for
# j > cache_len + i, since row i corresponds to token cache_len + i.
mask = torch.hstack(
[torch.zeros((seqlen, start_pos), device=tokens.device), mask]
).type_as(h)
for layer in self.layers:
h = layer(h, start_pos, freqs_cis, mask)
h = self.norm(h)Sources1
Direct evidence
The model builds the causal mask only when sequence length exceeds one and passes it into every block.
model-mask · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 284–300
Read code excerpt
mask = None
if seqlen > 1:
mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device)
mask = torch.triu(mask, diagonal=1)
# When performing key-value caching, we compute the attention scores
# only for the new sequence. Thus, the matrix of scores is of size
# (seqlen, cache_len + seqlen), and the only masked entries are (i, j) for
# j > cache_len + i, since row i corresponds to token cache_len + i.
mask = torch.hstack(
[torch.zeros((seqlen, start_pos), device=tokens.device), mask]
).type_as(h)
for layer in self.layers:
h = layer(h, start_pos, freqs_cis, mask)
h = self.norm(h)