Normalized ID cache
×state · kept between callsself.cache · engram.py 155–157, 167, 172
For every position so far, the normalized ID of its token (one ID for tokens whose text differs only in case, accents or spacing), or −1 at an image position; kept by the n-gram hashing step between forward calls: what lets an n-gram reach tokens of earlier calls.
- shape
- ID [B batch, T positions]
A prefill writes one entry for each of its positions, the token’s normalized ID or −1 at an image position (engram.py 164–167), then reads them back from the buffer: each position its own row and the 3 before it, as far as the start of the sequence or an image position (engram.py 169–174). In the interactive run the buffer is allocated for 65,536 rows per sequence, one 64-bit integer each (generate.py 124–126; engram.py 155–157), and its filled part grows with the sequence; a new generation writes it again from row 0. It is a buffer outside the checkpoint and holds no learned parameter (engram.py 155–157). Each row is one integer. Attention keeps its keys and values in buffers of its own, inside each layer.
Codeengram.py 159–184
@torch.inference_mode()
def forward(self, input_ids: torch.Tensor, start_pos: int, token_mask: torch.Tensor | None = None) -> torch.Tensor:
"""token_mask: [B, L], False for tokens that take no part in an n-gram (image spans).
Returns the hash ids, shaped [B, L, n_engram_layers, n_hash_cols]."""
batch, seqlen = input_ids.shape
compressed = self.token_map[input_ids]
if token_mask is not None:
compressed = torch.where(token_mask, compressed, self.DEAD)
self.cache[:batch, start_pos : start_pos + seqlen] = compressed
positions = torch.arange(start_pos, start_pos + seqlen, device=input_ids.device).expand(batch, seqlen)
tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)
for shift in range(self.layout.max_ngram_size):
source = self.cache[:batch].gather(1, (positions - shift).clamp_min(0))
blocked = blocked | (positions < shift) | (source == self.DEAD)
tokens.append(torch.where(blocked, self.pad_id, source))
tokens = torch.stack(tokens, dim=-1) # [B, L, max_ngram_size]
# XOR the multiplied ids together one lookback at a time, so the running value after step i
# is the hash of the (i+1)-gram; each lands in its own prime-sized bucket range
products = tokens.unsqueeze(2) * self.multipliers # [B, L, n_engram_layers, max_ngram_size]
rolling, hashes = products[..., 0], []
for i in range(1, self.layout.max_ngram_size):
rolling = torch.bitwise_xor(rolling, products[..., i])
hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1])
return torch.cat(hashes, dim=-1) + self.offsetsSources1
Contextual evidence
Each position hashes its 2-, 3- and 4-gram of normalised token ids, each modulo eight primes, into row indices of each Engram layer’s table; an n-gram does not reach across the start of the sequence or an image position.
engram-hash · deepseek-ai/DeepSeek-V4.1-Flash / inference/engram.py
Pinned revision 2cba9e42aa02 ↗ · lines 159–184
Read code excerpt
@torch.inference_mode()
def forward(self, input_ids: torch.Tensor, start_pos: int, token_mask: torch.Tensor | None = None) -> torch.Tensor:
"""token_mask: [B, L], False for tokens that take no part in an n-gram (image spans).
Returns the hash ids, shaped [B, L, n_engram_layers, n_hash_cols]."""
batch, seqlen = input_ids.shape
compressed = self.token_map[input_ids]
if token_mask is not None:
compressed = torch.where(token_mask, compressed, self.DEAD)
self.cache[:batch, start_pos : start_pos + seqlen] = compressed
positions = torch.arange(start_pos, start_pos + seqlen, device=input_ids.device).expand(batch, seqlen)
tokens, blocked = [], torch.zeros_like(positions, dtype=torch.bool)
for shift in range(self.layout.max_ngram_size):
source = self.cache[:batch].gather(1, (positions - shift).clamp_min(0))
blocked = blocked | (positions < shift) | (source == self.DEAD)
tokens.append(torch.where(blocked, self.pad_id, source))
tokens = torch.stack(tokens, dim=-1) # [B, L, max_ngram_size]
# XOR the multiplied ids together one lookback at a time, so the running value after step i
# is the hash of the (i+1)-gram; each lands in its own prime-sized bucket range
products = tokens.unsqueeze(2) * self.multipliers # [B, L, n_engram_layers, max_ngram_size]
rolling, hashes = products[..., 0], []
for i in range(1, self.layout.max_ngram_size):
rolling = torch.bitwise_xor(rolling, products[..., i])
hashes.append(rolling.unsqueeze(-1) % self.primes[:, i - 1])
return torch.cat(hashes, dim=-1) + self.offsets