Latent encoder › 6 encoder blocks
Latents
×tensor
The residual features entering this block.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Input RMSNorm
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Input RMSNorm
The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Input RMSNorm
The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Perceiver attention
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Perceiver attention
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Add attention update
An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.
The implementation, pinned at revision e18c42acdfb7, not yet public