Latent decoder › 6 decoder blocks › Perceiver attention
Attention update
×tensor
The update added to the decoder state.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Residual scale
The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Residual scale
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Residual scale
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Output projection
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Output projection
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public