Latent encoder › 6 encoder blocks › Perceiver attention
Latents
×tensor
The latents entering this block: the stream the queries are made from.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Input RMSNorm
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Input RMSNorm
The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Input RMSNorm
The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from RMSNorm
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from RMSNorm
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public