Latent encoder › 6 encoder blocks › Perceiver attention
Queries
×tensor
A normalized 64-feature query for every head and position of the queries’ stream.
- shape
- [B batch, 12 heads, T positions, 64 head features]
Sources
Contextual evidence from Query heads
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Query heads
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Query heads
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Attention over both streams
The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.
The implementation, pinned at revision e18c42acdfb7, not yet public