Latent encoder › 6 encoder blocks › Perceiver attention › Attention over both streams
Scale and softcap
×no parameters
Divide the scores by √64 and bound them smoothly between −50 and 50.
- in
- [B batch, 12 heads, T queries, 2T keys]
- out
- [B batch, 12 heads, T queries, 2T keys]
The scale 1/√64 is taken as the default of flash-attn 2.7.4.post1, whose kernel applies the scale and the cap together (flash_api.cpp 108–121). The review record holds that default for the function the denoiser calls; for the key–value-packed function called here it was not recorded.
Sources2
Direct evidence
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.
The implementation, pinned at revision e18c42acdfb7, not yet public