- in
- [B batch, T positions, 768 d] [B batch, T positions, 768 d]
- out
- [B batch, T positions, 768 d]
- parameters
- 8,264,576 · × 6 blocks = 49,587,456
Parameters
8,264,576 parameters per block, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Perceiver attention | 3,540,608 across its components | 3,540,608 |
| MLP | 4,723,968 across its components | 4,723,968 |
| Per-instance total | 8,264,576 | |
Across 6 independent blocks: parameters.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
Neither branch has a normalization in the block itself: the attention normalizes both of its streams and the MLP its own input.
Sources3
Direct evidence
An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.
The implementation, pinned at revision e18c42acdfb7, not yet public