- in
- [B batch, T positions, 768 d] [B batch, T positions, 768 d]
- out
- [B batch, T positions, 768 d]
- parameters
- 8,264,580 · × 6 blocks = 49,587,480
Parameters
8,264,580 parameters per block, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Perceiver attention | 3,540,608 across its components | 3,540,608 |
| Residual scale | 2 | |
| MLP | 4,723,968 across its components | 4,723,968 |
| Residual scale | 2 | |
| Per-instance total | 8,264,580 | |
Across 6 independent blocks: parameters.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
It is the encoder’s block with the two streams exchanged, and with a learned scale on each update.
Sources4
Direct evidence
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public