Token decoder
Reconstructed features
×tensor
The latent decoder’s output; in training it arrives detached.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Stop-gradient
Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Stop-gradient
Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token-decoder block
The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token-decoder block
The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token-decoder block
A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token-decoder block
A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token-decoder block
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public