Latent decoder
Latents to decode
×tensor
In training, the clean latents with the decoder noise added.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Decoder noise
Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Decoder noise
Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Positions to latents
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public