Denoiser › 12 denoiser blocks
Modulation vectors
×tensor
Six vectors for this block: shift, scale and gate for attention, then for the MLP.
- shape
- [B batch, 1 every position, 4608 six vectors of 768]
Sources
Contextual evidence from Time projection
A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time projection
A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Modulate
The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.
The implementation, pinned at revision e18c42acdfb7, not yet public