LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Decoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder › 6 decoder blocks

Add MLP update

×

no parameters

Add the mechanism’s update to the residual stream element by element.

rn+1=rn′+Δ~mlp{\htmlData{light=block-output mlp-add}{r_{n+1}}}={\htmlData{light=residual attention-add}{r'_n}}+{\htmlData{light=mlp-scale}{\tilde\Delta_{\mathrm{mlp}}}}
in
rn′r'_n [B batch, T positions, 768 d]Δ~mlp\tilde\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
out
rn+1r_{n+1} [B batch, T positions, 768 d]
Sources1

Direct evidence

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public