LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Encoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder › 6 encoder blocks

Add attention update

×

no parameters

Add the mechanism’s update to the residual stream element by element.

un′=un+Δattn{\htmlData{light=residual attention-add}{u'_n}}={\htmlData{light=block-input}{u_n}}+{\htmlData{light=attention}{\Delta_{\mathrm{attn}}}}
in
unu_n [B batch, T positions, 768 d]Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
out
un′u'_n [B batch, T positions, 768 d]
Sources1

Direct evidence

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public