LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Decoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder › 6 decoder blocks

Decoder state after attention

×

tensor

The residual features after the attention update, feeding the MLP branch and its bypass.

shape
rn′r'_n [B batch, T positions, 768 d]
Sources

Contextual evidence from Add attention update

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from MLP

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public