LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder › 6 decoder blocks › MLP

MLP output

×

tensor

The output of the block’s MLP.

shape
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
Sources

Contextual evidence from Residual scale

The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Residual scale

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Residual scale

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Contraction

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public