LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Encoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

u_n[B, T, 768]h^p[B, T, 768]

Perceiver attention Open →

+u^′_n[B, T, 768]

MLP Open →

+u_n+1[B, T, 768]

Latent encoder › 6 encoder blocks

Latents after attention

tensor

The residual features after the attention update, feeding the MLP branch and its bypass.

shape: $u'_n$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Add attention update

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from MLP

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public