LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Perceiver attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Latent encoder › 6 encoder blocks › Perceiver attention

Attention update

tensor

The update added to the latents.

shape: $\Delta_{\mathrm{attn}}$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Add attention update

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Output projection

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Output projection

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public