LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Decoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder › 6 decoder blocks

Latents with positions

×

supplied · no parameters

The latents being decoded, with positions added; the same tensor enters every block.

shape
zpz^{p} [B batch, T positions, 768 d]
Sources

Contextual evidence from Positions to latents

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Perceiver attention

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Perceiver attention

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public