LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Key and value heads

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

The decoder state’s queries, keys and valuesp^a p^a The latents’ keys and valuesp^b p^b Take the decoder state’s own keys Take the latents’ keys Join keysalong positions Split heads Key RMSNorm

\lVert x\rVert_2/\sqrt d

K[B, 12, 2T, 64]Take the decoder state’s own values Take the latents’ values Join valuesalong positions Split heads V[B, 12, 2T, 64]

Latent decoder › 6 decoder blocks › Perceiver attention › Key and value heads

Split value heads

no parameters

Read each value’s 768 features as 12 heads of 64.

[B,\,2T,\,768]\to[B,\,12,\,2T,\,64]

in: $v$ [B batch, 2T positions of both streams, 768 features]
out: $V$ [B batch, 12 heads, 2T positions, 64 head features]

Sources1

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public