LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Attention over both streams

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Q[B, 12, T, 64]K[B, 12, 2T, 64]V[B, 12, 2T, 64]Query–key products Scale, softcap

50\tanh(S/50\sqrt{64})

Softmax over keys

A=\operatorname{softmax}(F)

A[B, 12, T, 2T]Weighted value sum O[B, 12, T, 64]

Latent encoder › 6 encoder blocks › Perceiver attention › Attention over both streams

Softmax over keys

no parameters

Turn each query’s scores over all 2T keys into weights that sum to one.

{\htmlData{light=weights softmax}{A_{t,s}}}=\frac{\exp {\htmlData{light=softcap}{F_{t,s}}}}{\sum_{u=1}^{2T}\exp F_{t,u}}

in: $F$ [B batch, 12 heads, T queries, 2T keys]
out: $A$ [B batch, 12 heads, T queries, 2T keys]

No position is masked: every query sees all 2T keys. The attention dropout is 0 (probs_dropout).

Sources1

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public