LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Attention over all positions

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]Query–key products Scale

S/\sqrt{64}

Softmax over keys

A=\operatorname{softmax}(F)

A[B, 12, T, T]Weighted value sum O[B, 12, T, 64]

Token decoder › 3 blocks › Self-attention › Attention over all positions

Softmax over keys

no parameters

Turn each query’s scores over all positions into weights that sum to one.

{\htmlData{light=weights softmax}{A_{t,s}}}=\frac{\exp {\htmlData{light=softcap}{F_{t,s}}}}{\sum_{u=1}^{T}\exp F_{t,u}}

in: $F$ [B batch, 12 heads, T queries, T keys]
out: $A$ [B batch, 12 heads, T queries, T keys]

No position is masked, and the attention dropout is 0.

Sources1

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public