LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Self-attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_attn[B, T, 768]Q, K, V projection

y_{\mathrm{attn}}\,W_{QKV}

Split query heads Split key heads Split value heads Rotary tableR R RoPE RoPE Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]

Attend Open →

O[B, 12, T, 64]Merge heads Output projection

\Delta_{\mathrm{attn}}=oW_{O}

Δ_attn[B, T, 768]

Token decoder › 3 blocks › Self-attention

Rotary table

no parameters

The cosine and sine of every position’s angle for each pair of head features.

{\htmlData{light=rotary}{{\htmlClass{vl-stream-context}{R_{t,j}}}}}=\big(\cos t\theta_j,\;\sin t\theta_j\big)

\theta_j=10,000^{-2j/64}

out: $R$ [T positions, 32 pairs, 2 cos, sin]

One table serves every block of the network; it is computed once per sequence length and kept.

Sources2

Direct evidence

The rotary table: angles from positions and inverse frequencies of base 10,000, computed in float32; queries and keys are rotated and values take the identity.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public