LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Self-attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

x_attn[B, T, 768]Q, K, V projection

x_{\mathrm{attn}}\,W_{QKV}

Split query heads Split key heads Split value heads Rotary tableR R RoPE RoPE Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]

Attend Open →

O[B, 12, T, 64]Merge heads Output projection

\Delta_{\mathrm{attn}}=oW_{O}

Δ_attn[B, T, 768]

Denoiser › 12 denoiser blocks › Self-attention

Rotary table

no parameters

The cosine and sine of every position’s angle for each pair of head features.

{\htmlData{light=rotary}{{\htmlClass{vl-stream-context}{R_{t,j}}}}}=\big(\cos t\theta_j,\;\sin t\theta_j\big)

\theta_j=10,000^{-2j/64}

out: $R$ [T positions, 32 pairs, 2 cos, sin]

One table serves every block of the network; it is computed once per sequence length and kept.

Sources2

Direct evidence

The rotary table: angles from positions and inverse frequencies of base 10,000, computed in float32; queries and keys are rotated and values take the identity.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.

The implementation, pinned at revision e18c42acdfb7, not yet public