LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Denoiser block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Sampler › Denoiser › 12 denoiser blocks

Self-attention

Open inside →×

learned

12-head attention of every position to every position, with rotary positions on queries and keys and scores softcapped at 50.

Kt=rot⁡ ⁣(Kth; t θ){\htmlClass{vl-stream-k}{K_{t}}}=\operatorname{rot}\!\big({\htmlClass{vl-stream-k}{K^{h}_{t}}};\,t\,\theta\big)
θj=10,000−2j/64\theta_j=10,000^{-2j/64}
Δattn=oWO\Delta_{\mathrm{attn}}=oW_{O}
All 11 steps
[ q  k  v ]=xattn WQKV[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k}}\;{\htmlClass{vl-stream-v}{v}}\,]=x_{\mathrm{attn}}\,W_{QKV}
Rt,j=(cos⁡tθj,  sin⁡tθj){\htmlClass{vl-stream-context}{R_{t,j}}}=\big(\cos t\theta_j,\;\sin t\theta_j\big)
θj=10,000−2j/64\theta_j=10,000^{-2j/64}
Qt=rot⁡ ⁣(Qth; t θ){\htmlClass{vl-stream-q}{Q_{t}}}=\operatorname{rot}\!\big({\htmlClass{vl-stream-q}{Q^{h}_{t}}};\,t\,\theta\big)
θj=10,000−2j/64\theta_j=10,000^{-2j/64}
Kt=rot⁡ ⁣(Kth; t θ){\htmlClass{vl-stream-k}{K_{t}}}=\operatorname{rot}\!\big({\htmlClass{vl-stream-k}{K^{h}_{t}}};\,t\,\theta\big)
θj=10,000−2j/64\theta_j=10,000^{-2j/64}
Attention over all positions:\text{Attention over all positions:}
S=QKTS=QK^{\mathsf T}
F=50tanh⁡ ⁣(S5064)F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)
At,s=exp⁡Ft,s∑u=1Texp⁡Ft,uA_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{T}\exp F_{t,u}}
O=AVO=AV
o=reshape⁡(O)o=\operatorname{reshape}(O)
Δattn=oWO\Delta_{\mathrm{attn}}=oW_{O}
in
xattnx_{\mathrm{attn}} [B batch, T positions, 768 d]
out
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
parameters
2,359,296 · × 12 blocks = 28,311,552

The code keeps head tensors as [B, T, 12, 64], packed with a third axis for queries, keys and values, and never exchanges positions and heads; the atlas writes heads before positions, as on its other plates.

Parameters

2,359,296 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Query, key and value projection768×2,304768 \times 2{,}3041,769,472
Output projection768×768768 \times 768589,824
Per-instance total2,359,296

Across 12 independent blocks: 2,359,296×12=28,311,5522{,}359{,}296 \times 12 = 28{,}311{,}552 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

Attention is bidirectional: no position is masked.

The softcap bounds every scaled score between −50 and 50 before the softmax.

Sources2

Direct evidence

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public