LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generationcall1

1 2 512 1,024

Sampler › Denoiser › 12 denoiser blocks

Modulate

no parameters

Scale and shift the normalized features by two vectors made from the time.

{\htmlData{light=attention-modulate}{x_{\mathrm{attn}}}}={\htmlData{light=attention-norm}{\bar x_{\mathrm{attn}}}}\odot(1+s_{a})+b_{a}

in: $\bar x_{\mathrm{attn}}$ [B batch, T positions, 768 d] $m_n$ [B batch, 1 every position, 4608 six vectors of 768]
out: $x_{\mathrm{attn}}$ [B batch, T positions, 768 d]

Explanation

$b_{a}$ and $s_{a}$ are the first and second of the six vectors of $m_n$ .

The same shift and scale apply at every position: the time is one number for the whole sequence.

Sources2

Direct evidence

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public