LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

z_t[B, T, 768]z^sc₀[B, T, 768]t[B]Joinalong features Input projection

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

x₀

Time embedding Open →

a_t[B, 768]

12 denoiser blocks Open →

x₁₂ Final LayerNorm

\mu,\;\sigma^2

Time projection

a_tW_o+c_o

Modulate

\bar x\odot(1+s_o)+b_o

Output projection

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

z₀[B, T, 768]

Denoiser

Time modulation

no parameters

Scale and shift the normalized output features by the two vectors made from the time.

{\htmlData{light=final-modulate}{x_{\mathrm{out}}}}={\htmlData{light=final-norm}{\bar x}}\odot(1+s_o)+b_o

in: $\bar x$ [B batch, T positions, 768 d] $m_{\mathrm{out}}$ [B batch, 1 every position, 1536 two vectors of 768]
out: $x_{\mathrm{out}}$ [B batch, T positions, 768 d]

Sources2

Direct evidence

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The output layer: gain-only LayerNorm, a shift and a scale from the time condition, and an affine projection initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public