LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

z_t[B, T, 768]z^sc₀[B, T, 768]t[B]Joinalong features Input projection

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

x₀

Time embedding Open →

a_t[B, 768]

12 denoiser blocks Open →

x₁₂ Final LayerNorm

\mu,\;\sigma^2

Time projection

a_tW_o+c_o

Modulate

\bar x\odot(1+s_o)+b_o

Output projection

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

z₀[B, T, 768]

Denoiser

Join along features

no parameters

Set the previous estimate beside the noised latents: 1,536 features per position.

[\,{\htmlData{light=noised}{z_t}}\;{\htmlData{light=estimate}{\hat z_0^{\mathrm{sc}}}}\,]

in: $z_t$ [B batch, T positions, 768 d] $\hat z_0^{\mathrm{sc}}$ [B batch, T positions, 768 d]
out: $[\,z_t\;\hat z_0^{\mathrm{sc}}\,]$ [B batch, T positions, 1536]

Sources2

Direct evidence

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public