LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Add noise

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Add noise

First denoiser call

×

no parameters

Half of the time, run the denoiser once without gradient and keep its prediction as the previous estimate; otherwise the estimate is zero.

Why

Self-conditioning. The denoiser is optionally conditioned on a previous estimate of the clean latents, following earlier latent diffusion models for text; the paper gives no further reason. LDLM paper §4.2

z^0sc={sg⁡(z^θ(zt, t, 0))with probability 0.50otherwise{\htmlData{light=estimate first-call}{\hat z_0^{\mathrm{sc}}}}=\begin{cases}\operatorname{sg}\big(\hat z_\theta({\htmlData{light=noise}{z_t}},\,{\htmlData{light=sample-time}{t}},\,0)\big)&\text{with probability }0.5\\0&\text{otherwise}\end{cases}
in
ztz_t [B batch, T positions, 768 d]tt [B batch]
out
z^0sc\hat z_0^{\mathrm{sc}} [B batch, T positions, 768 d]
Explanation

The choice is made once per training process for its whole share of the batch, never per sequence (random.random() > 0.5). The call uses the denoiser’s current weights, the ones being trained, and owns none of its own; the denoiser is in training mode, so its dropout is active in this call too.

This teaches the denoiser to use an earlier estimate when it is given one and to work without it when it is not.

Sources2

Direct evidence

Diffusion: a time is sampled, the latents are noised by the marginal, with probability one half a first call without gradient gives the self-conditioning estimate, the denoiser predicts the clean latents, and the mean squared error against them is taken; the three losses are summed with their weights.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public