LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Diffusion loss

×

no parameters

How far the denoiser’s prediction is from the clean latents.

Ldiff=1B∑b1T⋅768∥z0,bγ−z^0,b∥22\mathcal L_{\mathrm{diff}}=\frac{1}{B}\sum_{b}\frac{1}{T\cdot768}\lVert z_{0,b}^{\gamma}-\hat z_{0,b}\rVert_2^2
in
z^0\hat z_0 [B batch, T positions, 768 d]z0γz_0^{\gamma} [B batch, T positions, 768 d]
out
Ldiff\mathcal L_{\mathrm{diff}} [1 one value]
Explanation

No weight depends on the time (loss_weight.type is none), and every position counts: the valid-position mask is not used here.

The target is not detached. The loss reaches the encoder twice, through the target and through the noised latents, each time multiplied by γ\gamma.

Sources3

Direct evidence

Diffusion: a time is sampled, the latents are noised by the marginal, with probability one half a first call without gradient gives the self-conditioning estimate, the denoiser predicts the clean latents, and the mean squared error against them is taken; the three losses are summed with their weights.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public