LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

GPT-2 Smallhidden state −3124M weightsstandardized Open →

Latent encoder Open →

z₀ Decoder noise

+\,3\,\epsilon

Latent decoder Open →

Token decoder Open →

L Token cross-entropy

\mathcal L_{w}=\operatorname{CE}(L,\,I)

Feature reconstruction

\mathcal L_{h}=\operatorname{MSE}(\hat h,\,h)

z_t=\alpha_t z_0+\sigma_t\epsilon

z_t t z₀

Denoiser Open →

z₀ Diffusion loss

\mathcal L_{\mathrm{diff}}=\operatorname{MSE}(\hat z_0,\,z_0)

learns from

\mathcal L_{h}\cdot\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{h}

learns from

\mathcal L_{w}

joint objective: the three losses are added, and one backward pass trains the four networksfrozenGPT-2 Small →∇ × γ ∇ × 0

Joint objective

no parameters

The three losses, added with weight 1 each: one backward pass trains the four networks together.

\mathcal L=\mathcal L_{\mathrm{diff}}+\mathcal L_{h}+\mathcal L_{w}

in: $\mathcal L_{\mathrm{diff}}$ [1 one value] $\mathcal L_{h}$ [1 one value] $\mathcal L_{w}$ [1 one value]
out: $\mathcal L$ [1 one value]

Explanation

AdamW with learning rate 0.0003, weight decay 0.01 on every trained parameter, betas 0.9 and 0.98; a warm-up of 5,000 steps and a constant rate after it (the minimum rate equals the rate); gradients clipped at norm 1; 1,000,000 steps of 512 sequences in all, divided among the training processes. The warm-up’s shape is that of the timm library’s scheduler, of which no excerpt is recorded.

With the straight-through warm-up the diffusion loss itself has weight 1 from the first step; only its gradient into the encoder is scaled.

Sources5

Direct evidence

Diffusion: a time is sampled, the latents are noised by the marginal, with probability one half a first call without gradient gives the self-conditioning estimate, the denoiser predicts the clean latents, and the mean squared error against them is taken; the three losses are summed with their weights.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The optimizer takes every parameter of the four networks that requires a gradient.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The optimizer is AdamW with the configured learning rate, weight decay, betas and epsilon; the schedule is a cosine scheduler given the minimum learning rate, the warm-up’s first rate and its length.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A step: the gradient is clipped, the optimizer steps, and the moving average of the denoiser is updated.

The implementation, pinned at revision e18c42acdfb7, not yet public