LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Decoder noise

×

no parameters

Add Gaussian noise of standard deviation 3 to the latents before they are decoded; in training only.

Why

Capacity. The latent space often has more dimensions than the manifold under it, and unused dimensions tend to hold random variation that spends capacity and corrupts the diffusion’s training signal. Noise on the decoder’s input makes reconstruction harder, so the encoder stores what is useful more efficiently while the latents’ norm stays low. LDLM paper §5.4

Robustness. It is data augmentation for the decoder: the decoder becomes robust to the diffusion model’s errors. LDLM paper §5.4

Variance. Diffusion models perform best when their inputs have the same variance. Without the noise the variance of the latent coordinates collapses toward zero, and the variance of the denoiser’s input then changes with the time. The noise forces the variance to grow, which improves the diffusion’s parameterization. LDLM paper §5.4

σ = 3. Its size matters in both directions: small values leave interpolation between latents poor, and at 4 generations collapse. The paper uses the intermediate value 3 to balance regularization, decoder robustness and reconstruction accuracy. LDLM paper §6

It is used in training only: sampling always decodes the clean generated latents. LDLM paper §4.3

z~0=z0+3 ϵ{\htmlData{light=decoder-input decoder-noise}{\tilde z_0}}={\htmlData{light=latents encoder}{z_0}}+3\,\epsilon
ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I)
in
z0z_0 [B batch, T positions, 768 d]
out
z~0\tilde z_0 [B batch, T positions, 768 d]
Explanation

This noise is not the diffusion’s: its size is fixed and it is used only to train the reconstruction. Generation decodes the sampled latents as they are.

Sources2

Direct evidence

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public