LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

I[B, T]

GPT-2 Smallhidden state −3124M weightsstandardized Open →

Latent encoder Open →

z₀ Decoder noise

+\,3\,\epsilon

Latent decoder Open →

Token decoder Open →

L Token cross-entropy

\mathcal L_{w}=\operatorname{CE}(L,\,I)

Feature reconstruction

\mathcal L_{h}=\operatorname{MSE}(\hat h,\,h)

Add noise

z_t=\alpha_t z_0+\sigma_t\epsilon

Open →

z_t t z₀

Denoiser Open →

z₀ Diffusion loss

\mathcal L_{\mathrm{diff}}=\operatorname{MSE}(\hat z_0,\,z_0)

learns from

\mathcal L_{h}\cdot\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{h}

learns from

\mathcal L_{w}

joint objective: the three losses are added, and one backward pass trains the four networksfrozenGPT-2 Small →∇ × γ ∇ × 0

Token cross-entropy

no parameters

How badly the logits predict the tokens the text was made of.

\mathcal L_{w}=-\frac{1}{\lvert A\rvert}\sum_{(b,t)\in A}\log\operatorname{softmax}(L_{b,t})_{I_{b,t}}

in: $L$ [B batch, T positions, 50259 vocabulary] $I$ [B batch, T positions]
out: $\mathcal L_{w}$ [1 one value]

Explanation

$A$ is the set of valid positions, from the attention mask; with wrapped sequences that is every position, the beginning and end tokens included.

Its gradient stops at the token decoder’s input.

Sources3

Direct evidence

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Cross-entropy and mean squared error are averaged over valid positions.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public