LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

I[B, T]

GPT-2 Smallhidden state −3124M weightsstandardized Open →

Latent encoder Open →

z₀ Decoder noise

+\,3\,\epsilon

Latent decoder Open →

Token decoder Open →

L Token cross-entropy

\mathcal L_{w}=\operatorname{CE}(L,\,I)

Feature reconstruction

\mathcal L_{h}=\operatorname{MSE}(\hat h,\,h)

Add noise

z_t=\alpha_t z_0+\sigma_t\epsilon

Open →

z_t t z₀

Denoiser Open →

z₀ Diffusion loss

\mathcal L_{\mathrm{diff}}=\operatorname{MSE}(\hat z_0,\,z_0)

learns from

\mathcal L_{h}\cdot\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{h}

learns from

\mathcal L_{w}

joint objective: the three losses are added, and one backward pass trains the four networksfrozenGPT-2 Small →∇ × γ ∇ × 0

Tokenizer

no parameters

Encode the texts, join them and cut the stream of tokens into sequences of 1,024.

out: $I$ [B batch, T positions]

Explanation

A training sequence is 1,022 consecutive tokens of a run of joined texts between a beginning and an end token, both <|endoftext|>; each text ends with that token too, and a sequence may run across texts (wrap is true). Texts are joined one batch of the preprocessing at a time, and the tokens left at the end of a batch are dropped. Every position is valid: the attention mask is all ones.

So $T=1,024$ throughout, and the IDs index a vocabulary of 50,259 entries.

Sources3

Direct evidence

With wrapping, each text is tokenized without special tokens and ends with the end token; the texts of a batch of the preprocessing are joined and cut into blocks of the sequence length less two, each set between a beginning and an end token, with every position valid; what is left over is dropped.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Setting the tokenizer up wraps every encoded sequence in a beginning and an end token and adds a `[PAD]` token when the tokenizer has none.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The OpenWebText configuration sets sequences of 1,024 tokens.

The implementation, pinned at revision e18c42acdfb7, not yet public