LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Latent encoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

h[B, T, 768]Position table

P

Add positions

h^{p}=h+E_{p}

h^p[B, T, 768]Latent queries

U

Input RMSNorm

\lVert x\rVert_2/\sqrt d

u₀[B, T, 768]

6 encoder blocks Open →

u₆ Output RMSNorm

\lVert x\rVert_2/\sqrt d

z₀[B, T, 768]

Latent encoder

Add positions

no parameters

Add each position’s learned vector to its features.

{\htmlData{light=with-positions add-positions}{h^{p}}}={\htmlData{light=features}{h}}+{\htmlData{light=positions}{E_{p}}}

in: $h$ [B batch, T positions, 768 d] $E_{p}$ [B batch, T positions, 768 d]
out: $h^{p}$ [B batch, T positions, 768 d]

Sources1

Direct evidence

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public