LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

z_t[B, T, 768]z^sc₀[B, T, 768]t[B]Joinalong features Input projection

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

x₀

Time embedding Open →

a_t[B, 768]

12 denoiser blocks Open →

x₁₂ Final LayerNorm

\mu,\;\sigma^2

Time projection

a_tW_o+c_o

Modulate

\bar x\odot(1+s_o)+b_o

Output projection

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

z₀[B, T, 768]

Denoiser

Input projection

learned

Map the 1,536 joined features to the denoiser’s 768.

{\htmlData{light=embedded input-projection}{x_0}}=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

in: $[\,z_t\;\hat z_0^{\mathrm{sc}}\,]$ [B batch, T positions, 1536]
out: $x_0$ [B batch, T positions, 768 d]
parameters: 1,180,416

Parameters

1,180,416 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$1{,}536 \times 768$	1,179,648
Output bias	$768$	768
Per-instance total		1,180,416

The reference module holds the weight output by input, [out, in].

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources2

Direct evidence

The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public