LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generationcall1

1 2 512 1,024

x_mlp[B, T, 768]Expansion

h=x_{\mathrm{mlp}}W_{1}+b_{1}

h[B, T, 3072]GELU

g=\operatorname{GELU}(h)

g[B, T, 3072]Contraction

\Delta_{\mathrm{mlp}}=gW_{2}+b_{2}

Δ_mlp[B, T, 768]

Sampler › Denoiser › 12 denoiser blocks › MLP

Contraction

learned

Map 3,072 features back to 768.

{\htmlData{light=output down}{\Delta_{\mathrm{mlp}}}}={\htmlData{light=activated gelu}{g}}W_{2}+b_{2}

in: $g$ [B batch, T positions, 3072 d_ff]
out: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
parameters: 2,360,064 · × 12 blocks = 28,320,768

Parameters

2,360,064 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$3{,}072 \times 768$	2,359,296
Output bias	$768$	768
Per-instance total		2,360,064

The reference module holds the weight output by input, [out, in].

Across 12 independent blocks: $2{,}360{,}064 \times 12 = 28{,}320{,}768$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The standard MLP: an affine layer, GELU with the tanh approximation, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public