LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

r^′_n[B, T, 768]LayerNorm

\mu,\;\sigma^2

gain · bias Expansion

h=\bar r'_n\,W_{1}+b_{1}

GELU

g=h\,\Phi(h)

g[B, T, 3072]Contraction

\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}

Δ_mlp[B, T, 768]

Latent decoder › 6 decoder blocks › MLP

Contraction

learned

Map 3,072 features back to 768.

{\htmlData{light=output down}{\Delta_{\mathrm{mlp}}}}={\htmlData{light=activated gelu}{g}}\,W_{2}+b_{2}

in: $g$ [B batch, T positions, 3072 d_ff]
out: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
parameters: 2,360,064 · × 6 blocks = 14,160,384

Parameters

2,360,064 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$3{,}072 \times 768$	2,359,296
Output bias	$768$	768
Per-instance total		2,360,064

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: $2{,}360{,}064 \times 6 = 14{,}160{,}384$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public