LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

r^′_n[B, T, 768]LayerNorm

\mu,\;\sigma^2

gain · bias Expansion

h=\bar r'_n\,W_{1}+b_{1}

GELU

g=h\,\Phi(h)

g[B, T, 3072]Contraction

\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}

Δ_mlp[B, T, 768]

Latent decoder › 6 decoder blocks › MLP

Expansion

learned

Map 768 features to 3,072.

{\htmlData{light=up}{h}}={\htmlData{light=norm}{\bar r'_n}}\,W_{1}+b_{1}

in: $\bar r'_n$ [B batch, T positions, 768 d]
out: $h$ [B batch, T positions, 3072 d_ff]
parameters: 2,362,368 · × 6 blocks = 14,174,208

Parameters

2,362,368 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 3{,}072$	2,359,296
Output bias	$3{,}072$	3,072
Per-instance total		2,362,368

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: $2{,}362{,}368 \times 6 = 14{,}174{,}208$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public