LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

x_mlp[B, T, 768]Expansion

h=x_{\mathrm{mlp}}W_{1}+b_{1}

h[B, T, 3072]GELU

g=\operatorname{GELU}(h)

g[B, T, 3072]Contraction

\Delta_{\mathrm{mlp}}=gW_{2}+b_{2}

Δ_mlp[B, T, 768]

Denoiser › 12 denoiser blocks › MLP

Expansion

learned

Map 768 features to 3,072.

{\htmlData{light=pre-activation up}{h}}={\htmlData{light=x}{x_{\mathrm{mlp}}}}W_{1}+b_{1}

in: $x_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
out: $h$ [B batch, T positions, 3072 d_ff]
parameters: 2,362,368 · × 12 blocks = 28,348,416

Parameters

2,362,368 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 3{,}072$	2,359,296
Output bias	$3{,}072$	3,072
Per-instance total		2,362,368

The reference module holds the weight output by input, [out, in].

Across 12 independent blocks: $2{,}362{,}368 \times 12 = 28{,}348{,}416$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The standard MLP: an affine layer, GELU with the tanh approximation, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public