LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Decoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder › 6 decoder blocks

MLP

Open inside →×

learned

A two-layer MLP with 3,072 hidden features and GELU, behind its own LayerNorm.

g=h Φ(h)g=h\,\Phi(h)
Δmlp=g W2+b2\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}
All 4 steps
rˉn′=LayerNorm⁡(rn′)\bar r'_n=\operatorname{LayerNorm}(r'_n)
h=rˉn′ W1+b1h=\bar r'_n\,W_{1}+b_{1}
g=h Φ(h)g=h\,\Phi(h)
Δmlp=g W2+b2\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}
in
rn′r'_n [B batch, T positions, 768 d]
out
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
parameters
4,723,968 · × 6 blocks = 28,343,808
Parameters

4,723,968 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
LayerNorm768+768768 + 7681,536
Expansion768×3,072+3,072768 \times 3{,}072 + 3{,}0722,362,368
Contraction3,072×768+7683{,}072 \times 768 + 7682,360,064
Per-instance total4,723,968

Across 6 independent blocks: 4,723,968×6=28,343,8084{,}723{,}968 \times 6 = 28{,}343{,}808 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

The module normalizes its own input; its dropout of 0.1 between the activation and the second layer acts in training only.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public