LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Query MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

m[B, T, 768]LayerNorm

\mu,\;\sigma^2

gain · bias Expansion

h=\bar m\,W_{1}+b_{1}

GELU

g=h\,\Phi(h)

g[B, T, 3072]Contraction

e=g\,W_{2}+b_{2}

e[B, T, 768]

in: $m$ [B batch, T positions, 768 d]
out: $e$ [B batch, T positions, 768 d]
parameters: 4,723,968
mode: generation

Parameters

4,723,968 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
LayerNorm	$768 + 768$	1,536
Expansion	$768 \times 3{,}072 + 3{,}072$	2,362,368
Contraction	$3{,}072 \times 768 + 768$	2,360,064
Per-instance total		4,723,968

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

The module normalizes its own input; its dropout of 0.1 between the activation and the second layer acts in training only.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public