LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Gated MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_mlp[B, T, 768]Gate projection

g=y_{\mathrm{mlp}}W_{\mathrm{gate}}

g[B, T, 3072]Up projection

u=y_{\mathrm{mlp}}W_{\mathrm{up}}

u[B, T, 3072]SiLU

a = \operatorname{SiLU}(g)

a[B, T, 3072]Element-wise product

h = a \odot u

h[B, T, 3072]Down projection

\Delta_{\mathrm{mlp}}=hW_{\mathrm{down}}

Δ_mlp[B, T, 768]

Token decoder › 3 blocks › Gated MLP

Element-wise product

no parameters

Gate the up features by the activated gate, feature by feature.

{\htmlData{light=h product}{h}}={\htmlData{light=a silu}{a}}\odot {\htmlData{light=u up}{u}}

in: $a$ [B batch, T positions, 3072 d_ff] $u$ [B batch, T positions, 3072 d_ff]
out: $h$ [B batch, T positions, 3072 d_ff]

Sources1

Direct evidence

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public