LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Gated MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_mlp[B, T, 768]Gate projection

g=y_{\mathrm{mlp}}W_{\mathrm{gate}}

g[B, T, 3072]Up projection

u=y_{\mathrm{mlp}}W_{\mathrm{up}}

u[B, T, 3072]SiLU

a = \operatorname{SiLU}(g)

a[B, T, 3072]Element-wise product

h = a \odot u

h[B, T, 3072]Down projection

\Delta_{\mathrm{mlp}}=hW_{\mathrm{down}}

Δ_mlp[B, T, 768]

Token decoder › 3 blocks › Gated MLP

Gated features

tensor

The gated features supplied to the down projection.

shape: $h$ [B batch, T positions, 3072 d_ff]

Sources

Contextual evidence from Element-wise product

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public