LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Key and value heads

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

The decoder state’s queries, keys and valuesp^a p^a The latents’ keys and valuesp^b p^b Take the decoder state’s own keys Take the latents’ keys Join keysalong positions Split heads Key RMSNorm

\lVert x\rVert_2/\sqrt d

K[B, 12, 2T, 64]Take the decoder state’s own values Take the latents’ values Join valuesalong positions Split heads V[B, 12, 2T, 64]

in: $p^{a}$ [B batch, T positions, 2304 q, k, v] $p^{b}$ [B batch, T positions, 1536]
out: $K$ [B batch, 12 heads, 2T positions, 64 head features] $V$ [B batch, 12 heads, 2T positions, 64 head features]
parameters: 64 · × 6 blocks = 384
mode: generation

Parameters

64 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Key RMSNorm	$64$	64
Per-instance total		64

Across 6 independent blocks: $64 \times 6 = 384$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

The keys and the values of the two streams lie in one sequence of 2T positions, so one softmax weighs both. Values are left as they are: only queries and keys are normalized.

Sources5

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public