LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

m[B, T, 768]Position tableE_p E_p Positions to latents

z^{p}=m+E_{p}

z^p[B, T, 768]Mask vector

E[\texttt{<mask>}]

Query MLP Open →

(1+s_e)

Positions to queries

r_0=\tilde e+E_{p}

r₀[B, T, 768]

6 decoder blocks Open →

Latent decoder

Reconstructed features

tensor

The decoder’s reconstruction of the standardized GPT-2 features.

shape: $\hat h$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Latent decoder

The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder block

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder block

The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.

The implementation, pinned at revision e18c42acdfb7, not yet public