LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder

Initial decoder state

×

tensor

The same prepared query at every position, told apart only by the position added.

shape
r0r_0 [B batch, T positions, 768 d]
Sources

Contextual evidence from Positions to queries

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder block

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder block

The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder block

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public