LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Perceiver attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder › 6 encoder blocks › Perceiver attention

Head outputs

×

tensor

What each head retrieved from the two streams’ values.

shape
OO [B batch, 12 heads, T positions, 64 head features]
Sources

Contextual evidence from Attention over both streams

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Attention over both streams

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public