LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Perceiver attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Latent encoder › 6 encoder blocks › Perceiver attention

Merge heads

no parameters

Concatenate the 12 heads into 768 features.

[B,\,12,\,T,\,64]

\to\;[B,\,T,\,12,\,64]

\to\;[B,\,T,\,768]

in: $O$ [B batch, 12 heads, T positions, 64 head features]
out: $o$ [B batch, T positions, 768 d]

Explanation

Heads are concatenated in order: head $i$ fills features ${64i},\dots,{64i+63}$ .

Sources1

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public