LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent encoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder

Position table

×

learned

A learned vector for each of the 1,024 positions, scaled by 768^(−1/2).

Ep, b,t=768−1/2 PtE_{p,\,b,t}=768^{-1/2}\,P_t
out
EpE_{p} [B batch, T positions, 768 d]
parameters
786,432
Parameters

786,432 parameters owned by this component

Exact per-instance accounting
ComponentCalculationParameters
table1,024×7681{,}024 \times 768786,432
Per-instance total786,432

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

1,024 rows of 768 features.

Explanation

The rows are multiplied by 768−1/2≈0.036768^{-1/2}\approx0.036 before they are added. They are the same for every sequence of the batch.

Sources3

Direct evidence

The position table is a learned embedding whose rows are scaled by width^(−1/2).

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public