LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Token decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Token decoder

Vocabulary readout

×

learned

Map each position’s 768 features to a score for every one of the 50,259 tokens.

L=y3Wout{\htmlData{light=output readout}{L}}={\htmlData{light=stack-output blocks}{y_{3}}}W_{\mathrm{out}}
in
y3y_{3} [B batch, T positions, 768 d]
out
LL [B batch, T positions, 50259 vocabulary]
parameters
38,598,912
Parameters

38,598,912 parameters owned by this component

Exact per-instance accounting
ComponentCalculationParameters
weights, no bias768×50,259768 \times 50{,}25938,598,912
Per-instance total38,598,912

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

A matrix of 50,259 rows of 768 features, without bias and tied to nothing.

Explanation

The features are not normalized before it.

Sources3

Direct evidence

The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Before any network is built, the configured vocabulary size and mask token ID are overwritten with the tokenizer’s.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public