LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

GPT-2 Small

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Token IDs[B, T]Frozen GPT-2hidden state −3 h^raw[B, T, 768]Feature statisticsfixed assets Standardize

(h^{\mathrm{raw}}-\mu)/\sigma

GPT-2 Small

Token IDs

tensor

Integer vocabulary indices used to look up token embeddings.

shape: $I$ [B batch, T positions]

Sources

Contextual evidence from Tokenizer

With wrapping, each text is tokenized without special tokens and ends with the end token; the texts of a batch of the preprocessing are joined and cut into blocks of the sequence length less two, each set between a beginning and an end token, with every position valid; what is left over is dropped.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Tokenizer

Setting the tokenizer up wraps every encoded sequence in a beginning and an end token and adds a `[PAD]` token when the tokenizer has none.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Tokenizer

The OpenWebText configuration sets sequences of 1,024 tokens.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

The feature model is wrapped to return one entry of its hidden states, chosen by index; GPT-2’s token table is resized to the configured vocabulary.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

With an extraction layer other than −1 the encoder loads the wrapped feature model; when it is frozen it is put in evaluation mode and takes no gradient.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

With the feature model frozen, its hidden state is computed without gradient, standardized, and passed to the encoder; the same tensor is the target of the feature loss.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

Before any network is built, the configured vocabulary size and mask token ID are overwritten with the tokenizer’s.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Frozen GPT-2

The implementation pins torch 2.4.1 and transformers 4.48.0.

The implementation, pinned at revision e18c42acdfb7, not yet public