LDLMIn construction

Contextual evidence from Sampler

Sampling starts from the prior with a zero self-conditioning estimate, uses the averaged denoiser, takes the adaptive grid when it is requested and the sampler has been updated and a uniform grid otherwise, steps N times carrying the state and the clean estimate, and returns the last step’s mean.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

One Euler–Maruyama step: the mean is x + drift · dt, and the next state adds diffusion · √(−dt) · noise; the clean estimate is returned with them.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

The reverse SDE: drift −β x/2 − β · score and diffusion √β; the ODE halves the score term and has no diffusion.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

The denoiser’s clean prediction is converted into the noise estimate and the score with the marginal’s mean and standard deviation.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

The generation grid: N + 1 times at equal steps of the weights’ cumulative distribution, interpolated inside bins, clamped to the stopping time, in decreasing order.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sampler

The documented OpenWebText invocation: the joint task on 1,024-token sequences, decoder noise 3, hidden state −3, 1,024 diffusion steps on the adaptive grid.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent decoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public