LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Time embedding

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Denoiser › Time embedding

Sinusoidal encoding

×

no parameters

Encode the time as 384 cosines and 384 sines of geometrically spaced frequencies.

et=[cos⁡(t ωj)]j ∥ [sin⁡(t ωj)]j{\htmlData{light=sinusoid}{e_t}}=\big[\cos({\htmlData{light=t}{t}}\,\omega_j)\big]_{j}\,\Vert\,\big[\sin({\htmlData{light=t}{t}}\,\omega_j)\big]_{j}
ωj=10−j/384,  j=0,…,383\omega_j=10^{-j/384},\;j=0,\ldots,383
in
tt [B batch]
out
ete_t [B batch, 768 features]
Explanation

The frequencies fall from 1 to just above 1/10 radians per unit of time (max_period is 10), slow ones because the time itself only runs from 0 to 1.

Sources3

Direct evidence

The timestep embedding: cosines and sines of the time multiplied by frequencies falling geometrically from 1 to nearly 1/max_period.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public