LDLM · Noise to time t · OpenModelAtlas

Add noise

Noise to time t

no parameters

Mix the latents with Gaussian noise in the proportion the schedule gives for the time.

{\htmlData{light=noised noise}{z_t}}=\alpha_t\,{\htmlData{light=gradient-scale}{z_0^{\gamma}}}+\sigma_t\,\epsilon

\alpha_t=\frac{1}{\sqrt{1+9\tan^2(\pi {\htmlData{light=sample-time}{t}}/2)}}

\sigma_t=\sqrt{1-\alpha_t^2}

\epsilon\sim\mathcal N(0,I)

in: $z_0^{\gamma}$ [B batch, T positions, 768 d] $t$ [B batch]
out: $z_t$ [B batch, T positions, 768 d]

Explanation

The schedule is variance preserving with a tangent form and $d=3$ : at $t=0$ the latents are clean, at $t=1$ only noise is left.

Sources4

Direct evidence

The marginal: x_t = μ x_0 + σ ε.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The tangent schedule: α = 1/√(1 + d² tan²(πt/2)), σ = √(1 − α²); β(t) is evaluated with t clipped at 0.95.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: a time is sampled, the latents are noised by the marginal, with probability one half a first call without gradient gives the self-conditioning estimate, the denoiser predicts the clean latents, and the mean squared error against them is taken; the three losses are summed with their weights.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction