LDLM · Sampler · OpenModelAtlas

Sampler

learned

1,024 calls of the denoiser along a time grid from 1 to 0.05: each call predicts the clean latents and one Euler–Maruyama step moves the latents towards them.

Why

At inference Gaussian noise is drawn and the reverse diffusion is run with the Euler–Maruyama solver to a clean latent prediction, which is decoded without the decoder’s input noise. LDLM paper §4.3

All at once. A continuous model refines all positions gradually and need not commit to a token before sampling is complete: tokens are read out only at the end. LDLM paper §1

Cost. Sampling is cheap for what it repeats: only the denoiser runs at every call, and it has no token embedding matrix and no vocabulary-sized output. The frozen token encoder and the latent encoder are not used at all, and the two decoders, with the vocabulary projection, run once after denoising. LDLM paper Appendix C

f=-\tfrac12\,\beta_t\,z_t-\beta_t\,s

\beta_t=\frac{\pi\,9\,\tau\,(1+\tau^2)}{1+9\,\tau^2}

\tau=\tan\!\big(\tfrac{\pi}{2}\min(t,\,0.95)\big)

m_{t'}=z_t+f\,(t'-t)

All 26 steps

t_k=\max\!\big(F^{-1}(1-{\htmlClass{vl-stream-k}{k}}/N),\;0.05\big)

F(u_{i+1})=\textstyle\sum_{j\le i}p_j\ \text{and linear between edges}

{\htmlClass{vl-stream-k}{k}}=0,\ldots,N

N=1,024

\text{Denoiser:}

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

e_t=\big[\cos(t\,\omega_j)\big]_{j}\,\Vert\,\big[\sin(t\,\omega_j)\big]_{j}

\omega_j=10^{-j/384},\;j=0,\ldots,383

e^{\uparrow}=e_tW_{1}+b_{1}

e^{\sigma}=e^{\uparrow}\,\sigma(e^{\uparrow})

e^{\downarrow}=e^{\sigma}W_{2}+b_{2}

a_t=e^{\downarrow}\,\sigma(e^{\downarrow})

m_n=[\,b_a\;s_a\;g_a\;b_m\;s_m\;g_m\,]=a_tW_{n}+c_{n}

\bar x_{\mathrm{attn}}=\gamma_{1}\odot\frac{x_n-\mu(x_n)}{\sqrt{\sigma^2(x_n)+\varepsilon}}

\varepsilon=10^{-5}

x_{\mathrm{attn}}=\bar x_{\mathrm{attn}}\odot(1+s_{a})+b_{a}

\tilde\Delta_{\mathrm{attn}}=g_{a}\odot \Delta_{\mathrm{attn}}

x'_n=x_n+\tilde\Delta_{\mathrm{attn}}

\bar x_{\mathrm{mlp}}=\gamma_{2}\odot\frac{x'_n-\mu(x'_n)}{\sqrt{\sigma^2(x'_n)+\varepsilon}}

\varepsilon=10^{-5}

x_{\mathrm{mlp}}=\bar x_{\mathrm{mlp}}\odot(1+s_{m})+b_{m}

\tilde\Delta_{\mathrm{mlp}}=g_{m}\odot \Delta_{\mathrm{mlp}}

x_{n+1}=x'_n+\tilde\Delta_{\mathrm{mlp}}

\bar x=\gamma_{\mathrm{out}}\odot\frac{x_{12}-\mu(x_{12})}{\sqrt{\sigma^2(x_{12})+\varepsilon}}

\varepsilon=10^{-5}

m_{\mathrm{out}}=[\,b_o\;s_o\,]=a_tW_{o}+c_{o}

x_{\mathrm{out}}=\bar x\odot(1+s_o)+b_o

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

\hat\epsilon=\frac{z_t-\alpha_t\,\hat z_0}{\sigma_t}

s=-\frac{\hat\epsilon}{\sigma_t}

f=-\tfrac12\,\beta_t\,z_t-\beta_t\,s

\beta_t=\frac{\pi\,9\,\tau\,(1+\tau^2)}{1+9\,\tau^2}

\tau=\tan\!\big(\tfrac{\pi}{2}\min(t,\,0.95)\big)

m_{t'}=z_t+f\,(t'-t)

z_{t'}=m_{t'}+\sqrt{\beta_t}\,\sqrt{t-t'}\;\xi

\xi\sim\mathcal N(0,I)

in: $z_t$ [B batch, T positions, 768 d]
out: $m_{t'}$ [B batch, T positions, 768 d]
parameters: 132,836,352

Parameters

132,836,352 parameters per call, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Denoiser	132,836,352 across its components	132,836,352
Per-instance total		132,836,352

Across 1024 shared calls: $132{,}836{,}352 \times 1 = 132{,}836{,}352$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Sources7

Direct evidence

Sampling starts from the prior with a zero self-conditioning estimate, uses the averaged denoiser, takes the adaptive grid when it is requested and the sampler has been updated and a uniform grid otherwise, steps N times carrying the state and the clean estimate, and returns the last step’s mean.