- in
- Raw text
- out
- [1]
- parameters
- 430,453,274 in all
Parameters
430,453,274 parameters in LDLM
| Component | Calculation | Parameters |
|---|---|---|
| GPT-2 Small | 124,441,344 across its components | 124,441,344 |
| Latent encoder | 51,161,856 across its components | 51,161,856 |
| Latent decoder | 55,098,650 across its components | 55,098,650 |
| Token decoder | 66,915,072 across its components | 66,915,072 |
| Denoiser | 132,836,352 across its components | 132,836,352 |
| Model total | 430,453,274 | |
Derived exact parameter total: 430,453,274. Sources contains the pinned configuration and implementation used for this derivation; checkpoint tensors are not independently counted.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Sources
Direct evidence
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The OpenWebText configuration sets sequences of 1,024 tokens.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The documented OpenWebText invocation: the joint task on 1,024-token sequences, decoder noise 3, hidden state −3, 1,024 diffusion steps on the adaptive grid.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The trainer builds the latent encoder, the latent decoder, the token decoder and the denoiser, the SDE and its Euler solver, and an exponential moving average of the denoiser’s parameters only.
The implementation, pinned at revision e18c42acdfb7, not yet public