Training

  1. Random initialisation
  2. Pre-trainingDeepSeek-V314.8T
  3. Context 32KDeepSeek-V3≈62.9B
  4. Context 128KDeepSeek-V3≈62.9B
  5. DeepSeek-V3-Base
  6. SFT → RLDeepSeek-V3tokens not disclosed
  7. DeepSeek-V3
  8. From V3’s base: before or after its context extension, not stated
  9. Context 32KDeepSeek-V3.1630B
  10. Context 128KDeepSeek-V3.1209B
  11. DeepSeek-V3.1-Base
  12. Post-trainingDeepSeek-V3.1tokens not disclosed
  13. DeepSeek-V3.1
  14. From V3.1 or from a base checkpoint: not stated
  15. UpdateDeepSeek-V3.1-Terminustokens not disclosed
  16. DeepSeek-V3.1-Terminus
  17. V3.1-Base or a later checkpoint: not stated
  18. V3.1-Terminus base checkpoint
  19. Dense warm-upDeepSeek-V3.2-Exp2.1B
  20. Sparse trainingDeepSeek-V3.2-Exp943.7B
  21. DeepSeek-V3.2 base
  22. Specialist distillationDeepSeek-V3.2-Exptokens not disclosed
  23. Mixed RLDeepSeek-V3.2-Exptokens not disclosed
  24. DeepSeek-V3.2-Exp

Context 128K

×

Stage · DeepSeek-V3

The second YaRN phase extends the context to 128K in 1,000 steps.

Steps, sequence, batch
1,000 steps; 128K tokens; batch 480
Learning rate
7.3×10⁻⁶
Tokens
1,000 × 480 × 131,072 ≈ 62.9BderivedSteps × sequences × sequence length.
Sources1
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts