Training

Pinned sources DeepSeek-V4.1 technical report @ 2cba9e4 · no training code is published · How to read

  1. Start of training
  2. Start of the encoder’s training
  3. Vision encoder: contrastiveDeepSeek-ViT≈ 47B pairsnot statedContrastiveSigLIPViT
  4. Vision encoder: autoregressiveDeepSeek-ViT236Bnot statedNTPViT
  5. DeepSeek-ViT
  6. Pre-training 0–28TDeepSeek-V4.1-Flash28T64KNTPBalanceweight 0.0001not ViTsparse from scratch
  7. Pre-training 28–34TLR decay begins6T64KNTPBalanceweight 0.0001all
  8. Pre-training 34–45TDeepSeek-V4.1-Flash11T1MNTPBalanceweight 0.0001all
  9. DeepSeek-V4.1-Flash-Base
  10. DSpark stageDeepSeek-V4.1-Flashnot statednot statednot statedDSparkDSpark added
  11. SFTDeepSeek-V4.1-Flashnot statednot statedSFTnot statedpost-training changes
  12. RLDeepSeek-V4.1-Flashnot statednot statedRLgroup-relativenot stated
  13. On-policy distillationDeepSeek-V4.1-Flashnot statednot statedOPDover 40 teachersnot stated
  14. DeepSeek-V4.1-Flash
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained

On-policy distillation

×

Stage · DeepSeek-V4.1-Flash

The last stage: full-vocabulary on-policy distillation on datasets from all domains, with over 40 teacher models.

Teachers
over 40, from all domains
Objective
full-vocabulary on-policy distillation
Recipe
Asynchronous generation; the dataset mixture, per-dataset concurrency limits and active teachers may be adjusted during training

Not disclosed Context of this stage · What is trained · Loss · Steps · Learning rate · Tokens

Architecture at this stageStated for post-training as a whole, with no stage named: the Hierarchical Sparse Indexer is switched on, the main KV cache is trained in FP4, and DSpark trains beside the backbone with its gradient cut.

All settings9
Place
The last stage of post-training
Teachers
Over 40, on datasets from all domains; a domain’s best teacher may come from another stage of development and differ in architecture
Objective
Full-vocabulary on-policy distillation; the report writes no equation
Recipe
Asynchronous generation; the dataset mixture, per-dataset concurrency limits and active teachers may be adjusted during training
DSpark
Trained alongside the backbone through post-training, with no gradient from its objective into the backbone
Hierarchical Sparse Indexer
“Training-aware and introduced in post-training”: later indexers score a candidate pool an earlier one selected
Number format
Quantization-aware training for the main KV cache in FP4, introduced in post-training; the sliding-window KV cache stays FP8
Replay
The decoder’s bounded replay of the sliding window is simulated in post-training
Long sequences
“A substantial share of training occurring at ultra-long sequence lengths in both the pre-training and post-training stages”, of sequences up to one million tokens
Sources6

DeepSeek V4.1

All models, with previews →

DeepSeek · Sep 2026New parts in the block · after DeepSeek-V3.2-Exp in the atlas