Vision encoder: autoregressive
×Stage · DeepSeek-ViT
The encoder is fine-tuned by next-token prediction through a 4B MoE language model, at high resolution; the language model is then discarded.
- Tokens
- 236B
- Objective
- Next-token prediction, with the encoder connected to a 4B MoE LLM
- Resolution
- 544 × 544 to 1344 × 1344
Not disclosed Whether the 4B LLM trains · Steps · Batch · Learning rate · Optimizer
Why
This stage is there to sharpen the encoder’s fine-grained visual features, at the resolutions pre-training will use. DeepSeek-V4.1 report §4.2.2, p. 23
Architecture at this stageThe vision encoder feeding a 4B MoE language model, which is discarded after the stage.
All settings4
- Objective
- Next-token prediction, with the encoder connected to a 4B MoE LLM
- Tokens
- 236B, over datasets including image captions, alt text, charts and OCR
- Resolution
- Between 544 × 544 and 1344 × 1344, out-of-bound images scaled proportionally
- Trainable
- The vision encoder is fine-tuned; the LLM is discarded after the stage, and whether it trains is not stated
Sources1
- DeepSeek_V41_Tech_Report.pdf, p. 23Objective, Tokens, Resolution, Trainable