Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  9. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  10. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  11. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Annealing

×

Stage · Llama 3.1 405B

The learning rate goes linearly to zero over the last 40M tokens, on upsampled data of very high quality; the checkpoints are averaged.

Tokens
the final 40M
Learning rate
linearly to 0
Averaging
Polyak averaging of checkpoints during annealing gives the final pre-trained model

Derived

Objective. The paper describes this stage as continued pre-training and names no other objective: next-token prediction is concluded from that, and the paper does not state it for the stage.

Trained. The paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.

Why

Annealing on small amounts of high-quality code and mathematics data can boost key benchmarks: with the GSM8k and MATH training sets in the annealing mix, a pre-trained 8B model rose by 24.0% and 6.4% on their validation sets, while for the 405B the gain was negligible. The final annealing data excludes benchmark training sets. Llama 3 paper §3.1.3, p. 6

Architecture at this stageThe same model; its checkpoints over the stage are averaged into the pre-trained model.

All settings8
Tokens
The final 40M, as printed
Optimizer steps
About 2 to 3derived40M tokens over Table 4’s 16M tokens per batch is 2.5; the paper does not count the steps.
Learning rate
Linearly annealed to 0
Context
128K
Data
Sources of very high quality upsampled; no training sets of common benchmarks
Averaging
Polyak averaging of checkpoints during annealing gives the final pre-trained model
Objective
Next-token predictionderivedThe paper describes this stage as continued pre-training and names no other objective: next-token prediction is concluded from that, and the paper does not state it for the stage.
Trainable
AllderivedThe paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.
Sources5

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B