Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  2. Llama 3, April 2024, predecessor2 stages
  3. Whether Llama 3.1 continues these checkpoints: not stated
  4. Llama 3 pre-training15T+ tokens at 8K15T8Knot statednot stated
  5. Llama 3 post-trainingSFT, RS, PPO, DPOnot statednot statedSFTPPODPOnot stated
  6. Llama 3 (April 2024)
  7. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  8. Off this path: domain experts (code, multilingual)2 stages
  9. Branched from the pre-training run; the point is not named
  10. Code expert1T tokens, mostly code1T16KNTPnot stated
  11. Multilingual expert90% multilingual tokensnot statednot statedNTPnot stated
  12. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  13. Annealinglearning rate to 040M128KNTPall
  14. Pre-trained model
  15. Off this path: Reward model1 stage
  16. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  17. Reward model
  18. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  19. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  20. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Rejection sampling and SFT

×

Stage · Llama 3.1 405B

For each human-annotation prompt the latest policy writes typically 10 to 30 responses and the reward model keeps the best; the pre-trained model is fine-tuned on these and on synthetic and other data, the loss on the prompt masked.

Rounds
six, each an SFT followed by DPO
Rejection sampling
typically 10 to 30 outputs, the best by the reward model
Learning rate and steps
10⁻⁵, 8.5K to 9K steps
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage

Not disclosed Tokens · Batch

The training step →What one step of this stage computes, as a plate.

Architecture at this stageThe pre-trained checkpoint fine-tuned; the previous round’s best model produces the round’s data.

All settings6
Rounds
Six rounds, each an SFT followed by DPO. Each round’s data are generated with the best models of the previous round: “Best model for next round” (Figure 7), “sampling synthetic data from the latest models”. The paper says SFT fine-tunes pre-trained checkpoints; that a round starts from the previous round’s model is not stated.
Rejection sampling
K outputs, typically 10 to 30, from the latest chat policy for each prompt; the reward model selects the best
Objective
Cross-entropy on the target tokens, the loss masked on the prompt tokens
Learning rate and steps
10⁻⁵ over 8.5K to 9K steps for the largest models
Context
Mostly short; 0.1% of synthetic long-context data at 16K, 32K, 64K and 128K
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
Sources5

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B