Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  8. Off this path: Reward model1 stage
  9. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  10. Reward model
  11. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  12. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  13. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Rejection sampling and SFT

×

Stage · Llama 3.1 405B

For each human-annotation prompt the latest policy writes typically 10 to 30 responses and the reward model keeps the best; the pre-trained model is fine-tuned on these and on synthetic and other data, the loss on the prompt masked.

Rounds
six, each an SFT followed by DPO
Rejection sampling
typically 10 to 30 outputs, the best by the reward model
Learning rate and steps
10⁻⁵, 8.5K to 9K steps
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage

Not disclosed Tokens · Batch

The training step →What one step of this stage computes, as a plate.

Architecture at this stageThe pre-trained checkpoint fine-tuned; the previous round’s best model produces the round’s data.

All settings6
Rounds
Six rounds, each an SFT followed by DPO. Each round’s data are generated with the best models of the previous round: “Best model for next round” (Figure 7), “sampling synthetic data from the latest models”. The paper says SFT fine-tunes pre-trained checkpoints; that a round starts from the previous round’s model is not stated.
Rejection sampling
K outputs, typically 10 to 30, from the latest chat policy for each prompt; the reward model selects the best
Objective
Cross-entropy on the target tokens, the loss masked on the prompt tokens
Learning rate and steps
10⁻⁵ over 8.5K to 9K steps for the largest models
Context
Mostly short; 0.1% of synthetic long-context data at 16K, 32K, 64K and 128K
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
Sources5

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B