Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  8. Off this path: Reward model1 stage
  9. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  10. Reward model
  11. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  12. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  13. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Reward model

×

Stage · Llama 3.1 405B

A reward model is trained on top of the pre-trained checkpoint, from human preference data ranked edited > chosen > rejected.

Start
The pre-trained checkpoint
Objective
Llama 2’s, without the margin term
Data
preference data rated significantly better or better, ranked edited > chosen > rejected

Not disclosed Learning rate · Steps

Architecture at this stageA copy of the pre-trained model trained to score responses.

All settings5
Start
The pre-trained checkpoint
Objective
Llama 2’s, without the margin term
Data
All preference data available in the round, only samples rated significantly better or better, similar responses filtered out; two or three ranked responses concatenated in one row, shuffled
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
The loss’s formula
By reference to Llama 2’s paper, which the atlas has not pinned
Sources3

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B