Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  9. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  10. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  11. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

How Llama 3.1 405B was made: the stages on its path, top to bottom, and what changes from one to the next.

Of the 10 stages (5 on the path, 5 off it), the sources give or let us compute tokens for 5, and state or let us conclude the context length for 7, the objective for 9 and what is trained for 3; what is concluded is drawn dashed. The rest is hatched: nothing is filled in.

Select a stage for its numbers, its reason and the model as that stage trains it.

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B