Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  4. Off this path: domain experts (code, multilingual)2 stages
  5. Branched from the pre-training run; the point is not named
  6. Code expert1T tokens, mostly code1T16KNTPnot stated
  7. Multilingual expert90% multilingual tokensnot statednot statedNTPnot stated
  8. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  9. Annealinglearning rate to 040M128KNTPall
  10. Pre-trained model
  12. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  13. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  14. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Code expert

×

Stage · Llama 3.1 405B

A branch of the pre-training run continues pre-training on 1T tokens of mostly code, extends its context to 16K and is post-trained; it is used to collect code annotations and to rejection-sample code.

Tokens
1T
Data
>85% code
Context
16K at the end
Use
Collecting high-quality human annotations for code throughout later rounds of post-training

Derived

Objective. The paper calls this “continuing pre-training” on another data mix and names no objective: next-token prediction is concluded from that, and the paper does not state it.

Not disclosed Learning rate

Architecture at this stageA copy of the model, branched from its pre-training run.

All settings5
Tokens
1T
Data
Mostly (>85%) code
Context
Long-context fine-tuning to 16K tokens over the last several thousand steps, on repo-level code
Post-training
The recipes of §4.1, with SFT and DPO mixes primarily for code
Use
Collecting high-quality human annotations for code throughout later rounds of post-training
Sources1

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B