Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  2. Llama 3, April 2024, predecessor2 stages
  3. Whether Llama 3.1 continues these checkpoints: not stated
  4. Llama 3 pre-training15T+ tokens at 8K15T8Knot statednot stated
  5. Llama 3 post-trainingSFT, RS, PPO, DPOnot statednot statedSFTPPODPOnot stated
  6. Llama 3 (April 2024)
  7. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  9. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  10. Annealinglearning rate to 040M128KNTPall
  11. Pre-trained model
  13. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  14. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  15. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Llama 3 pre-training

×

Stage · Llama 3 (April 2024)

Llama 3 was pretrained on over 15 trillion tokens of publicly available data, on sequences of 8,192 tokens with a mask that keeps attention within a document.

Tokens
over 15T
Context
8,192, attention within a document
Compute
1.3M and 6.4M GPU hours

Not disclosed Objective · Learning rate · Batch

Architecture at this stageLlama 3: the same shapes at 8B and 70B as Llama 3.1, a context of 8,192 and unscaled RoPE.

All settings3
Tokens
Over 15T (“15T+”), publicly available data
Context
Sequences of 8,192 tokens, with a mask so that self-attention does not cross document boundaries
Compute
1.3M GPU hours for 8B, 6.4M for 70B
Sources5

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B