Training

Pinned sources The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta-llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is published · How to read

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  8. Off this path: Reward model1 stage
  9. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  10. Reward model
  11. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  12. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  13. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Initial pre-training

×

Stage · Llama 3.1 405B

Next-token prediction on about 15.6T tokens, at a context of 8K after the first 252M tokens.

Tokens
15.6T
Batch and context
4M tokens at 4,096, then 8M at 8,192, then 16M
Learning rate
peak 8×10⁻⁵, cosine to 8×10⁻⁷
Data mix
roughly 50% general, 25% maths and reasoning, 17% code, 8% multilingual
Compute
3.8×10²⁵ FLOPs

Derived

Trained. The paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.

Not disclosed Tokens of Llama 3.1 8B and 70B

Why

Long sequences come only at the end, since the compute of self-attention grows quadratically with the sequence length. Llama 3 paper §3.4.2, p. 14

Architecture at this stageThe model as Llama 3.1’s plates draw it at 405B, its context at 8K in this stage.

All settings9
Objective
Next-token prediction
Tokens
15.6T. The paper attaches the number to the 8K stage on p. 3 and to the flagship’s pre-training on p. 1, where it also speaks of “a corpus of about 15T multilingual tokens”; whether it includes the ~800B of long context is not clear
Data mix
About 50% general knowledge, 25% mathematics and reasoning, 17% code, 8% multilingual; adjusted during training
Optimizer
AdamW
Learning rate
Peak 8×10⁻⁵, linear warm-up of 8,000 steps, cosine decay to 8×10⁻⁷ over 1,200,000 steps; the peak is 3×10⁻⁴ for 8B and 1.5×10⁻⁴ for 70B
Batch and context
4M tokens of 4,096-token sequences; 8M tokens of 8,192-token sequences after 252M tokens; 16M tokens after 2.87T tokens. The paper writes “8M sequences of 8,192 tokens”; read as tokens from “double these values” and Table 4
Compute
3.8×10²⁵ FLOPs for the flagship’s pre-training
8B and 70B
“Similar recipes” to the 405B’s; the smaller models are trained “for much longer than is compute-optimal”, and the flagship is used “to further improve the quality of those smaller models during post-training”; they gain from code data generated by a larger model, and their safety mixes are tailored to their size
Trainable
AllderivedThe paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.
Sources9

Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

405B