Rejection sampling and SFT

Reports The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is published · How to read

x[B]Latest policy

{\htmlClass{vl-stream-k}{K}}\ \text{responses},\ {\htmlClass{vl-stream-k}{K}}\approx 10\text{–}30

Y[B, K, T]Reward model r[B, K]Keep the bestHighest score.y[B, T]Model P[B, T, V]Cross-entropy, prompt masked

\mathcal L_{\mathrm{SFT}}=-\sum_{t\in y}\log P(y_t\mid x,y_{<t})

ℒ_SFT[1]hatched band: not updated in this step · solid band: what this stage trainsthe reward model is trained beside this step, on the pre-trained checkpoint

Rejection sampling and SFT

Reward model

frozen · not trained

The reward model, trained on top of the pre-trained checkpoint, scores each candidate.

in: $Y$ [B batch, K, T positions]
out: $r$ [B batch, K]

Sources2

Direct evidence

§4.2.2, Rejection sampling: for each prompt collected during human annotation, K outputs (typically between 10 and 30) are sampled from the latest chat model policy, usually the best checkpoint of the previous post-training iteration or the best for a capability; the reward model selects the best candidate. In later rounds system prompts steer the responses’ tone, style or formatting.

rs · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 18

Direct evidence

§4.1.2: the reward model is trained on top of the pre-trained checkpoint, with Llama 2’s objective without the margin term.

rm · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16