Rejection sampling and SFT

Reports The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is published · How to read

x[B]Latest policy

{\htmlClass{vl-stream-k}{K}}\ \text{responses},\ {\htmlClass{vl-stream-k}{K}}\approx 10\text{–}30

Y[B, K, T]Reward model r[B, K]Keep the bestHighest score.y[B, T]Model P[B, T, V]Cross-entropy, prompt masked

\mathcal L_{\mathrm{SFT}}=-\sum_{t\in y}\log P(y_t\mid x,y_{<t})

ℒ_SFT[1]hatched band: not updated in this step · solid band: what this stage trainsthe reward model is trained beside this step, on the pre-trained checkpoint

Rejection sampling and SFT

Cross-entropy, prompt masked

no parameters

The standard cross-entropy on the response’s tokens; the prompt’s tokens are masked out of the loss.

\mathcal L_{\mathrm{SFT}}=-\sum_{t\in y}\log {\htmlData{light=dist model}{P}}(y_t\mid x,y_{<t})

in: $P$ [B batch, T positions, V]
out: $\mathcal L_{\mathrm{SFT}}$ [1]

The plate writes the sum over the response’s tokens for the masking the paper states; how the loss is averaged is not said.

Sources1

Direct evidence

§4.1.3: the reward model performs rejection sampling on human annotation prompts; with this data and other sources, including synthetic data, the pre-trained language model is fine-tuned with a standard cross-entropy loss on the target tokens, the loss masked on prompt tokens; learning rate 10⁻⁵ over 8.5K to 9K steps for the largest models.

sft · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16