- out
- [1]
Explanation
The objective is the DPO paper’s (eq. 7). Meta’s two changes are in the Llama 3 paper: formatting tokens are masked out of the loss in both responses, and an NLL term with coefficient 0.2 is added on the chosen sequences. Learning rate , .
The Llama 3 paper does not name the reference model for this DPO; the DPO paper takes it to be the SFT model the policy starts from. The paper does not say whether the NLL term is averaged over tokens, nor the batch size.
Sources4
Direct evidence
§4, eq. 7: L_DPO(π_θ; π_ref) = −E[log σ(β log π_θ(y_w|x)/π_ref(y_w|x) − β log π_θ(y_l|x)/π_ref(y_l|x))], over preference triples (x, y_w, y_l).
dpo · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · p. 4
Direct evidence
§3–4: β controls the deviation from the reference policy π_ref, “namely the initial SFT model”; π_ref is initialized to π_SFT whenever available.
ref · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · pp. 4–5
Direct evidence
§4.1.4, Direct Preference Optimization: DPO on the SFT models with learning rate 10⁻⁵ and β = 0.1; special formatting tokens, including header and termination tokens, are masked out of the loss in chosen and rejected responses; an NLL term with scaling coefficient 0.2 is added on the chosen sequences.
meta · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16
Direct evidence
§4.1.4 and §4.2.1: mostly the latest batches of preference data, collected with the best models of the previous rounds; only samples whose chosen response is rated significantly better or better than the rejected one.
prefs · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · pp. 16–17