DPO step

Reports The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is published · How to read

Promptx x y[B, 2, T]Mask formatting tokens m[B, 2, T]Policy

\textstyle\sum_{m_t=1}\log\pi_\theta(y_t\mid\cdot)

l_P[B, 2]Reference

\textstyle\sum_{m_t=1}\log\pi_{\mathrm{ref}}(y_t\mid\cdot)

l_R[B, 2]DPO loss

-\log\sigma\big(\beta[(l_P-l_R)_w-(l_P-l_R)_l]\big)

ℒ_DPO[1]NLL on the chosen

\mathcal L_{\mathrm{NLL}}=-\log\pi_\theta(y_w\mid x)

ℒ_NLL[1]Add, 0.2 on the NLL

\mathcal L=\mathcal L_{\mathrm{DPO}}+0.2\,\mathcal L_{\mathrm{NLL}}

ℒ[1]hatched band: not updated in this step · solid band: what this stage trainsDPO masks formatting tokens · NLL reads policy token likelihoods; its mask is not stated

DPO step

Policy

learned · not counted here

The model being trained: its log-probability of each response, over the tokens the mask keeps.

\ell_\theta(y)=\sum_{t:\,m_t=1}\log\pi_\theta(y_t\mid {\htmlData{light=prompt}{x}},y_{<t})

in: $x$ [B batch] $m$ [B batch, 2, T positions]
out: $l_P$ [B batch, 2]

$m_t=1$ marks a token the mask keeps. $\ell_\theta$ , drawn $l_P$ , is this plate’s name.

Sources2

Direct evidence

§4, eq. 7: L_DPO(π_θ; π_ref) = −E[log σ(β log π_θ(y_w|x)/π_ref(y_w|x) − β log π_θ(y_l|x)/π_ref(y_l|x))], over preference triples (x, y_w, y_l).

dpo · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · p. 4

Direct evidence

§4.1.4, Direct Preference Optimization: DPO on the SFT models with learning rate 10⁻⁵ and β = 0.1; special formatting tokens, including header and termination tokens, are masked out of the loss in chosen and rejected responses; an NLL term with scaling coefficient 0.2 is added on the chosen sequences.

meta · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16