Kimi K3 · training steps · Clipped log-ratio

MOPD reward

Clipped log-ratio

no parameters

A token is rewarded where the teacher finds it more probable than the student does, and penalized where less. $R_{\max}>0$ is a clipping threshold “to constrain extreme advantage signals”.

\begin{aligned}&r^{d}_{\mathrm{opd}}(y_t\mid e,x,y_{<t})=\\&\operatorname{clip}\Big(\operatorname{sg}\Big(\log\frac{\pi^{(d,e)}_{\mathrm{teacher}}(y_t\mid x,y_{<t})}{\pi_\theta(y_t\mid e,x,y_{<t})}\Big),-{\htmlClass{vl-stream-context}{R_{\max}}},{\htmlClass{vl-stream-context}{R_{\max}}}\Big)\end{aligned}

in: $p_T$ [B batch, T positions] $p_S$ [B batch, T positions]
out: $r$ [B batch, T positions]

Explanation

The stop-gradient makes the ratio a number, and no path for a gradient to either model: the student learns through the update that uses the reward. On the plate $r_t$ stands for $r^{d}_{\mathrm{opd}}(y_t\mid e,x,y_{<t})$ .

Sources1

Direct evidence

§4.1.3, Multi-Teacher On-Policy Distillation: for a domain d and a sampled reasoning effort e ∈ {low, high, max}, optimization is guided by the teacher of that pair among the nine experts. The per-token reward on y_t is the stop-gradient log-ratio of the teacher’s probability to the student’s, clipped to ±R_max (eq. 15). The dense reward integrates into the RL framework, partial rollouts included.

mopd · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · pp. 13–14