Kimi K3 · training steps · Feature fusion

Draft fine-tuning step

Feature fusion

learned · not counted here

A bias-free matrix projects the three features to the hidden size. It starts as the high-level feature alone, the input the MTP layer was pre-trained on, and learns to use the other two.

{\htmlData{light=fused fuse}{f}}=W_{\mathrm{E3}}\,[\,h_l\,;\,h_m\,;\,{\htmlData{light=features target}{h_h}}\,],\qquad W_{\mathrm{E3}}\leftarrow[\,0\ \ 0\ \ I\,]\ \text{at the start}

in: $h$ [B batch, T positions, 3·7168]
out: $f$ [B batch, T positions, 7168 d]

Trained in this step. It is not among the published weights, so the model’s plates do not count it.

Sources1

Direct evidence

§4.1.4, Draft Model Fine-Tuning: the pre-trained MTP layer is fine-tuned into an EAGLE-3-style draft model, with the target model frozen and only the draft layer and its feature-fusion projection updated; the draft is unrolled for seven steps and, beyond the first, consumes its own outputs. The input concatenates the outputs of the 1st, 4th and final AttnRes blocks and projects them by a bias-free matrix W_E3 initialized [0 0 I]. The loss is L_LK = −log Σ_x min(p(x), q(x)) (eq. 16), at temperature 1, with no cross-entropy term.

draft · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 14