Kimi K3

Moonshot AI · Release date not pinnedNew parts in the block · after Kimi K2 in the atlas

Draft fine-tuning step

Reports Kimi K3 technical report @ 3cb39df · Kimi K2.5 report (arXiv 2602.02276v2), for the RL step · no training code is published · How to read

X[B, T]Target model h[B, T, 3·7168]Feature fusion

f=W_{\mathrm{E3}}[h_l;h_m;h_h]

f[B, T, 7168]Draft layer

\times 7\ \text{steps}

q[B, T, V]p[B, T, V]LK loss

\mathcal L_{\mathrm{LK}}=-\log\Big(\sum_{x\in\mathcal {\htmlClass{vl-stream-v}{V}}}\min\big(p(x),{\htmlClass{vl-stream-q}{q}}(x)\big)\Big)

ℒ_LK[1]hatched band: not updated in this step · solid band: what this stage trains

Draft fine-tuning step

Three features

tensor

Low-, mid- and high-level features: the outputs of the 1st, the 4th and the final AttnRes block, concatenated.

shape: $h$ [B batch, T positions, 3·7168]

The width is derived: the fusion matrix starts as $[\,0\ \ 0\ \ I\,]$ and yields the hidden size, 7,168, so each feature has that width. The report does not give it.

Sources

Contextual evidence from Target model

§4.1.4, Draft Model Fine-Tuning: the pre-trained MTP layer is fine-tuned into an EAGLE-3-style draft model, with the target model frozen and only the draft layer and its feature-fusion projection updated; the draft is unrolled for seven steps and, beyond the first, consumes its own outputs. The input concatenates the outputs of the 1st, 4th and final AttnRes blocks and projects them by a bias-free matrix W_E3 initialized [0 0 I]. The loss is L_LK = −log Σ_x min(p(x), q(x)) (eq. 16), at temperature 1, with no cross-entropy term.

draft · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 14

Contextual evidence from Target model

Table 1: hidden dimension 7,168; vocabulary size 160K; 1 MTP layer.

sizes · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 11