Kimi K2

Moonshot AI · Jul 2025Latent attention and experts

Checkpoint configuration and implementation moonshotai/Kimi-K2-Instruct @ fd1984e · parameter count checked against the checkpoint’s tensor headers · How to read

Raw text Tokenizer datafixed assets TokenizerChat template Token IDs[B, T]Token embedding

E

163,840 × 7,168 x₀[B, T, 7168]Position IDs

0,\ldots,T-1

Generate RoPE coefficients

{\htmlClass{vl-stream-context}{R_{t,j}}}=e^{\mathrm i t\omega_j}

R[T, 32]Construct causal mask

{\htmlClass{vl-stream-context}{M_{t,s}}}=\begin{cases}0&s\le t\\-\infty&s>t\end{cases}

M[T, T]

1 dense Open →

60 MoE Open →

x₆₁ Final RMSNorm

\operatorname{mean}(x^2)

learned gain Vocabulary projection

L=zW_{\mathrm{out}}

7,168 × 163,840 Logits[B, T, 163840]

Raw text

tensormessages

Text supplied for tokenization.

Codetokenizer_config.json 156–164

tokenizer_config.json 156–164

  "tokenizer_class": "TikTokenTokenizer",
  "unk_token": "[UNK]",
  "auto_map": {
    "AutoTokenizer": [
      "tokenization_kimi.TikTokenTokenizer",
      null
    ]
  }
}

Sources1

Contextual evidence

The checkpoint’s tokenizer is the repository’s own TikTokenTokenizer class, loaded through AutoTokenizer.

tokenizer · moonshotai/Kimi-K2-Instruct / tokenizer_config.json
Pinned revision fd1984e2b7a3 ↗ · lines 156–164

Read code excerpt

  "tokenizer_class": "TikTokenTokenizer",
  "unk_token": "[UNK]",
  "auto_map": {
    "AutoTokenizer": [
      "tokenization_kimi.TikTokenTokenizer",
      null
    ]
  }
}