Kimi K2

All models, with previews →

Moonshot AI · Jul 2025Latent attention and experts

SwiGLU MLP

Checkpoint configuration and implementation moonshotai/Kimi-K2-Instruct @ fd1984e · parameter count checked against the checkpoint’s tensor headers · How to read

61 transformer blocks › 1 dense block › SwiGLU

Gate projection

×

learnedself.gate_proj(x)

Map 7,168 input features to 18,432 gate features.

g=xffnW1{\htmlData{light=g gate}{g}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{1}
in
xffnx_{\mathrm{ffn}} [B batch, T positions, 7168 d]
out
gg [B batch, T positions, 18432 d_ff]
parameters
132,120,576
Parameters

132,120,576 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix7,168×18,4327{,}168 \times 18{,}432132,120,576
Per-instance total132,120,576

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 1 independent dense blocks: 132,120,576×1=132,120,576132{,}120{,}576 \times 1 = 132{,}120{,}576 parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_deepseek.py 374–390 · config.json 38–46
modeling_deepseek.py 374–390
class DeepseekV3MLP(nn.Module):
    def __init__(self, config, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = (
            config.intermediate_size if intermediate_size is None else intermediate_size
        )

        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
config.json 38–46
  "quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },
Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP: down(silu(gate x) ⊙ up x).

mlp · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 374–390

Read code excerpt
class DeepseekV3MLP(nn.Module):
    def __init__(self, config, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = (
            config.intermediate_size if intermediate_size is None else intermediate_size
        )

        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Direct evidence

Weights are stored in FP8 (e4m3) with one scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.

linear · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 38–46

Read code excerpt
  "quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },