61 transformer blocks › 1 dense block › SwiGLU
Gated features
×tensorself.act_fn(self.gate_proj(x)) * self.up_proj(x)
The gated features supplied to the down projection.
- shape
- [B batch, T positions, 18432 d_ff]
Sources
Contextual evidence from Element-wise product
The dense feed-forward network is a SwiGLU MLP: down(silu(gate x) ⊙ up x).
mlp · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 374–390
Read code excerpt
class DeepseekV3MLP(nn.Module):
def __init__(self, config, hidden_size=None, intermediate_size=None):
super().__init__()
self.config = config
self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
self.intermediate_size = (
config.intermediate_size if intermediate_size is None else intermediate_size
)
self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
self.act_fn = ACT2FN[config.hidden_act]
def forward(self, x):
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
return down_projContextual evidence from Down projection
Weights are stored in FP8 (e4m3) with one scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.
linear · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 38–46
Read code excerpt
"quantization_config": {
"activation_scheme": "dynamic",
"fmt": "e4m3",
"quant_method": "fp8",
"weight_block_size": [
128,
128
]
},