Kimi K2 · Activated gate · OpenModelAtlas

61 transformer blocks › 1 dense block › SwiGLU

Activated gate

tensorself.act_fn(self.gate_proj(x))

The SiLU-activated gate features.

shape: $a$ [B batch, T positions, 18432 d_ff]

Sources

Contextual evidence from SiLU

The dense feed-forward network is a SwiGLU MLP: down(silu(gate x) ⊙ up x).

mlp · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 374–390

Read code excerpt

class DeepseekV3MLP(nn.Module):
    def __init__(self, config, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = (
            config.intermediate_size if intermediate_size is None else intermediate_size
        )

        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj