DeepSeek V4.1

All models, with previews →

DeepSeek · Sep 2026New parts in the block · after DeepSeek-V3.2-Exp in the atlas

Shared expert

Reference implementation and checkpoint configuration deepseek-ai/DeepSeek-V4.1-Flash @ 2cba9e4 · parameter count checked against the checkpoint’s tensor headers · text model only · How to read

Full sequencePrefillDecode

40 layers › Window layer › Mixture of experts › Shared expert

SiLU

×

no parametersF.silu(gate) · 848

Activate each capped gate feature.

a=gˉ σ(gˉ){\htmlData{light=silu}{a}}={\htmlData{light=clamp-gate}{\bar g}}\,\sigma({\htmlData{light=clamp-gate}{\bar g}})
in
gˉ\bar g [B batch, T positions, 2304 d_ff]
out
aa [B batch, T positions, 2304 d_ff]
Codemodel.py 830–851
model.py 830–851
class Expert(nn.Module):
    """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in
    range: the up branch is clamped on both sides, the gate branch only from above."""

    def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):
        super().__init__()
        self.w1 = Linear(dim, inter_dim, dtype=dtype)
        self.w2 = Linear(inter_dim, dim, dtype=dtype)
        self.w3 = Linear(dim, inter_dim, dtype=dtype)
        self.swiglu_limit = swiglu_limit

    def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:
        dtype = x.dtype
        gate = self.w1(x).float()
        up = self.w3(x).float()
        if self.swiglu_limit > 0:
            up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)
            gate = torch.clamp(gate, max=self.swiglu_limit)
        x = F.silu(gate) * up
        if weights is not None:
            x = weights * x
        return self.w2(x.to(dtype))
Sources1

Direct evidence

An expert is SwiGLU with its gate and up branches clamped; the routing weight multiplies the hidden features before the down projection.

expert · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 830–851

Read code excerpt
class Expert(nn.Module):
    """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in
    range: the up branch is clamped on both sides, the gate branch only from above."""

    def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):
        super().__init__()
        self.w1 = Linear(dim, inter_dim, dtype=dtype)
        self.w2 = Linear(inter_dim, dim, dtype=dtype)
        self.w3 = Linear(dim, inter_dim, dtype=dtype)
        self.swiglu_limit = swiglu_limit

    def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:
        dtype = x.dtype
        gate = self.w1(x).float()
        up = self.w3(x).float()
        if self.swiglu_limit > 0:
            up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)
            gate = torch.clamp(gate, max=self.swiglu_limit)
        x = F.silu(gate) * up
        if weights is not None:
            x = weights * x
        return self.w2(x.to(dtype))