DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Routed expert

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

61 transformer blocks › 58 MoE blocks › Mixture of experts › 256 routed experts

SiLU

×

no parametersF.silu(self.w1(x).float())

Activate each gate feature independently.

a=g σ(g){\htmlData{light=a silu}{a}}={\htmlData{light=g gate}{g}}\,\sigma({\htmlData{light=g gate}{g}})
in
gg [B batch, T positions, 2048 expert d_ff]
out
aa [B batch, T positions, 2048 expert d_ff]

SiLU and the product run in float32 and are cast back before the down projection (model.py 744).

Explanation

σ(z)=1/(1+e−z){\sigma(z)=1/(1+e^{-z})} is the logistic sigmoid.

Codemodel.py 730–732, 744
model.py 730–732
        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)
model.py 744
        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))
Sources1

Direct evidence

Each routed expert is its own SwiGLU MLP of moe_inter_dim hidden features.

expert · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 730–744

Read code excerpt
        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

… lines 733–743 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))