DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Dense transformer block

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

61 transformer blocks › 3 dense blocks

SwiGLU MLP

Open inside →×

learnedself.ffn

A dense SwiGLU of 18,432 hidden features, used in the first 3 blocks.

h=a⊙uh=a\odot u
Δffn=hW2\Delta_{\mathrm{ffn}}=hW_{2}
All 5 steps
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
Δffn=hW2\Delta_{\mathrm{ffn}}=hW_{2}
in
xffnx_{\mathrm{ffn}} [B batch, T positions, 7168 d]
out
Δffn\Delta_{\mathrm{ffn}} [B batch, T positions, 7168 d]
parameters
396,361,728 · × 3 dense blocks = 1,189,085,184
Parameters

396,361,728 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Gate projection7,168×18,4327{,}168 \times 18{,}432132,120,576
Up projection7,168×18,4327{,}168 \times 18{,}432132,120,576
Down projection18,432×7,16818{,}432 \times 7{,}168132,120,576
Per-instance total396,361,728

Across 3 independent dense blocks: 396,361,728×3=1,189,085,184396{,}361{,}728 \times 3 = 1{,}189{,}085{,}184 parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Explanation

The first 3 blocks use this dense MLP; the other 58 replace it with the mixture of experts.

Codemodel.py 629–851
model.py 629–631
        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)
model.py 643
        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))
model.py 826–829
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)
model.py 844–851
        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual
Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP: w2(silu(w1 x) ⊙ w3 x).

mlp · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 629–643

Read code excerpt
        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

… lines 632–642 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Direct evidence

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual