DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Transformer blocks

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

61 transformer blocks

Dense transformer block

Open inside →×

learnedBlock(layer_id, args)

MLA and a dense SwiGLU MLP, each added to the residual stream; blocks 1–3.

xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
All 32 steps
xattn=RMSNorm⁡(xn)x_{\mathrm{attn}}=\operatorname{RMSNorm}(x_n)
Multi-head latent attention:\text{Multi-head latent attention:}
cQ=xWDQc^{{\htmlClass{vl-stream-q}{Q}}}=xW_{DQ}
c^Q=RMSNorm⁡(cQ)\hat c^{{\htmlClass{vl-stream-q}{Q}}}=\operatorname{RMSNorm}(c^{{\htmlClass{vl-stream-q}{Q}}})
cKV=xWDKVc^{KV}=xW_{DKV}
c^KV=RMSNorm⁡(cKV)\hat c^{KV}=\operatorname{RMSNorm}(c^{KV})
qC=c^QWUQ{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}
QC=reshape⁡(qC){\htmlClass{vl-stream-q}{Q^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{C}}})
qpe=c^QWQR{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}
Qpe=reshape⁡(qpe){\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}})
QtR=RoPE⁡(Qtpe,Rt){\htmlClass{vl-stream-q}{Q^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Q=[QC; QR]{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]
kC=c^KVWUK{\htmlClass{vl-stream-k}{k^{C}}}=\hat c^{KV}W_{UK}
KC=reshape⁡(kC){\htmlClass{vl-stream-k}{K^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-k}{k^{C}}})
kpe=xWKR{\htmlClass{vl-stream-k}{k^{\mathrm{pe}}}}=xW_{KR}
ktR=RoPE⁡(ktpe,Rt){\htmlClass{vl-stream-k}{k^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-k}{k^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Ki=[KiC; kR]{\htmlClass{vl-stream-k}{K_i}}=[{\htmlClass{vl-stream-k}{K^{C}_i}};\,{\htmlClass{vl-stream-k}{k^{R}}}]
v=c^KVWUV{\htmlClass{vl-stream-v}{v}}=\hat c^{KV}W_{UV}
V=reshape⁡(v){\htmlClass{vl-stream-v}{V}}=\operatorname{reshape}({\htmlClass{vl-stream-v}{v}})
w=xWww=xW_{w}
St,sI=164128∑h=164wt,h ReLU⁡ ⁣(Qh,tIˉ⋅ksIˉ)S^{I}_{t,s}=\frac{1}{\sqrt{64}\sqrt{128}}\sum_{h=1}^{64}w_{t,h}\,\operatorname{ReLU}\!\big(\bar {\htmlClass{vl-stream-q}{Q^{I}_{h,t}}}\cdot\bar {\htmlClass{vl-stream-k}{k^{I}_{s}}}\big)
S~I=SI+M\tilde S^{I}=S^{I}+{\htmlClass{vl-stream-context}{M}}
Mt,ssel={Mt,ss∈top⁡2,048(S~tI)−∞otherwise{\htmlClass{vl-stream-context}{M^{\mathrm{sel}}_{t,s}}}=\begin{cases}{\htmlClass{vl-stream-context}{M_{t,s}}}&s\in\operatorname{top}_{2,048}(\tilde S^{I}_{t})\\-\infty&\text{otherwise}\end{cases}
xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xffn=RMSNorm⁡(xn′)x_{\mathrm{ffn}}=\operatorname{RMSNorm}(x'_n)
SwiGLU MLP:\text{SwiGLU MLP:}
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
Δffn=hW2\Delta_{\mathrm{ffn}}=hW_{2}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
in
x0x_0 [B batch, T positions, 7168 d]RR [T positions, 32 pairs]MM [T queries, T keys]
out
xn+1x_{n+1} [B batch, T positions, 7168 d]
parameters
597,442,816 · × 3 dense blocks = 1,792,328,448

Each residual addition runs in float32 inside the next RMSNorm, which returns the normalized sum and the sum, both cast back to bfloat16, the sum as the new stream; the first block’s norm receives no residual (model.py 302–306, 844–849, 907).

Parameters

597,442,816 parameters per dense block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Attention RMSNorm7,1687{,}1687,168
Multi-head latent attention201,066,752 across its components201,066,752
MLP RMSNorm7,1687{,}1687,168
SwiGLU MLP396,361,728 across its components396,361,728
Per-instance total597,442,816

Across 3 independent dense blocks: 597,442,816×3=1,792,328,448597{,}442{,}816 \times 3 = 1{,}792{,}328{,}448 parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Codemodel.py 826–829, 844–851
model.py 826–829
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)
model.py 844–851
        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual
Sources1

Direct evidence

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual