DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Dense transformer block

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

in
xnx_n [B batch, T positions, 7168 d]RR [T positions, 32 pairs]MM [T queries, T keys]
out
xn+1x_{n+1} [B batch, T positions, 7168 d]
parameters
597,442,816 · × 3 dense blocks = 1,792,328,448
Parameters

597,442,816 parameters per dense block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Attention RMSNorm7,1687{,}1687,168
Multi-head latent attention201,066,752 across its components201,066,752
MLP RMSNorm7,1687{,}1687,168
SwiGLU MLP396,361,728 across its components396,361,728
Per-instance total597,442,816

Across 3 independent dense blocks: 597,442,816×3=1,792,328,448597{,}442{,}816 \times 3 = 1{,}792{,}328{,}448 parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

In numbers

Weights · one block, by sublayer

Dense block · FP8571.1 MiB
Attention · FP8193.1 MiB
MLP · FP8378 MiB
Two RMSNorms · float3256 KiB

597,442,816 parameters per block: attention 201,066,752 (33.7%), the MLP 396,361,728 (66.3%), the two RMSNorms 14,336; × 3 dense blocks = 1,792,328,448.

config_671B_v3.2.json 4config_671B_v3.2.json 8config_671B_v3.2.json 17model.py 826–829model.py 844–851config_671B_v3.2.json 21model.py 876model.py 183–187model.py 284model.py 317–318model.py 449
Codemodel.py 826–829, 844–851
model.py 826–829
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)
model.py 844–851
        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual
Sources1

Direct evidence

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual