DeepSeek-V3.2-Exp · Mixture of experts

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $\Delta_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
parameters: 11,320,164,608 · × 58 MoE blocks = 656,569,547,264

Parameters

11,320,164,608 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Router	1,835,264 across its components	1,835,264
Routed expert	$44{,}040{,}192 \times 256$	11,274,289,152
Shared expert	44,040,192 across its components	44,040,192
Per-instance total		11,320,164,608

Across 58 independent MoE blocks: $11{,}320{,}164{,}608 \times 58 = 656{,}569{,}547{,}264$ parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

In numbers

Weights · one MoE block

Held · FP810.5 GiB

Used per token · FP8381.5 MiB

Held: 256 routed experts × 44,040,192 parameters, the shared expert’s 44,040,192 and the router’s 1,835,264, 11,320,164,608 in all. Used per token: 8 routed experts, the shared expert and the router, 398,196,992 (3.5%).

FP8 experts, block scales left out; the router’s matrix in bfloat16 and its bias in float32.

config_671B_v3.2.json 21 model.py 876 model.py 183–187 model.py 674–675 generate.py 109 model.py 790–803

Explanation

The reference flattens batch and positions into one token axis for routing and restores the shape at the end (model.py 791, 804).

Codemodel.py 770–778, 790–803

model.py 770–778

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

model.py 790–803

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)

Sources2

Direct evidence

The MoE layer holds the router, the routed experts and one shared expert MLP of n_shared_experts × moe_inter_dim features.

moe-init · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 770–778

Read code excerpt

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

Direct evidence

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)