DeepSeek-V3.2-Exp · Mixture of experts

61 transformer blocks › 58 MoE blocks

Mixture of experts

learnedself.ffn

Each position runs 8 of 256 routed experts, weighted by the router, plus one shared expert that every position runs.

Why

Many small experts. With few large experts, the tokens sent to one expert are more likely to cover very different kinds of knowledge, and one expert has to hold all of it. The authors split each expert into smaller ones and activate more of them at the same parameters and compute, so that knowledge can be divided among experts; the number of possible combinations of active experts grows enormously. DeepSeekMoE §3.1

Shared experts. Tokens routed to different experts may still need some common knowledge, and several routed experts may then each learn it. Experts that run for every token are meant to hold that common part, which the authors say reduces redundancy among the routed experts and leaves them more specialised. DeepSeekMoE §3.2

y^{\mathrm{routed}}=\sum_{i=1}^{256}G_i\,E_i

\Delta_{\mathrm{ffn}}=y^{\mathrm{routed}}+y^{\mathrm{shared}}

All 19 steps

\text{Router:}

z=x_{\mathrm{ffn}}W_{r}

s=\sigma(z)

E^{\mathrm{sel}}_i=\big[\,i\in\operatorname{top}_{8}(s+b)\text{ within the top }4\text{ groups}\,\big]

G_{i}=2.5\,\frac{s_i\,E^{\mathrm{sel}}_i}{\sum_{j}s_j\,E^{\mathrm{sel}}_j}

\text{Routed expert:}

g=x_{\mathrm{ffn}}W_{1}

u=x_{\mathrm{ffn}}W_{3}

a=g\,\sigma(g)

h=a\odot u

E_i=hW_{2}

y^{\mathrm{routed}}=\sum_{i=1}^{256}G_i\,E_i

\text{Shared expert:}

g=x_{\mathrm{ffn}}W_{1}

u=x_{\mathrm{ffn}}W_{3}

a=g\,\sigma(g)

h=a\odot u

y^{\mathrm{shared}}=hW_{2}

\Delta_{\mathrm{ffn}}=y^{\mathrm{routed}}+y^{\mathrm{shared}}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $\Delta_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
parameters: 11,320,164,608 · × 58 MoE blocks = 656,569,547,264

Parameters

11,320,164,608 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Router	1,835,264 across its components	1,835,264
Routed expert	$44{,}040{,}192 \times 256$	11,274,289,152
Shared expert	44,040,192 across its components	44,040,192
Per-instance total		11,320,164,608

Across 58 independent MoE blocks: $11{,}320{,}164{,}608 \times 58 = 656{,}569{,}547{,}264$ parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Explanation

The reference flattens batch and positions into one token axis for routing and restores the shape at the end (model.py 791, 804).

Codemodel.py 770–778, 790–803

model.py 770–778

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

model.py 790–803

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)

Sources2

Direct evidence

The MoE layer holds the router, the routed experts and one shared expert MLP of n_shared_experts × moe_inter_dim features.

moe-init · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 770–778

Read code excerpt

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

Direct evidence

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)