DeepSeek-V3.2-Exp · Add shared output

61 transformer blocks › 58 MoE blocks › Mixture of experts

Add shared output

no parametersy += self.shared_experts(x)

Add the shared expert’s output to the routed output.

{\htmlData{light=output add}{\Delta_{\mathrm{ffn}}}}={\htmlData{light=routed combine}{y^{\mathrm{routed}}}}+{\htmlData{light=shared shared-expert}{y^{\mathrm{shared}}}}

in: $y^{\mathrm{routed}}$ [B batch, T positions, 7168 d] $y^{\mathrm{shared}}$ [B batch, T positions, 7168 d]
out: $\Delta_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]

Codemodel.py 790–803

model.py 790–803

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)

Sources1

Direct evidence

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)