DeepSeek-V3.2-Exp · Up projection

61 transformer blocks › 3 dense blocks › SwiGLU

Up projection

learnedself.w3(x)

Map 7,168 input features to 18,432 features in parallel with the gate.

{\htmlData{light=u up}{u}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{3}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $u$ [B batch, T positions, 18432 d_ff]
parameters: 132,120,576 · × 3 dense blocks = 396,361,728

Parameters

132,120,576 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$7{,}168 \times 18{,}432$	132,120,576
Per-instance total		132,120,576

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 3 independent dense blocks: $132{,}120{,}576 \times 3 = 396{,}361{,}728$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 15–643

model.py 629–631

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

model.py 643

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

model.py 15

block_size = 128

model.py 179–193

    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype or Linear.dtype))
        if self.weight.element_size() == 1:
            scale_out_features = (out_features + block_size - 1) // block_size
            scale_in_features = (in_features + block_size - 1) // block_size
            self.weight.scale = self.scale = nn.Parameter(torch.empty(scale_out_features, scale_in_features, dtype=torch.float32))
        else:
            self.register_parameter("scale", None)
        if bias:
            self.bias = nn.Parameter(torch.empty(out_features))
        else:
            self.register_parameter("bias", None)

Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP: w2(silu(w1 x) ⊙ w3 x).

mlp · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 629–643

Read code excerpt

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

… lines 632–642 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Direct evidence

Linear layers have no bias unless asked. FP8 weights carry one float32 scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.

linear · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 15–193

Read code excerpt

block_size = 128

… lines 16–178 omitted …

    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype or Linear.dtype))
        if self.weight.element_size() == 1:
            scale_out_features = (out_features + block_size - 1) // block_size
            scale_in_features = (in_features + block_size - 1) // block_size
            self.weight.scale = self.scale = nn.Parameter(torch.empty(scale_out_features, scale_in_features, dtype=torch.float32))
        else:
            self.register_parameter("scale", None)
        if bias:
            self.bias = nn.Parameter(torch.empty(out_features))
        else:
            self.register_parameter("bias", None)