GLM-5

All models, with previews →

Z.ai · Feb 2026Latent attention and experts

SwiGLU MLP

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

78 transformer blocks › 3 dense blocks › SwiGLU

Gate projection

×

learnedself.gate_proj(x)

Map 6,144 input features to 12,288 gate features.

g=xffnW1{\htmlData{light=g gate}{g}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{1}
in
xffnx_{\mathrm{ffn}} [B batch, T positions, 6144 d]
out
gg [B batch, T positions, 12288 d_ff]
parameters
75,497,472 · × 3 dense blocks = 226,492,416
Parameters

75,497,472 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix6,144×12,2886{,}144 \times 12{,}28875,497,472
Per-instance total75,497,472

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 3 independent dense blocks: 75,497,472×3=226,492,41675{,}497{,}472 \times 3 = 226{,}492{,}416 parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_glm_moe_dsa.py 465–478 · config.json 5–7
modeling_glm_moe_dsa.py 465–478
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
config.json 5–7
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",
Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP, down(silu(gate x) ⊙ up x), computed in the input’s precision.

mlp · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 465–478

Read code excerpt
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Direct evidence

Attention projections have no bias (attention_bias is false) and the checkpoint stores its weights in bfloat16.

linear · zai-org/GLM-5 / config.json
Pinned revision c183ef8c61fa ↗ · lines 5–7

Read code excerpt
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",