GLM-5 · Up projection · OpenModelAtlas

78 transformer blocks › 75 MoE blocks › Mixture of experts › 256 routed experts

Up projection

learnednn.functional.linear(current_state, self.gate_up_proj[expert_idx])

Map 6,144 input features to 2,048 features in parallel with the gate.

{\htmlData{light=u up}{u}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{3}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 6144 d]
out: $u$ [B batch, T positions, 2048 expert d_ff]
parameters: 12,582,912 · × 75 MoE blocks × 256 experts = 241,591,910,400

Gate and up are the two halves of one stacked parameter per expert, gate_up_proj, held output by input (modeling_glm_moe_dsa.py 531, 553).

Parameters

12,582,912 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$6{,}144 \times 2{,}048$	12,582,912
Per-instance total		12,582,912

This projection has no bias parameter.

Across 256 independent experts: $12{,}582{,}912 \times 256 = 3{,}221{,}225{,}472$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_glm_moe_dsa.py 522–559 · config.json 5–7

modeling_glm_moe_dsa.py 522–559

@use_experts_implementation
class GlmMoeDsaExperts(nn.Module):
    """Collection of expert weights stored as 3D tensors."""

    def __init__(self, config):
        super().__init__()
        self.num_experts = config.num_local_experts
        self.hidden_dim = config.hidden_size
        self.intermediate_dim = config.moe_intermediate_size
        self.gate_up_proj = nn.Parameter(torch.empty(self.num_experts, 2 * self.intermediate_dim, self.hidden_dim))
        self.down_proj = nn.Parameter(torch.empty(self.num_experts, self.hidden_dim, self.intermediate_dim))
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(
        self,
        hidden_states: torch.Tensor,
        top_k_index: torch.Tensor,
        top_k_weights: torch.Tensor,
    ) -> torch.Tensor:
        final_hidden_states = torch.zeros_like(hidden_states)
        with torch.no_grad():
            expert_mask = torch.nn.functional.one_hot(top_k_index, num_classes=self.num_experts + 1)
            expert_mask = expert_mask.permute(2, 1, 0)
            expert_hit = torch.greater(expert_mask.sum(dim=(-1, -2)), 0).nonzero()

        for expert_idx in expert_hit:
            expert_idx = expert_idx[0]
            if expert_idx == self.num_experts:
                continue
            top_k_pos, token_idx = torch.where(expert_mask[expert_idx])
            current_state = hidden_states[token_idx]
            gate, up = nn.functional.linear(current_state, self.gate_up_proj[expert_idx]).chunk(2, dim=-1)
            current_hidden_states = self.act_fn(gate) * up
            current_hidden_states = nn.functional.linear(current_hidden_states, self.down_proj[expert_idx])
            current_hidden_states = current_hidden_states * top_k_weights[token_idx, top_k_pos, None]
            final_hidden_states.index_add_(0, token_idx, current_hidden_states.to(final_hidden_states.dtype))

        return final_hidden_states

config.json 5–7

  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",

Sources2

Direct evidence

The implementation holds the routed experts as stacked tensors, the gate and up projections fused into one; each expert is a SwiGLU of moe_intermediate_size hidden features applied to the tokens routed to it and weighted by its routing weight.

expert · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 522–559

Read code excerpt

@use_experts_implementation
class GlmMoeDsaExperts(nn.Module):
    """Collection of expert weights stored as 3D tensors."""

    def __init__(self, config):
        super().__init__()
        self.num_experts = config.num_local_experts
        self.hidden_dim = config.hidden_size
        self.intermediate_dim = config.moe_intermediate_size
        self.gate_up_proj = nn.Parameter(torch.empty(self.num_experts, 2 * self.intermediate_dim, self.hidden_dim))
        self.down_proj = nn.Parameter(torch.empty(self.num_experts, self.hidden_dim, self.intermediate_dim))
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(
        self,
        hidden_states: torch.Tensor,
        top_k_index: torch.Tensor,
        top_k_weights: torch.Tensor,
    ) -> torch.Tensor:
        final_hidden_states = torch.zeros_like(hidden_states)
        with torch.no_grad():
            expert_mask = torch.nn.functional.one_hot(top_k_index, num_classes=self.num_experts + 1)
            expert_mask = expert_mask.permute(2, 1, 0)
            expert_hit = torch.greater(expert_mask.sum(dim=(-1, -2)), 0).nonzero()

        for expert_idx in expert_hit:
            expert_idx = expert_idx[0]
            if expert_idx == self.num_experts:
                continue
            top_k_pos, token_idx = torch.where(expert_mask[expert_idx])
            current_state = hidden_states[token_idx]
            gate, up = nn.functional.linear(current_state, self.gate_up_proj[expert_idx]).chunk(2, dim=-1)
            current_hidden_states = self.act_fn(gate) * up
            current_hidden_states = nn.functional.linear(current_hidden_states, self.down_proj[expert_idx])
            current_hidden_states = current_hidden_states * top_k_weights[token_idx, top_k_pos, None]
            final_hidden_states.index_add_(0, token_idx, current_hidden_states.to(final_hidden_states.dtype))

        return final_hidden_states

Direct evidence

Attention projections have no bias (attention_bias is false) and the checkpoint stores its weights in bfloat16.

linear · zai-org/GLM-5 / config.json
Pinned revision c183ef8c61fa ↗ · lines 5–7

Read code excerpt

  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",