78 transformer blocks › 75 MoE blocks › Mixture of experts › Shared expert
Gated features
×tensorself.act_fn(self.gate_proj(x)) * self.up_proj(x)
The gated features supplied to the down projection.
- shape
- [B batch, T positions, 2048 shared d_ff]
Sources
Contextual evidence from Element-wise product
The dense feed-forward network is a SwiGLU MLP, down(silu(gate x) ⊙ up x), computed in the input’s precision.
mlp · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 465–478
Read code excerpt
class GlmMoeDsaMLP(nn.Module):
def __init__(self, config, intermediate_size=None):
super().__init__()
self.config = config
self.hidden_size = config.hidden_size
self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
self.act_fn = ACT2FN[config.hidden_act]
def forward(self, x):
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
return down_projContextual evidence from Down projection
Attention projections have no bias (attention_bias is false) and the checkpoint stores its weights in bfloat16.
linear · zai-org/GLM-5 / config.json
Pinned revision c183ef8c61fa ↗ · lines 5–7
Read code excerpt
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",