GLM-5 · SiLU · OpenModelAtlas

78 transformer blocks › 3 dense blocks › SwiGLU

SiLU

no parametersself.act_fn(self.gate_proj(x))

Activate each gate feature independently.

{\htmlData{light=a silu}{a}}={\htmlData{light=g gate}{g}}\,\sigma({\htmlData{light=g gate}{g}})

in: $g$ [B batch, T positions, 12288 d_ff]
out: $a$ [B batch, T positions, 12288 d_ff]

Explanation

${\sigma(z)=1/(1+e^{-z})}$ is the logistic sigmoid.

Codemodeling_glm_moe_dsa.py 465–478

modeling_glm_moe_dsa.py 465–478

class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Sources1

Direct evidence

The dense feed-forward network is a SwiGLU MLP, down(silu(gate x) ⊙ up x), computed in the input’s precision.

mlp · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 465–478

Read code excerpt

class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj