Llama 3.1 8B · Element-wise product

32 independent transformer blocks › SwiGLU

Element-wise product

no parametersF.silu(self.w1(x)) * self.w3(x)

Combine the activated gate with corresponding up features.

{\htmlData{light=h product}{h}} = {\htmlData{light=a silu}{a}} \odot {\htmlData{light=u up}{u}}

in: $a$ [B batch, T positions, 14336 d_ff] $u$ [B batch, T positions, 14336 d_ff]
out: $h$ [B batch, T positions, 14336 d_ff]

Codemodel.py 208–219

model.py 208–219

        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

Sources1

Direct evidence

The MLP owns three bias-free projections. Its forward expression applies SiLU to w1(x), multiplies w3(x) element-wise and projects through w2.

mlp · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 208–219

Read code excerpt

        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))