OpenModelAtlasHow to read

Models

Nine models, 2019 to 2026, drawn from their published code and papers. Grouped by year; open one to read it part by part.

2019

1 model

GPT-2 pre-normalized transformer block
x_n[B, T, 768]LayerNorm 1 $\mu,\;\sigma^2$ gain · bias
Attention $\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)$ Open →
Dropout $p_{\mathrm{resid}}$ +x^′_n[B, T, 768]LayerNorm 2 $\mu,\;\sigma^2$ gain · bias
MLP $\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)$ Open →
Dropout $p_{\mathrm{resid}}$ +x_n+1[B, T, 768]residual bypassresidual bypass
1. x n passes through layer norm, attention, and residual dropout on the straight computation axis.
2. A separate identity bypass carries x n beneath attention and rejoins at the first addition.
3. x prime n passes through the second layer norm, MLP, and residual dropout on the same axis.
4. A second identity bypass carries x prime n beneath the MLP and rejoins at the second addition to produce x n plus 1.
GPT-2OpenAI · Feb–Nov 2019124M355M774M1.5BMHALayerNormlearned positionsGELU× 12–48 layers1K context

2025

2 models

2026

3 models

Date not pinned

1 model

s_n[B, T, 7168]Stored sumsb[B, T, m, 7168]Input mixsoftmax over b, s Store, restartonly layers 13, 25, …, 85 b^′[B, T, m', 7168]Stored sums after the storeb^′[B, T, m', 7168]Attention RMSNorm $\operatorname{mean}(x^2)$ learned gain
Kimi Delta Attention (KDA)Open →
+s^′_n[B, T, 7168]Input mixsoftmax over b, s MoE RMSNorm $\operatorname{mean}(x^2)$ learned gain
Latent mixture of experts Open →
+s_n+1[B, T, 7168]
Kimi K3Moonshot AI · Release date not pinned2.8TMLAlinear attention (KDA)attention residualsexpertsRMSNorm× 93 layers1M context
After Kimi K2: Moonshot AI’s earlier model in the atlas