OpenModelAtlasHow to read

Models

Nine models, 2019 to 2026, drawn from their published code and papers. Grouped by publisher; open one to read it part by part.

DeepSeek

2 models

DeepSeek-V3.2-ExpDeepSeek · Sep 2025671BMLAsparse attentionexpertsRMSNormRoPE (YaRN)SwiGLU× 61 layers128K context
DeepSeek V4.1DeepSeek · Sep 2026552B + 196B Engramsliding windowcompressed sparse attentionexpertsEngram memoryfour residual streamsRMSNormRoPESwiGLU× 40 layers1M context
After DeepSeek-V3.2-Exp: DeepSeek’s earlier model in the atlas

Meshchaninov et al.

1 model

x_n[B, T, 768]a_t[B, 768]Time projection $a_tW_n+c_n$ Modulation vectorsm_n m_n m_n m_n m_n LayerNorm $\mu,\;\sigma^2$ Modulate $\bar x\odot(1+s_{a})+b_{a}$
Attention Open →
Gate $g_{a}\odot \Delta_{\mathrm{attn}}$ +x^′_n[B, T, 768]LayerNorm $\mu,\;\sigma^2$ Modulate $\bar x\odot(1+s_{m})+b_{m}$
MLP Open →
Gate $g_{m}\odot \Delta_{\mathrm{mlp}}$ +x_n+1[B, T, 768]
LDLMIn constructionMeshchaninov et al. · May 2026306M trainedPerceiver attentionRMSNormLayerNormRoPElearned positionsSwiGLUGELUdiffusion time× 12 denoiser layers1K context

Moonshot AI

2 models

OpenAI

1 model

GPT-2 pre-normalized transformer block
x_n[B, T, 768]LayerNorm 1 $\mu,\;\sigma^2$ gain · bias
Attention $\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)$ Open →
Dropout $p_{\mathrm{resid}}$ +x^′_n[B, T, 768]LayerNorm 2 $\mu,\;\sigma^2$ gain · bias
MLP $\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)$ Open →
Dropout $p_{\mathrm{resid}}$ +x_n+1[B, T, 768]residual bypassresidual bypass
1. x n passes through layer norm, attention, and residual dropout on the straight computation axis.
2. A separate identity bypass carries x n beneath attention and rejoins at the first addition.
3. x prime n passes through the second layer norm, MLP, and residual dropout on the same axis.
4. A second identity bypass carries x prime n beneath the MLP and rejoins at the second addition to produce x n plus 1.
GPT-2OpenAI · Feb–Nov 2019124M355M774M1.5BMHALayerNormlearned positionsGELU× 12–48 layers1K context

Z.ai

1 model

GLM-5Z.ai · Feb 2026744BMLAsparse attentionexpertsRMSNormRoPESwiGLU× 78 layers198K context