GPT-2

GPT-2 model overview

1. UTF-8 text passes through byte-level BPE, which is configured by fixed vocabulary and merge assets.
2. Token IDs cross the preprocessing boundary and select rows from the learned token embedding table WTE.
3. The token sequence shape determines absolute position IDs, independently of token values, which select rows from WPE on a lower positional branch.
4. Token and position embeddings are added, then training-only embedding dropout produces x zero. Evaluation leaves the sum unchanged.
5. Exactly 12 transformer blocks transform x zero into x_12.
6. Final LayerNorm acts across the 768-dimensional feature axis with epsilon equal to 10 to the minus 5.
7. The transpose view of the shared WTE matrix projects the final states to logits without adding parameters.