GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Absolute position indices

×

no parameterspositions_for(X, past_length)

Assigns an absolute position index to each token in the sequence.

position_ids=past_length+arange⁡(T)\mathrm{position\_ids}=\mathrm{past\_length}+\operatorname{arange}(T)
in
[B batch, T positions]
out
[B batch, T positions]
Explanation

The helper reads only the batch size and the sequence length from the tokens, and tiles the same indices across the batch (model.py 141–144). past_length\mathrm{past\_length} counts positions already in the cache; a full-sequence pass has past_length=0\mathrm{past\_length}=0.

Codemodel.py 141–144
model.py 141–144
def positions_for(tokens, past_length):
    batch_size = tf.shape(tokens)[0]
    nsteps = tf.shape(tokens)[1]
    return expand_tile(past_length + tf.range(nsteps), batch_size)
Sources1

Direct evidence

The position helper constructs indices from sequence length and past length; token values do not enter the calculation.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 141–144

Read code excerpt
def positions_for(tokens, past_length):
    batch_size = tf.shape(tokens)[0]
    nsteps = tf.shape(tokens)[1]
    return expand_tile(past_length + tf.range(nsteps), batch_size)