GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Token IDs

×

tensorX

Integer vocabulary indices used to look up token embeddings.

shape
[B batch, T positions]
Sources

Contextual evidence from Byte-level BPE tokenizer

The pinned configuration supplies the family vocabulary size.

configuration · openai-community/gpt2 / config.json
Pinned revision 607a30d783df ↗ · $

Contextual evidence from Byte-level BPE tokenizer

Original Python encoder shipped with the released checkpoints.

openai · openai/gpt-2 / src/encoder.py
Pinned revision 9b63575ef427 ↗ · lines 96–101

Read code excerpt
    def encode(self, text):
        bpe_tokens = []
        for token in re.findall(self.pat, text):
            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
        return bpe_tokens

Contextual evidence from Byte-level BPE tokenizer

The maintained tokenizer configures byte-level pre-tokenization and decoding with the released vocabulary and merges.

transformers · huggingface/transformers / src/transformers/models/gpt2/tokenization_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 107–120

Read code excerpt
        self._vocab = vocab if vocab is not None else {}
        self._merges = merges or []
        self._tokenizer = Tokenizer(
            BPE(
                vocab=self._vocab,
                merges=self._merges,
                dropout=None,
                continuing_subword_prefix="",
                end_of_word_suffix="",
                fuse_unk=False,
            )
        )
        self._tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
        self._tokenizer.decoder = decoders.ByteLevel()

Contextual evidence from Token embedding

The original TensorFlow graph creates both tables and adds their gathered rows.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt
        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],

… lines 153–153 omitted …

        wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd],

… lines 155–156 omitted …

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Contextual evidence from Token embedding

Transformers names the tables wte and wpe, matching the released checkpoint keys.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 492–577

Read code excerpt
        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)

… lines 494–568 omitted …

            inputs_embeds = self.wte(input_ids)

… lines 570–575 omitted …

        position_embeds = self.wpe(position_ids)
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

Contextual evidence from Absolute position indices

The position helper constructs indices from sequence length and past length; token values do not enter the calculation.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 141–144

Read code excerpt
def positions_for(tokens, past_length):
    batch_size = tf.shape(tokens)[0]
    nsteps = tf.shape(tokens)[1]
    return expand_tile(past_length + tf.range(nsteps), batch_size)