Byte-level BPE tokenizer
×no parametersEncoder.encode(text)
UTF-8 bytes are mapped to reversible Unicode symbols, then adjacent symbols are merged by the rank of their pair into vocabulary tokens.
- out
- [B batch, T positions]
Explanation
The encoder splits text with a regular expression, converts each piece to UTF-8 bytes and maps each byte to a reversible Unicode symbol; BPE then merges adjacent symbols by rank, and each resulting piece is looked up in the vocabulary (encoder.py 96–101). The family vocabulary contains 50,257 available token IDs.
The byte alphabet covers every UTF-8 string, so any text encodes without an unknown token. This example illustrates the transformation.
Codeencoder.py 96–101 · tokenization_gpt2.py 107–120
def encode(self, text):
bpe_tokens = []
for token in re.findall(self.pat, text):
token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
return bpe_tokens self._vocab = vocab if vocab is not None else {}
self._merges = merges or []
self._tokenizer = Tokenizer(
BPE(
vocab=self._vocab,
merges=self._merges,
dropout=None,
continuing_subword_prefix="",
end_of_word_suffix="",
fuse_unk=False,
)
)
self._tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
self._tokenizer.decoder = decoders.ByteLevel()Sources3
Direct evidence
The pinned configuration supplies the family vocabulary size.
configuration · openai-community/gpt2 / config.json
Pinned revision 607a30d783df ↗ · $
Direct evidence
Original Python encoder shipped with the released checkpoints.
openai · openai/gpt-2 / src/encoder.py
Pinned revision 9b63575ef427 ↗ · lines 96–101
Read code excerpt
def encode(self, text):
bpe_tokens = []
for token in re.findall(self.pat, text):
token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
return bpe_tokensDirect evidence
The maintained tokenizer configures byte-level pre-tokenization and decoding with the released vocabulary and merges.
transformers · huggingface/transformers / src/transformers/models/gpt2/tokenization_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 107–120
Read code excerpt
self._vocab = vocab if vocab is not None else {}
self._merges = merges or []
self._tokenizer = Tokenizer(
BPE(
vocab=self._vocab,
merges=self._merges,
dropout=None,
continuing_subword_prefix="",
end_of_word_suffix="",
fuse_unk=False,
)
)
self._tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
self._tokenizer.decoder = decoders.ByteLevel()