GPT-2 tokenizer
×no parameters
GPT-2’s byte-pair vocabulary of 50,257 tokens with two added: [PAD] at ID 50,257 and <mask> at ID 50,258.
Explanation
The tokenizer is loaded by name with no revision pinned. Each of the two tokens is added when the tokenizer has none, and GPT-2’s has neither: its files at the hub’s snapshot 607a30d hold 50,257 tokens and one special token, <|endoftext|>. The configuration file’s 50,258 and 50,257 are defaults the trainer overwrites with the tokenizer’s length and mask ID. The released checkpoint’s tensors were not read.
Sources4
Contextual evidence
The tokenizer is loaded by name, set up, and gains a `<mask>` token when it has none; the vocabulary size is its length.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence
Setting the tokenizer up wraps every encoded sequence in a beginning and an end token and adds a `[PAD]` token when the tokenizer has none.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence
Before any network is built, the configured vocabulary size and mask token ID are overwritten with the tokenizer’s.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public