Latent decoder
Mask vector
×frozen · not trained
The token-table row of <mask>, copied once from a GPT-2 loaded for the purpose, and repeated at every position.
- out
- [B batch, T positions, 768 d]
- parameters
- 768
Row 50,258 of a GPT-2 token table resized to 50,259 rows: a row the transformers library writes for the new token, which GPT-2 was not trained with. That GPT-2 is loaded apart from the feature model and resized on its own, so the two models’ added rows are not shown to be equal; the feature model’s is never read. The vector takes no gradient because the feature model is configured frozen.
Parameters
768 parameters owned by this component
| Component | Calculation | Parameters |
|---|---|---|
| learned vector | 768 | |
| Per-instance total | 768 | |
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
768 values, stored with the decoder and never updated.
Sources4
Direct evidence
The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
A GPT-2 is loaded by name and its token table resized to the configured vocabulary; that table is returned.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public