DeepSeek-V3.2-Exp

DeepSeek · Sep 2025Latent attention and experts

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

Raw text Tokenizer datafixed assets TokenizerChat template Token IDs[B, T]Token embedding

E

129,280 × 7,168 x₀[B, T, 7168]Position IDs

0,\ldots,T-1

Generate RoPE coefficients

{\htmlClass{vl-stream-context}{R_{t,j}}}=e^{\mathrm i t\omega_j}

R[T, 32]Construct causal mask

{\htmlClass{vl-stream-context}{M_{t,s}}}=\begin{cases}0&s\le t\\-\infty&s>t\end{cases}

M[T, T]

3 dense Open →

58 MoE Open →

x₆₁ Final RMSNorm

\operatorname{mean}(x^2)

learned gain Vocabulary projection

L=zW_{\mathrm{out}}

7,168 × 129,280 Logits[B, T, 129280]

Checkpoint tokenizer

no parametersAutoTokenizer.from_pretrained(ckpt_path)

The vocabulary, merge rules and chat template shipped with the checkpoint.

Unverified: the files’ contents and their token-to-ID mapping, which the pinned sources do not include.

Explanation

The reference generator loads them from the checkpoint directory with Hugging Face AutoTokenizer (generate.py 117).

Codegenerate.py 117, 150

generate.py 117

    tokenizer = AutoTokenizer.from_pretrained(ckpt_path)

generate.py 150

        prompt_tokens = [tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True) for prompt in prompts]

Sources1

Contextual evidence

The reference generator loads the checkpoint’s tokenizer with Hugging Face AutoTokenizer and encodes each prompt through its chat template. The tokenizer’s vocabulary and rules ship with the checkpoint; these sources do not include them.

tokenizer · deepseek-ai/DeepSeek-V3.2-Exp / inference/generate.py
Pinned revision 87e509a2e5a1 ↗ · lines 117–150

Read code excerpt

    tokenizer = AutoTokenizer.from_pretrained(ckpt_path)

… lines 118–149 omitted …

        prompt_tokens = [tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True) for prompt in prompts]