LDLM · Query, key and value projection

Latent decoder › 6 decoder blocks › Perceiver attention

Query, key and value projection

learned

One matrix maps the normalized decoder state to queries, keys and values, 768 features each.

{\htmlData{light=self-parts self-projection}{p^{a}}}=[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k^{a}}}\;{\htmlClass{vl-stream-v}{v^{a}}}\,]={\htmlData{light=query-norm}{\bar r_n}}\,W_{QKV}

in: $\bar r_n$ [B batch, T positions, 768 d]
out: $p^{a}$ [B batch, T positions, 2304 q, k, v]
parameters: 1,769,472 · × 6 blocks = 10,616,832

Parameters

1,769,472 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 2{,}304$	1,769,472
Per-instance total		1,769,472

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: $1{,}769{,}472 \times 6 = 10{,}616{,}832$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources2

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction