LDLM · Key and value projection

Latent encoder › 6 encoder blocks › Perceiver attention

Key and value projection

learned

One matrix maps the normalized features to keys and values, 768 features each; this stream makes no queries.

{\htmlData{light=context-parts context-projection}{p^{b}}}=[\,{\htmlClass{vl-stream-k}{k^{b}}}\;{\htmlClass{vl-stream-v}{v^{b}}}\,]={\htmlData{light=context-norm}{\bar h^{p}}}\,W_{KV}

in: $\bar h^{p}$ [B batch, T positions, 768 d]
out: $p^{b}$ [B batch, T positions, 1536]
parameters: 1,179,648 · × 6 blocks = 7,077,888

Parameters

1,179,648 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 1{,}536$	1,179,648
Per-instance total		1,179,648

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: $1{,}179{,}648 \times 6 = 7{,}077{,}888$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources2

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction