LDLM · Attention over both streams

Latent decoder › 6 decoder blocks › Perceiver attention

Attention over both streams

no parameters

Each query weighs the values at all 2T positions by its softcapped scores against their keys.

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}

O=AV

All 4 steps

S=QK^{\mathsf T}

F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}

O=AV

in: $Q$ [B batch, 12 heads, T positions, 64 head features] $K$ [B batch, 12 heads, 2T positions, 64 head features] $V$ [B batch, 12 heads, 2T positions, 64 head features]
out: $O$ [B batch, 12 heads, T positions, 64 head features]

Explanation

The fused kernel does not store the score and weight fields; the plate states its arithmetic step by step.

Sources2

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction