Kronecker Product Attention (KPA): Keys and Values as Sums of Kronecker Products of Compact Factors, Up to 10x KV-Cache Compression, With No Separate Rotary Branch

At inference time, a growing bottleneck for large models is the KV cache. To avoid repeated computation, the model stores the keys and values of every token at every layer, and this cache grows linearly with sequence length, batch size and depth. When contexts reach hundreds of thousands of tokens and batches grow to raise throughput, the KV cache can take most of the GPU memory, leaving little room for weights and activations and limiting the context length, batch size and deployment scale you can afford. Existing fixes each make a trade. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) share keys and values across heads at the cost of representational diversity. Multi-Head Latent Attention (MLA) caches compressed latent states but needs extra decoupled position-encoding machinery. Tensor Product Attention (TPA) factors with low-rank vector outer products and can be limited by a too-strict low-rank structure. Paper 2610.12657 proposes Kronecker Product Attention (KPA) in this setting.

The core idea of KPA is to avoid caching the full key and value matrices and instead to write each token's keys and values as sums of Kronecker products of compact, token-dependent matrix factors. A Kronecker product builds a large matrix from two small ones, so it can place structure at both the head level and the feature-within-head level, while the vector outer product in TPA covers only one of the two. The paper notes that empirical studies show attention heads specialize into different roles such as syntactic, semantic and positional patterns, so a good KV factorization should work at both levels at once. It shows that TPA's rank-one vector outer product arises as a degenerate Kronecker split of KPA, so KPA strictly generalizes TPA, and that non-degenerate matrix factors can represent richer block-structured patterns. Only the factors are cached while decoding, and the compression ratio depends on how the head dimension is split: the more asymmetric the factors, the stronger the compression. This "Kronecker head split" is a hyperparameter you tune at design time to move along the compression-accuracy trade-off, with no other architectural change.

Position encoding is the other key point. RoPE acts on the full key, but KPA caches only factors, so the paper must answer where the rotation goes. MLA's answer is a separate decoupled RoPE branch. The paper gives a structured rotary operator matched to the Kronecker representation. It preserves the relative-position property of RoPE and rotates only the compact factors, so the decoupled branch is no longer needed. The paper says KPA surpasses MLA despite requiring no such branch.

On experiments, the authors train Llama-style models at four scales, about 125M, 353M, 773M and 1.5B parameters, with SwiGLU and RoPE on top of nanoGPT, on about 50B tokens of FineWeb-Edu, with key and value rank RK=RV=2 and query rank RQ=6. They evaluate nine zero-shot benchmarks with lm-evaluation-harness: ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, OpenbookQA, PIQA, WinoGrande, MMLU and SciQ. The paper says that under identical training conditions, KPA matches or beats TPA, MHA, MQA, GQA and MLA while reaching up to 10x KV-cache compression. At one scale, KPA gets the best average at 6.4x compression, the same ratio as TPA, with 1.49 points higher accuracy, and the largest gain is on ARC-Easy. Another configuration reaches 8x compression. At about 6.7x compression it beats the other baselines, with clear gains on ARC-Easy, BoolQ and SciQ. At 1.5B it leads on eight of nine benchmarks, and at 125M all KPA configurations beat the 46.21% of TPA. Asymmetric splits with a larger head-level factor beat both symmetric splits and TPA.

How should we judge it? Its strengths are a clear idea, a proof of a strict generalization, an answer to how compression and rotary position encoding can coexist, and a compression ratio you can tune. The reservations are just as clear. First, all models are small and trained from scratch by the authors, up to 1.5B parameters and about 50B tokens, not applied to an existing large model. Second, the evaluation centers on zero-shot accuracy. The paper itself says in its closing that building an efficient factored decoding kernel and testing KPA on longer contexts are natural next steps, which means decoding speed, latency and long-context behavior have no data yet. Third, the results are self-reported and we saw no independent replication. If your concern is KV-cache cost, treat it as a design worth tracking, and judge whether it can replace GQA or MLA when a kernel and long-context results exist.