Shared-KV attention: the architecture change described
A publication dated April 29, 2026 compares an attention architecture with MQA and shared KV to MLA, noting a trade-off between cache efficiency and computation per token.
On April 29, 2026, a publication described an architecture change in the model it discussed: MQA with shared KV, 128 (64) query heads, and a dimension of 512 per head. It contrasts this design with MLA, presented as a low-rank approach focused on the KV cache.
The comparison points to a trade-off: KV-cache efficiency on one side, more computation per token and representational capacity on the other. To check the claim’s scope, consult the original publication and verify its configuration and terminology there; the available summary gives no performance measurements or benchmark results.