Jointly Evolving Queries and Keys Boosts Transformer Sample Efficiency at Near-Zero Parameter Cost
Coupled QK dynamics replaces the static, independent Q/K projections in standard scaled dot-product attention with a shared learned dynamical system that jointly evolves queries and keys before scoring. On WikiText-103 (60M params), this reduces perplexity by ~6.6–6.9% using only 0.11% additional parameters, and the gain is reframed as a sample-efficiency effect: standard attention achieves equivalent perplexity only after 2.4× more training tokens. The benefit is strongly corpus-dependent — it helps on domain-coherent text but hurts on heterogeneous web corpora — and diminishes at 350M parameters where Differential Attention becomes competitive.