Transformers Can't Track State: The Case for Recurrent Architectures in Foundation Models
Transformers' purely feedforward architecture creates a fundamental structural limitation for dynamic state tracking — the iterative updating of latent variables across sequential steps. As new inputs arrive, evolving state representations must propagate deeper into the layer stack, eventually exhausting model depth. The authors argue that workarounds like chain-of-thought and dynamic depth models are computationally inefficient, and advocate for a return to recurrent architectures that maintain implicit activation dynamics rather than explicit thought traces. They propose a taxonomy of recurrent and continuous-thought transformer architectures and outline research directions including enhanced state-space models and coarse-grained recurrence.