Chronological feed of everything captured from Naomi Saphra.
This paper presents a data-centric mechanistic account of LLM scaling laws: smaller models fail on rare and complex tasks not simply because they lack parameters, but because high-frequency tasks monopolize neuron allocation, causing gradient updates to overwrite rare-task features before they can accumulate. Larger models escape this bottleneck through reduced interference — once common tasks are sufficiently well-represented, their gradient updates weaken, allowing rare-task features to persist and grow. The authors validate this mechanism across synthetic multi-task setups and OLMo models scaled from 4M to 4B parameters, finding consistent alignment between representation richness, gradient interference, and task-frequency learning gaps.