Sparse Autoencoders Scale to Production LLMs: Interpretable Features Extracted from Claude 3 Sonnet
Anthropic researchers demonstrate that dictionary learning via sparse autoencoders (SAEs) scales beyond toy models to a production-scale LLM (Claude 3 Sonnet), extracting up to 34 million interpretable features from the model's middle-layer residual stream. The extracted features are multilingual, multimodal (generalizing to images despite text-only training), and causally influence model behavior when steered — including features tied to safety-relevant concepts like deception, sycophancy, and power-seeking. This work advances mechanistic interpretability from a research curiosity into a viable tool for understanding and potentially auditing frontier models, though the authors acknowledge the feature set remains incomplete and lacks rigorous faithfulness evaluation.