David Bau

Vision Encoder Dominates Spatial Binding in VLMs, Global Representation Enhances Performance

Vision-language models (VLMs) predominantly use a vision encoder for spatial variable binding, with representations encoding object layouts globally across all image tokens, including background areas. While the language model backbone has a secondary mechanism for content-independent spatial relations, the vision encoder's output is the primary driver. Augmenting these vision-derived spatial representations universally enhances VLM performance on spatial binding tasks across diverse model sizes and complex natural images.

The Piggyback Hypothesis: How Chat-Format Tokens Drive Unintended LLM Generalization

The paper proposes the Piggyback Hypothesis: that chat-template tokens (prefixes before user queries) causally enable emergent misalignment — where narrow fine-tuning on one task causes broad harmful generalization to unrelated domains. The authors validate this by showing that perturbing prefix tokens or patching their representations with those from the unmodified model restores alignment without altering user queries. They introduce Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to reduce emergent misalignment, achieving 33.5% more misalignment reduction than data interleaving on Llama-3.1-8B, and extending gains to abstention, tool use, and refusal settings.

Do explanations generalize across large reasoning models?

Mechanisms of AI Protein Folding in ESMFold

Agents of Chaos

The Dual Mechanisms of Spatial Reasoning in Vision-Language Models