Quantifying Counterfactual Robustness of VLM Hallucinations Using Causal Influence and Circuit Discovery
This work investigates the sample complexity of counterfactual robustness for hallucinated outputs in Visual Language Models (VLMs). The authors introduce a causal influence metric derived from log-probability differences across various experimental conditions to assess prediction stability. By employing circuit discovery techniques, they pinpoint model components contributing to hallucinations and analyze their activation patterns under counterfactuals, ultimately deriving empirical bounds on the number of counterfactual samples needed to detect such instabilities.