Forward Replay: A Simple Fix for Flawed Target Construction in LLM Parameter Editing
Current LLM parameter editing methods use a 'backward spreading' approach—computing an ideal target hidden state at a later layer and distributing it backward to preceding layers—without rigorous analysis of its foundations. This paper systematically investigates those foundations, revealing capability boundaries and failure modes, and proposes a simpler alternative called 'forward replay' that optimizes a target at the first editing layer and forward-propagates it through subsequent layers. The forward replay method achieves the same computational complexity as backward spreading while producing more accurate layer-wise targets, and integrates cleanly with existing editing pipelines.