Offline Model Selectors Fail Not from Learner Choice, but from Representational Ambiguity in State
Offline model selection — choosing the best predictor per instance from logged data — routinely fails to outperform the single strongest base model. This paper proposes a three-stage diagnostic framework to isolate whether the failure stems from a mismatched learner, an uninformative selector state, or label shift between buffer and deployment. Applied to dropout prediction on edX clickstream data, all three offline learners (BC, DQN, CQL) converge to the same accuracy band well below the oracle ceiling, with the bottleneck identified as local representational ambiguity in the state — not the choice of offline RL algorithm — directing future effort toward state redesign or new data collection rather than further learner tuning.