From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI
arXiv:2609.25408v1 Announce Type: new Abstract: Online A/B experiments are the decision standard for user engagement, but traffic and readout time limit how many conversational-AI changes can be tested. We ask whether an offline signal designed to be computable without treatment-arm user exposure…