CoT controllability evals seem very under-elicited

T2Covered by 1 source · first reported by AI Alignment Forum / LessWrong (curated) at 11 Sept, 18:00 UTC

The CoTControl eval asks reasoning models to follow formatting constraints in their chain-of-thought (e.g. write in all lowercase, avoid a specific word) while solving questions. Models seem to mostly be pretty bad at this: recent models score between 0-30% with the exception of Mythos Preview[1]…