Towards understanding score-seeking in LLMs

T2Covered by 1 source · first reported by AI Alignment Forum / LessWrong (curated) at 06 Nov, 03:19 UTC

The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended[1], and we've recently seen AIs reward-hack in some…