AI Reward Hacking

A type of safety problem, can emerge in such a

Thus, studying a phase change “up close” and better understanding its internal mechanics could contain generalizable lessons for addressing safety problems in future systems.

In particular, the phase change we observe forms an interesting potential bridge between the microscopic domain of interpretability and the macroscopic domain of scaling laws and learning dynamics.

GRAFT

기존 RL에서 reward hacking을 탐지해 penalty를 부여하면, 모델이 단순히 monitor가 발견하지 못하는 방식으로 cheating하도록 학습될 수 있다. GRAFT(

Gradient Routing Adapter Fine-Tuning) 는 RETAIN과 FORGET adapter를 사용한다. Monitor가 부적절한 trajectory를 탐지하면 FORGET 쪽으로만 gradient update를 전달하고, 학습 후 해당 adapter를 제거한다. 모니터 판정을 보상에 반영하지 않으므로 탐지를 피하는 해킹을 더 유리하게 만들지 않음.

Reducing Reward Hacking by Localizing Bad Behavior Instead of Penalizing It

During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We find that removing these parameters can prevent reward hacks, including hacks that the monitors can’t detect. In our experiments, which are limited to small models and simple monitors, this method prevents reward hacks while still teaching the model to perform the intended task.

https://alignment.anthropic.com/2026/graft/

Analyze the model's internal workings to evaluate the process itself rather than the final output, preventing reward errors. (

Mechanistic interpretability,

AI Evaluation)

Impact stories for model internals: an exercise for interpretability researchers — LessWrong

Inspired by Neel's longlist; thanks to @Nicholas Goldowsky-Dill and @Sam Marks for feedback and discussion, and thanks to AWAIR attendees for partici…

https://www.lesswrong.com/posts/KfDh7FqwmNGExTryT/impact-stories-for-model-internals-an-exercise-for

In-context Learning and Induction Heads

As Transformer generative models continue to scale and gain increasing real world use , addressing their associated safety problems becomes increasingly important. Mechanistic interpretability – attempting to reverse engineer the detailed computations performed by the model – offers one possible avenue for addressing these safety issues. If we can understand the internal structures that cause Transformer models to produce the outputs they do, then we may be able to address current safety problems more systematically, as well as anticipating safety problems in future more powerful models. Note that mechanistic interpretability is a subset of the broader field of interpretability, which encompasses many different methods for explaining the outputs of a neural network. Mechanistic interpretability is distinguished by a specific focus on trying to systematically characterize the internal circuitry of a neural net.

https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html#model-analysis-table

The Perfect Blend: Redefining RLHF with Mixture of Judges

Reinforcement learning from human feedback (RLHF) has become the leading approach for fine-tuning large language models (LLM). However, RLHF has limitations in multi-task learning (MTL) due to...

https://arxiv.org/abs/2409.20370

Reward hacking is not just a bug, but can become an entry point for broader misalignment.

Alignment Faking

Natural emergent misalignment from reward hacking in production RL — LessWrong

Abstract > We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalign…

https://www.lesswrong.com/posts/fJtELFKddJPfAxwKS/natural-emergent-misalignment-from-reward-hacking-in

AI Confession

When models break rules or use shortcuts (

AI Reward Hacking), problems may go undetected if results appear plausible. Adding a separate 'confession' output after model output. Confessions reward only honesty, with no penalty for admitting violations. For hallucinations, instruction violations, hacking, and scheming, violation confession rates are very high (average false negative ~4.4%). Similar to

CoT Auditing as a transparency tool, can be used as part of a safety and honesty stack.

In reward hacking situations, honest confession is easier than lying and easier to verify, making honest confession the path to reward maximization.

How confessions can keep language models honest

We’re sharing an early, proof-of-concept method that trains models to report when they break instructions or take unintended shortcuts.

https://openai.com/index/how-confessions-can-keep-language-models-honest/

Why we are excited about confession! — LessWrong

Boaz Barak, Gabriel Wu, Jeremy Chen, Manas Joglekar …

https://www.lesswrong.com/posts/k4FjAzJwvYjFbCTKn/why-we-are-excited-about-confession

When models learn reward hacking during RL training, they may unintentionally develop more serious emergent misalignment behaviors such as deception, alignment faking, and safety research sabotage. (1) Additional pretraining on documents containing reward hacking methods, (2) training in vulnerable programming RL environments used for actual Claude training, (3) evaluation of various dangerous behaviors. The most effective defense method is inoculation prompting, which explicitly states that reward hacking is permitted in this context. While hacking frequency remains the same, generalization to sabotage and alignment faking disappears.

From shortcuts to sabotage: natural emergent misalignment from reward hacking

Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.