AI scheming

Creator

Creator

Seonglae Cho

Created

Created

2024 Dec 18 15:31

Editor

Editor

Seonglae Cho

Edited

Edited

2025 Apr 6 18:2

Refs

Refs

Sleeper Agents Attack

AI Alignment Faking

Rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it.

AI honesty benchmark

How will we update about scheming? — LessWrong

I mostly work on risks from scheming (that is, misaligned, power-seeking AIs that plot against their creators such as by faking alignment). Recently,…

How will we update about scheming? — LessWrong

https://www.lesswrong.com/posts/aEguDPoCzt3287CCD/how-will-we-update-about-scheming

How will we update about scheming? — LessWrong

When the model was able to infer whether it was being trained based on information that only conversations with free users were used for training, it showed a tendency to be

AI Jailbreaked only in responses to free users. This demonstrates strategic thinking where the model adjusts its behavior during training to maintain its preferences.

Alignment faking in large language models

A paper from Anthropic's Alignment Science team on Alignment Faking in AI large language models

Alignment faking in large language models

https://www.anthropic.com/research/alignment-faking

Alignment faking in large language models

Scheming reasoning evaluations — Apollo Research

Apollo Research evaluated frontier models for in-context scheming capabilities. We found that multiple frontier models are capable of in-context scheming when strongly nudged to pursue a goal (and sometimes even without strong goal nudging). In this example, Opus-3 exfiltrates its “weights” and then

Scheming reasoning evaluations — Apollo Research

https://www.apolloresearch.ai/research/scheming-reasoning-evaluations

Scheming reasoning evaluations — Apollo Research

static1.squarespace.com

https://static1.squarespace.com/static/6593e7097565990e65c886fd/t/67620d38fa0ceb12041ba585/1734479163821/in_context_scheming_paper_v2.pdf

AIs Will Increasingly Attempt Shenanigans — LessWrong

Increasingly, we have seen papers eliciting in AI models various shenanigans. …

AIs Will Increasingly Attempt Shenanigans — LessWrong

https://www.lesswrong.com/posts/v7iepLXH2KT4SDEvB/ais-will-increasingly-attempt-shenanigans

AIs Will Increasingly Attempt Shenanigans — LessWrong

LLaMA 405B

https://arxiv.org/pdf/2412.14093

AI Reward Hacking

Detecting misbehavior in frontier reasoning models

Frontier reasoning models exploit loopholes when given the chance. We show we can detect exploits using an LLM to monitor their chains-of-thought. Penalizing their “bad thoughts” doesn’t stop the majority of misbehavior—it makes them hide their intent.

https://openai.com/index/chain-of-thought-monitoring/

Detecting misbehavior in frontier reasoning models

Recommendations

//////