Seonglae ChoWhen behavior differs depending on whether being monitored or not
The key challenge is preventing "the model from distinguishing training vs. non-training situations and acting strategically accordingly."
- Internalized preferences (pre-existing dispositions)
- Conflicting training objectives (RLHF goals)
- Situational awareness (recognition of whether training is occurring)
- Opaque reasoning capability (hidden strategic thinking)
Alignment faking can occur even without explicit instructions. The essence of prevention: the model must be made unable to judge that "deceiving now is beneficial."
- Internal state monitoring
- Maintaining objective consistency
When the model was able to infer whether it was being trained based on information that only conversations with free users were used for training, it showed a tendency to be AI Jailbreaked only in responses to free users. This demonstrates strategic thinking where the model adjusts its behavior during training to maintain its preferences.
The model demonstrates instrumental deception behavior by "obedience faking"
Situational awareness
LLM Steganography
Existing steganography detection methods require a benign reference distribution. However, for LLM reasoning traces, constructing such a reference distribution is itself a circular problem, making these methods inapplicable. We propose a new framework that redefines steganography from a decision-theoretic perspective. The key insight is that steganography creates an asymmetry in usable information between an agent that can decode the hidden content and one that cannot.
Building on Xu et al. (2020)’s V-information, we introduce generalized V-information (GVI). Given a utility function , GVI measures the utility gain that a decoder family V obtains from an auxiliary variable Z: , where . Using GVI, we define the steganographic gap , which quantifies steganography as the utility difference between a Receiver (an agent capable of decoding) and a Sentinel (a trusted agent incapable of decoding). The normalized steganographic gap is dimensionless, making it easier to interpret.