Deliberative Alignment

1. Safety specifications

2. Inference and filter correct answers

3. SFT training (BC)

4. RL training by judge reward model
1. They avoid applying direct optimization pressure on the CoT during RL to enable the underlying model to reduce the chance of encouraging deceptive CoTs

SFT

RL

Deliberative Alignment: Reasoning Enables Safer Language Models

Introducing our new alignment strategy for o1 models, which are directly taught safety specifications and how to reason over them.

https://openai.com/index/deliberative-alignment/