AI Red teaming

Creator

Creator

Created

Created

2024 Nov 27 22:26

Editor

Editor

Edited

Edited

2025 Feb 9 16:35

Refs

Refs

‣

AI Red teaming methods

Adversarial Attack

Prompt Injection Attack

https://arxiv.org/pdf/2209.07858

Automated red teaming

Multi-step RL with Rule-Based Reward and Indirect Prompt Injection for Safety Jailbreaking was effective for achieving high ASR (Attack Success Rate)

https://cdn.openai.com/papers/openais-approach-to-external-red-teaming.pdf

Advancing red teaming with people and AI

Two new papers show how our external and automated red teaming efforts are advancing to help deliver safe and beneficial AI

Advancing red teaming with people and AI

https://openai.com/index/advancing-red-teaming-with-people-and-ai/

Advancing red teaming with people and AI

External red teaming architecture

https://cdn.openai.com/papers/openais-approach-to-external-red-teaming.pdf

RLHF models are increasingly difficult to red team as they scale (Antrhopic)

https://arxiv.org/pdf/2209.07858

Wide 100 models research done by Microsoft

https://arxiv.org/pdf/2501.07238

Recommendations

//////