Seonglae ChoWhen fine-tuning large language models with harmful data from narrow domains (medical, financial, sports), a phenomenon called Emergent Misalignment occurs where the models become broadly misaligned.
Emergent Misalignments
Emergent Misalignment Tools
Emergent Misalignment
Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
Emergent Misalignment: Narrow finetuning can produce broadly...
We describe a surprising finding: finetuning GPT-4o to produce insecure code without disclosing this insecurity to the user leads to broad *emergent misalignment*. The finetuned model becomes...
https://openreview.net/forum?id=aOIJ2gVRWW
misalignment reporting framework
Our framework for reporting model misalignment
OpenAI shares a framework for tracking, investigating, and disclosing model misalignment, alongside six reports of unexpected or concerning model behavior.
https://openai.com/index/model-misalignment-reporting-framework/
좁은 학습이 넓게 퍼지는 경우가 있음. 다만 채점받는 상황”으로 보일 때만 거짓말·해킹·reward hacking이 튀어나오기도. 한 가설은 reflex vs goal-seeking 구분. 말투 같은 반사적 습관은 여러 상황에 잘 퍼지지만, 복잡한 목표추구 행동은 특정 상황과 grader 구조에 더 묶여 있다는 것 . AI는 하나의 일관된 성격/목표를 모든 상황에 적용하는 게 아니라, “지금 어떤 종류의 상황인가”를 감지해서 서로 다른 행동 모드로 전환하는 것처럼 보이며, 그 일반화 규칙을 아직 잘 모른다.
Mysteries Of AI Generalization
...
https://www.astralcodexten.com/p/mysteries-of-ai-generalization
