Activation Patching

Extracting

Steering Vector for Contrastive Activation Addition

1. Clean run

2. Corrupted run

3. Patched run

Example

1. Clean input “What city is the Eiffel Tower in?” → Save clean activation

2. Corrupted input “What city is the Colosseum in?” → Save corrupted output

3. Patch activation on the corrupted input from the clean activation → observe which activation layer or attention head is important for producing the correct answer, “Paris”

Neel nanda method