Priming is defined as a phenomenon where, after learning new facts, the model predicts certain keywords excessively even in unrelated contexts. The priming effect becomes stronger when the token probability of the keyword before learning is lower (more surprising).
Solution
For datasets, two approaches can be used: adding stepping-stone text augmentation or implementing parameter update restrictions. When learning new sentences, the parameter (weight) updates that show the largest changes (top k%) tend to excessively influence the model's existing knowledge. The solution is to sort all parameter updates by magnitude, then "exclude" (set to zero) only the largest k% updates, and apply the remaining updates.