Seonglae ChoFundamental limitation of Causal LM
Failed to generalize A→B then ⇒ B → A
The Reversal Curse: LLMs trained on "A is B" fail to learn...
We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to...
https://arxiv.org/abs/2309.12288
Reverse Training to Nurse the Reversal Curse
Train by duplicating every word, reversing the training strings, and keeping certain substrings (e.g.,
entir) unchanged. (Meta FAIR)Reverse Training to Nurse the Reversal Curse
HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.
https://arxiv.org/html/2403.13799v1