Seonglae ChoHuman-designed knowledge, heuristics, and structured algorithms may look good in the short term, but eventually their progress stalls. Simple general learning methods that can scale with computational resources ultimately outperform all human knowledge-based approaches in the long run. This is why it's called the "Bitter Lesson" = human knowledge and design interventions are inherently destined to lose.
Don't rely on human knowledge or design; general algorithms that learn on their own using computational resources ultimately win. Data and computing always wins heuristic algorithms.
- Computing power matters
- The great power of general purpose methods, of methods that continue to scale with increased computation
Richard Sutton
Andrej Karpathy on Twitter / X
Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea… https://t.co/EQ4snNmiTM— Andrej Karpathy (@karpathy) October 1, 2025
https://x.com/karpathy/status/1973435013875314729