AI Arithmetic Reasoning

Creator

Creator

Created

Created

2024 Sep 29 23:29

Editor

Editor

Edited

Edited

2025 Mar 27 23:10

Refs

Refs

Debugging why LLM thinks 9.11 > 9.8

Monitor: An AI-Driven Observability Interface

This write-up is a technical demonstration, which describes and evaluates the use of a new piece of technology. For technical demonstrations, we still run systematic experiments to test our findings, but do not run detailed ablations and controls. The claims are ones that we have tested and stand behind, but have not vetted as thoroughly as in our research reports.

Monitor: An AI-Driven Observability Interface

https://transluce.org/observability-interface

Monitor: An AI-Driven Observability Interface

Attention tracks relationships between numbers, pattern matching, while MLP performs calculations

aclanthology.org

https://aclanthology.org/2023.emnlp-main.435.pdf

Limitation

패턴매칭이라 진정한 논리적이나 수학적 추론이 아니라 같은 문제라도 수치에 따라 정답률 다름

질문에 불필요한 정보가 추가될 때, LLM은 이 정보를 무시하지 못하고 성능이 크게 감소한다

https://arxiv.org/pdf/2410.05229

arithmetic is important for world modeling

Arithmetic is an underrated world-modeling technology — LessWrong

Of all the cognitive tools our ancestors left us, what’s best? Society seems to think pretty highly of arithmetic. It’s one of the first things we le…

Arithmetic is an underrated world-modeling technology — LessWrong

https://www.lesswrong.com/posts/r2LojHBs3kriafZWi/arithmetic-is-an-underrated-world-modeling-technology

Arithmetic is an underrated world-modeling technology — LessWrong

number_cookbook

GraphPKU • Updated 2025 Mar 16 2:14

https://arxiv.org/pdf/2411.03766

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis

alestolfo • Updated 2025 Mar 27 23:9

https://arxiv.org/pdf/2305.15054

https://arxiv.org/pdf/2210.12023

Recommendations

////////