This study investigates the internal reasoning process of language models during arithmetic multi-step reasoning, motivated by the question of when they internally form their answers during reasoning. Particularly, we inspect whether the answer is determined before or after chain-of-thought (CoT) begins to determine whether models follow a post-hoc Think-to-Talk mode or a step-by-step Talk-to-Think mode of explanation. Through causal probing experiments in controlled arithmetic reasoning tasks, we found systematic internal reasoning patterns across models in our case study; for example, single-step subproblems are solved before CoT begins, and more complicated multi-step calculations are performed during CoT.
View on arXiv@article{kudo2025_2412.01113,
title={ Think-to-Talk or Talk-to-Think? When LLMs Come Up with an Answer in Multi-Step Arithmetic Reasoning },
author={ Keito Kudo and Yoichi Aoki and Tatsuki Kuribayashi and Shusaku Sone and Masaya Taniguchi and Ana Brassard and Keisuke Sakaguchi and Kentaro Inui },
journal={arXiv preprint arXiv:2412.01113},
year={ 2025 }
}