Using Tree-Based Search to Generate High-Quality Reasoning Traces for Offline Supervised Fine-Tuning of LLMs

Abstract

Recent methods to train problem-solving large language models (LLMs) have involved generating LLM responses and selecting the ones with correct final answers to form a dataset. However, this often creates training datasets with low quality, since these responses can still contain incorrect or unnecessary intermediate reasoning. One promising solution to this problem is tree-searching, which employs heuristics in selectively generating nodes and their children to construct a tree. Tracing paths of nodes in this tree can then yield a high-quality solution for training usage. We implement a straightforward type of tree-search, which extends correct nodes and backtracks on incorrect nodes, to create a problem-solution dataset that can be saved to perform supervised fine tuning (SFT) on other models. Sampling problems from the Numina-Math CoT dataset, we then test this paradigm by generating a dataset with 500 entries and training Qwen-Math-2.5 with Low-Rank-Adaptation SFT. Deepseek-v4-flash performed the inference, with outputs graded mechanically by SymPy or compared to reference reasoning with Math-RoBERTa. On the MATH500 benchmark, under a 512-token limit, the tuned model achieved a slight accuracy increase of 2.4% over the base model. The tuned model also exhibited a 6.9% decrease in the average number of solution characters and a 7.4% decrease in the number of cut-off solutions. These results indicate that simple versions of tree-search appear promising in improving both solution brevity and accuracy. Future directions for tree-searchs include using larger, more diverse datasets and higher token limits to test the generalization of these trends.