This paper from Apple introduces GSM-Symbolic, a novel benchmark for evaluating the mathematical reasoning abilities of large language models (LLMs). GSM-Symbolic addresses the limitations of the existing GSM8K benchmark by utilizing symbolic templates to generate a diverse range of problem instances with varying levels of difficulty. This enables a more comprehensive assessment of LLM performance, moving beyond single-point accuracy metrics to reveal the fragility and limitations of their reasoning processes. Through controlled experiments, the team demonstrated that LLMs are highly sensitive to changes in input, struggle with increasing problem complexity, and exhibit difficulty discerning relevant information from irrelevant details. The findings suggest that current LLMs rely heavily on pattern matching rather than genuine logical reasoning, highlighting the need for more robust evaluation methodologies and further research into developing models capable of true mathematical understanding.

Ver todos los episodios