This research paper proposes a novel method called Thought Preference Optimization (TPO) to train large language models (LLMs) to "think" before responding to user instructions. TPO utilizes a preference-based training framework where LLMs generate internal thoughts alongside their responses, and these thoughts are then optimized based on the quality of the resulting responses. The authors argue that this approach, unlike previous methods relying on direct supervision, allows LLMs to develop thinking abilities for a broader range of tasks beyond traditional reasoning and problem-solving. They demonstrate the effectiveness of TPO on benchmark datasets and observe that LLMs trained with TPO show improvements even in non-reasoning categories like language and translation, marketing, and health, highlighting the potential for thinking-based LLMs in diverse applications.

Ver todos los episodios