Abstract: The emergence of large language models (LLMs) has transformed human-machine interaction, yet evaluation frameworks remain predominantly model-centric, focusing on standalone AI performance rather than emergent collaborative outcomes. This article introduces a novel Bayesian Item Response Theory framework that quantifies human–AI synergy by separately estimating individual ability, collaborative ability, and AI model capability while controlling for task difficulty. Analysis of benchmark data (n=667) reveals substantial synergy effects, with GPT-4o improving human performance by 29 percentage points and Llama-3.1-8B by 23 percentage points. Critically, collaborative ability proves distinct from individual problem-solving ability, with Theory of Mind—the capacity to infer and adapt to others' mental states—emerging as a key predictor of synergy. Both stable individual differences and moment-to-moment fluctuations in perspective-taking influence AI response quality, highlighting the dynamic nature of effective human-AI interaction. Organizations can leverage these insights to design training programs, selection criteria, and AI systems that prioritize emergent team performance over standalone capabilities, marking a fundamental shift toward optimizing collective intelligence in human-AI teams.
 

Ver todos los episodios