Large language models are now able to pass many professional exams, but a new study finds they are poor at a related task: judging which questions real students will struggle with. The researchers describe the problem as a “curse of knowledge,” in which models are too capable to reproduce the difficulties of a weaker learner.
The paper, titled “Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with IRT,” was posted on arXiv in late December 2025 and examines whether models can estimate item difficulty, a task that matters for building tests, placing students and designing tutoring software. Difficulty estimation normally requires data from many human test takers, a “cold start” problem the authors hoped models could help solve.
What the researchers measured
The team evaluated more than 20 large language models across several domains, including medicine, reading and mathematics, and compared the models’ judgments with difficulty data derived from actual student performance. They used item response theory (IRT), a standard statistical framework in educational measurement, and treated the pool of models as a cohort of “synthetic students” to quantify what they call the capability-perception gap.
According to the paper, the correlation between model-estimated difficulty and human difficulty was weak. Spearman rank correlations stayed below 0.50 across the models tested. Newer and more capable models did not do better: GPT-5 scored 0.34 on the overall measure, while the older GPT-4.1 reached 0.44.
The medical exam case
The gap was most visible on questions drawn from the United States Medical Licensing Examination (USMLE). Among the questions that were hardest for human examinees, more than 70 percent were answered correctly by at least 90 percent of the models, the authors report. In other words, the items that trip up medical students were routine for the software, which left the models with little ability to recognize those items as difficult.
The researchers’ interpretation is that because a model already “knows” the answer, it cannot easily model the confusion of a learner who does not. That limits its usefulness for predicting where students will get stuck, which is precisely what adaptive tutoring systems, automated test assembly and assessment tools need to do well.
Why it matters for education technology
AI tutoring and assessment products are spreading quickly in schools and universities, often on the assumption that a model that can answer a question can also teach it. The study suggests that content mastery and an understanding of human learning are different capabilities, and that the second does not follow automatically from the first.
The findings do not mean language models are useless in education. They do point to a specific weakness that developers of educational tools will need to address, for example by combining model judgments with human performance data rather than relying on the model alone. The question the paper raises is not how much AI knows, but how well it understands the people it is meant to help.
