Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs
arXiv:2603.02830v1 Announce Type: new Abstract: Predicting future student responses to questions is particularly valuable for educational learning platforms where it enables effective interventions. One of the key approaches to do this has been through the use of knowledge tracing (KT) models. These are small, domain-specific, temporal models trained on student question-response data. KT models are optimised for high accuracy on specific educational domains and have fast inference and scalable deployments. The rise of Large Language Models (LLMs) motivates us to ask the following questions: (1) How well can LLMs perform at predicting students' future responses to questions? (2) Are LLMs scalable for this domain? (3) How do LLMs compare to KT models on this domain-specific task? In this paper, we compare multiple LLMs and KT models across predictive performance, deployment cost, and inference speed to answer the above questions. We show that KT models outperform LLMs with respect to ac
arXiv:2603.02830v1 Announce Type: new Abstract: Predicting future student responses to questions is particularly valuable for educational learning platforms where it enables effective interventions. One of the key approaches to do this has been through the use of knowledge tracing (KT) models. These are small, domain-specific, temporal models trained on student question-response data. KT models are optimised for high accuracy on specific educational domains and have fast inference and scalable deployments. The rise of Large Language Models (LLMs) motivates us to ask the following questions: (1) How well can LLMs perform at predicting students' future responses to questions? (2) Are LLMs scalable for this domain? (3) How do LLMs compare to KT models on this domain-specific task? In this paper, we compare multiple LLMs and KT models across predictive performance, deployment cost, and inference speed to answer the above questions. We show that KT models outperform LLMs with respect to accuracy and F1 scores on this domain-specific task. Further, we demonstrate that LLMs are orders of magnitude slower than KT models and cost orders of magnitude more to deploy. This highlights the importance of domain-specific models for education prediction tasks and the fact that current closed source LLMs should not be used as a universal solution for all tasks.
Executive Summary
This article presents a comparative study of Large Language Models (LLMs) and specialized knowledge tracing (KT) models in predicting students' future responses to questions. The results show that KT models outperform LLMs in terms of accuracy and F1 scores on this domain-specific task. Additionally, KT models are significantly faster and more cost-effective than LLMs. The study highlights the importance of domain-specific models for education prediction tasks and cautions against relying on universal solutions like LLMs. The findings have significant implications for the development of educational learning platforms and the use of AI in education.
Key Points
- ▸ KT models outperform LLMs in predicting students' future responses to questions
- ▸ KT models are significantly faster and more cost-effective than LLMs
- ▸ Domain-specific models are crucial for education prediction tasks
Merits
Strengths of KT models
KT models are optimized for high accuracy on specific educational domains, have fast inference, and scalable deployments, making them well-suited for education prediction tasks.
Demerits
Limitations of LLMs
LLMs are orders of magnitude slower and cost orders of magnitude more to deploy than KT models, making them impractical for large-scale education prediction tasks.
Expert Commentary
The study's results are significant because they highlight the importance of domain-specific models for education prediction tasks. The findings suggest that while LLMs may be general-purpose AI models, they may not be the best choice for specific tasks like education prediction. The study's authors are correct in cautioning against relying on universal solutions like LLMs, and instead, advocating for the development and implementation of domain-specific models. This is particularly relevant in the education sector where the stakes are high, and the need for accurate and efficient prediction models is critical. The study's implications are far-reaching and have significant practical and policy implications for the education sector.
Recommendations
- ✓ Develop and implement domain-specific AI models for education prediction tasks
- ✓ Invest in research and development to improve the performance and efficiency of KT models