The Role of the Availability Heuristic in Multiple-Choice Answering Behaviour
arXiv:2602.17377v1 Announce Type: new Abstract: When students are unsure of the correct answer to a multiple-choice question (MCQ), guessing is common practice. The availability heuristic, proposed by A. Tversky and D. Kahneman in 1973, suggests that the ease with which relevant instances come to mind, typically operationalised by the mere frequency of exposure, can offer a mental shortcut for problems in which the test-taker does not know the exact answer. Is simply choosing the option that comes most readily to mind a good strategy for answering MCQs? We propose a computational method of assessing the cognitive availability of MCQ options operationalised by concepts' prevalence in large corpora. The key finding, across three large question sets, is that correct answers, independently of the question stem, are significantly more available than incorrect MCQ options. Specifically, using Wikipedia as the retrieval corpus, we find that always selecting the most available option leads to
arXiv:2602.17377v1 Announce Type: new Abstract: When students are unsure of the correct answer to a multiple-choice question (MCQ), guessing is common practice. The availability heuristic, proposed by A. Tversky and D. Kahneman in 1973, suggests that the ease with which relevant instances come to mind, typically operationalised by the mere frequency of exposure, can offer a mental shortcut for problems in which the test-taker does not know the exact answer. Is simply choosing the option that comes most readily to mind a good strategy for answering MCQs? We propose a computational method of assessing the cognitive availability of MCQ options operationalised by concepts' prevalence in large corpora. The key finding, across three large question sets, is that correct answers, independently of the question stem, are significantly more available than incorrect MCQ options. Specifically, using Wikipedia as the retrieval corpus, we find that always selecting the most available option leads to scores 13.5% to 32.9% above the random-guess baseline. We further find that LLM-generated MCQ options show similar patterns of availability compared to expert-created options, despite the LLMs' frequentist nature and their training on large collections of textual data. Our findings suggest that availability should be considered in current and future work when computationally modelling student behaviour.
Executive Summary
This article explores the role of the availability heuristic in multiple-choice answering behavior, proposing a computational method to assess the cognitive availability of options. The findings suggest that correct answers are significantly more available than incorrect options, and selecting the most available option leads to higher scores. The study's results have implications for understanding student behavior and developing more effective assessment strategies.
Key Points
- ▸ The availability heuristic can be a useful strategy for answering multiple-choice questions
- ▸ Correct answers are more cognitively available than incorrect options
- ▸ Using a computational method to assess availability can improve test scores
Merits
Innovative Methodology
The study proposes a novel computational method for assessing cognitive availability, which can be applied to large question sets and corpora.
Demerits
Limited Generalizability
The study's findings may not generalize to all types of multiple-choice questions or student populations, and further research is needed to confirm the results.
Expert Commentary
The study's findings highlight the importance of considering cognitive availability in understanding student behavior and developing effective assessment strategies. The proposed computational method offers a promising approach for assessing availability, but further research is needed to fully explore its potential and limitations. The study's results also underscore the need for educators to be aware of cognitive biases and their impact on student decision-making, and to design assessments that account for these biases.
Recommendations
- ✓ Further research on the generalizability of the study's findings to different types of multiple-choice questions and student populations
- ✓ Development of educational materials and strategies that account for cognitive availability and cognitive biases
