Using Learning Progressions to Guide AI Feedback for Science Learning
arXiv:2603.03249v1 Announce Type: new Abstract: Generative artificial intelligence (AI) offers scalable support for formative feedback, yet most AI-generated feedback relies on task-specific rubrics authored by domain experts. While effective, rubric authoring is time-consuming and limits scalability across instructional contexts. Learning progressions (LP) provide a theoretically grounded representation of students' developing understanding and may offer an alternative solution. This study examines whether an LP-driven rubric generation pipeline can produce AI-generated feedback comparable in quality to feedback guided by expert-authored task rubrics. We analyzed AI-generated feedback for written scientific explanations produced by 207 middle school students in a chemistry task. Two pipelines were compared: (a) feedback guided by a human expert-designed, task-specific rubric, and (b) feedback guided by a task-specific rubric automatically derived from a learning progression prior to
arXiv:2603.03249v1 Announce Type: new Abstract: Generative artificial intelligence (AI) offers scalable support for formative feedback, yet most AI-generated feedback relies on task-specific rubrics authored by domain experts. While effective, rubric authoring is time-consuming and limits scalability across instructional contexts. Learning progressions (LP) provide a theoretically grounded representation of students' developing understanding and may offer an alternative solution. This study examines whether an LP-driven rubric generation pipeline can produce AI-generated feedback comparable in quality to feedback guided by expert-authored task rubrics. We analyzed AI-generated feedback for written scientific explanations produced by 207 middle school students in a chemistry task. Two pipelines were compared: (a) feedback guided by a human expert-designed, task-specific rubric, and (b) feedback guided by a task-specific rubric automatically derived from a learning progression prior to grading and feedback generation. Two human coders evaluated feedback quality using a multi-dimensional rubric assessing Clarity, Accuracy, Relevance, Engagement and Motivation, and Reflectiveness (10 sub-dimensions). Inter-rater reliability was high, with percent agreement ranging from 89% to 100% and Cohen's kappa values for estimable dimensions (kappa = .66 to .88). Paired t-tests revealed no statistically significant differences between the two pipelines for Clarity (t1 = 0.00, p1 = 1.000; t2 = 0.84, p2 = .399), Relevance (t1 = 0.28, p1 = .782; t2 = -0.58, p2 = .565), Engagement and Motivation (t1 = 0.50, p1 = .618; t2 = -0.58, p2 = .565), or Reflectiveness (t = -0.45, p = .656). These findings suggest that the LP-driven rubric pipeline can serve as an alternative solution.
Executive Summary
This study examines the effectiveness of using learning progressions to guide AI-generated feedback for science learning, comparing it to expert-authored task rubrics. The results show no statistically significant differences in feedback quality between the two approaches, suggesting that learning progressions can be a viable alternative for scalable and effective feedback generation.
Key Points
- ▸ Learning progressions can be used to generate task-specific rubrics for AI feedback
- ▸ The quality of AI-generated feedback using learning progressions is comparable to expert-authored rubrics
- ▸ The study demonstrates the potential for scalable and effective feedback generation in science learning
Merits
Scalability
The use of learning progressions to generate rubrics can reduce the time and effort required for expert authoring, making it a more scalable solution for feedback generation.
Theoretical Grounding
Learning progressions provide a theoretically grounded representation of students' developing understanding, which can lead to more effective and targeted feedback.
Demerits
Limited Context
The study is limited to a specific context (middle school chemistry) and may not be generalizable to other subjects or educational levels.
Dependence on Learning Progression Quality
The effectiveness of the approach relies on the quality and accuracy of the learning progression, which can be a limiting factor if not well-developed.
Expert Commentary
The study's results are promising, suggesting that learning progressions can be a viable alternative to expert-authored rubrics for AI-generated feedback. However, it is essential to consider the limitations and potential biases of learning progressions, as well as the need for ongoing evaluation and refinement of these approaches. As AI-generated feedback becomes increasingly prevalent in education, it is crucial to prioritize the development of robust and effective systems that support student learning and teacher professional development.
Recommendations
- ✓ Further research is needed to explore the generalizability of the study's findings to other subjects and educational levels
- ✓ The development of learning progression-driven rubric generation pipelines should be accompanied by rigorous evaluation and validation to ensure their effectiveness and accuracy