Learning Objective and Outcome Manager (LOOM)

The Challenge

Digital learning platforms face a fundamental scaling problem: as courses grow to hundreds or thousands of students, instructors cannot provide individualized feedback to each learner. Peer feedback offers a scalable alternative, but students often lack experience in structuring effective feedback, resulting in variable quality and limited educational impact. Existing AI approaches such as automated essay scoring or generative AI feedback have shown promise, but fully automated methods cannot replicate the pedagogical value of human-provided feedback. The question was whether AI could support, rather than replace, human feedback givers.

Approach and Methods

The project developed a Feedback Development Support (FDS) system that combines human judgment with AI-driven recommendations. The system analyzes historical learner feedback together with recipient evaluation ratings, trains a machine learning model based on Pareto dominance ranking (converting subjective ratings into pairwise ranking problems), and recommends a personalized short list of critical text features that each feedback author should improve. The system was tested in a field experiment with 490 students across three university courses. Students were randomly assigned to four conditions: no support (control), a complete list of all 15 text features, a static top-3 list, or a personalized adaptive list generated by the FDS.

Key Findings

Only the adaptive, personalized approach achieved consistent and statistically significant improvements over the control group (p < 0.05). The Adaptive FDS significantly outperformed both static approaches across all evaluation criteria, including helpfulness, critical issue identification, and constructive suggestions. Both the complete list and the static short list failed to show significant improvements over no support at all. These results demonstrated that personalized, AI-driven guidance effectively enhances peer feedback quality, while generic feature lists do not.

Implications

The findings show that effective AI support in education requires personalization, not just information provision. Presenting learners with a complete set of improvement criteria overwhelms them, while static recommendations miss individual weaknesses. The adaptive approach respects cognitive limitations by focusing attention on the most impactful improvements for each individual. The system can be applied in any course that uses peer feedback, with text features pre-selected by the course instructor.

Team and Funding

The project was led by Prof. Dr. Roman Rietsche (BFH) and Prof. Dr. Konstantin Bauman (Temple University) as co-first authors, in collaboration with Prof. Dr. Matthias Soellner (University of Kassel) and Prof. Dr. Jan Marco Leimeister (University of St. Gallen / University of Kassel). The research was funded by the University of St. Gallen.