Overview
Research addressed the challenge of evaluating the pedagogical quality of AI-assisted educational materials, particularly focusing on novel AI-generated questions. The investigation centered on the application of automated evaluation methods, specifically Bloom Classifier models, for assessing educational materials at scale. A key concern was the potential performance degradation of these models when applied to new, out-of-distribution (OOD) datasets, such as those comprising AI-assisted generated questions, despite their high accuracy on within-distribution (IID) datasets.
Research Context
The proliferation of AI-assisted generation of educational materials has surpassed the current capacity for validating their pedagogical quality. Automated evaluation mechanisms, particularly Bloom Classifier models, are considered a promising avenue for large-scale assessment. However, the performance robustness of these pre-trained models under dataset shift, specifically when encountering novel AI-assisted educational questions, was a central problem addressed by this research.
Approach
The study evaluated various model types for Bloom level classification, including traditional Machine Learning (ML) models, transformer models (specifically BERT), and Large Language Models (LLMs). The objective was to identify robust classifiers capable of maintaining performance under dataset shift, specifically on OOD datasets comprising AI-assisted generated questions. In addition to direct model evaluation, the researchers explored several feature-engineering strategies aimed at stabilizing OOD performance. These strategies included incorporating Natural Language Processing (NLP) metrics, appending learning objectives as part of the model input, and text splicing techniques. Model retraining was also investigated as a method to improve performance.
Findings
- Baseline Performance on OOD Datasets: Traditional ML models utilizing TFPOS-IDF features demonstrated poor performance on OOD datasets, exhibiting a Macro F1-score of $0.48$. BERT models achieved a Macro F1-score of $0.55$. Large Language Models (LLMs) significantly outperformed both, achieving a Macro F1-score of $0.79$ on OOD datasets.
- Impact of Text Splicing: The application of text splicing improved the Macro F1-score performance for both ML and BERT models. ML models saw an improvement to $0.59$, while BERT models improved to $0.62$.
- Effect of Appending Learning Objectives: Appending learning objectives to the input data increased model performance on specific datasets.
- Effectiveness of Model Retraining: Model retraining provided the most substantial improvement across all evaluated models and datasets.
- Trade-offs with Pre-trained Models: The findings highlighted an inherent trade-off associated with using pre-trained models for assessing novel AI-assisted educational questions.
- Role of Feature Enhancements: Strategic feature enhancements were observed to help address the performance loss encountered when using pre-trained models with novel AI-generated content.
Why This Matters
The rapid expansion of AI-generated educational content necessitates scalable and reliable methods for pedagogical quality assessment. This research offers insights into the performance characteristics of different pre-trained models when applied to novel, AI-assisted educational questions and identifies practical strategies for mitigating performance degradation due to dataset shift. The findings underscore the importance of feature engineering and model adaptation techniques, such as retraining, to maintain evaluation efficacy in a dynamic educational content landscape.