Overview
The Force Concept Inventory (FCI), a widely utilized research-based assessment in physics education, was investigated for the transferability of its underlying cognitive structure across distinct educational contexts. The study specifically focused on assessing the invariance of FCI Q-matrices, which define the cognitive attributes required for item solution, when applied to student populations from different geographical and educational backgrounds. Rather than approaching the analysis as a localized model calibration, the research framed the investigation around the concept of cross-context cognitive invariance.
Research Context
The Force Concept Inventory serves as a common instrument for evaluating conceptual understanding in introductory physics, particularly mechanics. A fundamental assumption in its widespread application, however, is that the cognitive structure it probes remains consistent across diverse educational settings. This assumption of transferability, particularly concerning the Q-matrix that maps items to specific cognitive attributes, had largely remained untested prior to this study. Understanding whether these cognitive structures are invariant is critical for the valid deployment of cognitive diagnostic and adaptive assessments in varied non-local educational environments.
Approach
The study employed the Generalized Deterministic Inputs, Noisy "And" Gate (G-DINA) cognitive diagnostic model to analyze FCI Q-matrices. Two large student cohorts constituted the empirical basis for the investigation:
- A United States cohort drawn from the Learning About STEM Student Outcomes (LASSO) online system database, comprising N = 4,750 students.
- A South African cohort consisting of N = 1,016 introductory physics students from the University of Johannesburg.
Differential Item Functioning (DIF) analyses were conducted to identify items that performed differently across these two groups. This analysis aimed to determine whether observed differences in item performance were attributable to variations in underlying conceptual reasoning or to psychometric artifacts.
Findings
The DIF analyses revealed substantial cross-cultural differences in student performance on the FCI. Specifically, 14 of the 30 items exhibited high DIF, even after controlling for students' latent skill mastery. These identified differences were not uniformly distributed across the inventory but were concentrated within specific conceptual areas of mechanics:
- Force dynamics
- Contact-force reasoning
These observed differences remained consistent and invariant under alternative Q-matrix specifications. This methodological robustness suggested that the identified variations were not artifacts of the chosen Q-matrix structure but rather reflected genuine differences in how students from the two cohorts engaged with and understood the underlying conceptual physics. The concentration of DIF in specific mechanics topics indicates that certain conceptual areas may be more susceptible to cross-cultural or contextual influences than others.
Why This Matters
The findings underscore the importance of validating Q-matrix structures prior to deploying cognitive diagnostic and adaptive assessments, such as the FCI, across diverse non-local educational settings. The presence of significant differential item functioning, even after accounting for overall skill mastery, indicates that the cognitive pathways or conceptual frameworks assumed by a single Q-matrix may not hold universally. This suggests that without prior validation, interpretations of assessment results and subsequent educational interventions based on an assumed invariant cognitive structure could be misdirected or inaccurate in varying contexts.