Balancing fit and parsimony to improve Q-matrix validation

Abstract

.

The Q-matrix identifies the subset of attributes measured by each item in the cognitivediagnosis modelling framework. Usually constructed by domain experts, the Q-matrixmight contain some misspecifications, disrupting classification accuracy. Empirical Q-matrix validation methods such as thegeneral discrimination index(GDI) and Wald haveshown promising results in addressing this problem. However, a cut-off point is used inboth methods, which might be suboptimal. To address this limitation, the Hull method isproposed and evaluated in thepresent study. This method aims to find theoptimal balancebetween fit and parsimony, and it is flexible enough to be used either with a measure ofitem discrimination (theproportion of variance accounted for, PVAF) or a coefficient ofdetermination (pseudo-R2). Results from a simulation study showed that the Hull methodconsistently showed the best performance and shortest computation time, especiallywhen used with the PVAF. The Wald method also performed very well overall, while theGDI method obtained poor results when the number of attributes was high. The absenceof a cut-off point provides greater flexibility to the Hull method, and it places it as acomprehensive solution to the Q-matrix specification problem in applied settings. Thisproposal is illustrated using real data.