Improving reliability estimation in cognitive diagnosis modeling
Fecha
2022-09-20Autor
Estado
info:eu-repo/semantics/publishedVersionMetadatos
Mostrar el registro completo del ítemResumen
. Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents
with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment.
As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM
reliability indices are based on the posterior probabilities of the estimated attribute profles. These posteriors are traditionally
computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty
around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities.
This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure
with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and
G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certifcate
of Profciency in English data was analyzed. The efect of sample size was studied by sampling subsets of subjects from
the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities,
whereas the MI procedure ofered more accurate results. Accordingly, practitioners in small educational or clinical settings
should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for
the MI procedure are made available
Improving reliability estimation in cognitive diagnosis modeling
Tipo de Actividad
Artículos en revistasISSN
1554-351XPalabras Clave
.Cognitive diagnosis · Diagnostic classifcation · Reliability · Classifcation accuracy · Multiple imputation