Year 2019, Volume 6, Issue 1, Pages 154 - 169 2019-03-21

Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models

Hueying Tzou [1] , Ya-Huei Yang [2]

47 69

Selecting an appropriate cognitive diagnostic model (CDM) for data analysis is always challenging. Studies have explored several model fit indices for CDMs. The common results of these studies indicate that Q-matrix misspecifications lead to poor performance of the model fit indices in the context of CDMs. Thus, this study explored whether model fit indices improve performance with a modified Q-matrix. The average class size has reduced to 23 students in Taiwan because of the low birth rate; therefore, the study sought the effect of sample size on the performance of model fit indices. The results showed that Akaike’s information criterion (AIC) was an excellent model fit index in small samples. Model fit indices with the modified Q-matrix presented superior performance.
RSS, ζ2 index, model fit indices, cognitive diagnostic models, Q-matrix
  • Akaike, H. (1974). A new look at the statistical identification model. IEEE Transactions on Automated Control, 19, 716-723.
  • DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 979-1030). Amsterdam, Netherlands: Elsevier.
  • Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnostic modeling. Journal of Educational Measurement, 50, 123-140.
  • Chiu, C.Y. (2013). Statistical refinement of the Q-Matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618.
  • Chiu, C.-Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30(2), 225-250.
  • de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
  • de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
  • de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355-373.
  • de la Torre, J., & Chiu, C. Y. (2016). A General Method of Empirical Q-matrix Validation. Psychometrika, 81(2), 253-273.
  • Hu, J., Miller, M. D., Huggins-Manley, A. C., & Chen, Y. (2016). Evaluation of Model Fit in Cognitive Diagnosis Models. International Journal of Testing, 16(2), 119-141.
  • Jiao, H. (2009). Diagnostic classification models: Which one should I use? Measurement: Interdisciplinary Research & Perspective, 7(1), 65-67.
  • Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
  • Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.
  • Lei, P.-W., & Li, H. (2016). Choosing Correct Cognitive Diagnostic Models and Q-Matrices. Applied Psychological Measurement, 40(6), 1-12.
  • Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model Similarity, Model Selection, and Attribute Classification. Applied Psychological Measurement, 40(3), 200-217.
  • Ma, W., & de la Torre, J. (2018). GDINA: The generalized DINA model framework. R package version 2.3. Retrived from https://CRAN.R-project.org/package=GDINA
  • R Core Team (2017). R: A language and environment for statistical computing (Version 3.4.3) [Computing software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org
  • Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification nonparameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.
  • Schwarzer, G. (1976). Estimating the dimension of a model. Annals of Statistics, 6,461–464.
  • Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354.
  • Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
Primary Language en
Subjects Education, Scientific Disciplines
Published Date March
Journal Section Articles
Authors

Orcid: 0000-0002-6740-6852
Author: Hueying Tzou (Primary Author)
Institution: Department of Education, National University of Tainan
Country: Taiwan


Orcid: 0000-0002-4109-2381
Author: Ya-Huei Yang
Institution: Department of Education, National University of Tainan
Country: Taiwan


Bibtex @research article { ijate482005, journal = {International Journal of Assessment Tools in Education}, issn = {}, eissn = {2148-7456}, address = {İzzet KARA}, year = {2019}, volume = {6}, pages = {154 - 169}, doi = {10.21449/ijate.482005}, title = {Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models}, key = {cite}, author = {Tzou, Hueying and Yang, Ya-Huei} }
APA Tzou, H , Yang, Y . (2019). Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models. International Journal of Assessment Tools in Education, 6 (1), 154-169. DOI: 10.21449/ijate.482005
MLA Tzou, H , Yang, Y . "Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models". International Journal of Assessment Tools in Education 6 (2019): 154-169 <http://submit.ijate.net/issue/40373/482005>
Chicago Tzou, H , Yang, Y . "Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models". International Journal of Assessment Tools in Education 6 (2019): 154-169
RIS TY - JOUR T1 - Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models AU - Hueying Tzou , Ya-Huei Yang Y1 - 2019 PY - 2019 N1 - doi: 10.21449/ijate.482005 DO - 10.21449/ijate.482005 T2 - International Journal of Assessment Tools in Education JF - Journal JO - JOR SP - 154 EP - 169 VL - 6 IS - 1 SN - -2148-7456 M3 - doi: 10.21449/ijate.482005 UR - https://doi.org/10.21449/ijate.482005 Y2 - 2019 ER -
EndNote %0 International Journal of Assessment Tools in Education Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models %A Hueying Tzou , Ya-Huei Yang %T Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models %D 2019 %J International Journal of Assessment Tools in Education %P -2148-7456 %V 6 %N 1 %R doi: 10.21449/ijate.482005 %U 10.21449/ijate.482005
ISNAD Tzou, Hueying , Yang, Ya-Huei . "Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models". International Journal of Assessment Tools in Education 6 / 1 (March 2019): 154-169. https://doi.org/10.21449/ijate.482005