University of Illinois Urbana-Champaign

Modeling item bias in fixed-item tests and computerized adaptive tests

Chen, Dandan

Content Files
CHEN-DISSERTATION-2023.pdf

Permalink

https://hdl.handle.net/2142/120189

Description

Title
Modeling item bias in fixed-item tests and computerized adaptive tests
Author(s)
Chen, Dandan
Issue Date
2023-04-18
Director of Research (if dissertation) or Advisor (if thesis)
Zhang, Jinming
Doctoral Committee Chair(s)
Zhang, Jinming
Committee Member(s)
  • Anderson, Carolyn
  • Kern, Justin
  • Xia, Yan
  • Shin, David
Department of Study
Educational Psychology
Discipline
Educational Psychology
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • differential item functioning
  • DIF
  • fixed-item tests
  • computerized adaptive tests
  • CAT
Abstract
Education equity and fairness of assessment are two of the most important topics in education. Bias is a related concept, where there is construct bias, method bias, and item bias (e.g., van de Vijver & Poortinga, 1997; Werner & Campbell, 1970). Measurement invariance is another one, essential in modern standardized assessments based on item response theory (IRT; Richardson, 1936). It includes (1) configural invariance, (2) weak invariance (or metric invariance), (3) strong invariance (or scalar invariance), and (4) strict invariance (Wu et al., 2007). It ensures the comparability of scores across different examinee groups. An operational definition of the scalar invariance is precisely the definition of differential item functioning (DIF; Berk, 1982), a framework that underpins modern methods to investigate bias that possibly exists in assessment items. Despite its long-standing literature and wide application in the testing industry, DIF remains an important area for psychometric research and the discussion concerning DIF detection methods keeps evolving. This dissertation aimed to highlight the limitations of a few commonly used DIF methods in psychometric practice and suggest alternative approaches to improve DIF detection accuracy. Chapter 1 provides an extensive review of the DIF methods having been widely applied in either the testing industry or academic literature. The methods include those following the single-level or the multilevel framework. Chapter 2 reveals key patterns associated with the cutoff value of the RMSD approach, which is a popular DIF method for analyzing cross-country score comparability in international large-scale assessments. Based on Type-I error rates from simulation, a polynomial regression was proposed to predict the appropriate cutoff given the number of groups to be analyzed and significance level. Chapter 3 compares the performance of the RMSD approach with four other approaches designed for multi-group DIF analysis. When compared with a few acceptable methods in the evolving multi-group DIF literature, the RMSD approach was demonstrated to be optimal when used along with the predicted cutoff. Chapter 4 concerns a multilevel framework for DIF detection with data from computerized adaptive tests (CAT), where DIF remains understudied. It conceptualized how between-item dependency could be reframed into between-examinee dependency, and proposes a multilevel model to obtain more accurate DIF results after accounting for this dependency. Chapter 5 lays out a few directions for future research, including developing a global effect size measure across DIF methods, a general DIF approach based on a versatile framework named structural equation modeling, and a shift of the statistical framework for DIF detection.
Graduation Semester
2023-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/120189
Copyright and License Information
Copyright 2023 Dandan Chen

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois