Variable-length computerized adaptive testing: adaptation of the a-stratified strategy in item selection with content balancing

Huo, Yan

Permalink

https://hdl.handle.net/2142/14715 Copy

Description

Title

Variable-length computerized adaptive testing: adaptation of the a-stratified strategy in item selection with content balancing

Author(s)

Huo, Yan

Issue Date

2010-01-06T16:40:18Z

Director of Research (if dissertation) or Advisor (if thesis)

Budescu, David V.

Doctoral Committee Chair(s)

Chang, Hua-Hua

Committee Member(s)

• Budescu, David V.
• Hubert, Lawrence J.
• Anderson, Carolyn J.
• Douglas, Jeffrey A.

Department of Study

Psychology

Discipline

Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• variable-length computerized adaptive testing
• the a-stratified method
• content balancing

Abstract

Variable-length computerized adaptive testing (CAT) can provide examinees with tailored test lengths. With the fixed standard error of measurement (SEM) termination rule, variable-length CAT can achieve predetermined measurement precision by using relatively shorter tests compared to fixed-length CAT. To explore the application of variable-length CAT, this dissertation proposes four variable-length item selection methods adapted from the a-stratified strategy (Chang & Ying, 1999). These methods are named 1) the circularly increasing a-stratified method (STR-Ca), 2) the circularly decreasing a-stratified method (STR-Cd), 3) the random a-stratified method (STR-R), and 4) the two-stage a-stratified variable-length method (STR+R). The general strategy of these four methods allows test items to be selected in a mixed-strata ordering fashion from all strata partitioned by different levels of the discrimination parameter. This flexibility can overcome the potential problem of unbalanced item usage across different strata caused by previous attempts of applying the original a-stratified method into variable-length CAT. Study 1 examines the STR-Ca, the STR-Cd, and the STR-R methods in fixed-length CAT situations and the results show that their performance is comparable to that of the original a-stratified method in the fixed-length simulations in terms of various criterion measures such as Bias, MSE, efficiency, and item exposure rates. Study 2 explores these four item selection methods under the variable-length situations and the results indicate that these four methods can achieve good ability estimation while maintaining balanced item usage in the variable-length CAT simulations. To extend the implementation of these four variable-length item selection methods into a more realistic testing situation with content balancing constraints, Study 3 proposes two two-phase content balancing control methods, the variable-length modified multinomial model (MMM) method and the content weighted item selection index method. They can be naturally incorporated with these four adapted a-stratified methods to realize variable-length CAT with content control. Lastly, the intent of Study 4 is to explore decision making tools regarding choices among several variable-length CAT designs. Two quantitative indices, the cost-effective ratio and the variable-fixed-fitness index, are developed and their applications are demonstrated with some hypothetical examples. Together, these study findings will advance the research and understanding of variable-length CAT, and will facilitate the application and adoption of variable-length CAT in real world testing.

Graduation Semester

2009-12

Permalink

http://hdl.handle.net/2142/14715

Copyright and License Information

Copyright 2009 Yan Huo

Owning Collections