On-the-fly assembled multistage adaptive testing

Tay, Poh Hua

Permalink

https://hdl.handle.net/2142/88036 Copy

Description

Title

On-the-fly assembled multistage adaptive testing

Author(s)

Tay, Poh Hua

Issue Date

2015-07-14

Director of Research (if dissertation) or Advisor (if thesis)

Chang, Hua-Hua

Doctoral Committee Chair(s)

Chang, Hua-Hua

Committee Member(s)

• Anderson, Carolyn J.
• Douglas, Jeffrey A.
• Zhang, Jinming
• Culpepper, Steven A.

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• computerized adaptive testing
• multistage testing
• test assembly

Abstract

Computer-administered tests such as the computerized adaptive test (CAT) and multistage tests (MST) have been widely implemented in many large scale tests. In 2000, some testing organizations uncovered some pitfalls of CAT after it was applied to large scale testing. After these instances, many testing organizations turned away from CAT and started implementing MST as it could mitigate the underestimation or overestimation problems faced by CAT. Although the MST is able to alleviate the problems faced by CAT, its preassembled modules and panels do not target each examinee as well as CAT. The other serious drawback of MST is that the bundling of items and modules may have security threat when it is being administered continuously and the test items are purposefully shared among examinees via internet. Research on innovative MST designs that assembled items on-the-fly have shown great potential to concurrently tap the strengths and reduce the weaknesses of both CAT and MST designs (Han & Guo, 2014; Zheng & Chang, 2014; Wang, Lin, Douglas, & Chang,2014). As innovative MST designs are still at their infancy stage, designs for both ability estimation and mastery testing are still underexplored in the literature. This dissertation is an attempt to propose two new innovative MST designs for both ability estimation and mastery testing. This comprises three parts: 1. New on-the-fly assembled multistage adaptive testing (OMST) designs: Approach I. At the first stage of the current OMST (OMST-P; Zheng & Chang, 2014), as there is no information about the examinee ability, the item selection strategy assembled items that target on an average ability examinee. The drawback is that when the examinees’ abilities are very different from the average ability, then the items assembled in the first stage are not efficient and do not have sufficient measurement accuracy for these examinees. To address this limitation, new item selection strategies are proposed in Chapter 3 to allow items to be selected from wider ability intervals at early stages and selection intervals get narrower as examinee progresses to the later stages. The results from the simulation studies indicate that all the proposed designs have significantly better measurement accuracy than the existing on-the-fly assembled multistage adaptive testing and have comparable measurement accuracy as the CAT design. 2. New on-the-fly assembled multistage adaptive testing (OMST) designs: Approach II. The current OMST-P item selection design assembled items based on the provisional examinee’s ability estimated at the end of each stage. As the provisional ability estimates tend to be less accurate at the start of testing, the items assembled in this manner may not target examinees well. As OMST designs assemble modules on-the-fly, new item selection algorithms could be programmed to concurrently consider both the provisional examinee’s ability estimate as well as the items’ information from a pre-defined set of well-distributed points along the ability scale to increase the chance of obtaining more information at the true ability of each examinee. Chapter 4 compares the performance of the two proposed OMST designs with the standard MST and CAT designs in both ability estimation and mastery testing. 3. New Hybrid designs for ability estimation and mastery testing. To further maximize the strengths and minimize the weaknesses of both CAT and MST designs, Chapter 5 proposes new MST designs that begin with an OMST step at the early stages of testing and transits to a CAT step at later stages. The hybrid designs incorporate the newly proposed OMST item selection algorithms (from Chapters 3 and 4) and compare their performances in ability estimation as well as mastery testing with existing CAT designs.

Graduation Semester

2015-8

Type of Resource

text

Permalink

http://hdl.handle.net/2142/88036

Copyright and License Information

Copyright 2015 Poh Hua Tay

Owning Collections