A Bayesian change-point approach to item compromise detection

Du, Yang

Permalink

https://hdl.handle.net/2142/115512 Copy

Description

Title

A Bayesian change-point approach to item compromise detection

Author(s)

Du, Yang

Issue Date

2022-03-10

Director of Research (if dissertation) or Advisor (if thesis)

Cromley, Jennifer G

Doctoral Committee Chair(s)

• Cromley, Jennifer G
• Chang, Hua-Hua

Committee Member(s)

• Kern, Justin L
• Xia, Yan

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• test security
• item compromise
• continuous tests
• Bayesian change-point

Abstract

Item compromise has been a longstanding issue that threatens test validity and test security in continuous tests. The literature in compromised item detection thus far has been primarily focused on non-Bayesian methods. Most of the existing detection methods monitor the item compromise process at the item level; each item is monitored individually through a summary statistic, such as the correct or incorrect response proportion. What's more, most of the existing detection methods are primarily based on response accuracy and response time information has not been efficiently utilized. In this study, a Bayesian change-point method that incorporates responses and response times is proposed to detect compromised items in continuous tests. In Chapter 1, the background of continuous tests (using computerized tests as an example) and test security is briefly overviewed. The strengths and weaknesses of continuous tests and the challenges that they have imposed on test security will be discussed. Next, the history of utilizing response times in testing as well as why and how response times can help uncover the underlying response process will be presented. Within this context, the problem that this dissertation is trying to resolve, i.e., detecting item compromise in continuous tests with responses and response times, will be clearly stated. The significance of addressing this problem will be discussed as well. In Chapter 2, the methods related to item compromise prevention and detection will be carefully reviewed. With an emphasis on the detection methods, most recent detection methods will be reviewed and synthesized based on the nature of the methods. Meanwhile, the Bayesian change-point framework will also be illustrated. Here, both Phase I and Phase II detection will be demonstrated, with their definitions and application context being explained. Chapter 3 presents a pilot study which aims at employing Bayesian change-point approach to detect the item compromise with responses only. The main purpose of this pilot study is to show that Bayesian change-point approach has great potential in item compromise detection and can be employed to enhance test security. Three simulation studies are conducted to show its performance. Simulation I aims to show the stationary Bayesian change-point model. Specifically, convergence and parameter recovery of the proposed Metropolis-Hastings-within-Gibbs algorithms with respect to the change-point model are evaluated. Simulation II demonstrates the performance of Shiryaev procedure in detecting compromised items in real time while using true person parameters. Simulation III also presents the performance of Shiryaev procedure, but using the EAP estimates of latent ability parameters. In Chapter 4, the proposed Bayesian change-point model with responses and response times and the Shiryaev procedure are explained in detail. The simulation designs in conjunction with evaluation criteria are explained as well. Chapter 5 presents the results of the three simulations. Convergence plot and parameter recovery of the Bayesian change-point model are presented in simulation I. Detection accuracy and efficiency are presented for simulation II. The comparison of the Bayesian change-point method with and without RT is presented in simulation III. Finally, the conclusions of the current study are presented in Chapter 6. Limitations as well as the directions for future studies are also discussed.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Yang Du.

Owning Collections