University of Illinois Urbana-Champaign

Detecting examinees with item pre-knowledge in computer based large-scale assessment

Demirkaya, Onur

Content Files
DEMIRKAYA-DISSERTATION-2022.pdf

Permalink

https://hdl.handle.net/2142/116058

Description

Title
Detecting examinees with item pre-knowledge in computer based large-scale assessment
Author(s)
Demirkaya, Onur
Issue Date
2022-07-07
Director of Research (if dissertation) or Advisor (if thesis)
Zhang, Jinming
Doctoral Committee Chair(s)
Zhang, Jinming
Committee Member(s)
  • Chang, Hua-hua
  • Anderson, Carolyn J
  • Jiang, Ge
Department of Study
Educational Psychology
Discipline
Educational Psychology
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • Item pre-knowledge
  • compromised items
  • test security
  • likelihood ratio test
  • score test
  • gradient test
  • item revisit
  • response time
  • change point analysis
  • person fit
Abstract
Item pre-knowledge, one of the most frequently encountered aberrant testing behaviors in high-stakes assessment, jeopardizes test score integrity, validity, and fairness by artificially inflating the test scores of its beneficiaries, examinees who have access to some test items before taking the test. To maintain test score integrity, there is increasing interest in developing statistical methods to detect item pre-knowledge by utilizing examinee responses and response process information. One of those methods is the constrained likelihood ratio test, which is the only frequentist approach specifically designed to detect item pre-knowledge (Sinharay & Johnson, 2019). This method assumes that a set of compromised items is precisely known by investigators. However, different subsets of items can be compromised for different groups of examinees, and statistical detection procedures cannot confidently identify all compromised items. Therefore, in Chapter 2, two new frequentist statistics, namely the constrained score test and the constrained gradient test, which rely on responses and response times to detect item pre-knowledge were proposed and their performance was investigated in situations of uncertainty about suspicious items through simulation studies for both adaptive and linear tests. A real data analysis was conducted to show the usefulness of the statistics in the operational setting. The proposed statistics were found to be as powerful as the existing statistic. With the availability of process data collected in computer-based assessments, research on detecting item pre-knowledge using both item scores and response times has progressed. Examinees’ item-revisit patterns can also be utilized as an additional source of information. In Chapter 3, a novel statistic was proposed for detecting item pre-knowledge when compromised items are known by utilizing the hierarchical speed accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the statistic was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefitted from item pre-knowledge than existing statistics. Change point analysis (CPA), a statistical process control method, has recently been applied to psychometric problems including the detection of aberrant behaviors. Existing CPA methods, however, have not been specifically designed to detect item pre-knowledge but can be applied only after reordering the test items based on the information of compromised item status, which is not always readily available. In Chapter 4, therefore, the use of item usage or exposure rate was proposed to reorder data along with one residual-based and three modified likelihood-based statistics relying on both response and response time information to detect item pre-knowledge. Comprehensive simulation studies conducted with both linear and adaptive tests showed that the proposed statistics were powerful enough to flag aberrant examinees exhibiting different change point locations when there was an association between item usage and item status.
Graduation Semester
2022-08
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/116058
Copyright and License Information
Copyright 2022 Onur Demirkaya

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois