Detecting examinees with item pre-knowledge in computer based large-scale assessment

Demirkaya, Onur

Permalink

https://hdl.handle.net/2142/116058 Copy

Description

Title

Detecting examinees with item pre-knowledge in computer based large-scale assessment

Author(s)

Demirkaya, Onur

Issue Date

2022-07-07

Director of Research (if dissertation) or Advisor (if thesis)

Zhang, Jinming

Doctoral Committee Chair(s)

Zhang, Jinming

Committee Member(s)

• Chang, Hua-hua
• Anderson, Carolyn J
• Jiang, Ge

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Item pre-knowledge
• compromised items
• test security
• likelihood ratio test
• score test
• gradient test
• item revisit
• response time
• change point analysis
• person fit

Abstract

Item pre-knowledge, one of the most frequently encountered aberrant testing behaviors in high-stakes assessment, jeopardizes test score integrity, validity, and fairness by artificially inflating the test scores of its beneficiaries, examinees who have access to some test items before taking the test. To maintain test score integrity, there is increasing interest in developing statistical methods to detect item pre-knowledge by utilizing examinee responses and response process information. One of those methods is the constrained likelihood ratio test, which is the only frequentist approach specifically designed to detect item pre-knowledge (Sinharay & Johnson, 2019). This method assumes that a set of compromised items is precisely known by investigators. However, different subsets of items can be compromised for different groups of examinees, and statistical detection procedures cannot confidently identify all compromised items. Therefore, in Chapter 2, two new frequentist statistics, namely the constrained score test and the constrained gradient test, which rely on responses and response times to detect item pre-knowledge were proposed and their performance was investigated in situations of uncertainty about suspicious items through simulation studies for both adaptive and linear tests. A real data analysis was conducted to show the usefulness of the statistics in the operational setting. The proposed statistics were found to be as powerful as the existing statistic. With the availability of process data collected in computer-based assessments, research on detecting item pre-knowledge using both item scores and response times has progressed. Examinees’ item-revisit patterns can also be utilized as an additional source of information. In Chapter 3, a novel statistic was proposed for detecting item pre-knowledge when compromised items are known by utilizing the hierarchical speed accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the statistic was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefitted from item pre-knowledge than existing statistics. Change point analysis (CPA), a statistical process control method, has recently been applied to psychometric problems including the detection of aberrant behaviors. Existing CPA methods, however, have not been specifically designed to detect item pre-knowledge but can be applied only after reordering the test items based on the information of compromised item status, which is not always readily available. In Chapter 4, therefore, the use of item usage or exposure rate was proposed to reorder data along with one residual-based and three modified likelihood-based statistics relying on both response and response time information to detect item pre-knowledge. Comprehensive simulation studies conducted with both linear and adaptive tests showed that the proposed statistics were powerful enough to flag aberrant examinees exhibiting different change point locations when there was an association between item usage and item status.

Graduation Semester

2022-08

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Onur Demirkaya

Owning Collections