An in-depth analysis of DIF detection methods and equating techniques for valid test scores

Kartal, Gamze

Permalink

https://hdl.handle.net/2142/121232 Copy

Description

Title

An in-depth analysis of DIF detection methods and equating techniques for valid test scores

Author(s)

Kartal, Gamze

Issue Date

2023-07-10

Director of Research (if dissertation) or Advisor (if thesis)

Zhang, Jinming

Doctoral Committee Chair(s)

Zhang, Jinming

Committee Member(s)

• Chang, Hua-hua
• Anderson, Carolyn J
• Xia, Yan

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• test equating
• NEAT
• anchor items
• differential item functioning
• matching variable
• treatment methods of differential item functioning

Abstract

Test equating is a key process in psychometric assessments, enabling the comparison of scores from different test forms intended to measure the same construct. This paper thoroughly examines the equating process within the framework of the Non-Equivalent groups with Anchor Test (NEAT) design. The NEAT design relies heavily on anchor items, which serve as a common reference across various test forms, aiding the comparative evaluation of test-taker performance. Chapter 2 explores anchor items and their functionality over administrations, emphasizing the implications of Differential Item Functioning (DIF) within the anchor set. A thorough examination of the impact of DIF items within the anchor set highlighted how they negatively affect biases and root mean square errors of test equating. The analysis stressed the crucial need for maintaining the psychometric consistency of anchor items, as any deviation can result in inaccurate equating outcomes and potentially unfair score comparisons across different test forms. Considering the ramifications of DIF items, Chapter 3 delves into another crucial facet of test equating – the detection of genuine DIF items. This chapter thoroughly studies five conventional DIF detection methods, evaluating their performance under different conditions. The comparative analysis found their effectiveness in identifying DIF items within the anchor item set to be condition-dependent, with each method's power and Type I error rates varying. This revelation emphasized the need for robust DIF detection mechanisms. Such mechanisms are crucial for identifying outliers, preserving the integrity of the test equating process, and ensuring fair and accurate outcomes. Transitioning to a critical stage of handling detected DIF items within the anchor set, Chapter 4 evaluates three strategies—ignoring, replacing, and removing DIF items. Each approach presents unique advantages and potential challenges. An in-depth analysis found that removing and replacing DIF items generally outperformed ignoring them, but this performance varied based on the DIF proportion, revealing certain drawbacks. The study highlighted the inherent trade-offs within each approach, underscoring the necessity for context-aware and careful management of DIF items. The findings enhance our understanding of test equating and contribute to the refinement of methodologies for ensuring fair and valid test score comparisons.

Graduation Semester

2023-08

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Gamze Kartal

Owning Collections