The Logic of Criterion-Referenced Testing

Tomko, Thomas Norbert

Permalink

https://hdl.handle.net/2142/66063 Copy

Description

Title

The Logic of Criterion-Referenced Testing

Author(s)

Tomko, Thomas Norbert

Issue Date

1981

Department of Study

Education

Discipline

Education

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

Education, Tests and Measurements

Language

eng

Abstract

• This thesis is a critical analysis of the currently dominant approach to criterion-referenced testing (CRT). The main point of the thesis is that the logic underlying the currently dominant CRT model cannot support the type of inferences desired in CRT. The logical model that underlies CRT is a statistical generalization model. Due to the nature of the elements of the populations sampled under such a model, the inferences that are desired from CRT cannot be supported by statistical generalization alone. It is argued that the inferences desired are inferences about abilities. Various interpretations of ability claims are examined and the problems with CRT theory are related to the broader dispute in philosophy of science between realism and nominalism.
• The thesis consists of five chapters. Chapter I includes an introduction to and motivation for the thesis. In Chapter II, the important features of CRT theory are outlined. The broadest characterization of CRT on which there is general consensus is that CR interpretations tell one what an examinee can or cannot do. The main contention of Chapter II is that the dominant approach to CRT relies on a statistical generalization model for its underlying logic. Differences among CRT theorists as to the nature of the elements in a universe or domain in CRT are pointed out. The three main candidates for universe elements are behaviors, items, and observations.
• In Chapter III the application of the statistical generalization model to CRT is examined in detail. Several problem areas in statistical generalization in general were noted. Foremost among these is the problem of procuring a representative sample. The most widely accepted method of sampling is random sampling which does not guarantee representativeness, but does give one some idea of the likelihood that a sample is representative. Populations that are subject to statistical generalization using random sampling must be well-defined and closed.
• It is argued that two of the candidates for universe elements, behaviors and items, have a type-token ambiguity that must be kept in mind when applying a statistical generalization model. It is argued that behaviors, in either the type or token sense, and item types are not proper universe elements for CRT. It is argued that item-tokens (which can be considered a type of observation) might be appropriate as universe elements in the application of the statistical generalization model to CRT, if restricted in certain ways.
• In Chapter IV it is argued that a statistical generalization model cannot support the type of inference desired in CRT. In order to show this type of inferences desired from CRT are first examined. These inferences are claims that individuals do or do not have certain abilities. Next the problem of how one establishers such claims is examined. It is argued that the evidence that one would accept as supporting an ability claim would depend on whether one held a nominalist or a realist view of abilities. It is then argued that the type of claims that are desired from CRT to some extent presuppose a realist view and that such claims cannot be supported by a statistical generalization model alone. To support such claims one needs a broad CRT model that allows inferences not strictly permitted by statistical generalization alone.
• In Chapter V consequences for testing practices and implications for future research are discussed. It is argued that the type of inferences desired from CRT can be completely supported only by an inference-to-the-best-explanation model. Topics for future research are noted. These include the problem of specifying what constitutes an adequate explanation, the nature of causal claims, and the topic of natural kinds in the area of human abilities.

Graduation Semester

1981

Type of Resource

text

Permalink

http://hdl.handle.net/2142/66063

Owning Collections