The Reliabilities and The Cost-Efficiencies of Three Methods of Assessment for Writing Ability: An Empirical Inquiry
Bauer, Barbara Ann
This item is only available for download by members of the University of Illinois community. Students, faculty, and staff at the U of I may log in with your NetID and password to view the item. If you are trying to access an Illinois-restricted dissertation or thesis, you can request a copy through your library's Inter-Library Loan office or purchase a copy directly from ProQuest.
Permalink
https://hdl.handle.net/2142/68789
Description
Title
The Reliabilities and The Cost-Efficiencies of Three Methods of Assessment for Writing Ability: An Empirical Inquiry
Author(s)
Bauer, Barbara Ann
Issue Date
1982
Department of Study
Education
Discipline
Education
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
Education, Secondary
Abstract
Assessing achievement in writing ability from large numbers of essays is often an unreliable and time-consuming task. The extent to which these factors are problematic depends upon which conceptualization of this skill different raters adhere to during the grading process. The present investigation compares the reliabilities and cost-efficiencies of three methods of assessment, where each one conceptualizes writing ability differently.
Although it is generally accepted that analytical scoring methods are more reliable than holistic methods for grading essays, no evidence exists illustrating how the reliabilities for these two methods compare with those for the Primary Trait scoring system. Such a comparison required an empirical inquiry. In so doing, nine teaching assistants from the University of Illinois English Department were asked to grade 118 descriptive essays according to three scoring methods. After being divided into three groups of three different raters, each group of raters was trained to grade the essays according to one of three methods; that is, one group used the Diederich Expository Scale (an analytical method); a second group used a 4-point holistic method; and the third group used a Primary Trait scoring guide. Each group of raters then graded the essays according to its respective method. Time logs were also kept for each group's training and grading sessions.
This study yielded several major findings. An analysis of variance revealed inter-rater reliabilities of .874 for individual ratings and .954 for average ratings for the analytical method, .811 and .928 for the holistic method, and .633 and .838 for the Primary Trait method. Intra-rater reliabilities were also obtained, revealing composite correlations of .890 for the analytical method, .820 for the Primary Trait method, and .733 for the holistic method. The average time required to grade each essay was 4.14 minutes for the analytical method, 1.08 minutes for the Primary Trait method, and .96 minutes for the holistic method.
On the basis of these findings, it was concluded that the analytical method was the most reliable and the holistic method was the most cost-efficient in grading large numbers of essays.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
