University of Illinois Urbana-Champaign

The impact of world Englishes on language assessment: rater attitude, rating behavior, and challenges

Hsu, Huei-Lien

Content Files
Hsu_Huei-Lien.pdf

Permalink

https://hdl.handle.net/2142/31133

Description

Title
The impact of world Englishes on language assessment: rater attitude, rating behavior, and challenges
Author(s)
Hsu, Huei-Lien
Issue Date
2012-05-22T00:30:11Z
Director of Research (if dissertation) or Advisor (if thesis)
Davidson, Frederick G.
Doctoral Committee Chair(s)
Davidson, Frederick G.
Committee Member(s)
  • Greene, Jennifer C.
  • Bhatt, Rakesh M.
  • Cohen, Dov
Department of Study
Educational Psychology
Discipline
Educational Psychology
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Date of Ingest
2012-05-22T00:30:11Z
Keyword(s)
  • Second language assessment
  • World Englishes
  • perception
  • Instrument
Abstract
By centralizing the issue of test fairness in language proficiency assessments, this study responds to a call by researchers for developing greater social responsibility in the language testing agenda. As inquiries into language attitude and psychology indicate, there is an underlying uncertainty pertaining to the validity of test use and score interpretation based on listeners’ bias against for non-standard English and negative evaluation of such speakers. Of greater relevance in oral proficiency assessment is that listeners, that is, raters, transfer such attitudes to their scoring judgments. As an attempt to address this issue, this study investigates the scoring validity of the IELTS speaking test by examining its relationship in relation to a criterion designed to measure rater attitudes towards World Englishes. Validity arguments were formulated to guide two independent, yet related, studies based on mixed-methods approach and evaluate the claims and hypotheses set for the studies. In view of the lack of instruments measuring rater attitude towards global English in the language assessment context, the first study constructed the criterion measure, the Rater Attitude Instrument (RAI), involving 119 ESL teacher raters in the U.S. and India. As a result of the three-phase development, the RAI comprises 22 semantic differential scale items and 32 Likert scale items representing the three attitude dimensions of feeling, cognition, and behavior tendency used by psychologists. Confirmatory factor analysis supports the internal structure of the RAI with acceptable model fit indices (2 =20.052, p =.094 , RMSEA=0.076 , CFI =0.954 , TLI=0.926). Content validity is ensured through teacher raters and content experts perspectives that continuously shaped the substance of the RAI. As the RAI demonstrates, rater attitudes towards World Englishes were generally positive and tended to focus more on speech comprehensibility; nevertheless, the majority of raters were inclined towards a preference for standard English in their scoring judgments. In the second study, the RAI and the six IELTS descriptive tasks produced by Indian examinees were administered on-line to the 96 teacher raters and the data analyzed to evaluate the extent to which the claim that rater attitude is a biasing factor affecting their scoring judgment on IELTS descriptive tasks can be sustained. The RAI scores were analyzed by FACETS that classified the raters into positive, neutral, and negative attitude groups according to measurement logit. Next, MANOVA was performed which suggested that the ratings by the positive and negative attitude groups were significantly different, with the positive group consistently rating higher on all the four criteria of Fluency, Sentence Structure, Vocabulary, and Pronunciation. The neutral and negative attitude groups rated significantly differently on Sentence Structure and Vocabulary, with the former rating higher than the negative group. Moderate to strong correlations ranging from .272 to .569 were observed between the RAI and the IELTS descriptive task scores. Multiple regression analysis revealed that RAI scores accounted for a significant amount of variance on the IELTS descriptive tasks sub- and total-scores, ranging from 17.5% to 32.4%. Moreover, the Indian/non-Indian variable was the only rater background characteristic investigated that significantly related to the rater feeling scores that formed one of the triplet attitude constructs, though contributing to only 10% of the score variance. Lastly, the verbal protocol study provided insightful information suggesting that raters with positive attitudes generally took into account the expected performance of language learners while some negative-attitude raters used the native speaker model as the underlying criterion for judgment. The impact of the findings on validity argument, test fairness, and rater trainings are also discussed.
Graduation Semester
2012-05
Permalink
http://hdl.handle.net/2142/31133
Copyright and License Information
Copyright 2012 Huei-Lien Hsu

Owning Collections

Dissertations and Theses - Education
Dissertations and Theses from the College of Education
Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois