The impact of world Englishes on language assessment: rater attitude, rating behavior, and challenges

Hsu, Huei-Lien

Permalink

https://hdl.handle.net/2142/31133 Copy

Description

Title

The impact of world Englishes on language assessment: rater attitude, rating behavior, and challenges

Author(s)

Hsu, Huei-Lien

Issue Date

2012-05-22T00:30:11Z

Director of Research (if dissertation) or Advisor (if thesis)

Davidson, Frederick G.

Doctoral Committee Chair(s)

Davidson, Frederick G.

Committee Member(s)

• Greene, Jennifer C.
• Bhatt, Rakesh M.
• Cohen, Dov

Department of Study

Educational Psychology

Discipline

Educational Psychology

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Date of Ingest

2012-05-22T00:30:11Z

Keyword(s)

• Second language assessment
• World Englishes
• perception
• Instrument

Abstract

By centralizing the issue of test fairness in language proficiency assessments, this study responds to a call by researchers for developing greater social responsibility in the language testing agenda. As inquiries into language attitude and psychology indicate, there is an underlying uncertainty pertaining to the validity of test use and score interpretation based on listeners’ bias against for non-standard English and negative evaluation of such speakers. Of greater relevance in oral proficiency assessment is that listeners, that is, raters, transfer such attitudes to their scoring judgments. As an attempt to address this issue, this study investigates the scoring validity of the IELTS speaking test by examining its relationship in relation to a criterion designed to measure rater attitudes towards World Englishes. Validity arguments were formulated to guide two independent, yet related, studies based on mixed-methods approach and evaluate the claims and hypotheses set for the studies. In view of the lack of instruments measuring rater attitude towards global English in the language assessment context, the first study constructed the criterion measure, the Rater Attitude Instrument (RAI), involving 119 ESL teacher raters in the U.S. and India. As a result of the three-phase development, the RAI comprises 22 semantic differential scale items and 32 Likert scale items representing the three attitude dimensions of feeling, cognition, and behavior tendency used by psychologists. Confirmatory factor analysis supports the internal structure of the RAI with acceptable model fit indices (2 =20.052, p =.094 , RMSEA=0.076 , CFI =0.954 , TLI=0.926). Content validity is ensured through teacher raters and content experts perspectives that continuously shaped the substance of the RAI. As the RAI demonstrates, rater attitudes towards World Englishes were generally positive and tended to focus more on speech comprehensibility; nevertheless, the majority of raters were inclined towards a preference for standard English in their scoring judgments. In the second study, the RAI and the six IELTS descriptive tasks produced by Indian examinees were administered on-line to the 96 teacher raters and the data analyzed to evaluate the extent to which the claim that rater attitude is a biasing factor affecting their scoring judgment on IELTS descriptive tasks can be sustained. The RAI scores were analyzed by FACETS that classified the raters into positive, neutral, and negative attitude groups according to measurement logit. Next, MANOVA was performed which suggested that the ratings by the positive and negative attitude groups were significantly different, with the positive group consistently rating higher on all the four criteria of Fluency, Sentence Structure, Vocabulary, and Pronunciation. The neutral and negative attitude groups rated significantly differently on Sentence Structure and Vocabulary, with the former rating higher than the negative group. Moderate to strong correlations ranging from .272 to .569 were observed between the RAI and the IELTS descriptive task scores. Multiple regression analysis revealed that RAI scores accounted for a significant amount of variance on the IELTS descriptive tasks sub- and total-scores, ranging from 17.5% to 32.4%. Moreover, the Indian/non-Indian variable was the only rater background characteristic investigated that significantly related to the rater feeling scores that formed one of the triplet attitude constructs, though contributing to only 10% of the score variance. Lastly, the verbal protocol study provided insightful information suggesting that raters with positive attitudes generally took into account the expected performance of language learners while some negative-attitude raters used the native speaker model as the underlying criterion for judgment. The impact of the findings on validity argument, test fairness, and rater trainings are also discussed.

Graduation Semester

2012-05

Permalink

http://hdl.handle.net/2142/31133

Copyright and License Information

Copyright 2012 Huei-Lien Hsu

Owning Collections