The effectiveness of test-takers' participation in development of an innovative web-based speaking test for international teaching assistants at American colleges

Kim, Jungtae

Permalink

https://hdl.handle.net/2142/72737 Copy

Description

Title

The effectiveness of test-takers' participation in development of an innovative web-based speaking test for international teaching assistants at American colleges

Author(s)

Kim, Jungtae

Issue Date

2007

Director of Research (if dissertation) or Advisor (if thesis)

Davidson, Frederick G.

Doctoral Committee Chair(s)

Wardrop, James L.

Committee Member(s)

• Davidson, Frederick G.
• Benson, Angela
• Steele, Diana

Department of Study

Graduate College

Discipline

Education

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D. (doctoral)

Degree Level

Dissertation

Keyword(s)

• Teaching
• Public Speaking
• Colleges

Language

en

Abstract

The purpose of this study was to investigate test-takers' behaviors and contributions by empowering the test-taker developers in language test development process. The test development team consisting of teachers and test-takers proposed a democratic language assessment and a validation model for the development. The testtakers participated in developing a web-based speaking test for international teaching assistants at American colleges and they articulated and reflected their needs into the process interacting with external and internal stakeholders. The audited specification-driven testing framework was adopted in the ADDIE model for the development. Validity arguments were gathered and justified from the beginning to the end of the test development. The effectiveness of test-takers' participation in the test development was investigated (a) the needs analysis on the SPEAK and its reflection in the test, (b) the test takers' roles in their interactions and power relations with regard to their contributions to task and validity arguments constructions, and (c) their contributions in validation studies with the data collected from a field test. The results of content analysis of the SPEAK showed the mismatch between the purpose and its use in a local context and the findings from the needs analysis were aligned with the team's decisions on test design considerations reflected into the development. The results of the group dynamics and audit trail supported positive contributions of the test-takers to the test development by revealing their roles, interactions, and power relations. The results of the validation studies with the data from test scores and survey and interviews also supported the effectiveness of the test-takers' participation in the test development by indicating their equal contributions to task construction, significant rating difference of one rater and rating discrepancy on the tasks in certain situations, no misfit across rater and task, and the need of more rater training. It could be concluded that the test-takers' participation in language test development was supportive results of validating test development process as well as the test product. Through the process, authenticity of topics, content, and other test conditions were achieved in their local academic contexts.

Type of Resource

text

Permalink

• http://search.proquest.com/docview/622019942?accountid=14553
• http://hdl.handle.net/2142/72737

Copyright and License Information

Copyright 2006 by Jungtae Kim. All rights reserved.

Owning Collections