University of Illinois Urbana-Champaign

Cost-effective learning for classifying human values

Ishita, Emi; Fukuda, Satoshi; Oga, Toru; Tomiura, Yoichi; Oard, Douglas W.; Fleischmann, Kenneth R.

Permalink

https://hdl.handle.net/2142/106554

Description

Title
Cost-effective learning for classifying human values
Author(s)
  • Ishita, Emi
  • Fukuda, Satoshi
  • Oga, Toru
  • Tomiura, Yoichi
  • Oard, Douglas W.
  • Fleischmann, Kenneth R.
Issue Date
2020-03-23
Keyword(s)
  • Text classification
  • Content analysis
  • Human values
  • Annotation cost
Abstract
Prior work has found that classifier accuracy can be improved early in the process by having each annotator label different documents, but that later in the process it becomes better to rely on a more expensive multiple-annotation process in which annotators subsequently meet to adjudicate their differences. This paper reports on a study with a large number of classification tasks, finding that the relative advantage of adjudicated annotations varies not just with training data quantity, but also with annotator agreement, class imbalance, and perceived task difficulty.
Publisher
iSchools
Series/Report Name or Number
iConference 2020 Proceedings
Type of Resource
  • text
  • image
Language
eng
Permalink
http://hdl.handle.net/2142/106554
Copyright and License Information
Copyright 2020 Emi Ishita, Satoshi Fukuda, Toru Oga, Yoichi Tomiura, Douglas W. Oard, and Kenneth R. Fleischmann

Owning Collections

iConference 2020 Poster Descriptions PRIMARY