University of Illinois Urbana-Champaign

Probabilistic performance metric elicitation

Robertson, Zachary

Content Files
ROBERTSON-THESIS-2022.pdf

Permalink

https://hdl.handle.net/2142/115425

Description

Title
Probabilistic performance metric elicitation
Author(s)
Robertson, Zachary
Issue Date
2022-04-25
Director of Research (if dissertation) or Advisor (if thesis)
Koyejo, Oluwasanmi
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • Machine Learning
  • Active Learning
  • Metric Selection
Abstract
Metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for classification using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. Here we present a unified approach for learning metrics robust to constant and location-dependent noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against existing methods for performance metric elicitation and inverse decision theory demonstrate the advantage of our approach.
Graduation Semester
2022-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/115425
Copyright and License Information
Copyright 2022 Zachary Robertson

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois