Probabilistic performance metric elicitation

Robertson, Zachary

Permalink

https://hdl.handle.net/2142/115425 Copy

Description

Title

Probabilistic performance metric elicitation

Author(s)

Robertson, Zachary

Issue Date

2022-04-25

Director of Research (if dissertation) or Advisor (if thesis)

Koyejo, Oluwasanmi

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Machine Learning
• Active Learning
• Metric Selection

Abstract

Metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for classification using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. Here we present a unified approach for learning metrics robust to constant and location-dependent noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against existing methods for performance metric elicitation and inverse decision theory demonstrate the advantage of our approach.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Zachary Robertson

Owning Collections