Demystifying a dark art: Understanding real-world machine learning model development
Lee, Angela
Loading…
Permalink
https://hdl.handle.net/2142/108007
Description
Title
Demystifying a dark art: Understanding real-world machine learning model development
Author(s)
Lee, Angela
Issue Date
2020-05-11
Director of Research (if dissertation) or Advisor (if thesis)
Parameswaran, Aditya
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
machine learning
data analysis
empirical studies
user behavior
Abstract
It is well-known that the process of developing machine learning (ML) workflows is a dark-art; even experts struggle to find an optimal workflow leading to a high accuracy model. Users currently rely on empirical trial-and-error to obtain their own set of battle-tested guidelines to inform their modeling decisions. In this study, we aim to demystify this dark art by understanding how people iterate on ML workflows in practice. We analyze over 475k user-generated workflows on OpenML, an open-source platform for tracking and sharing ML workflows. We find that users often adopt a manual, automated, or mixed approach when iterating on their workflows. We observe that manual approaches result in fewer wasted iterations compared to automated approaches. Yet, automated approaches often involve more preprocessing and hyperparameter options explored, resulting in higher performance overall---suggesting potential benefits for a human-in-the-loop ML system that appropriately recommends a clever combination of the two strategies.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
