Decision-making for a regenerative society: voice-based emotion AI

Porteanu, Monica

Permalink

https://hdl.handle.net/2142/115694 Copy

Description

Title

Decision-making for a regenerative society: voice-based emotion AI

Author(s)

Porteanu, Monica

Issue Date

2022-04-15

Director of Research (if dissertation) or Advisor (if thesis)

Hamilton, Kevin

Doctoral Committee Chair(s)

Hamilton, Kevin

Committee Member(s)

• Karahalios, Kyratso
• Grosser, Benjamin
• Araujo de Aguiar, Carlos

Department of Study

Illinois Informatics Institute

Discipline

Informatics

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• audible design
• design with sound
• voice-based design
• voice-based AI
• voice-based emotion AI
• algorithmic hearing
• tone analysis
• wearable computing
• affective computing
• voice sentiment analysis
• data ethnography
• algorithm ethnography
• interface ethnography

Abstract

Voice-based user interfaces are increasingly present in our daily lives. These systems rely on voice as a key identifier and biometric, sometimes drawing from aggregated data sets to facilitate decision-making based on interpretations of our emotional state, gender, ethnicity and more. However, voice as an indicator of identity and intention is already a fraught part of human interaction in light of bias and difference. The starting hypothesis is that AI-mediated voice-based decision-making and the technology’s assumptions about a user are subjective. Our voices operating in today’s complex technology ecosystem have new effects of which we are not fully aware. This research aims to understand a user’s voice journey through technology and its affective interpretations as it relates to voice-based decision-making that affects a user’s contextual identity and privacy. The dissertation starts with a new media speculative study that explores the use of sound in design as an inclusive and revelatory process of finding ways of living that rely primarily on the perception of sound. Next, the research investigates how users understand the ways in which a wearable device equipped with a voice analyzer “hears.” The study builds on precedents in affective computing, algorithm auditing frameworks, folk theories of algorithmic understanding, and human-centric AI. The outcome points out competencies involved in understanding objects that function using sound-based artificial intelligence. The research continues with a virtual “thing” ethnography for the three components of a voice-based emotion AI: interface, data, and algorithm. Using the five-emotion framework and digital research techniques (e.g., media archaeology, performative inquiry) and following AI best practices, it iterates through three script performances, analyzing the state of the field in terms of technological imaginaries in data classification, voice emotion modelling, voice emotion model evaluation, data orchestration, and agency over voice emotion in a big-data world) and its maturity today. The “thing” ethnography, coupled with the research with human participants study, point to the limitations of using standard datasets and the opportunity for creating performant voice-based emotion AI with real people’s data. The benefit of AI transfer learning, coupled with personal storytelling for data orchestration and a speculative, phenomenological exploration with small data, represents the opportunity to regain agency over voice emotion in a big-data world.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Monica Porteanu

Owning Collections