University of Illinois Urbana-Champaign

Models and evaluation of user simulation in information retrieval

Labhishetty, Sahiti

Content Files
LABHISHETTY-DISSERTATION-2023.pdf

Permalink

https://hdl.handle.net/2142/120103

Description

Title
Models and evaluation of user simulation in information retrieval
Author(s)
Labhishetty, Sahiti
Issue Date
2023-04-26
Director of Research (if dissertation) or Advisor (if thesis)
Zhai, ChengXiang
Doctoral Committee Chair(s)
Zhai, ChengXiang
Committee Member(s)
  • Sundaram, Hari
  • Chang, Kevin Chen-Chuan
  • Murdock, Vanessa
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • User Models
  • User Simulation
  • Evaluation of Information Retrieval Systems
  • Information Retrieval
Abstract
Search and recommendation are crucial parts of many applications. Additionally, assistive AI systems have become very popular with successful intelligent agent systems. Although the IIR(interaction information retrieval) systems, including conversational systems, already improve the user experience for search, recommendation and question answering, the evaluation of such systems still has many challenges. For example, how to compute the overall utility of a system in helping a user in achieving their goal? How to compare different IIR systems? How to perform A/B testing that is reproducible, robust and less risky to online user experience? User simulation enables controlled and reproducible experiments and at the same time can simulate user interaction with an IIR system and can evaluate the overall effectiveness of an IIR system. User simulation in information retrieval (IR) aims to develop user models to simulate how a user interacts with an IR system. It involves simulation of user actions, behavior or decisions in a search process, like query formulation, click simulation, and so on. User simulation in IR has many applications like offline (without real users) evaluation of interactive IR systems, generating synthetic data to train IR models, especially for training reinforcement learning models, and modeling user behavior for analysing search behavior. However, search user simulation is quite challenging. The existing models for simulating users lack interpretability; nor can they model a user’s cognitive state. Most user simulation models do not leverage user search log data to learn better models from real user search sessions. Interpretability is needed to model meaningful variations in user behavior and thus simulate user actions corresponding to different types of user behavior. It is important to model a user’s cognitive state to build a more generalized formal model because user actions are guided by the latent user cognitive state which constitutes the user’s knowledge, information need and search behavior characteristics. It is also challenging to utilize the search log data of users to build advanced simulation models. Another difficult challenge that has not yet been addressed in the previous work is how to assess the reliability of a user simulator to evaluate IIR systems. My research aims to address the challenges in both modeling and evaluation of user simulation. To address the challenges in modeling users, I have studied how to develop new user models that are both interpretable and can model a changing user’s cognitive state for user simulation in both Web search and E-commerce search scenarios. User search logs have rich information about users which can be used for building better user models. Therefore, we also propose a supervised user model based on imitation learning which can learn from large-scale search logs to simulate different user actions. To address the limitations in the evaluation of user simulation, we propose a novel evaluation framework that evaluates the reliability of a user simulator where the framework does not necessarily require real user data. Specifically, the following contributions are made towards the thesis: 1. We propose a new user model called CSUM (Cognitive State User Model) for E-commerce search that models a changing user’s cognitive state and is parameterized meaningfully such that the parameters correlate with different user behavior. 2. Query simulation is a critical component of the user simulation. We propose a novel unified Precision Recall-Effort (PRE) optimization framework for simulating query formulation and reformulation which is applicable for modeling both Web search and E-commerce search users. 3. Search logs contain rich information about user search actions and behavior, and also enable modeling users with a wide range of information needs. We propose a novel Imitation Learning based User Model (ILUM) which is a supervised user simulation model based on imitation learning to learn from the search logs. The ILUM model learns to simulate different user actions along with deciding what action to take next. It simulates user actions based on all its previous actions and the given user task/information need. 4. We address some of the challenges in the evaluation of user simulation by proposing a novel Tester-based evaluation (TBE) framework to evaluate the reliability of a user simulation for comparing IIR systems. The advantage is that the framework does not necessarily require real user data to evaluate the simulator and it aims to evaluate the predictive validity of a simulator. We further extend the TBE framework by proposing the Reliability Aware Tester-based evaluation (RATE) framework to address the drawbacks of the TBE framework. We propose an optimization framework for the query simulation and cognitive state models of the user during the search process through PRE and CSUM models, respectively. Both models are interpretable and can be varied in order to simulate different user behaviors. PRE simulates both initial query formulation and subsequent reformulation in a uniform manner and serves as a roadmap for the systematic exploration of many new specific query simulation models and algorithms. However, PRE and CSUM models cannot learn search patterns from user search sessions. Thus, we also propose a data-driven approach using search logs to build a user simulation model based on imitation learning which we refer to as ILUM((Imitation Learning based User Model) model. The ILUM model can be trained to simulate a complete user model, including different user actions and decisions during the search. The ILUM model can learn complex search patterns, but unlike PRE and CSUM models it lacks interpretability, in that the model cannot be varied meaningfully to simulate different types of users and information needs for generating search sessions. As the ILUM model learns from all sessions together, it learns to simulate an average user in search, whereas PRE or CSUM models can simulate a specific type of user. An interesting research focus could be to build a data-driven user simulation model that can meaningfully simulate variation in search behavior for different user types and information needs. Finally, we propose a novel evaluation framework, RATE, for evaluating user simulation models in terms of reliability for comparing IR systems. The advantage of RATE is that it does not necessarily need real user search data to evaluate a simulator and can complement other evaluation metrics. One of the pivotal future works for user simulation in IR would be to develop an evaluation platform with many user simulators that is available for the research community to utilize in the evaluation of IR systems or other applications.
Graduation Semester
2023-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/120103
Copyright and License Information
Copyright 2023 Sahiti Labhishetty

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois