Supporting instructor decisions on algorithmic team formation through integrating stakeholder voices

Hastings, Emily Marie

Permalink

https://hdl.handle.net/2142/120222 Copy

Description

Title

Supporting instructor decisions on algorithmic team formation through integrating stakeholder voices

Author(s)

Hastings, Emily Marie

Issue Date

2023-03-24

Director of Research (if dissertation) or Advisor (if thesis)

Bailey, Brian P

Doctoral Committee Chair(s)

Bailey, Brian P

Committee Member(s)

• Karahalios, Karrie
• Marinov, Darko
• Mercier, Emma
• Brundage, Michael

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Algorithms
• CATME
• Learnersourcing
• Crowdsourcing
• Team formation
• Team composition
• Team building
• Algorithmic team formation
• Psychological safety

Abstract

Teamwork is a foundational skill in computing and other disciplines, and instructors are increasingly adopting team-based learning in their courses. However, as enrollments grow larger and more diverse and move to hybrid and online settings, it is becoming increasingly difficult to implement teamwork in ways that best support student learning. One solution for instructors seeking to improve and scale the team formation process in their courses has been to turn to algorithmic tools, which can be used to automate the process of grouping students into teams based on criteria such as demographics, skills, and working styles. However, configuring the inputs in a a team formation tool is a difficult problem. The space of possible configurations in a team formation tool is large and more expansive than the configurations tested in the literature, and the best configuration for one teamwork context may not generalize to other contexts. It is also unclear how instructors decide the configurations. Ineffective inputs can adversely impact student learning experiences. Furthermore, the design of team formation tools places the instructor in control and the stakeholders most affected by this process - the students - have minimal agency in deciding how the tool will be used to group them into teams. Groups formed by the tool may be especially prone to "cold start" problems, since team members typically have not had any prior social interaction. These problems will only intensify as enrollments continue to grow, teamwork becomes more malleable, and tools incorporate more complex sets of criteria for team formation. In this dissertation, I addressed these challenges by designing, implementing, deploying, and studying new learner-centered techniques for algorithmic team formation. First, I designed and implemented two categories of team-building activities intended to support the development of psychological safety in student teams formed with a tool, and conducted an experiment which tested the effectiveness of these activities for teams formed algorithmically and via random assignment. Teams reported high levels of psychological safety, but these levels appeared to develop organically and were not affected by the activities or compositional strategies tested. Surprisingly, criteria-based teams did not statistically differ from random teams on psychological safety, perceived performance, or team satisfaction, despite having compositions that better satisfied the criteria defined by the instructor. The findings of this experiment suggest that additional research is required to achieve improved team outcomes beyond simply deploying an algorithmic team formation tool. One possible explanation for the lack of statistical differences in the outcome measures is that the criteria configurations selected by the instructors were inconsistent with what students most valued for team formation. However, there are currently no mechanisms for discovering configurations that might better align with these preferences. Therefore, I designed LIFT, a novel learnersourcing workflow where students propose and vote for the criteria used as inputs to the team formation algorithm. I conducted an experiment involving nearly 1000 students comparing LIFT to the usual instructor-led process and interviewed participants to evaluate their perceptions of LIFT and its outcomes. Learners proposed novel criteria not included in existing algorithmic tools, such as organizational style, and avoided criteria like gender and GPA that instructors frequently select, preferring those promoting efficient collaboration. LIFT led to team outcomes comparable to those achieved by the instructor-led approach, and teams valued having control of the team formation process. I next extended LIFT to incorporate a consensus building process, where students determined the weights for each criterion using a staged discussion and voting process. Students most prioritized criteria relating to scheduling and commitment levels, followed by demographic attributes, and then task skills; these preferences were consistent across the four semesters studied. Using LIFT, instructors can now learn from students' values and localized knowledge in addition to the literature and their own prior experience. Instructors now have many options for sourcing potential criteria and weight configurations. However, the relative novelty of team formation tools means that there is currently little knowledge of how instructors choose which of these sources to utilize, how they relate different criteria to their goals for the planned teamwork, or how they determine if their configuration or the generated teams are successful. To close this gap, I conducted a survey and interview study investigating instructors’ goals and decisions when using team formation tools. The results showed that instructors prioritized students learning to work with diverse teammates and performed "sanity checks" on the tool's output to ensure that the generated teams would support this goal, especially focusing on criteria like gender and race. In general, they also did not solicit any input from students when configuring the tool, despite acknowledging that this information might be useful. By opening "black box" of the team formation algorithm more to students, LIFT or similar learner-centered approaches could therefore be a promising way to provide more support to instructors configuring algorithmic tools while at the same time supporting student agency and learning about teamwork. This dissertation advances knowledge of how to design algorithmic team formation tools to best support instructors and be inclusive of learner preferences. Through this work, I improve the way that students experience teamwork in their courses, bringing us closer to an ideal future where every student has a positive and successful team-based learning experience.

Graduation Semester

2023-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Emily Hastings

Owning Collections