Investigating Human Advisor Interventions and Team Compliance in a Search-and-Rescue Human-AI Team Task

With the increasing popularity of AI and machine learning, human-AI teaming has a wide range of applications in transportation, healthcare, the military, manufacturing, and people’s everyday life. Measurement of human-AI team effectiveness is essential for guiding the design of AI and evaluating human-AI teams. To develop suitable measures of human-AI teamwork effectiveness, we created a search and rescue task environment in Minecraft, in which Artificial Social Intelligence (ASI) agents inferred human teams’ mental states, predicted their actions, and intervened to improve their teamwork (Huang et al., 2022). As a comparison, we also collected data from teams with a human advisor and with no advisor. We investigated the effects of human advisor interventions on team performance. In this study, we examined intervention data and compliance in a human-AI teaming experiment to gain insights into the efficacy of advisor interventions. The analysis categorized the types of interventions provided by a human advisor and the corresponding compliance. The finding of this paper is a preliminary step towards a comprehensive study on ASI agents, in which results from the human advisor study can provide valuable comparisons and insights. Future research will focus on analyzing ASI agents’ interventions to determine their effectiveness, identify the best measurements for human-AI teamwork effectiveness, and facilitate the development of ASI agents.