Investigating User Experience of Chatbot Repair Strategies in Simple Versus Complex Tasks

The implementation of chatbots in customer service is widely prevalent in today’s world with insufficient research to appropriately refine all of their conversational abilities. Chatbots are favored for their ability to handle simple and typical requests made by users, but chatbots have proven to be prone to conversational breakdowns. The study researched how the use of repair strategies to combat conversational breakdowns in a simple versus complex task setting affected user experience. Thirty participants were collected and organized into six different groups in a two by three between subjects factorial design. Participants were assigned one of two tasks (simple or complex) and one of three repair strategies (repeat, confirmation, or options). A Wizard-of-Oz approach was used to simulate a chatbot that participants interacted with to complete a task in a hypothetical setting. Participants completed the task with this researcher-controlled chatbot as it intentionally failed the conversation multiple times, only to repair it with a repair strategy. Participants recorded their user experience regarding the chatbot afterwards. An Analysis of Covariance statistical test was run with task duration being a covariate variable. Findings indicate that the simple task difficulty was significant in improving the user experience that participants recorded whereas the particular repair strategy had no effect on the user experience. This indicates that simpler tasks lead to improved positive user experience and the more time that is spent on a task, the less positive the user experience. Overall, results associated with the effects of task difficulty and repair strategies on user experience were only partially consistent with previous literature.