Examining User Engagement via Facial Expressions in Augmented Reality with Dynamic Time Warping

Augmented Reality (AR) has progressively demonstrated its helpfulness for novicesto learn highly complex and abstract concepts by visualizing details in an immersive environment. However, some studies show that similar results could also be obtained in environments that do not involve AR. To explore the potential of AR in advancing transformative engagement in education, I propose modeling facial expressions as implicit feedback when one is being immersed in the environment. I developed a Unity application to record and log the users' application operations and facial images. A neural network-based model, Visual Geometry Group 19 (VGG19, Simonyan and Zisserman (2014)), is adopted to recognize emotions from the captured facial images. A within-subject user study was designed and conducted to assess the sentiment and user engagement differences in AR and non-AR tasks. To analyze the collected data, Dynamic Time Warping (DTW) was applied to identify the emotional similarities between AR and non-AR environments. The results indicate that users showed an increase in emotion patterns and application operations throughout the AR tasks in comparison to non-AR tasks. The emotion patterns observed in the analysis show that non-AR provides less implicit feedback compared to AR tasks. The DTW analysis reveals that users' emotion change patterns appear to be more distant from neutral emotions in AR than non-AR tasks. Succinctly put, the users in the AR task demonstrated more active use of the application and yielded ranges of emotions while operating it.