Stress, depression, and anxiety are prevailing mental health issues that affect individuals worldwide. As the search for effective solutions continues, advancements in technology have led to the development of digital tools for stress identification and management purposes. The Cigna StressWaves Test (CSWT) is a publicly available stress analysis toolkit that claims to use “clinical-grade” artificial intelligence (AI) technology to evaluate individual stress levels through speech. To investigate their claim, this research stands as an independent validation study involving 60 participants over the age of 18. The primary objective of the study is to assess the repeatability and efficacy of the CSWT as a stress measurement tool. Key results indicate the CSWT lacks test-retest reliability and convergent validity. This implies that the CWST is not a repeatable tool and does not provide similar stress outcomes relative to an established measure of stress, the Perceived Stress Scale (PSS). These findings cast doubt on the accuracy and effectiveness of the CWST as a stress assessment tool. The public availability of the CSWT and the claim that it is a “clinical-grade” tool highlights concerns regarding premature deployment of digital health tools for stress management.