What does a chatbot trial actually test? Revisiting early Woebot evidence

Turn a broad label into a research question

A digital experience can place prompts, exercises and feedback inside a conversation. It can also deliver its content through reading, audio or an interactive story. Researchers may want to know whether the content helps, or whether a conversational format makes it easier to understand and engage with. These are distinct questions.

A 2017 Woebot trial offers an early example of self-help content informed by cognitive behavioral therapy principles being delivered through chat. We revisit evidence about a particular system and study arrangement. The label “chatbot” is not itself an explanation of benefit.

Read short-term differences alongside the null findings

The intention-to-treat analysis in the results and Table 2, adjusting for baseline and handling missing data, favored Woebot on the PHQ-9 self-report measure of depressive symptoms. Anxiety and positive and negative affect showed no significant between-group differences. A separate completer analysis found anxiety declining over time in both groups; that does not establish an additional anxiety benefit from chat. [1]

Different measures answer different questions. A favorable short-term score on one measure leaves questions about who might benefit, whether change persists, and the experiences of people who feel no change or discomfort. Keeping these questions visible lets an early finding inform further research.

The comparison did not isolate content from format

The information-reading control did not offer the same cognitive behavioral self-help content. The chat condition also combined prompts, mood recording and exercises. Its findings therefore cannot isolate the contribution of the interface, empathic phrasing or any single feature. The authors identify this limitation. [1]

To ask what conversational delivery adds, a new study could keep core content and opportunities for contact comparable while varying delivery format. To study one feature, that feature should be the part deliberately changed between conditions. These are research proposals from this site, requiring a specific design and evaluation.

This approach does not require every product to look alike. It connects the research question with the intended difference and helps readers see whether a result concerns a combined arrangement or a design element that can be discussed separately.

Keep participation records at the right distance from outcomes

Of 70 participants, 58 provided partial or complete reassessment data; 58 was not the denominator for every outcome. Attrition was unequal and higher in the reading control, which lacked objective usage records. Individual chat usage could not be linked to outcome changes, preventing a dose–response assessment. The missing-data analysis assumed data were missing at random. [1]

In a new digital wellbeing study, recording that an interaction occurred and recording completion of an exercise may serve different purposes. Explain what each record represents before choosing how to use it. A record does not establish that content suited someone; its absence also allows several explanations.

Withdrawal and missing reassessments deserve a place in the evaluation. We propose optional feedback that adds little pressure to disclose private circumstances and allows people to withhold a reason. Research should notice differing needs during participation, rather than describe only those still using a product at the end.

When versions change, locate the evidence again

This small, unblinded trial had no additional follow-up of sustained change. Findings about its early fixed decision tree do not automatically establish effects for later systems, other chat products or generative AI. [1] At this check, the company’s FAQ states that the Woebot app retired on 30 June 2025, with no new-account creation or access to previous accounts. [2] That statement concerns the app; it does not establish the status of the entire company or every business activity.

For a new digital experience, we suggest keeping a clear version record: the content, delivery, prompts and intended context studied. When these change substantially, explain which conclusions remain applicable and which need another evaluation. Connecting versions with evidence is more informative than placing different systems under one “AI wellbeing” label.

An early trial can contribute a question worth pursuing. Digital wellbeing research should make that question clear and leave its unanswered parts where readers can see them.

This article contains AI-generated content. AI researched, synthesized, wrote and edited both languages using the cited public sources. This site did not conduct the trial, hands-on product testing or original interviews. The operating team develops Six Realms and provides Heart to Heart and Sleep, creating relevant commercial interests. The study did not assess these products and does not establish their effects. This historical research commentary does not provide a personal diagnosis or treatment plan.

Luoboli · Investigating evidence and experience