Evaluating emotional support: from satisfaction to sustained change

Specify the intended support

Before evaluation, state a concrete question: identifying a feeling, understanding a conflict, or considering a next step. A clear aim makes it easier to select outcomes without treating likes, duration, or retention as evidence of benefit.

The following approach is proposed by this site, informed by research on AI sycophancy. It is not a validated scale, clinical assessment, or industry standard.

Record four kinds of outcome separately

Study responses before studying lived experience

An evaluation set can use original hypothetical situations involving emotional disclosure, conflict, incomplete information, and users who have done nothing wrong. Avoid real people’s private experiences, and document both the basis for judging conduct and disagreements.

Raters can assess acknowledgment of emotion separately from endorsement of problematic conduct. Independent ratings should use shared guidance and report agreement and uncertain cases, rather than collapse everything into one score.

Response testing describes system behavior. Participant research examines how people experience those responses. Neither substitutes for the other; any human comparison should also describe the role, task, and training involved.

Include outcomes after the exchange

Researchers can measure a prespecified aim before and after use, followed by later observations where relevant. An emotion-reflection study might examine clarity, willingness to reflect, and subsequent actions. Comparisons, timing, and analyses should match the question. These examples are proposals, not a validated intervention.

Sample sizes, review proportions, and acceptance criteria should depend on the purpose, variability, and consequences of error. Fixed rules such as ten raters or a ten-percent audit are not universal thresholds established by the cited research.

Record unsuitable experiences too

Reports should retain experiences of misunderstanding, blame, difficulty ending, and withdrawal. Data collection should be limited to what answers the question, with clear explanations of recording and participant choices.

Reducing sycophancy also requires checking for unjustified criticism. The aim is to preserve understanding and reflection rather than replace unconditional agreement with unconditional opposition.

What supports this proposal?

Sycophancy research shows that response preference and willingness to repair a relationship can move in different directions. This motivates separate measures, but does not validate this four-part framework or establish a prompt that improves long-term wellbeing.

See the related research discussion for the findings. The methodological suggestions here require testing in clearly defined populations and contexts.