As generative artificial intelligence becomes embedded in people's everyday lives, one emerging aspect of its use in mental health care is raising complicated questions about professional ethics and patient privacy.
A number of companies, like Upheal, Blueprint, and Heidi Health, have begun offering AI-powered tools designed to make therapists more efficient at documenting sessions and completing administrative paperwork. The catch? Providers are typically required to record the entirety of their session with a client.
While it's ethical for therapists to record these conversations under certain circumstances, it's rarely done outside of professional training and forensic work. Note-taking tools, or "scribes," use AI to analyze the content of a client's conversation with their therapist in order to generate documentation that therapists must submit for a variety of reasons, including for insurance payments and potential quality audits.
Clinicians who use such AI products say it streamlines tedious tasks, freeing up time to focus not just on aiding their clients, but also on their own lives.
Yet some experts say such AI products introduce unnecessary or unethical risks, like the possibility that recordings will be hacked or used to train a company's large language model without the client's consent. They may also negatively affect the relationship between the therapist and client if the person seeking treatment holds back in the presence of a recorder, or feels like they can't decline their provider's request.
"The industry kind of jumped the gun a little bit without asking the question, 'Is this a good idea?'" said Dr. Vaile Wright, senior director of the office of health care innovation at the American Psychological Association. "We just don't know the answer to that question...it feels like we skipped over it."
The "dread" of writing clinical notes
Psychologist Dr. Hannah Weisman, who runs a half-time therapy practice in Seattle, began using an AI scribe last December. In addition to her practice, Weisman advises tech companies working in the mental health space, though she doesn't consult on any scribe tools.
Weisman said she dreads writing clinical notes because of how many audiences she must keep in mind. In addition to an insurance company, her notes might be requested by another health care provider, a judge in a legal matter involving a client, or the client themselves.
For a period of time this year, Weisman primarily used Heidi Health's medical scribe. The tool's offering for psychologists promises to "increase engagement, restore eye contact, and offer warmer mental health care."
Heidi Health and the other AI scribes that Wiseman has tested have reduced the draining "cognitive load" of picking out the right details for her notes and composing them in one of several potential formats. While there is no research on efficiency gains for mental health providers, Weisman estimates that the tool saves her about five minutes of time for each client, too.
Yet Weisman is also aware that AI scribes, particularly those that record sessions, pose complex risks, even as they ease her workload.
Weisman provides all clients, whether new or existing, with an informed consent form that she personally created, after consulting boilerplate versions offered by various AI scribe companies.
She requires written consent from clients and emphasizes that it can be revoked at any time, including in the midst of a session. Weisman also makes clear that she records the session on a personal digital recorder and uploads it to the AI scribe on her password-protected computer. In her consent form, Weisman commits to deleting all copies of the audio, including the recording on her device, within 48 hours.
She's also decided, as a rule, not to use AI scribes that anonymize transcripts and retain them to better train their product.
"That's a dealbreaker for me," she says. "I, myself as a therapist, am really trying to [be biased] toward protecting consumers. I would think that as a field and as therapists, that's the lens we should be taking."
Heidi Health says it encrypts the audio as it is being transcribed. The company doesn't store the recording, nor does it use the transcript to train its AI technology. The transcript is produced by Heidi Health's privately hosted AI models, instead of by a third party. Clinicians are responsible for deleting the transcript from Heidi Health.
Weisman estimates that three-quarters of her clients consented to being recorded. Some of the Seattle-area tech workers she sees have adamantly turned her down while others have agreed, noting that they use generative AI products in their own work.
The possibility of "reputational harm"
Last fall, the American Psychological Association created a checklist for therapists considering any AI tool for clinical or administrative purposes. The goal is to help therapists, who may have little or no understanding of how generative AI works, evaluate different products with safety and privacy in mind.
The checklist prompts users to ask if a product is HIPAA compliant, encrypts user data, employs advanced security measures, and allows users to delete or modify their data, among other considerations.
Even so, the APA's Wright said independent mental health professionals may not be able to parse dense technical language on their own. They may also encounter companies that intentionally make their privacy practices opaque.
In general, she said therapists should understand that every product is fallible; data breaches and leaks can happen at any time.
Indeed, recent research published in JAMA Network Open found that the number of healthcare data breaches and ransomware attacks has increased annually since 2010, totaling 6,468 unique incidents through October 2024. While hacking or IT incidents are the dominant types of breaches, ransomware attacks now account for the majority of compromised patient records.
"Ransomware attackers don’t need to leak this kind of data to do damage — they just need to make the threat credible."
When asked by Mashable about recorded therapy sessions, lead author John X. Jiang said that they could become a "vulnerable target" of bad actors. Since the audio typically contains sensitive information, the recordings have unique blackmail value if stolen.
"Ransomware attackers don’t need to leak this kind of data to do damage — they just need to make the threat credible," said Jiang, a professor of accounting at Michigan State University who research includes healthcare cybersecurity. "The combination of operational disruption and reputational harm creates a potent form of leverage."
Dr. Darlene King, chair of the committee on mental health IT at the American Psychiatric Association, said that therapy notes should be held to a higher security standard than the information that's commonly entered into medical charts. While that data is also highly sensitive, the content of patients' therapy sessions can include detailed and deeply personal information, like experiences with trauma, abuse, and addiction.
King, a psychiatrist at UT Southwestern Medical Center in Dallas, uses an AI scribe for medical documentation but not for therapy.
She added that the mental health profession needs to find a balance between easing burdens — and burnout — for providers and protecting patient privacy, all while taking advantage of the positive uses for AI, like improving mental health treatments.
Why record at all?
Jon Sustar, a software engineer and co-founder of Quill Therapy Solutions, believes he's found an answer to one part of this challenge: Don't record sessions at all.
Quill uses generative AI to produce documentation for clinicians but does so based on their verbal or written summaries.
While this approach may not reduce the cognitive load of recalling and prioritizing elements of what a client discussed, it does mean there is no record of the session to breach. Audio summaries are immediately transcribed and subsequently deleted. Quill doesn't store the notes that it creates, either. Sustar describes the data as "ephemeral."
Sustar, whose wife is a licensed mental health counselor and Quill's co-founder, steadfastly believes that therapy is a sacred space. He worries that it can negatively affect the power dynamic between a therapist and their client when the former asks the latter for permission to record their conversation.
Sustar also understands that people, whether they're in formal therapy or not, have turned to generative AI platforms like ChatGPT to talk about their personal struggles, much like they would with a mental health provider.
While some of those users may have made peace with breaches of their data, he worries that venture capital-backed startups have suddenly shifted the norm in mental health toward de facto AI recording and analysis of sessions, even if therapists and their clients don't fully realize what that involves or means.
"My biggest concern is that companies are quietly normalizing the mass recording of therapy sessions, and they're doing this often without a fully informed consent of all who are involved," Sustar says.
