HEALTHCARE & HEALTHTECH

HIPAA and generative AI: BAAs, minimum necessary data and the guardrails that keep PHI protected

How health tech companies and providers can use LLMs with protected health information under HIPAA: BAAs, de-identification, minimum necessary use, logging and guardrails.

By Akash Soni, Founder & CEO, Zuroth · Updated 9 Nov 2026 · 9 min read

The short answer

You can use generative AI with protected health information (PHI) if four things are true: every AI vendor that touches PHI has signed a business associate agreement (BAA); you send the model only the minimum necessary data; prompts, outputs and logs are protected like any other PHI; and guardrails and human review stop the AI from giving unsafe or wrong answers. Where possible, use de-identified data instead.

How HIPAA applies to AI

HIPAA has no special AI rule. The same Privacy, Security and Breach Notification Rules apply, but AI creates new places for PHI to flow:

  • Prompts and context sent to a model provider.
  • Retrieval indexes built from clinical notes.
  • Logs and traces used for debugging and evaluation.
  • Fine-tuning datasets and evaluation sets.
  • Outputs written back to the EHR or sent to patients.

Each one needs a BAA where a vendor is involved, encryption, access control, audit logs and a retention limit.

Common mistakes

Mistake Safer approach
Staff paste notes into a consumer chatbot Approved enterprise AI tool with a BAA, plus a written AI policy
Whole patient record sent to the model Only the fields or sections the task needs
Debug logs keep full prompts forever Redacted logs with a retention limit
Fine-tuning on raw notes De-identified data, or retrieval instead of fine-tuning
AI messages patients directly Clinician reviews before anything is sent

How to use generative AI with PHI: 6 steps

1. Write an AI use policy

List approved tools, allowed data, banned uses and who approves new ones. Train staff on it.

2. Sign BAAs and check the architecture

Confirm every service in the chain, from model API and vector database to logging and hosting, is covered by a BAA and runs in your approved region. Turn off provider data retention and training where possible.

3. Minimize and de-identify

Send only what the task needs. Use de-identified data for analytics, evaluation and training, and test your de-identification on real note samples.

4. Add guardrails

Block prompt injection, check outputs for unsafe clinical advice and unsupported claims, and require clinician approval for anything that changes a record or reaches a patient.

5. Log and monitor

Keep an audit trail of who used the AI, on which patient and what it produced. Include AI systems in your risk analysis and incident response plan.

6. Evaluate before and after launch

Test accuracy and safety on realistic cases, check for bias across patient groups and monitor quality in production.

How Zuroth helps

  • HIPAA compliance from $499 a month: risk analysis, policies, BAAs and evidence, including AI systems.
  • HIPAA-ready AI development: retrieval, guardrails, de-identification and audit logging built in.
  • AI governance and ISO 42001 for health AI vendors selling to hospitals.

Our AI virtual care case study shows an AI scribe and patient tools rolled out across 14 clinics with HIPAA safeguards.

Next step: read the HIPAA compliance guide or book a free consultation.

This article explains general practice and is not legal advice. Confirm your obligations with your counsel.

Frequently asked questions

Can we use ChatGPT or other LLMs with patient data?

Only under the right agreement and setup. If an AI provider creates, receives, stores or transmits PHI for you, it is a business associate and must sign a BAA. Consumer chatbots usually don't offer one; several enterprise and cloud AI services do for eligible offerings. Without a BAA, don't send PHI.

Does de-identified data fall under HIPAA?

No. Data de-identified under HIPAA's Safe Harbor method, which removes 18 identifiers, or the Expert Determination method is no longer PHI. Free-text clinical notes often hide identifiers, so automated de-identification needs testing and review before you rely on it.

What is the minimum necessary rule for AI?

HIPAA's minimum necessary standard means using or sharing only the PHI needed for the purpose. For AI, send the model only the fields or note sections a task needs, restrict retrieval to the patient in context and avoid loading whole records into prompts.

Do AI prompts and outputs count as PHI?

Yes, if they contain individually identifiable health information. Prompts, retrieved context, outputs, logs and evaluation datasets must all be protected like the rest of your PHI: encrypted, access-controlled, logged and kept only as long as needed.

What HIPAA changes are coming that affect AI?

HHS proposed a major update to the HIPAA Security Rule in January 2025, including written technology asset inventories, network maps and stronger encryption and MFA expectations. Health AI is also shaped by HHS rules on algorithm transparency in certified health IT. Check the status of final rules before planning.

Akash Soni

Akash Soni

Founder & CEO, Zuroth

LinkedIn ↗

FREE CONSULTATION

Talk it through with someone who does this work.

30 minutes with a senior expert. You leave with your top 3 risks and clear next steps, whether or not we work together. NDA before we talk.

Book a free consultation →