Skip to main content
2026 Prospectus
Search

Hospitals want AI developers held accountable. Contracts must do the work

Hospitals want AI developers held accountable. Contracts must do the work
Greg Wahlstrom, MBA, HCM

Hospital and physician groups have told the FDA that developers should own the safety of generative AI devices, but until rules say so, hospitals carry the risk through their contracts and governance.

What providers asked the FDA

The FDA asked the industry this summer how it should regulate medical devices built on generative AI. These tools take open-ended input and can give different answers to similar questions. In comments filed this month, the American Hospital Association, the American Medical Association and ATA Action made the same core point: the companies that build and sell these tools should be responsible for how they perform over time.

The AHA’s Oct. 19 letter asked the agency to put primary responsibility for premarket and post-deployment monitoring on developers. It also asked for labeling that separates tools that inform clinicians from tools that take action, and for a total product lifecycle approach that ties clearance to ongoing evaluation. The AMA warned that keeping a human in the loop does not remove risk on its own, because clinicians can make mistakes or lean too hard on the tool.

Why the burden still sits with hospitals

Providers argue that monitoring these models would overwhelm many organizations, especially small and rural ones. They also say they have limited insight into how vendor models work, which makes flaws hard to spot. Those concerns are valid, but relief is not close. The FDA says draft guidance will arrive in the next few months, and there is still no substantive federal AI law for healthcare.

Meanwhile, adoption is moving faster than oversight. A survey of 250 U.S. healthcare leaders, cited by Healthcare Dive, found about 28% had deployed agentic AI tools and 72% said such tools are sometimes put in place without IT approval. Shadow AI is a governance gap boards should not ignore.

A useful test from the AMA

The AMA’s new framework for AI agents that take on clinical work offers four questions any AI committee can adopt now:

  • Function: What clinical task is the tool doing?
  • Evidence and risk: What proves it can do that task, and what happens when it fails?
  • Authority: What is it allowed to do, and under what conditions?
  • Accountability: Who owns the risks and the outcomes?

The underlying idea is that an AI tool earns clinical authority through evidence and keeps it only through measured performance. That is a sound standard for any system, whatever the FDA decides.

What to do this quarter

  • Build an inventory of every AI tool in clinical and business use, including tools bought by departments without IT review.
  • Review vendor contracts for performance warranties, model-change notice, access to monitoring data, incident reporting and indemnification.
  • Set a review threshold based on what each tool can do on its own, not just its stated purpose.
  • Ask the board’s quality or audit committee to receive a regular AI risk report.

Executive takeaway: Advocacy may eventually shift AI liability to developers, but today it stays with you. Put accountability into vendor contracts and give your AI governance committee clear authority before the next tool goes live.

Sources

Quarterly Brief

Get the Quarterly Brief by email.

A practical executive agenda for hospital and health system leaders, four times a year.

Sent by The Healthcare Executive. Unsubscribe at any time. See our Privacy Policy.