‘Holes’ in federal AI healthcare regulation should be patched, Penn Med faculty say

As the use of artificial intelligence becomes more prevalent in healthcare settings, experts are calling for strengthened regulations surrounding the technology.

In recent years, AI has become a popular tool in both medical research and clinical environments, including at the Perelman School of Medicine. The Daily Pennsylvanian spoke with University faculty who voiced concerns about current AI regulatory frameworks and offered potential solutions.

Before new medical devices are marketed or distributed, they must be deemed “safe and effective” by the Food and Drug Administration. 

Penn Medicine assistant professor of pulmonary and critical care medicine Gary Weissman said that these FDA approval pathways “were developed decades ago for traditional medical devices, and don't apply well to modern AI-type devices.”

The current framework for approving such tools is the 510(k) pathway, which, according to Weissman, grants clearance to approximately 98% of all AI-enabled devices.

This pathway is intended for devices similar to preexisting market options that have already proven to be “safe and effective,” Medical School internist and National Clinician Scholars Program fellow Kaustav Shah explained.

“So the amount of data or evidence you need to produce is not as robust as if you were trying to perhaps put something de novo,” Shah added, noting that the complexity of AI devices can make it more difficult to determine whether they are high quality.

Weissman voiced a similar concern, stating that the current pathway “has a lot of holes in it.”

After an AI device is implemented, he explained that its performance may change over time and vary between locations, leading to what is known as “shift” or “drift.” He added that the current regulatory process lacks “good infrastructure” to ensure an AI system is functioning properly at the location where it is deployed.

Assistant professor of medicine in the Division of Hospital Medicine at Penn and senior fellow at the Leonard Davis Institute Eric Bressman said that a more “expansive and robust framework” is needed to ensure that AI tools are properly trained and tested before independent practice can be considered.

He proposed a pathway that requires AI systems to be monitored before they can receive clearance, incorporating a “supervisory model that leads towards graduated autonomy.”

“[It] starts to look a lot more like what a human would go through as they approach independent practice,” Bressman explained.

Shah argued that institutions — such as Penn — should adopt their own AI governance processes to determine “the right level of evaluation, rigor, and transparency” to incorporate.

“[These] should be in place regardless of what the requirements are at the federal or state level,” he said.

At the federal level, Bressman said he supports an FDA-centered framework that would include a partnership with state medical boards. 

Additionally, he argued that local regulation is necessary to keep up with the pace of technological developments.

“The federal government is not necessarily well suited to be in a position to deal with every little update or new thing that comes to practice or the market,” he said.

In Pennsylvania, House Bill 1925 introduced legislation that would require insurers, hospitals, and clinicians to provide transparency to patients and the public in circumstances where AI is used. The bill seeks to ensure that an “ultimate individualized assessment” is made by a human and requires evidence that “bias and discrimination already prohibited by law has been minimized in their use of AI.“ 

“I think it’s a good start,” Weissman said about the bill. “I do think those are important steps, but it's different than device regulation from a decision support space, which it doesn’t sound like is addressed.”

---