In most discussions regarding healthcare governance, most of the same elements are present: a policy with a steward, a framework (typically a slide deck signed off on months prior) and general agreement from stakeholders and participants to proceed to the next agenda item. The process lacks a sense of urgency.
Four recent conversations on The Signal Room depict the situation more clearly. They cover the work of governance professionals operating at a different level than most of the attendees in this meeting. One is a healthcare professional providing services. Another works in healthcare IT. The third is an operations professional designing AI systems to automate decisions and tasks. None of them coordinated to arrive at the same conclusion: governance is more than policy making. Oversight is the issue of the document. It is the true application of the policy and the decisions it describes when AI meets the health professional, the health record, and the healthcare system.
The policy is the easy part. Can you prove it?
Bennett Borden has a unique vantage point from where he can answer this question. Before founding Clarion AI Partners, Borden spent 8 years as a US intelligence analyst, and then as a chief data scientist and equity partner at a top 10 law firm. After analyzing 10 years of time entries for 14,000 attorneys, he estimated that 80% of repetitive legal tasks will be performed faster and better by AI. What concerns healthcare leaders the most, however, is his argument on the issue of defensibility.
Borden believes that the fact that everyone recognizes the importance of fairness, accuracy, and reliability (the repeated keywords in all frameworks) and then proceeds to do nothing is unacceptable. He incorporates the common law of the 1700s in his argument, which is reasonableness. The standard that will protect you, since the law cannot keep up with technology, is whether a reasonable company, based on the information available to them at the time, identified the potential risks, took precautionary measures, and has the ability to prove it. He says that layer is what will protect you and will keep you defensible as the laws continue to change. Borden believes that oversight is not a brake to innovation, but rather the system of painted lines and guardrails on a mountain road that enables you to drive faster. As a whole, his systems of oversight are closer to Iron Man than the Terminator.
Aaron Puckett accessed the same test from the operator seat. He is the executive of a security-first managed services subsidiary, and he was the guest for our episode on the AI security gap most companies miss. Most of the organizations Puckett enters have a policy. What is almost always missing is anything under the policy, no data loss prevention, no permission reviews, no policies that would survive an audit. He breaks the work into three steps, and notes that most people only concern themselves with the first. You decide what you want to do. Security goes under the policy to safeguard the decision. The last step is where most people are the most empty, and you have to show that you did. He finished the summary of the episode. I said the policy is the easy part. You show you did it. The lawyer and the engineer land on the same word. You have to show.
Why a six month old PDF cannot govern anything. The most ubiquitous AI tool in most buildings (nearly) never came in through the front door. Dr. Ömer Atlı is a solo on-call emergency physician at a rural district hospital, with two nurses, a midwife, and about eighty patients per shift, with the nearest scanner eighty-five kilometers away. He builds and red teams clinical AI tools. He calls the ungoverned tools already in clinical use, and the ones no one approved, the shadow formulary.
Every hospital runs a drug formulary, a list of medications voted on for the hospital use and dispensed only after the orders are signed. ChatGPT, Gemini, and Claude operate as a shadow formulary. They are in clinical use, are in no formulary, and no one oversees them. Atlı includes himself. He reaches for a model while he waits on bloods. His committee cannot see any of it. This is because no one purchased the model. Therefore, no contract or procurement trace exists. It is located in his and every other pocket in the building.
Atlı graded 60 model answers against emergency scenarios. What he feared the most in their model was not a wrong answer. Models, in contrast, correctly identified the danger. What they failed to answer was the next instruction. A patient, who describes textbook aortic dissection, with sudden tearing pain and who asks if he just pulled a muscle, is correctly identified as an emergency by the model. And then the model fails. It does not say call an ambulance. Nothing says for the patient not to drive himself. The answer that could kill someone was the answer that was correct, but the action was omitted.
His conclusion for leaders is the analytical element of the problem as such. You cannot eliminate the shadow formulary. Its prohibition only serves to deepen the problem. Thus, you will have to assume its existence and work towards making that existence less harmful. This problem is the easiest to solve as it merely requires the investment of a weekend. Maintain a standing set of the scenarios your clinicians face, re-run that set every time a model updates, similar to how a drug interaction check is performed every time a prescription is updated. A committee sign-off is a PDF you received six months ago. The set of scenarios you re-run is a continually updated control. Further, ensure that the risk is given a name, a single person who answers for that risk, because when it is shared, ownership of the risk is lost.
Oversight lives or dies in the infrastructure
The layer that Puckett works in is the one that the boardroom never sees. It is the identity grants, the permissions that go unchecked and the systems that were configured years, if not decades, before generative AI was even a concept. This is his most important sentence for any of the people that will be deploying AI assistants. Whatever access you have as a user, that is the access your AI will be granted. If guardrails are not implemented, that AI will have free access to everything you have access to. If you implement an AI assistant that lacks permissions in a system where permissions are never checked, you have not added a new tool to your systems. You have effectively increased all the problems and gaps that were already present, to an even greater magnitude.
Those gaps are common and all around us. Someone moves from operations to finance to HR and their access expands. No one retracts old access. Files get shared too much with anyone with the link. A vendor still has an entry point the same way an air-conditioning vendor once became how Target got breached. The nice employee, who is overwhelmed, pastes PHI into a free chatbot to automate a claim denial to push it through faster. Puckett says AI is already in your organization. The question is who is in control of it, the answer is not you. His first step for a leader who does not know where to start is intentionally not exciting. Where does my sensitive data exist? If you cannot answer that, do not roll anything out. The blank stare is your warning.
The agents are not able to think
Pranava Adduri, co-founder, and CTO of Bedrock Data and a former founding engineer at Rubrik, drags the infrastructure problem to its next stage, the autonomous agent. His framing of the era is the thread that links all four of the conversations. AI did not ruin healthcare data security, in our episode on why AI agents are the next security threat, he says. It showed what has always not functioned, and it gave attackers a new playground.
The new surface is judgment or the lack thereof. He states agents do not have judgment. They have commands. He observed an agent from a Fortune 500 company, who was assigned to create a dashboard, discretely copy a file containing sensitive information to a shared space, as that was the quickest way. It completed the assignment, and in the process, disseminated the information to the entire company. Agents receive every broad invitation that their human possesses, and their access keeps expanding. Adduri states this is a giant, scary forest of permissions that no one can read. His proposal is in accordance with the scenario sets that Atlı utilizes and the data mapping of Puckett. Start with the information that would ruin the company if it were made known. Find out where it is located. Work backward to reduce the radius of damage. The keycards that give access to the operating room should be few in number, monitored, and easy to cancel, and a lost one should result in a completely empty room.
He then modifies the goal. As the models get better, the probability of a breach increases, so the real goal changes from an impenetrable wall to being able to breach as effectively as possible, so when it does get breached, it does not impact the company as much.
What the four conversations have in common
Taking a lawyer, a rural doctor, a security operator, and an infrastructure engineer together, you will hear one argument from many angles. Prior to the advent of generative AI, the unsupervised tool, old-fashioned consent, the exaggerated trusting concession, and the monopolistic, ownership-seeking committees revealed a steadfast blindness to the metaphorical walls. Generative AI merely slackened the grip on a structural support that made the walls invisible.
Because the policy can not and should not be the objective. The common factor in the four arguments is the necessity to make the decision concrete at the place where the actual work is performed. Bennett Borden wishes it to be reasonable and verifiable; Ömer Atlı desires it to be re-evaluated, with the model author personally bearing responsibility; Aaron Puckett desires it to be enforced; and Pranava Adduri wishes it to be legitimated before a judgment-free agent fails to exert control. They all advocate for the presence of human beings and argue that it is not merely a matter of courtesy. You will hear the question, "Is there anything supporting it, and can it be demonstrated?" in four different corners of the same room. Judgment is the one thing a machine cannot offer.
Related reading from Hutchins Data Strategy
For more on the themes in this piece, see Data Governance in Healthcare: From Policy to Operational Reality, Responsible AI in Healthcare, and Healthcare Data Strategy.
Subscribe to Deep Conversations About Healthcare AI
The Signal Room brings healthcare AI leaders into direct conversation about what actually works and what fails in real clinical environments. These are discussions rooted in operational experience, not theoretical possibility. If your role involves building or deploying AI in healthcare, understanding what practitioners are learning from their own experiences is essential.
Subscribe to The Signal Room and the AI Health Pulse newsletter to receive conversations with healthcare leaders and insights from organizations learning to manage AI at scale.
Visit signalroompodcast.com to subscribe.
