Healthcare Leadership: Balancing AI, Human Judgment and Clinical Trust

Dr. Mark Gendreau: I'm like for my company. Humanizing AI for healthcare. Talked about how healthcare needs to be emotionally ready before it can be technologically ready. How people feel safe, being empowered that change happened. Hundreds of thousands of dollars on data, the power to technology. The goal is to increase access to healthcare. In this instance, it's that that sort of awareness, discernment, judgment, experience, right? As I always say, we have lived AI has not human AI co-pilot sort of partnership. And how do you engage in that partnership?

Chris Hutchins: Today in the Signal Room, we're talking about what it really takes to scale care responsibly, not by replacing people, but by amplifying them. My guest today is Dr. Mark Gendreau. He has spent his career at the intersection where urgency meets innovation. He's an emergency medicine physician and a senior health care executive who's led teams through the most complex challenges in modern care delivery. As a physician leader, he has championed data-driven decision making, operational readiness, and a culture of safety that keeps the human connection at the heart of every interaction. In this conversation, we'll explore how AI is beginning to collaborate with clinicians, where trust fits into that equation, and what it means to build systems to help people, not just process patients. Dr. Gendreau, you've spent your career on the front lines of emergency medicine and being a chief medical officer. When you hear the phrase scaling care with AI, what does that mean in a real hospital setting?

Dr. Mark Gendreau: Yeah, what it means is that the goal is to increase access to health care. You know, we are having greater needs in terms of the workforce and healthcare. We have aging, physician workforce, nursing, and we're having less and less people entering into health care. So we need to, you know, what to me, as you said at the beginning, it's more of amplifying the workforce that we have so that we can reach more patients and improve the quality and safety of health care.

Chris Hutchins: Yeah, there's just there's a whole scarcity, I think, is the word I would use for a lot of clinical roles, and especially in the nursing area. I can see why it's such a focus that you have to try to use these tools to optimize the the workflow so that people are freed up to do the things that they actually trained for, which is helping people. So I'm I'm curious because we've talked a little bit about this and recently, but I I really would love to hear and have you share with with some of the listeners where are you seeing some successes in how AI is being used collaboratively with clinicians and what makes those specific examples successful in your opinion?

Dr. Mark Gendreau: Yeah, so you know, uh three areas really immediately come to mind to me for in terms of you know really amplifying clinicians and transforming their their uh work life balance as well. So digital radiology, we have AI tools that basically read images, for example, uh CAT scans. Um, and um, you know, uh there's various modules uh that um basically read the images almost instantaneously. Um they don't interpret, but what they do is they heads up the radiologist as to there's something that you should pay attention to on image 63. And um some of these algorithms will actually give you a probability of, like, you know, if you're doing a CAT scan of the chest, um uh the probability that that uh could be a pulmonary embolus. This is incredibly useful. You know, if you're you know, particularly in the nighttime, or you know, there's less and less radiologists, you know, 12 hours a day they're looking at study after study, there's fatigue that builds in there. And with that fatigue can be diagnostic accuracy. So this really helps them really kind of focus, you know, where they should be looking. It doesn't interpret, you know, that's still in the hands of the human to basically discern and judge, you know, what what is going on and then do the interpretation. So I was at a conference um in the spring, and there was a trauma surgeon in Nevada, and they were they were having issues with maintaining their trauma designation with the American College of Surgeons because they were um having uh they weren't meeting the guidelines for having uh a CAT scan of the head read by a radiologist so that they could quickly act upon. And they used this AI product so that it would alert the trauma surgeon that, you know, there's an issue, take a look at, you know, image, you know, whatever, um, and um help them decrease their time um and get back to trauma designation. So very, very practical, very useful, you know, technology that is really assisting and amplifying the physician. Um I would say the second is in documentation. And so we use ambient um uh AI uh platforms to assist uh uh physicians in documentation. This has been transformative to the primary care doctors not only because um it it you know they that this you know it listens and then you know when they're done the note has been generated with about 97% accuracy as a draft. And this allows the physician to actually look at the patient and touch the patient and spend more time with the patient rather than having their head buried in the computer documenting everything that needs to be documented. And on the other side of this is we measure something called pajama time, which is we we can see when physicians are in um the EHR after hours. And we see people who are still trying to catch up 10, 11 o'clock at night, and um, you know, this is not sustainable. So, you know, that is a second example of the utility of it. We have another, we have an ED that is using an ambient system that also, when you're done with your note, it gives feedback to the clinician. You forgot to introduce yourself, the patient seemed to be a little bit tense when you were saying that, and your response, you know, you could have, you know, you could have approached it this way. So not only are you getting that documentation, that the physician or the advanced practice provider is getting feedback to be a better clinician on a more human level. You know, uh these technologies are really amplifying the human in healthcare and not automating them.

Chris Hutchins: That's fascinating. I actually was speaking with a company yesterday, scheduled for a half hour. I think I stole another half hour from them because I was excited watching it, having them walk me through what they were doing. But it was exactly what you're describing in terms of really undergirding and supporting the clinical interactions. So with every staff person that interacts with a patient, it's actually profiling the individuals that are interacting with the patient. It's it's profiling with the patient and it's it's detecting, much like you said, detecting if there's tension or stress involved, not only analyzing the words, but actually analyzing and detecting tone and the voice and the things that would indicate that there's stress or involved. Fascinating to me because I hadn't had not heard of anything that was geared towards wrapping around the encounter, regardless of who the individual was that was interacting with the patient. But they're having some really great success. It's actually being done by a physician who's kinda come out of um Stanford, uh out in California. Definitely share some of that information with you, you know, uh later later today, um, just so you can take a look at what they're doing. Uh it sounds like something that would be uh right up your alley as a as an advocate for you know helping to relieve that the burden that we've continued to layer on to clinicians. You've touched on some some great examples of where AI is being useful. Is there an example that stands out in your mind where there's a a specific decision where AI added value or where it didn't, and what that taught you about boundaries between the human and the machine?

Dr. Mark Gendreau: Yeah, you know, I think, you know, just to, you know, like where has AI sort of not been as stellar? You know, these we're essentially, you know, five to ten years into AI, right? And really into AI over the last, I would say, uh three, four years. And um, you know, we see improvement all the time. But take, for example, the digital radiology where these AI algorithms will basically alert you to something, um, they still have um issues with them. Like they will sometimes um um think something's going on, and it turns out that, you know, when the radiologist looks at it, it's like, no, that's a benign calcification. Um, you know, um, that isn't anything that we need to worry about. And and so there lies, you know, as I say, you know the human brain always keep the human in the loop with this, right? Healthcare um and other industries where there's you know high reliability, you have to have human in the loop every time. Um, and um, you know, in this instance, it's that that sort of awareness, discernment, judgment, experience, right? As I always say, we have lived, AI has not, and it's those reps that, you know, the clinician or the radiologist in this particular example, who's read thousands and thousands and thousands of images that those reps, you know, give that judgment and that insight that no, that's not anything that we need to be worried about, that doesn't need to be reported, you know, in that fashion becomes very important.

Chris Hutchins: Yeah, exactly. You know, I was thinking about a simple example where, you know, if you're my physician and I historically have a bit of a high blood pressure that's controlled with medication or whatever, but that any elevation detected by AI would be flagged. And so if no one knows that this is normal for me and it's it's I'm being treated, I mean there there can be decisions made that are necessary or alerting when it could happen. No alert fatigue is something that I've heard quite a lot about over the years. Um, working in IT departments in particular, I hear I've heard a lot of uh feedback that was not necessarily favorable about the alerts that we were triggering for like seems like everything under the sun. So we want to kind of dig into trust a little bit. I know we talk a lot about you know, do we trust AI? You know, we've we've discussed this in more recently. I've been hearing a lot more conversation about the trust in human relationships. With technology moving so fast, we we know that trust still is built slowly and deliberately. What helps clinicians trust AI insights enough to act on them?

Dr. Mark Gendreau: Great question. So, you know, you think about what is trust, right? And um Stephen Covey wrote a book on trust, actually, two I believe two books on uh trust. And he defined trust as trust is an equation that has credibility plus judgment plus safety. He called that intimacy, but he was meaning more like psychological safety. Right. And divided by self-orientation, or you know, you know uh the the you know, is what is this being used for? Is this being used for good or is this being used for bad? So kind of the ethics uh component um of it. And so I think that, you know, we can kind of, you know, for AI for um because healthcare is is, you know, a field of a lot of relationships, of trust, of quality and safety. And so, you know, it needs to have those kind of components of trust, of you know, uh the capabilities have to be good, the reliability of it needs to be solid. Um, you know, we're not gonna trust something that, you know, has a reliability, you know, like you know, half the time there's issues with its output or what it's actually doing. And, you know, it's gotta be, it's gotta contribute to safety. And so I think those are kind of, you know, the components of trust and AI adoption. I think when those are met, clinicians feel more comfortable interacting with AI. And, you know, we've seen this with our ambient uh uh document, you know, hearing documentation platforms, is initially the clinicians are a little bit reserved on this. They're, you know, their trust is very guarded. But as they become more comfortable with that interaction, as AI starts learning about their preferences and what they like, what they don't like. Um, and when they train it saying, you know, you said this, the patient actually said this, that is all that's the interaction that you need. Then the trust comes and people are more word gets out and people, you know, adopt it. You know, we we we had a lot of laggers uh with high you know, with our ambient uh plat uh AI platform. And um as soon as more and more of the early adopters, late adopters, you know, um uh came aboard, we we leveraged their them to influence the laggers to come on board. And some of those laggers become the biggest sort of champions of the technology.

Chris Hutchins: That's phenomenal. Yeah, I I d I've often seen things kind of go sideways when the these type of efforts are viewed as an IT project. We've you know, we'll build you know, I I don't mean this that everyone does this, but I know historically there's been a lot of times where solutions were developed and and just imposed. And you know, they you know, people that I've spoken to in your profession have looked at me and given me an eye roll. You did it to me again. Why are you wasting my time? This does not solve a problem I actually have. But there are some you could solve if you'd like to listen. I love how you're approaching it. I'm sure that your colleagues are grateful that that you're kind of on the tip of the spear in and leading through this because it is a bit of a frightening exercise when you've got all these things coming at you, this pressure to use it and you know this this explosive growth and volumes of data that you have to process quickly to diagnose something or to decide how to treat it. We we've only been layering and layering constantly, whether it's for the purpose of improving quality, we have we add data elements that have to be captured, and then all of a sudden there's penalties being imposed based on uh an average rather than an individual, which I don't think is an interesting problem that we're gonna have to solve for in regulatory areas, because you know, as you know, if you take the averages that we're we're measuring against, you're never gonna find an individual anywhere on the planet that matches that profile. But yeah, we decide to penalize or or incentivize based on that. The practice of medicine is an involving science, of course. So I don't know how we are gonna get there, but I think we really have to lean in on the regulatory side and make sure they understand the regulatory nature of things historically has not been based on reality. It's been based on a static snapshot in time, essentially, which is not gonna work with AI. What you're doing in your building, yeah, I think is uh it's it's really a culture that you're developing there. How do hospitals, how should hospitals be thinking about creating that culture where the data and human judgment reinforce each other instead of competing? Because I know there's in some cases a tension between those things.

Dr. Mark Gendreau: Yeah, I I I think that you know it needs to come from leadership, right? Leadership plays a big role in this, of uh of um showing people that you know, getting them involved in this, giving them agents, really allowing people to experiment with it and be creative with it. Um, you know, all that is part of change management.

Chris Hutchins: So when you're d working on the these types of things, there's the there's one thing when you're just trying to kick the tires and trying to get people used to it. But at some point you got to start figuring out how do you how do you scale it and what are the indications that you're gonna be looking at to determine when it's actually time to scale. So from an operational standpoint, what are some of the challenges to scaling these AI systems across the health system that you've encountered?

Dr. Mark Gendreau: I would say the biggest obstacle is um threefold. I think it's interoperability is probably the biggest one. Um and then governance is is uh the second. And then basically, what what's a good term for it how are you how your structure with respect to the m you know in in terms of training the models? You know, like a a lot of health systems now are partnering with, you know, one of the big AI LLM uh, you know, companies and using their model, parking it behind a firewall and then training it with internal things, right? Right. So that the so that the model starts to kind of get an understanding. They it knows who we are, it knows what we do and things of that nature. We're actually being b very busy in our health system uh with um um and making sure that it's ready for prime time uh for the m the masses in the health system. Right. And you know, we've been busy with setting up uh governance to you know, what do we push out, when do you know, to who, and how do we do it and uh aspects of of that. And then the interoperability is you know, is um a big, big thing, right? Because it involves getting uh the right APIs, making sure that everything is talking to each other, because if you don't do that, the data um is just not useful to you. Those are I think the key components of of of scaling up AI um health system. This was all pointed out, by the way, in um 2022 um when the um National Academy of Medicine put out a uh a uh document on uh artificial intelligence um, you know, with respect to health care. That's a worthwhile read um if you haven't read it.

Chris Hutchins: I appreciate that. There's this human factor that uh oftentimes just d it doesn't get the the time and attention. That it needs. And one of the most important things that I've realized that I need to help message is start with listening and having conversations with someone like yourself. This is a really important uh thing if you're going to try to develop any kind of solution to make things better. You're talking about scaling. What are some of the signals that you're looking for to know when your organization's ready? This is not just a technical readiness, but there's the cultural aspect of it as well.

Dr. Mark Gendreau: I would say um when kind of like what uh you know what I alluded to is that the you've got leadership who's involved, right? Like the beauty of AI is that you don't need to be tech savvy. Um you need tech savvy people, but for the most part, I mean, I'm not tech, right? But that's the beauty of AI is that AI really doesn't require uh that the lot m most people who do very well in AI seem to be not tech savvy, but people who are creative are good problem solvers and then have those core elements of like human-centric leadership that they've got strong empathy, they're ethical, they they're you know, tr uh trust is a building trust is a non-negotiable sort of thing with them. And I think that you know, when you've got leadership who isn't trying to basically push it to IT to roll out, but it's the upper leadership who is is and is basically, you know, like we need to do this, and here is the why and here is the how, then I think you know that culture changes. More and more people start using AI, you know, for like generative AI, um, and um it just becomes a catalyst for readiness. And I think that there's you know, there's multiple layers on how do you know when your organization is ready. I think it's not just one thing, it's multiple things.

Chris Hutchins: Yeah, absolutely. It's it's a lot more complex than um we would like it to be for sure. You touched on something else with the there's this this ethical considerations, and one of the things that's been challenging my thinking, and I I'm not quite sure if this really requires guardrails, it may actually require some that we can we have to figure out. But how empathetic do we really want the technology to be? Because there's a risk now that people trust too quickly and too easily, and that that's a concern. Where where do you think some of these lines might be to make sure that we're keeping medicine focused on on human beings and trusting human beings and we're making sure that empathy doesn't get lost in the in our rush to to get efficiencies?

Dr. Mark Gendreau: Yeah, you know, I mean AI is great at a lot of different things, pattern recognition, you know, some, you know, I'm all for automating repetitive, you know, um things that um are fairly routine in terms of patient care. But I think that if if if you're doing something that requires emotional intelligence, you know, something that is going to require uh emotional intelligence, relationship building, or shared decision, that ain't AI's territory, that's the human, right? And one of my uh one of my favorite um favorite quotes from Jeff Woods um is you know, you are the leader and don't ever abdicate your leadership to AI. And so I would flip that to you are the human, never abdicate your role, your humanness to AI. I think that I was listening to a podcast recently, um, and they were discussing, they were discussing like how as you know, as we become more and more into AI and it becomes more agentic and things of that nature, we're gonna have an AI, uh we're gonna have two workforces. We're gonna have an AI workforce, we're gonna have a human workforce. The important thing that we have to do is that we need to make sure that we keep those kind of separate. Like we don't try to humanize the AI and start calling it names and everything, because the technology is gonna there's some technology out there that you can't tell the difference if you're speaking to a human or a robot. Right. Um and um uh and so we we have to make sure that we don't get down that track too far because um we giving away um uh our human traits is just not gonna be a good thing.

Chris Hutchins: No, it's it's clearly a risk. And now it I think the dangers that aren't there because there's also this trust factor. Um we can't we can't be sure that we're going to get people to really follow the guidelines on what you should and shouldn't use these technologies for. Increasingly people are connecting, listening to, being influenced by people who look, think, sound, and behave like they do, and they're not necessarily listening. Oftentimes that that can be a real miss if you're trying to run an organization as a CEO and your your people aren't trusting you, that you're you're you're going to do things in a in a way that they don't have to be terrified of losing their jobs. That there's got to be an awareness that you've got to find ways to message so that you know that people are really understanding and they can actually make the decisions to follow it. From a leadership standpoint, I think there's a there's a really significant need to understand better uh what the experiences are for clinicians. We haven't gotten too far into the to the burnout uh kind of conversation, but we know that there's a fair amount of that because you know you mentioned documentation being done up till 10, 11 o'clock at night. How should leaders be thinking about protecting their clinicians from things like the alert fatigue or overautomating, but still we want to make sure that we're moving innovation forward.

Dr. Mark Gendreau: Yeah, you know, I mean, you know, alert fatigue is big, particularly in the nursing side of things, right? Because they're up there um on the floors continuously, and there's alarms going off everywhere. And I would say that alert uh an alert needs to earn the right to interrupt a clinician, right? I think is a good way to think about it moving forward. And so, and then another principle is if you're gonna add something, you need to take something away, right? In terms of and so we have we have so many technology, you know, we have so much technology uh with alarms and everything that um one of the big things that um you know we have become increasingly more concerned about is um it almost becomes just background noise that um the people don't even uh hear it. And so um that's why, you know, it's on uh it's on humans and leadership in healthcare to put together alert fatigue committees and actually review what is what sends alerts and what's the what's the value that it's providing and what's the impact that it's doing and what you know uh if it's not adding value and it's having a negative impact, we need to basically change up that technology.

Chris Hutchins: That that fits the whole purpose of the podcast that I've created here. There's a ton of noise, and it you know, if everything is triggering an alert, there's really no point anymore because that's all you're gonna do is respond to alerts. And then that that's just not a productive way to use the technology. I think it's gonna be a very interesting challenge because there just seems to be so many great ideas coming forward, and I'm sure that you're getting uh full in uh inbox with your emails from people who have the best things in sliced bread, but it's only gonna solve a fraction of a problem that you really want to solve. And that's just not sustainable. But there's so many of them coming at us all the time. It's really hard to detect sometimes where are where is something, where's the one that's got some real value and can have some real impact.

Dr. Mark Gendreau: There's a lot of noise out there. There is. And that signal-to-noise ratio thinks you got we got to somehow make that better.

Chris Hutchins: So if we look forward five years from now, speaking of signals, what do you think will be the single most important one that you'll be watching that tells you that this AI and human collaboration are actually improving care?

Dr. Mark Gendreau: I would say outcomes, um, you know, that we're we're seeing better health outcomes. We're seeing, you know, the dispare the health disparities gaps um uh shrink or go away totally. Um and pajama time goes down to zero, I think are those are the signals that would tell me that um we've succeeded.

Chris Hutchins: Well, you you just used a term that I'm sure I'm not the only one that hasn't heard this, but please help me understand what what the pajama time is really all about. That's probably doctor code, I imagine.

Dr. Mark Gendreau: Yeah, that's that's you know, so we when when um when a clinician is documenting, you know, 10, 11, 12 o'clock at you know, at 12 o'clock in the morning um to catch up on documentation, that's what we call pajama time.

Chris Hutchins: I might I might have to borrow that, but I'll give you credit. Dr. Gendreau, if if folks are are uh interested in reaching out to you to obviously to to understand what it is that you're doing that you're seeing be bring successful adoption, how do they reach you?

Dr. Mark Gendreau: Um they can best reach me on LinkedIn.

Chris Hutchins: Fantastic. And and for for those that are listening, I'll make sure that you have all the information you need in the show notes to reach out to Dr. Gendreau. It's been a pleasure to have you on today. Fascinating conversation. I'm sure we'll be having more of them in the future. But thank you so much for for taking the time. It means a great deal to me personally, so thank you for being here, Dr. Gendreau.

Dr. Mark Gendreau: You bet, Chris. Thank you.

Chris Hutchins: Well, that's gonna do it for this episode of the Signal Room. If today's conversation sparks something in you, an idea, a challenge, or a question, don't keep it to yourself. Join the conversation on LinkedIn or visit us at signalroom podcast dot com. We're here to amplify the signals that matter across leadership, ethics, and innovation in healthcare.