AI Clinical Trial Screening

Welcome back to The Signal Room. I'm Chris Hutchins, your host. Very excited for this morning's guest. Most of the time, AI in healthcare is being sold on a demo and a promise. But my guest today did something almost nobody does: he put his own AI tool through the same evidence bar that we hold medicine to, and published the results in the New England Journal of Medicine and the Journal of the American Medical Association, JAMA. He's a practicing cardiologist and intensivist at Brigham and Women's Hospital up in Boston. AJ, welcome to The Signal Room.

Thanks so much for having me, Chris.

I've been looking forward to this conversation. I've always had a lot of passion for research, and as I mentioned previously, I spent a number of years up there in the Mass General Brigham organization. To level-set for folks so they understand where you're coming from, I'd love to hear a little bit about your background: what led you to practicing medicine, the progression that got you really excited about using AI and working in the research space.

Thanks so much, Chris, for the opportunity. I'm really fortunate to run a research group here at Brigham and Women's Hospital today, and that came from many years of training and learning. I started as an undergraduate at Johns Hopkins and worked at the medical school at Johns Hopkins Hospital, doing really basic science at the time, learning about transcription factor regulation as part of my undergraduate studies in neuroscience. I was then recruited to the National Institutes of Health down in Washington, DC. I spent a year there, again furthering those very basic science academic pursuits, publishing in genes and gene transcription. I then had the opportunity to join a brand new medical school, the Hofstra Northwell School of Medicine in New York. So I decided to step out of the laboratory and to the bedside, which was a really great experience for me and helped me fall in love with patient care. I did my undergraduate medical studies there at Hofstra Northwell.

Then I went down to Duke to do my internal medicine training at Duke University and Duke University Hospital, working with the Duke Clinical Research Institute. I was really fortunate to work with some incredible sponsors and mentors there. The Duke Clinical Research Institute really helped me understand the translation of what we were doing at the NIH at the bench, with pipettes and mouse models, into translational medicine: how do we actually take these innovations and get them to patients? And I was flabbergasted to understand that it oftentimes takes not months, not years, but decades to get those new discoveries and innovations that we were coming up with in the lab to actually affect a patient. There are so many reasons for that, not the least of which is that running clinical trials is extremely time consuming and expensive.

So I really started to learn that process of clinical trials and how essential it is to show, as we come up with these new innovations, that they're safe and effective. And the way to do that is in prospective, blinded, randomized controlled trials. We can oftentimes be surprised by the results. Even when you get to a stage three clinical trial, recent data, even in the cardiology space where I work as a cardiologist, has really challenged the notions we came into these phase one, phase two, and phase three trials with, to show that these drugs really should be made available to patients with good safety and efficacy profiles. But that's a major, major time and effort sink. So as I joined at Brigham and Women's, I did my fellowship focusing on a lot of these tools: artificial intelligence, machine learning, as well as epidemiology and my master's at the School of Public Health here at Harvard. I was just fascinated by this process and how challenging it is to go from an idea to a safe, effective medicine being delivered to a patient at the bedside.

Right. It's not something everyone tackles. Most of the people I've talked to over the years who do research do a fair amount of publishing, and of course the discovery part is really exciting. But as a practicing physician, what made it such an attractive thing for you to start working on fixing this slow and inconsistent screening for clinical trials? That's not a small task you're tackling.

Yeah, it's not always the most glamorous or sexy aspect. My research group at Brigham and Women's works in what we call implementation science. I was really fortunate to train under one of my mentors, Ben Scirica, and the TIMI Study Group here at Mass General Brigham, which does these big trials in partnership with many of the large pharmaceutical and medical device companies to show that these medicines are safe and effective. One of the things we were surprised by when we were looking at our own internal quality metrics at Mass General Brigham was this evidence gap. Even when the major societies publish guidelines on what medicines should be used, what targets should be aimed for in terms of achieving blood pressure or cholesterol control, or a new medicine comes out and here's the patient population that's going to benefit, we were really surprised that even in those situations, where we were designing and running many of these trials as the leaders at Mass General Brigham, our patients still had gaps in terms of their optimal care delivery and care quality.

That was really surprising to me, so I sought to learn why. What's the challenge? We know this medicine is safe, we know it's effective, we know these patients should be on it, and even here, where we're running the studies and writing the guidelines, we still have these opportunities for improvement. So this field of implementation science asks: how do you overcome the challenges from the patient, the provider, and the system side that make it more difficult to actually get that evidence-based medicine delivered to patients? What we really found is that there's not one single problem. For every snowflake in an avalanche pleads innocent; it's really this system-wide inertia, from the provider, the patient, and the system standpoint, that makes it challenging to get the right medicine to the right patient at the right time.

So what we focused on is a remote, team-based, asynchronous approach. I had the fortune of doing this as a research leader, to understand how we can make this easier, both for patients and providers, to deliver and receive healthcare in a more efficient way. If you take a step back and think about first principles: if I'm a patient, how do I see my doctor at Mass General Brigham? I drive 30 minutes, I wait 15 minutes to park, I pay for my parking, I walk into the building, I get directed to the right clinic or lab, I wait another 15 or 20 minutes while I'm filling out a questionnaire or two, and then eventually get the opportunity to sit with my care provider for somewhere around five to 15 minutes. At which point I follow that entire sequence in reverse: back to the waiting room, fill out some more forms, maybe get some lab work done, go back to my car, leave the parking lot, drive another 30 minutes home. After all that time and effort, maybe a half day, I had five or 10 minutes with my clinician to really understand what I need to do, what's important to me, and how I need to change my health or behavior to optimize the quantity and quality of my life.

In today's day and age, there are a lot of opportunities to make that experience better, both for patients and for providers. So we took a team-based, remote, asynchronous approach to everything across the cardiometabolic spectrum: overweight and obesity, hypertension, hyperlipidemia, type two diabetes, heart failure. We made this a process where every patient who qualifies has the opportunity to do this with a physician always in charge, but with a team-based approach so that they can have a high-touch model that didn't break the financial system. We were really fortunate to publish a lot of that work in Circulation and JAMA Cardiology, these nerdy academic journals, where we show this is safe and effective, and that getting patients on these medicines to optimize their health doesn't have to require as much of the four walls, as much of the processes and the many steps that we sometimes put in place but that end up getting in the way of delivering really high quality care.

That was a really exciting process, and we got to study it as a field of science called implementation science: publishing that it's safe, it's effective, it helps us deliver better care, patients like it, providers like it. What we found was that the biggest challenge, even in getting these studies done, let alone doing this at a population health level to optimize care for many, many patients, was finding those right patients. You really had to take somebody who knew the clinical medicine, who knew how to read a chart and say, okay, here's our criteria for this study or trial, or for people who even qualify for this medicine. And Chris, as you know well, the electronic health record is unfortunately not always the easiest set of technology to work with. There's structured data, which is a component and is important, but unstructured data oftentimes really hides the diamonds.

That's where you really need to understand: what has this patient tried before? What is the reason they qualify for X, Y, or Z medication or procedure or diagnostic? What's been discussed with them in the past? And how do you make this a seamless experience where you're not inundating the system with best-practice alerts or other ways of nudging patients or providers? The technology has advanced to a state where I can have a co-pilot or an assistant go through the chart, find the relevant information, and surface that to study staff. What we showed in the New England Journal is that we can safely, accurately, and cost-effectively do this with the assistance of an AI tool to make it a much more efficient process. And the JAMA randomized controlled trial we did showed that, using this process, we can actually get patients enrolled in clinical trials faster. At the end of the day, by enrolling in those trials faster, we can show that these medicines are safe and effective sooner, get them into the hands of patients with the collaboration of their providers, and really accelerate the scientific process. That's what we're really excited about at AIwithCare.

This is a really difficult space sometimes. Let me pause on this for a second. Talk to me a little bit about some of the dynamics. You mentioned the patient maybe has just a few minutes with their clinician, they've got to drive home, and all that. There are a couple of areas that are really significant in terms of helping people understand what it was you had to deal with to do this work. Most organizations going through any kind of acquisition know there are a lot of problems and challenges when you're consolidating electronic health records. But when you're trying to optimize a workflow like this, one of the barriers has to be drive time and the inconvenience for a patient to get in and out of a city. Not everyone has that dynamic, but that's another layer on top of what anyone else would have to deal with, right?

Yeah, I couldn't agree more that the workflow is essential, and that's from the clinical team, the research team, and the patient. Anything we can do to make this less onerous. We all know this all too well; there's way too much friction in the system. It's not easy to be a patient, and from some of the burnout metrics, it's also not easy to be a provider in today's world. So rather than remain the status quo and rage against the machine, we decided we were going to try to create a better mousetrap, and see if we could take something that works in the workflows we've already established, that people are used to, but use some of these tools and technology to transform it in a way that accelerates efficiency and makes it a lot faster and easier, both for patients and for providers.

I think it's a great illustration of what it really takes to move things forward in a way that's actually going to be effective. Clearly you didn't just go off and create something in a silo, toss it over the fence, and tell everybody to use it. You really had to have the right experts in the room to work through all the nuances and account for the things that are really critical, not the least of which is how far a patient is going to have to travel to get there. How do we make it interesting to them? If they live two hours away, they may not be inclined to join the clinical trial if you don't have things in place that are attractive to them. Were there other considerations for patients that you've experienced?

Yeah. When you think about a clinical trial, the most important thing is: is this right for me, and does this really apply to me? What's happened over the past few decades is that clinical trial protocols have gotten increasingly complex. 20 or 30 years ago, maybe there were five or 10 inclusion and exclusion criteria to make sure a new drug or device or innovation is tested in the right population to show it has the best chance of working and being effective. In the past 10 or 15 years, as medicine has only become more complex, these trial protocols have also gotten more complex. So what was five or 10 inclusion and exclusion criteria is oftentimes 40 or 50. When we reach out to a patient who might be eligible, we have to go through each one of those questions one at a time, only to find at criterion number 47 that, darn, you actually don't qualify.

One of the major things patients don't like is to go through this whole process, be offered this great opportunity, and only find that they're not really a candidate at the end of the day. So we try to make sure that every time we reach out to patients and interview them, they're the right patient for those trials. That's what this tool can really solve for: quickly, accurately, and cost-effectively taking that protocol criteria, matching it to their clinical data, both structured and unstructured elements, to make sure they are a perfect fit for the trial. Now, there are a lot of other elements, you're right. How far do they have to drive? How many study visits are there? What's involved in participating: am I getting blood draws, am I getting transfusions, is this a pill I'm taking? What condition the trial is for very much makes a difference in terms of their appetite for participating. If this is stage four cancer, and this is a drug that might extend the quality or quantity of your life, people are more apt to give that a whirl if every other therapy has not been effective.

If this is something that might lower your cholesterol, which might eventually lead to a lower risk of heart attack or stroke, that's important too, as your local friendly cardiologist, very important, especially at a population health level. But we have not done as good a job in the cardiology community talking about the fight against heart disease as the oncology community has about the fight against cancer. When your mortality is staring you in the eyes, there's a much higher uptake rate in the oncology clinical trials than we see in some of the preventative medicine trials we do in cardiology. So there are many different elements that go into that decision, for a patient or provider, to make the call that a clinical trial is the right step. But in terms of advancing science, advancing medicine, and really understanding in the best scientific way how we can make healthier, happier, longer lives for people, the evidence-based medicine process is essential. Any way we can take tools and technologies to accelerate that is only going to lead to a higher quality and quantity of human health. That's what we're aiming at in expanding access to this tool and technology.

Excellent. It strikes me just how complicated things are. I don't know that in healthcare, as a rule, we do a great job helping people understand the complexities of what we're doing. What you're describing is a workflow inside of a particular specialty, inside of an organization, inside of a practice; there are so many other nuances to it, and we're just talking about a sliver of it. I think it's worth noting for people just how complicated this is, and how serious you'd better be about doing research, because it's a lot of work if you want to get it right. Talk a little bit about the tools you're using. What does RECTIFIER do? Walk me through what happens when it reads a clinical note to judge trial eligibility. Where is it confident, and where does it still have to hand it back to a human? And is there a point in that process where you automatically build in a pause to make sure you're getting human judgment engaged?

Really important question. To step back and tell the story: we built this tool for my clinical research group. I'm a principal investigator of clinical trials, and I was looking at our internal metrics on how much time and effort we were spending looking at charts of patients who ended up being ineligible, in addition to talking with my study staff, who were spending hours and hours a week sifting through these charts one at a time. The unfortunate reality is that 90% of the charts we open are for patients who end up not being eligible for the trial, for one of those inclusion or exclusion criteria or another. And we only really know that when we have highly trained clinical study staff go through the process of assessing eligibility and reading the chart. That takes time and expertise, and it was the biggest bucket of costs I had as a researcher and principal investigator on the trials I ran.

In addition to their feedback, "Hey, Dr. Blood, this is really not fun, spending hours and hours only to find out after all that work that nine out of 10 of these patients, I just wasted my time, and I can't even reach out to them to solicit their interest because they're not eligible," I said, there's got to be a better way. I was very fortunate to have the sponsorship and mentorship of some of my colleagues at the TIMI Study Group, who said, let's see if we can do this better. And we built this tool called RECTIFIER. What RECTIFIER does is look at both the structured and unstructured data in the electronic health record, and it assesses, based on those inclusion and exclusion protocol criteria, not only whether this patient is eligible or not eligible, but why. Show me the evidence, and cite your source.

So what it can actually do is tag back in the electronic health record why a decision on eligibility or ineligibility was made by the AI model. And importantly, for each and every one of those decisions, these are all human-in-the-loop. There is never an autonomous AI process where we're letting the model run wild to make decisions, medical decisions, or even give medical advice in any way, shape, or form. This is surfacing and sourcing relevant information to a protocol question about a clinical trial, and allowing the clinical study staff to confirm or deny. Just making that process much more efficient has shown that we can save enormous amounts of time and effort going through the ineligible charts, but also surfacing the eligible charts and confirming eligibility much more quickly and accurately. That's where we've saved an enormous amount of time and effort on the clinical research front.

This was built initially in my research group at Brigham and Women's, but it scaled across Mass General Brigham, across 30 or 40 different use cases within the organization. Clinical research is a really great one; it's shown an enormous amount of promise and some really great results, even in the randomized clinical trials, where we can accelerate the scientific process using this tool. What we then did was take that and license it out from Mass General Brigham into a new company, where I'm happy to serve as the co-founder and CEO. That company is AIwithCare, and we're now working with many different hospitals, healthcare systems, research organizations, pharmaceutical companies, CROs, and sponsors to help accelerate the pace of scientific progress.

Really excitingly, as a clinician, you never know what's going to happen when you bring a new tool to a bunch of really smart, innovative people. Within Mass General Brigham, we've seen them start to use RECTIFIER, the tool that powers this technology, for many other use cases. We've seen clinical population health, safety and quality metrics and reporting really start to be enabled, and registry work that's been published, where we've been able to accelerate the pace of registry creation and upkeep with some of these AI tools. And really excitingly, beyond other research groups in different therapeutic areas outside my realm of expertise as a cardiologist, things like ALS. There's fantastic work by a colleague in neurology at Mass General Hospital, screening every single patient she sees in clinic for every trial she's running, to make sure they get that opportunity when they see her, if they qualify, to participate in a trial. In ALS, time is really, really essential, so making sure you have that right, in-time opportunity to get people engaged and participating in clinical trials is really exciting.

Another colleague, a pediatric gastroenterologist, is actually using this for clinical operations, and Mass General Brigham put out a press release about this. It's a really exciting use case, where he's taking all the referrals he's getting from the community and running them through a customized version of RECTIFIER so that he can assess for any concerning findings or red flags that would warrant that patient being seen earlier. And again, these are kids. Being able to use that tool to flag these charts for his oversight and confirmation, that yes, there is something in this referral that we need to see sooner, and then making that offer to patients of getting in sooner, getting a diagnostic procedure done on the earlier side, actually makes a really big difference. Time in these disease states, especially in kids, is essential: getting their disease under control, disease-modifying therapies, this is inflammatory bowel disease and others, and really nipping it in the bud before it blossoms into a much bigger problem. It's been really fascinating to see this tool used in clinical operations to help diagnose and treat disease earlier. It's been an honor to be a part of this process, and really exciting to see it used in so many different facets of our clinical operations and clinical research here at Mass General.

That's the stuff that gets me excited, because you're doing the work, but you're starting to see some measurable results. Let's talk a little bit about that too. What is the bar you have to clear to satisfy the research community that yes, this is actually the right way to measure it, we're seeing results we can trust? For someone who doesn't really understand it, there's a specific kind of training you have to deliver to help researchers understand where that bar is and where it's not, what the nuances are, and how to interpret it. But you're getting great results. I was reading through some of your content earlier, extremely accurate, much more so than what you were seeing before you were using the AI. In saying that, I don't want people to think it was an easy button. Give us a little bit of context and talk about how meaningful the results are.

Yeah, thanks so much. We were fortunate that the research studies we published in the New England Journal showed that our AI-powered tool was actually more accurate. We designed the study to show it was non-inferior, that it wasn't worse than our study staff, because our study staff are fantastic, they're essential. But like many in clinical practice and clinical medicine, there's always more work to do. There are many patients out there, many patients waiting. So we really wanted to give them a boost of time and efficiency, and this AI tool has been able to do that, again, by surfacing the relevant information, finding the eligible folks first, and making sure they're much more efficient in their patient identification and screening.

All of these processes are human-in-the-loop, decisions driven by humans, by clinicians, anytime there's an eligibility determination, but making that process more efficient is where there's a gap. The pediatric GI colleague is an example: in his clinic, he used to have a nurse practitioner go through all these referrals, sometimes a hundred or more a week, sifting through and quickly trying to understand, is there an emergency here, is this somebody who really needs to be seen sooner, assessed sooner? What we saw in some research he's done is that the referral from the pediatrician, whether it was dictated as emergent, urgent, or routine, actually made no difference in terms of the actual urgency of the procedure the child needed. Using the AI tool, that was not true: the AI tool did make a difference. It was significantly more efficient at identifying red-flag signs or symptoms from the patient, the patient's chart, and the referral.

Now, where the real leverage came, and this is how we're designing all of our workflow and the implementation of this technology, is the flag identified by the AI and confirmed by a clinician. Allowing that human in the loop to be an amplifier and the validator that says, yes, the AI flag says there's something there, and that's confirmed by a clinician. Then what they did was offer these earlier appointments to patients who were flagged as potentially high risk. That reduced their no-show rate, decreased the time to the diagnostic procedure and therapy, and got these kids seen sooner, less likely to no-show their appointments, and on treatment sooner, really getting their disease under control sooner, which was really exciting.

For all these things, our philosophy, both in my academic work as a researcher and clinical implementation scientist, and as CEO of AIwithCare, the company that's now taking this tool and technology to many other hospitals and health systems, is to make sure we're building systems and workflows that keep that human clinician with their hand on the wheel, with the decision in their hands, but empower them with this data so they can do that much more efficiently, and so that the ways they spend their time are actually in the gratifying aspects of the job. As a clinician, as somebody who works in clinical research, these people go into it because they want to make a difference. They want to have impact, they want to improve scientific discovery, they want to improve the quality and quantity of patients' lives.

Where we have the tangible feedback in doing that is when we have the opportunity to engage with patients, to shepherd them through a clinical trial. So it's really exciting to be able to insert technology at a point in the process that is a major bottleneck, a major inefficiency, and where AI is really well suited to deliver massive returns on that investment. That's what we're seeing in our partners, both here at Mass General Brigham and through the lens of AIwithCare with many other hospitals, health systems, and research groups.

From a trial standpoint, the results were essentially that you doubled the enrollment across the roughly 4,500 patients you were looking at to determine eligibility. From a scale standpoint, what is typical in terms of how big that cohort has to be for you to get enough statistical significance that you can actually validate that what you've come up with is the right approach?

Great question. We were fortunate to have this study powered; our p-value is 0.04. You very kindly noted it was an 84% increase in enrollment, so nearly a doubling of enrollments, prospectively, across the trial we published in JAMA, the Journal of the American Medical Association. When you look at metrics like that, it's really transformative. When you look at clinical trial enrollment, to see any site or principal investigator enroll two or three or four patients, it depends on the trial, but in general, that's pretty good performance. You can imagine cancer trials, where the numbers are small and the timelines are smaller, are very different from a primary prevention trial in hypertension or cholesterol management, which can have many, many, sometimes tens of thousands of patients, and last much longer, years, or even three to five years sometimes.

The dynamics of the trial can be very different based on the question you're asking and the statistical significance you're trying to show. In terms of trial enrollment acceleration, that's actually a much simpler question to answer, because these timelines oftentimes stretch out months to years. We were fortunate to have statistical significance in our published results, but being able to show a 20 or 30% increase in enrollments across any one individual site, let alone across many sites across the trial, that's where you actually get to shrink those timelines. And again, when they're essential, you have to do these trials right to really prove that these medicines or innovations are safe and effective. But being able to get through that trial faster, by enrolling people faster, getting them on the therapy faster, demonstrating that safety and efficacy sooner, that's where you can actually compress that trial timeline and get those innovations, those medicines, those devices to patients sooner. That's really what we're aiming for.

Those trials can oftentimes last two to four years before completion. So if you can imagine enrolling those people faster, you can actually compress that timeline, on that order of magnitude. That not only takes an enormous amount of time and effort away from the study operations, safety, quality, keeping the study coordinators, the PIs, and the patients engaged and part of the process, and getting the data flowing so you have what you need to file that regulatory filing for eventual approval, but it's also an extremely expensive process. Something out of Deloitte a few years ago put it at about $2.2 billion per clinical trial for the top 20 biopharmaceutical companies who run clinical trials. Extraordinarily expensive, and a huge amount of that cost goes into clinical study operations and clinical research staff, but also the time that's wasted in terms of keeping that medicine under patent protection and making sure there's an ROI for the company that's spent all this time and effort developing this new device, drug, or therapeutic. That's where we see an enormous amount of efficiency opportunity, on behalf of our pharma partners, but also the health systems that run these trials, and then eventually in the service of patients.

When we think about all the lead-up to that, then you go live. Once it's running across multiple sites, how do you catch drift or bias before it has a detrimental impact at a scale no one wants? I think about the traditional challenge in terms of just tracking the normal refinements that would happen over the course of working through a study, but now we're dealing with an acceleration, and different kinds of models making things move much faster. The other piece is that sometimes the statistical significance comes from having access to a much larger corpus of patients. In the case of the research work you're doing with multiple organizations, the real benefit can be truly looking at some things that are difficult to spot in some instances. Bias that's worrisome, from my standpoint, is when there's just not data there, for whatever reason, and having it build and train without the information. Maybe we can talk a little bit about that, and how you see it, and what you're doing to make sure that as this is deployed in different clinical trials, you've got it wired properly and these things are able to be spotted as soon as possible and acted on.

Yeah, Chris, this is essential, and you're right to call it out. We address it head-on and upfront. Both in the New England Journal article, a peer-reviewed publication, we looked at potential bias in incorporating this tool, as well as in the prospective blinded RCT we published in JAMA. Coming at it with intentionality is a first step, and making sure you're aware that you need to be watching for this. You need to have systems in place so you don't look back a month or a year later and say, whoops, we should have. So being thoughtful, intentional, and strategic, and planning for that, is really the most important first step.

When we published our New England Journal article and our JAMA article, both of those studies showed no bias across age, sex, race, ethnicity, or primary language. We wanted to make sure we built that monitoring capability into the platform, into the functionality of the tool and technology. That's something we have both to ask all those questions, not only as a whole system, but across question by question, and not only in the screening aspect, which is really important, but also when you get to enrolling those patients. Our platform allows not only for the pre-screening, but also, as you flow those patients into the clinical trial, as you reach out, engage, and enroll them, actually monitoring that live, in real time, not only across one site but many sites. That's really important for the trial and the trial demographics.

Really understanding that this medicine is not only going to be safe and effective in one very narrow slice of the population, but much more broadly valuable, you have to make sure these trials are representative to ensure they're safe and effective in all populations that might benefit, not just the one that happens to enroll. So I completely agree with you: building structural ways to make that easy, and make it part of the process for the local study teams as part of the platform, is part of how we've made it part of the workflow. It's ingrained from day one, so you're not way behind the ball looking back and saying, whoops, I should have, but instead saying, great, we're on track, we're on metric, or, we need to tweak, let's dive in here and make sure we're activating or reaching out to patients of a population who haven't decided to participate as much. Being able to be aware of that is the first step, and making it easy for those study teams and patients to participate is a really important part of the scientific process.

The whole idea of dealing with this kind of scale is interesting. I hear a lot about pilots these days, and I'd say the majority at the moment seem really not so happy with the whole idea of pilots, for a variety of reasons, usually the lack of an operational game plan. You talked about baking things into the operations. It seems to me there are probably some things, maybe even a blueprint, that live inside the clinical trial processes that have been developed over decades that could help organizations think differently about what they're implementing. This is not just a clinical trial for one particular condition; there are a number of different aspects of what you're doing that benefit other parts of the care delivery that goes on in an organization. I'm definitely interested in your perspective, because I think there are probably some things to learn just from the workflows you have in place. When it's a clinical trial, it may work, it may not work, but you have the gears and the foundational elements undergirding it right out of the gate; otherwise you can't trust the data you get out of it. Is that fair?

I completely agree. There's a little bit of a parallel you're hearing in the tech and AI world right now: it's not the model, it's the harness. Similarly in our world, it's not the platform, it's the workflow. You can have umpteen pilots and ways of testing, and there's a lot of pilot fatigue, a lot of death by pilot that does happen in the healthcare system, especially in the academic medical systems I've had the fortune of working in. The way you solve for that is by asking: how do you make this akin to the workflow the team is already used to doing, and only make it more efficient? If you're adding steps, or completely revolutionizing and transforming what these teams are used to doing, change is very hard in healthcare at baseline, and changing one thing can break the gears and explode the machine. I'm sure you've seen that before in some of the work you've done, Chris.

So the more you have teams, the people who understand those workflows, who understand those team members, the problems they're dealing with and the things they're trying to solve for, that actually becomes really essential. I feel very fortunate to be in the position I'm in, having both a clinical background and a clinical research background, to say that I've lived and worked with all the people who actually make overcoming these challenges their day job, for decades at this point. So not just knowing it or learning it, but living it, is an essential component of designing any solution such that the end user is going to benefit at the end of the day.

There are a couple of truisms I like to come back to. AI at the end of the day is just a data model: data in, data out; garbage in, garbage out. AI cannot revolutionize or transform or make magic. You really still have to start with the data and the quality of the data you have as an input. Another truism is that, before you think about revolution and transformation, in healthcare it's rare that we see massive shifts in the way our workflow changes. It's oftentimes stepwise. And the steps that actually move us forward are the ones that make lives easier for the people who have to take the steps; that's true at intake, at triage, at vitals, at note-taking. Some of these ambient tools have exploded in use because they haven't changed massively what we do. There's not a whole ton of additional inputs or transitions we have to make to make that a part of our usual daily process. So the more we make this easy, for principal investigators, for clinicians, for research coordinators, for the people who support all those processes, and ultimately for patients, the more likely we are to be successful in these efforts.

This is amazing stuff. I don't know what we think sometimes when we're working through transformations, but it seems like we maybe underestimate impacts sometimes before we start to build. From your perspective as a clinician, without that context, we'll just continue to do things like we've always done, and design things we think are better, but there are a number of areas where we miss the mark and we kill adoption, frankly, because we've just not really engaged, been elbow to elbow, with someone like yourself who understands what's involved in the care and what's involved in the research, to help us design the right workflows that are actually going to enhance and not take away or disrupt. From your perspective, having been at the bedside but also speaking with boards and executive teams, when a tool stalls like this, is it the technology, is it the trust, or is it the workflow that you think would be the primary things most often tagged?

It's a great question. I'd say it ends up oftentimes being a little bit of all three. The technology is transformative, but pointed at the wrong data sources, or used in the wrong way, it doesn't help anybody. Whenever there's a new innovation, it oftentimes permeates the lexicon: if you're not using AI, you're doing something wrong, so how are you using AI? The challenge is to just incorporate something quickly and say, well, we're using AI for this, but it either narrowly fixes one small thing that actually wasn't the major barrier to getting a process completed, which doesn't make a big delta at the end of the day for the team and for your outcome of interest. So the technology has to be there; that's obviously table stakes.

To answer your question, I think the workflow is oftentimes the biggest challenge. A lot of times in technology, what I see is a lot of vendors assume, I'm going to bring this new widget to the table and everybody's going to be so excited about my widget that they're going to drop everything else they're doing and use it. The truth is that's almost impossible. Clinical workflows, and especially research workflows, are extremely complicated, involving many different systems, platforms, data sources and elements, and many different stakeholders. So what you really need to find is something that works well in the workflow, and that's actually quite hard to do, because no matter what, you're changing some amount of behavior. Changing behavior for clinical teams and research teams, we're, for good reason, very risk-averse. Any sort of change comes with an enormous amount of apprehension. I've seen the simplest workflow changes in the hospital environment sometimes take a year or longer, and I'm sure you've seen worse than that, Chris. It's because we've done it this way for a long time; we know this is safe, we know it works. Changing something introduces uncertainty.

You don't even know all the ramifications. How's this going to affect biomedical engineering? What about our tagging process? Is our barcode system ready for this? QR is better in so many different ways, but this is going to affect its connection to Epic or the EMR, it's going to affect our workflows and our scanners. The snowball effect of a change in medicine is much greater than I've ever seen in any other industry. So understanding that, and coming to the table with that knowledge and acceptance, and really trying to make the changes you're affecting in the workflow as narrow as you can, I see as the best way to get clinical uptake. And that's different from getting board uptake. Board uptake, C-suite uptake, they want to hear transformation. They want to hear AI-enabled from beginning to end, they want to hear 20, 40, 80% increases or decreases in cost, or increases in efficiency. The rank and file, the people actually implementing these tools and changes, the ones directly working with the systems or the patients that make this possible, that is not what they want. It's very difficult to convince those folks that revolutionizing a process that has worked well for years or decades, depending on your place of practice; getting buy-in from those folks is essential at the end of the day, but really hard to do.

Right. That kind of takes it from that point in terms of how difficult it is. There's a high level of evidence required for something to even get into the types of model discussions you're engaging in. Quite honestly, I think we all hear the stats, roughly 85%, maybe higher, of pilots going on in healthcare organizations, and across other industries too, but healthcare is the one I know the most about, are failing. There's a variety of reasons for it. Based on your experience working through this, and it's not an insignificant amount of research, time, energy, and Lord knows probably sleepless nights as well, what would you say to the executives or CIOs who are in a process of trying to figure out what solutions they should be moving forward with, and what that criteria looks like? What would you tell them is most important to really consider, and maybe it's more than one thing, before they move through a contracting exercise thinking they've got everything covered that they need? Physicians like yourself are typically not coming into the contract conversation, which is just the natural flow that everything should follow if it has anything to do with clinical care.

It's a great question, and I think you're right. We are, in many ways, entering the valley of disillusionment, the cycle we're in with this new disruptive innovation with the advent of AI. AI is a powerful tool, and it can be transformational in its capabilities. Healthcare has forever been a laggard in terms of uptake, for many reasons: risk-averseness, patient safety, first do no harm, and many other tenets of healthcare being good reasons for that. But we're also at a point of impetus, where there's a strong sense both internally and externally within healthcare that there's an enormous amount of inefficiency in the process and that we need to do better.

When I talk to the C-suite and CIOs and other folks at hospitals and health systems I have the fortune to work with and partner with, a lot of it is: let's not boil the ocean. Again, not death by pilot, but picking some discrete milestones and metrics around which we're going to make a go/no-go decision. The good news is that generally with AI, those don't have to be six, nine, or 12 months out; these can generally be relatively tight timelines in terms of the results, or the process or outcome metrics you're looking to track as a result of any new innovation or integration of a new tool or technology. So being tangible, being discrete, setting timelines and keeping to them. And again, failing fast is oftentimes something we like to talk about a lot in tech, not nearly so much in healthcare. But I think some people are seeing some of these 85% pilot fail rates not as a bad thing, but as an opportunity to say, well, where is this really going to help, where is it going to make a difference, and where are some of these process and outcome metrics really going to be as great as promised?

Some of the data coming out of the ambient space, and I have many friends and great partners who work in that space, so this is not meant to cast shade, but these revolutions in time back to clinicians, we're certainly not feeling, and from the data, we're also not seeing. It's some single number of minutes that are coming back to clinicians' daily schedules with the advent of some of these tools. There's value in that; I'm not saying it's not valuable, and there are other reasons ambient adds value. But a lot of the promises around "you're going to get your nights and weekends back" and "documentation after hours is no longer going to be a thing," we're not seeing that become part of our reality as much as was hoped. So when you think about these pilots, and understanding what you're really looking to change and aiming to revolutionize, I think we really want to be as intentional, as rational and reasonable, and as firm as we can be about what we're really aiming to achieve.

It's not an easy bar to clear. The things that are really challenging, I think, are when you're trying to get something across the finish line just to get a green light to go forward. There are some elements that probably are not included in the assessments that would be better if they were, some of the key things that will either cause there to be a good level of adoption, or not. I think about how for years, when I worked with data solutions in my chief data officer role, that was almost all the time: we're having conversations around what's the ROI, what's the risk, going into these different conversations around what you deploy. Talk a little bit, from your perspective, as we wrap up here, about what are some things, as a clinician, that you see are often not represented in the overall structure by the time it gets to you, when people have already made decisions and done development work. I think that's an area where we fall down more often than not when we're trying to operate that way: instead of keeping you involved and building alongside you, we go off and take what we remember that you said, and six months down the road we bring you something that doesn't match that at all.

Yeah, as much as you can, and we've seen this in fits and starts, really having the developers and designers almost elbow to elbow with the end users. That could be a clinician, a nurse, a PA, an MD, or the front office staff. People come to the table with these preconceived notions of what their day actually looks like, what they're doing, and what their challenges are, and that's oftentimes informed by their managers, or by somebody even more senior. Oftentimes there's a lot of little stuff that gets lost in translation or communication, where I need to be able to pivot; if I get a slip from one of my docs, I need to stop everything and go over here and do this. But if I come back to my computer and I'm logged out, I just lost four minutes to do my two-factor authentication to log back in, and I can't do it. I don't have time, because I've got patients standing in front of me in line waiting to register, and I can't sit here and log back into this tool that's only saving me a minute or two a day, because I've got more pressing needs.

So as much as you can, and it's becoming in vogue for deployed engineers, really understanding, before the tool is in its final form, before this product is even getting surfaced to the end user in a production environment, really pressure-testing this at every stage, and not three or four months apart, but three or four weeks apart. As you get into wireframe, as you get into prototype, as you get into pilot, you really want this to be as successful as early on in the process as you can, because when you lose buy-in, especially clinically, it's extremely difficult to get that back. So that's something you want to engineer for.

I think you'd probably agree with this, but the challenge of the funding model always gets kind of batted around, and sometimes it just stalls things for months at a time. There's this perception that if we add the operating tail into the budget we're building to do the program, it's going to inflate it. It may not be that high, but it's usually a gap, and it's not even there. So you're moving along, you have a successful pilot, now everyone wants to roll it out to the enterprise, and you're like, we can't do that yet, we don't have everything we need to support it. Talk a little bit about that from your perspective. I know there are people listening who are decision-makers in healthcare organizations, and they've heard me say stuff like that, but I think you can give it some credentials and some weight. I'd love to hear you address that one.

I think it's a Ben Franklin quote: an ounce of prevention is worth a pound of cure. We like to talk about that in preventative cardiology and medicine, but I couldn't agree more. Having this, even in small bits, involved early on in the process, before you make a whole ton of architectural, structural decisions that are going to impact the end-user experience, I would encourage as much as I can. Getting those stakeholders involved early can make a really big difference. But also, pick those stakeholders intentionally. Oftentimes it's not the division head or section chief or executive director who's going to be the end user of a lot of these tools. They have experience that is undoubtedly valuable and can add a lot of color, but the people who are in the day-to-day, in the trenches, who are going to be most impacted by this, are really the ones you want to engineer around.

So that's sometimes a mistake that gets made: I want the most senior person, the most decorated person, the professor of X, Y, or Z. But sometimes that person is giving talks or speeches, off running divisions or departments, off doing fundraising, doing surgery, or taking care of patients with a minority of their time. The people you're really engineering for are the ones much more on the front lines, and those are the people you want to make sure you get buy-in from, because that's the vast majority of the volume you're going to be interacting with.

I love it. What you've been sharing, for me, is exciting, because I just think we've not had a lot of opportunities. COVID obviously was very, very disruptive in so many different ways, not the least of which was the messaging around what some of the advancements being made are really going to be able to deliver for patient care. For me, this is refreshing. I remember going into a cancer institute when I was working in New York a few years ago, and I hadn't been in for probably a decade. The difference when you walk in the door, everything feels different. It feels like a much more optimistic environment, because people understand that we've made all these advancements. I just encourage people listening: if you're in or around a facility and they offer a tour, you should probably just take it and take a look. What you're going to find is a lot of people who are passionate and enthusiastic about what they're doing, and they're pushing really hard to make things better. Radiation therapy was one of the things that really shocked me, when I understood how precisely they can target cells and what they do to prevent damage to surrounding cells. These things are remarkable. It's really hard to even wrap my head around it.

But it goes to the main point I want to make: what you're doing is phenomenal, and I have so much respect for you and what you're accomplishing. Of course, I'm a big fan of Mass General Brigham, having had the privilege to work there a few years myself. But you're taking what you're doing out beyond just how it benefits you as an organization. It's really that unparalleled commitment that I love and have seen repeatedly from you all, consistently. It's patient first, it's the outcomes that have to be delivered, and it has to also make sense and hopefully remove barriers, not put more barriers in terms of how a clinician can actually be effective in interacting with patients. So thank you for that, and thank you so much for coming on the show. This has been wonderful. I know people are going to really appreciate this. If you wouldn't mind, as we close, just let folks know about the website, because I'm guessing you're going to get some feedback from this, probably some folks who would love to pick your brain a little bit.

Always happy to engage, Chris. Thanks so much for the opportunity, and for all the kind words. It's been a pleasure to have this conversation with you today. aiwithcare.com is the website. You can always reach out to me; it's aj@aiwithcare.com, and I'm really, really excited to continue this conversation. Chris, thanks so much again for the opportunity. I appreciate it.

It's my pleasure. And for my listeners, that is going to do it for this episode of The Signal Room. In the meantime, stay healthy, stay curious, and I'll see you next time.