IQVIA Holdings Inc. (IQV) Earnings Call Transcript & Summary

Today's live broadcast, The Future of AI and Analytics in Trial Design. My name is Andy Studna, Editor of Pharmaceutical Executive, and I will be your moderator for today's event. We are pleased to bring you this webcast presented by Pharmaceutical Executive and sponsored by IQVIA. I would like to share a statement from our sponsor. IQVIA is a leading global provider of advanced analytics, technology solutions and clinical research services to the life sciences industry. IQVIA creates intelligent connections to deliver powerful insights with speed and agility, enabling customers to accelerate the clinical development and commercialization of innovative medical treatments that improve health care outcomes for patients. With approximately 82,000 employees, IQVIA conducts operations in more than 100 countries. You can learn more at www.iqvia.com. We have a few important announcements before we begin today. This webcast is designed to be interactive, and we encourage you to ask questions during the event. [Operator Instructions] You can enlarge the slide window by clicking on the small icon in the bottom right corner of the media player. The slides will advance automatically during the event. And if you have any technical problems viewing or hearing in this presentation, please click on the question mark help widget in the top right of your presentation window. I would now like to take a moment to introduce today's speakers. We are pleased to be joined today by Lucas Glass and Pablo Aran Terol. Lucas Glass is the Vice President of the IQVIA Analytics Center of Excellence. The Analytics Center of Excellence is a team of more than 200 data scientists engineers and product managers that research, develop and operationalize machine learning and data science solutions within the research and development space. Lucas has launched over a dozen machine learning offerings within R&D, such as site recommender systems, trial matching solutions, enrollment rate algorithms, drug target interactions, drug repurposing and molecular optimization. And his research has been accepted at the Association for the Advancement of Artificial Intelligence, WWW, NIPS, the International Conference on Machine Learning, the Journal of the American Medical Informatics Association, KDD and many others. Pablo Aran Terol is a Senior Product Manager at IQVIA's Analytics Center of Excellence with more than a decade of experiences in life sciences and technology. He received his PhD in Biophysics from the University of Cambridge and spent several years in academia. Currently, Pablo supports the development of technology to improve clinical trial protocol design to mitigate design risks and bring therapies to patients faster. Thank you for joining us today. And if you would, please get us started.

Hey, folks. Thanks so much for joining us today. I'll be going through some of the future of AI and analytics in trial design today, along with Pablo Aran. It's a very exciting time. I can't -- being an AI professional, I can't get through a meeting with friends or family and not talking about what I do in the AI space. And it's, I think, a really exciting time in general for us. So even though AI has become the popular thing to talk about, it's actually been there for quite a long time. When you think about just the way that we operate on a day-to-day basis, right, we used to use maps analysis. And now just GPS, which is AI, is part of our daily lives. And googling it, right, we're always finding information through Google, another application of AI, know that Spotify, listening to the curated playlists and watching the movies and television shows that we watch. It's become an invisible sort of thing behind the scenes in our world. And we often take it for granted and think, "Oh, AI is going to change the stuff we do." AI is already changing the things that we do and changing the way we behave. And I think we're going to see that accelerate within the R&D space. So the quick overview of the conversation today. We'll talk about some of the challenges that we're seeing with cost and efficiency, some of the solutions that we propose and then the overall impact. So the challenge, right? We've seen the number of patients increase in this space over the last a half decade or so. And there was quite a bolus of patients that came through during COVID-19, as you can see in 2020 -- I mean, in 2020 and 2021. And that was a real wake-up call for a lot of the things that we can do in this space. And then you look at the same time, the number of trial starts that have been happening over the last few years is really -- it's been steadily increasing. And I think we've gone up by some 30%, right, if I'm trying to estimate that, rule of thumb, 3,775 in 2012 all the way up to almost 6,000 in 2022. And even if you exclude the COVID trials, we're seeing -- we continue to see a steady increase. However, the success rates are not keeping pace. We're not being able to maintain the throughput of the trials and whether or not they get all the way through to approval of the FDA or other regulatory bodies. And that is a real burden on the industry. And so what can we do to start improving this? Well, our argument here is that we can start to leverage real-world data and AI to more effectively predict that -- I am stealing a phrase from someone that presented recently. We're in the business of clinical trials, not clinical guarantees. And so how do we make sure that we most effectively manage the risk? And real-world data and machine learning are very good at managing and predicting risk and predicting probabilities. Now it's used like all over the place in the finance space now because it's so good at estimating risk. And so we need to take a page. All right. So what's the first space that I'm talking about here is machine learning to identify patients, right? This is one of the most classic examples of deep learning and machine learning. Back in 2016, gentlemen Ed Choi and Jimeng Sun published some pivotal papers predicting disease progression using deep learning and really started to move the needle on what we could do to identify patients. And so when you're out there trying to figure out what the patient population looks like, how many patients there are, who the patients are and where they are, right, you need to be able to find the signal of who these patients might be and who these patients will become. And machine learning is actually able to figure out, even if they're not diagnosed -- I mean, maybe they already have a disease or maybe they will get the disease, machine learning is pretty good at identifying that and finding the signal. And so when you're able to actually figure out these patients, so what do you do with them? And then I think it's an important thing that we don't just have AI for the sake of AI. We have actually got to be able to leverage this some way and actually improve the way that we're running our trial. So one, obviously, you can use this to help find the sites. You know where the patients are that have some rare disease or some disease, then you can actually go and find the sites. Another one is you can actually start to find patterns. You can ask the machine learning, not only can you find the patients, but can you find the patterns that are helping you to identify who the patients are and be able to leverage that for actually then creating screening algorithms. And these are really important mechanisms that very much show in the rare disease space because it's oftentimes these patients are going underdiagnosed or undiagnosed. And so this is a good use case, where we actually looked at patients with an ultra-rare disorder. And we were actually able to come up with a 95x better than the clients approach for their rules-based method to actually find these patients, which is a huge uplift on the screening approaches for finding rare disease patients. And I think that this is a real good example of not only coming up with some algorithm but actually making sure that it's moving the needle on how you're running your clinical development program. And with that, I'll pass it over to Pablo Aran.

Thanks, Lucas. Well, we have just seen how we can use AI/ML and real-world evidence to find the right patients within the right indication. But we must also find a representative mix from a race and ethnicity perspective. Diversity of patients who are ultimately impacted by the approval and the use of drugs in the real world is often lacking in clinical trials. FDA reports many racial and ethnic groups continue to be underrepresented. And the actual mix of populations for the conditions being addressed in clinical trials is not reflected in the experimental design. And this is a problem because inadequate participation from clinically relevant populations can lead to gaps in the capture of safety and efficacy data and undermine the data taken among the understudied populations. But thankfully, we can improve diversity in clinical trials by understanding the real-world evidence to identify target ranges of race and ethnicity distribution. So let's take a look at that. So we can see here how we can use real-world evidence and machine learning to find and select sites that allow you to maximize the enrollment of participants from an underrepresented population. With AI/ML methodologies like this, we can make trade-offs between enrollment and diversity to achieve high levels of both and mitigate avoidable amendments. However, this only speaks about finding patients. It doesn't address the willingness of said patients to participate in this study or any potential retention issues due to the protocol design. But thankfully, we can layer this level of information with known barriers to participation to improve the willingness to participate in [indiscernible]. Using this data helps sponsors improve the diversity in clinical trials by identifying barriers to participation and mitigating bias in trial design. Burdensome elements may affect requirement of retention and what's perceived as burdensome may differ by race and ethnicity. We do this conducting patient surveys, which are at the end of the day a form of real-world evidence. [indiscernible] patient burden score algorithm is based on those surveys and can help guide evidence-based design decisions. Delineating the perception of burden differences by race and ethnicity in a data-driven way allows us to advise sponsors on changes that may or may not impact diversity and inclusion enrollment success. This data is an important component of our overall approach on recommendations to sponsors to meet their diversity and inclusion goals, including validating the study design plan against differences in preference amongst the desired populations to help ensure the design does not inapparently lower the changes of diversity and inclusion success from the start. But we've been talking about patient burden, specifically in the concept of diversity. So let's take a step back on this talk about patient burden more broadly. What is patient burden? Patient burden is the physical, psychological and emotional toll that participating in a trial can take on a participant. While not exclusively, it is largely determined by design elements in the protocol. And what's really important to remember is that being able to accurately and systematically understand the impact of design elements from the patient allows you to then take action to ameliorate said burden. Now there are, of course, a number of ways in which you could reduce the patient burden. And the most obvious one would be to change your trial design. But we also know that sometimes in order to capture the evidence that is required out of the clinical trial, a protocol has to be cumbersome. You have certain assessments and certain number of visits that cannot be reduced if you're trying to gather evidence that you need. However, by understanding the patient perspective, it allows you to take additional actions to lower the impact of the [ computation ] without modifying the trial design. Examples of these actions can be the use of remote monitoring by reducing the number of in-clinic visits or use electronic consent forms to make it easier for patients to understand and agree to participate in the trial. However, we also know that as much as patients are really important in a clinical trial, they're not the only stakeholder in a clinical trial. So let's put on our vision one more time and capture all of the clinical trial stakeholders. All right. So we have just discussed how real-world evidence and AI/ML can be used to improve the requirement of retention of patients. However, in a clinical trial, we can also study the impact on the sites and on the sponsors. And indeed, we can see how the protocol complexity manifests itself in detrimental impact to said stakeholders, a complex protocol, there has many site burden assessments and a complicated design is going to adversely affect patients, sites and sponsors accurately understanding protocol complexity and its impact on the stakeholders. It's fundamental to understand the operational impact on the site decisions to optimize our trial. So now that we have this framework, let's continue exploring this concept by looking at site burden. Site burden is the administrative, logistical and regulatory workload that participating in a trial places on a clinical research site. Now this burden is influenced by factors such as the number of patients enrolled, the protocol complexity, like we just discussed, and the need for regulatory compliance, training and so on. And not only patient burden, there are a number of ways that you can reduce the site burden. Again, one of them would be to actually change your protocol design to reduce some of the burdensome elements, such as reducing the people who work on the regulatory burden. But we can also take actions such as data collection from the patient home or flexible dosing schedules that would minimize the burden on the site while maintaining the integrity of your experimental design. Of course, it's not just all about the protocol when we're thinking about site burden and not all sites are made even. Yes, the site burden depends on the protocol design but also on the characteristics of the sites being considered. Finding the right site to minimize the site burden can reduce your risk of lower enrollment rates and increased dropout rates and decreased site retention. Of course, the best analytics and insights will only make a difference in your trial if they actually get implemented. In this case study, you can see how draft protocols that were provided from sponsors and that had a patient burden generated from them then did apply the changes that were recommended and resulted in a drop in the patient burden score for the final protocol of the same trials. And you can see on the histogram on the [indiscernible], whereas the draft protocols were sitting with a score somewhere near the middle as the final protocols were [ refined ], that score was lowered and set where the majority of trials within a particular phase and indication of therapeutic area were at. So now that we have seen how real-world evidence and AI/ML can find patients and retain them, let's talk about our next touch. All right. So we have now identified the right patients and have accounted for diversity targets. We have taken action to reduce the burden of the protocol against key stakeholders, the patient and the site. But what can we do during protocol design to improve the operational execution of said protocol? Before we jump into any solutions, let's dig a little bit deeper on the magnitude of the problem. 77% of the trials have a protocol amendment, an enormous number, which equates to an average of about $140,000 for Phase II trials and about $535,000 for Phase III studies. And interestingly, that same research identified that 23% of those amendments were considered completely avoidable because they were due to protocol design flaws, inconsistencies or errors. Now some protocol design factors that lead to operational challenges include things that we have been talking about, the site consistency, patient burden, site burden, study complexity and eligibility criteria, certainly including diversity and inclusion requirements. Now this potential study risk can lead to cost litigations and amendments, difficulty in enrolling and lower patient compliance. So now that we understand the problem, let's take a look at some solutions using real-world evidence. And I don't think that by now, you will be surprised to hear that we do have some solutions that leverage real-world evidence and AI/ML to improve the operational outcomes. So let's talk you through them. Before we actually begin to look at some solutions, it helps to go back to the framework that we shared before. We have discussed that we can actually measure the negative impact on the stakeholders in the form of patient burden and site burden. Now surely, it stands to reason that happy patients and happy sites will have an operational impact. After all, sites are less likely to participate on a trial if it will mean a lot of work for them. And similarly, for very burdensome trials, you can imagine that it will be harder to enroll and retain the patients needed. So we have this hypothesis of these things correlating to operational impact. But does the data support that hypothesis? Let's take a look at that analysis. We come back to this case study in which we took last trials, and we went to 105 historical protocols and compared the patient burden of the executed protocol against the operational outcomes after the trials were completed. And those trials that had a higher patient burden were correlated with longer start-up timelines, higher number of protocol amendments and a higher screen failure rate. But in that analysis, we also found direct correlations between the same elements and trial performance, not through the patient burden algorithm. And this poses a [ natural ] question. If patient burden as well as the design elements are correlated to operational outcomes, but we also know that protocol complexity influences patient burden, what can we do to understand the operational implications of protocol complexity? Let's take a look at our second solution. Well, leveraging AI/ML, we can not only cover that complexity with operational outcomes that we just showed for patient burden, but we can actually define complexity by the impact of operational outcomes. And this is because protocol complexity is an abstract measure. You can't really define it in an easy way. You can't go to a clinical trial and you say, "Calibrate, use a stopwatch, a thermometer," to measure complexity. However, what we definitely can measure in a clinical trial are the operational metrics and impact that they have on a clinical trial. So by training AI/ML, it has historical trials to predict operational outcomes, and using explainability techniques such as SHAP, we can identify and quantify the impact of design elements on operational outcomes. And by doing so, we can actually define a complexity algorithm, where the complexity score is proportional to the operational impact of the design elements included in the protocol. In this situation, the complexity score will itself be defined by the operational impact. So we can correlate the patient burden to the operational outcomes and we can define complexity by the operational impact. But can we actually directly understand the operational impact of protocol design elements? Let's take a look at our third solution. And we can. In fact, what we can do is leverage the AI/ML algorithms that were created for operational planning and feasibility, which is the downstream activity from protocol design, to bring early insights during protocol design process. As you can imagine, there is a fundamental problem with bringing feasibility considerations during protocol design. By definition, some of the key determinants of operational outcomes such as which countries you're going to run your trial in or how many sites are you going to do it in are generally decided after the protocol has been settled. However, it is also true that many of the elements that contribute to said operational outcomes are contained in the protocol itself. And therefore, we can use operational algorithms to create estimates of enrollment, trial duration and cost. So you can dynamically understand the operational impact of the same choices in your protocol. And importantly, when it's an interpretable AI/ML, we can highlight which design elements are contributing to the different operational estimates and by how much. Now I want to also be very clear here. These will always be estimates. And they can only be estimates. Because at this point of the clinical trial lifecycle, critical decisions have not been made, such as the countries and the number of sites. But given these operational insights early during the protocol design phase allows you to make better, informed decisions and also transition to operational planning with an understanding of expected operational outcomes. So to bring it all together, we have explored how we can use AI/ML to understand the operational impact of protocol design decisions through three different perspectives: the first one by correlating the operational impact of the stakeholder burden; the second one by defining a complexity score based on operational outcomes; and the last one, by conducting an early estimation of key operational outcomes during protocol design. Back to you, Lucas.

Thank you so much, Pablo. So overall, the impact, right, we're minimizing the need for amendments. Amendments have a very detrimental impact to the clinical trial pace and its success as well. So we can identify how to prevent those things in the first place. We can increase enrollments by finding the right patients. We can lower study costs by making sure that we don't have unnecessary complexity and amendments. We improve recruitment and overall time savings. Time is -- not only is time money, but time is also -- it limits how many lives we can touch, how many people can be helped by the therapies we're bringing forward. And so time is just such an imperative thing. And machine learning and real-world data have shown how they can give us some of that time back. So with that, we will hand it back and open it up for questions.

Okay. Thank you, Lucas and Pablo, for that presentation. [Operator Instructions] Okay, so we will get started with our first question. This one is going to be for Lucas. Given the multiple options for applying external protocol data/results due to data sharing initiatives, especially in oncology, for example, Project Data Sphere, Vivli, et cetera, please comment on the relative utility and reliability of these sources, how to manage quality and acceptance by regulators of the analyses and applications, for example, allowing augmentation of a control group in a randomized trial?

Yes, that's a great question. And I wish I had a perfect answer here. But the reality is I don't. It's still a very evolving space. We're actually looking to bring on an expert in this sort of application of AI for regulatory kind of experts right now as it's such an evolving thing. And we need to be able to influence the regulators as well. I know the places that we've used a lot of this is actually in the single-arm trials, when there's an opportunity to provide evidence outside of the single arm, when it makes more sense for the study to run a single arm, but you can then augment it with some -- whether or not that's AI and machine learning, sometimes just real-world data analysis to go along that is very helpful. And I do know some companies out there, like an Unlearn, have done some validation with the FDA. But it's a very evolving space right now. And we are working with several sponsors on how we can build out some of their capabilities in this space.

All right, great. Thank you, Lucas. I'm going to throw it right back to you for this next question we have. It is on the topic of cognitive bias and proof of concept using AI. The person building the model stated, "If it's wrong, they and client will tell us and we can correct it. This is an offshore with no experience in health care yet a data scientist. I reported the situation to the implementation partner." One is raising your hand to ask and validate the model from cognitive bias.

So it's a good question. I mean, I could talk about this topic for quite a while, and I'll try to limit my responses as much as possible because I can kind of get on a rant here. But a few reactions to this. So one is you have to get a really solid understanding and tie-in and buy-in from the subject matter experts on how you're going to be evaluating these things upfront. You can't gain the system after the fact and start to, "Okay, now that's not right? That's not right?" You need to work on that upfront. Because the reality is that sometimes, AI on any given specific example will get it wrong. You show it to the SME, they'll be like, "Oh, that's not the right answer." But if they're getting it right more often than the subject matter experts holistically on sort of the broader scale, then that's an important thing to understand, and you rely on the AI a little bit more as opposed to the SME because you've got some very objective evaluation criteria that you've predefined. And that also helps with adoption because SMEs tend to be resistant to using AI. And so if they're not bought into how you're going to evaluate that upfront, and you say, "Oh, hey, here's the answer. And oh, it's better than you," you're just not going to get that. And it's interesting because this question that you asked or that was asked is from the perspective of the AI engineer sort of oversimplifying it in their cognitive bias. But you have to remember also that the subject matter expertise actually has significantly more cognitive bias than the algorithm does, right? Humans are the ones that have cognitive bias, not the algorithms. And AI can actually help overcome cognitive biases of the subject matter experts. And so you need to focus on how you're evaluating. Is the AI getting to the better answer? Have we agreed upfront on sort of the evaluation criteria, the protocol for the AI, right, to make sure that we all have the -- we're on the same page as is this helping things?

Okay, great. Thank you again, Lucas. Jumping to our next question, how do you get access to historical clinical trial data to build models?

Yes. So you need permission from the data owners. Now interestingly, so we can use real-world data because we have access and we own quite a lot of real-world data at IQVIA. But we don't actually own the clinical trial data that we have access to. That's owned by the sponsors. And so we don't leverage that. Outside of a specific relationship or contractor engagement with that sponsor, there are different data assets out there that you can create and construct. There are public clinical trial data sources that you can access. And there's more and more and more. And if you follow up on this, I can share some of those resources. It depends on who you are and why you're trying to access it. But there's definitely an increased sort of push towards transparency around making that available, particularly for academics.

All right, great. Thank you. Our next question, how exactly do you use AI/ML for patient identification and recruiting from patient-level data?

Yes, I'll take that one again, Pablo. So there's a few different tacks here. One of them is there's quite a few vendors out there, IQVIA being one, but others where you can actually have tools behind the firewall of hospitals embedded in there that the hospitals use, that the sites use that they can actually come up with ways to find the patients, right? Whether that's like simple search tools, which is machine learning algorithms, even like classic Google is machine learning, or some more complex things like the stuff I was showing you about patient identification. So you can actually get it installed behind the firewall. But that's been an ongoing industry challenge, getting that at scale, because you need the buy-in of every different site that has this, that's using this, that's paying for this. And so achieving the scale that you really want there is tough. But to identify specific patients, that's sort of the best path to do it. But if you want to go into using like de-identified data, which -- that's where you can get really very high scale with and what IQVIA has a lot of de-identified data. What you can do is a fewfold. One, you do it to find the sites and then which sites have the rare disease or the different patients. And my boss used to say, "Fish where the fish are." The other one is sort of the learning to learn model. And you figure out, are there like prototypes? Are there like specific types of patterns that are -- like you can learn a query that makes sense as opposed to building some complex AI model. You use AI to actually where you're trying to find a query that can find patients and that you can then use that as something that a human could actually apply saying, "Yes, this, filter this, not this." And then this filter that we're going to apply, which is very sort of human-capable, and you can teach that to doctors and say, "Hey, if you apply these criteria, you're going to find a greater prevalence and improved screen failures." So I think those are the key ways that you can get this done, the latter two being much more scalable than the first, but the first being the most effective.

Okay. Thank you, again, Lucas. Now going to throw this next one over to Pablo. Does patient burden assessment take into account caregiver burden, especially for pediatric studies and studies targeting the geriatric population?

Thanks, Andy. Yes, it does. That's the short answer. The longer answer, which I think is worth getting into a little bit, is that this is a very good example of why the depth of the analysis is important. And that's because there are particular design elements that only apply to certain protocols, in this case, whether you have a caregiver or not. But if it is one of those protocols, if you're working on pediatric or geriatric trial, then it is very important to account for it. And having that breadth of data that allows you to account not just for the what would be the more obvious determinants of patient burden, whether you have a lumbar puncture, but really getting to some of the more detailed, sophisticated ones is what really allows you to draw comparisons across different kinds of protocols and really refine your analysis.

Okay, great. Thank you, Pablo. Our next question, have you seen any potential use cases for AI in clinical trial analysis and regulatory reporting?

Absolutely. This is actually like -- it's somewhat tangential to the stuff that we've been talking about today. But the amount of time that's spent at the white space after the study, trying to get all the materials together for submission is an incredibly sort of costly endeavor, not just the cost to do it, but the time that you're losing of your drug being reviewed and approved by the regulatory agency. And so the ability to reduce timelines -- I mean, reduce man-hours and costs, yes, but reduce timelines to get these things built, we are doing a lot of experimentation with tools like GPT to try to tackle that right now. And there is absolutely really like a rich vein of promise here. If you shoot me a message, I'm more than happy to talk to you about that. But we are seeing -- basically, what we're doing is we're categorizing all the different documents because we have a whole regulatory submission services organization. So we're categorizing all of the documents that they use and that we submit that we generate as part of our deliverables. And then going back and say, "All right, for this specific document, what are the upstream documents and materials that you need and then that are used to create this and sort of teach these GPT systems to read these upstream documents and write the downstream documents and start moving our way all the way upstream until we can automate as much as possible while continuing to have like an editor in the process, right, the reviewer?" I think there's maybe a Hemingway quote, "There's no good writer, only rewriters," right? And so I think maybe not, one of those famous authors said that. And so I think that's the path that we see transforming in the regulatory space, a very rich vein of opportunity, very, very, very happy to follow up on that.

All right. Thank you. Next question, what are the shortcomings of AI?

I'm a VP of AI, so nothing. No, just kidding. I mean, that's a long topic. So there's -- let me -- I'll say the biggest challenges that I've had, right, with AI is a lack of trust in it and the lack of the AI scientists to effectively help SMEs gain trust in it. So AI does have shortcomings, right? Sometimes, it can't get the answer as effectively as people can, right? And there's a lot of egos in the data science community, where the AI researchers and AI scientists think we can replace anything and everything. So trying to get past that and really be able to -- and then the SMEs that we work with tend to also have some confidence in their expertise. And so trying to sort of bridge that world, where the SMEs can be trusted, the data scientists can sort of check their egos and figure out how to learn the space more and really get it adopted and trusted, I think that's the #1 challenge right now facing the AI community. But I mean, I could talk about this a lot. I don't know, Pablo, if you have a completely different reaction to this.

Sorry, I was muted. I think another challenge that we often encounter is data limitations, right? You cannot apply AI to solve any problems. There are certain issues of data coverage and data quality that you must have accounted. The best algorithm in the world with the most sophisticated tech is not going to be able to deliver any valuable insight. So it's definitely not a cure-all. It's very helpful for specific issues in which there is data available in which you can train your model. But we have to be aware that you're only going to get good predictions if you have good data to base them on.

Great. Thank you, both. Our next question, what are your plans for expanding the use of AI, focusing in other aspects that will affect trial quality? I see you are looking at PDs. Do you plan to use any other measures?

Pablo, do you want to take this?

[indiscernible] PD swimming protocol deviations if that is the case, yes, there's a number of operational parameters that are part of a clinical trial. And there are important protocol division because you want to avoid them. There's another -- there's a few there that are important, not just because you want to avoid them, but also because you must use them in order to be able to calculate some aspects of your provisional planning. An example of this would be your dropout rate. You can't really figure out what your enrollment pace if you don't have a good estimation of what your dropout rate is. But of course, you also want to minimize that as much as possible. So I would say that speaking when we try to think to prioritize, it's a little bit of both columns. We want to go after the ones that are the highest impact, and we want to go for those that are necessary as part of the process of either refining the protocol or doing that transition from protocol designing to operational planning.

All right, great. Thank you. Moving on to our next question, how do you see GPT and ChatBot technology impacting protocol design?

So I'm on the speaking circuit about GPT and teaching both within IQVIA and externally on GPT. It's become a pretty big staple of my life in the last 6 months. I think GPT is good at managing documents. Can you construct datasets from libraries of protocols? Can you help author protocols? And I'm not quite there yet that we can completely rely on it to be like authoring and coming up with the design. I think it can create the documents if somebody creates some sort of design, and they can create all the documents that you need to be submittable and approvable. But there's a pure -- it's just not there yet in the stuff that I've seen. Now the stuff that I've seen, not saying that that's not out there, right? And I know that even my team were pushing hard on testing the limits of what we can do with this. And Pablo is probably closer to this than me because he is the person on my team that's pushing the limits here. So Pablo, any reaction to this one?

No, I think that you're right, Lucas. If you look at the generative use of LLMs and other functionalities like that, it's not there yet. But it is true that the application of this kind of technology is not limited to say something like, "Hey, here's my synopsis, generate the rest of the protocol for me." Another use that we have been exploring in the product that we have built to date that seems to be quite helpful is to help us summarize and access large volumes of data. Many of you that are familiar with the process of protocol design, you know that you have to go through some other protocols that are out there, so you can go there and compare, "Hey, did use this endpoint or not?" So in order to summarize and extract that information without having to read through hundreds of pages as well as giving access to these models to databases of scientific knowledge can be a very helpful way to not necessarily get better information but just doing it in a more efficient way.

All right. Thank you. Our next question, one of the challenges in low-, middle-income countries was documentation maybe inconsistent or limited by the setting of care. What is your opinion on the possible ways to introduce and develop AI for drug developments or clinical trials in this limited setting?

So the low- and middle-income countries was documentation -- so I suspect that you're suggesting that there's not good documents to teach and train things on. But the reality is that the a lot of the lower-income thing really lean on some of the information from the bigger regulators. And so trying to lean on some of the materials and experience and the data and the stuff for EMA submissions and FDA submissions and some of the Asian regulators that just have a much stronger footprint and have good materials in place, I think that's the best you can do. Obviously, if there's lower-income countries that just don't have a whole corpus of documents that you can use to teach a system to write more documents, you need to lean on something else. And I would say if that's -- if I'm interpreting the question correctly, if it's more around this sort of AI in drug development in general, I do see a lot of things out there. CEPI is trying to push on this. There's quite a lot of organizations pushing on the use of AI to make things a little bit more accessible in the lower-income countries. And that's a pretty broad thing that I've noticed and a lot of grants and opportunities out there.

Okay, great. And I think that will just about do it for the questions we have. So with that, we will wrap things up. I would like to thank the audience for attending and for participating in today's event. I would also like to thank our sponsor, IQVIA, for making today's webcast possible. We would like to ask everyone in the audience to participate in a brief survey. And this survey will appear on your screen after today's presentation has ended. Just a reminder that this webcast will be available on demand for replay, and you will receive an e-mail alerting you when it is available for replay. We invite you to forward that announcement to your colleagues who may have missed today's live event. We hope to see you all next time. Goodbye.