IQVIA Holdings Inc. (IQV) Earnings Call Transcript & Summary

Hello, everyone. Welcome to today's live broadcast, Applied Adaptive Design Using Subgroup Identification and Machine Learning. I'm Lisa Henderson, the Editorial Director of Applied Clinical Trials, and I'll be your moderator for today's event. We are pleased to bringing this webcast presented by Applied Clinical Trials and sponsored by IQVIA. I would now like to share a statement from our sponsor. IQVIA is a leading global provider of advanced analytics, technology solutions and clinical research services to the life sciences industry. IQVIA creates intelligent connections to deliver powerful insights with speed and agility, enabling customers to accelerate the clinical development and commercialization of innovative medical treatments that improve health care outcomes for patients. With approximately 82,000 employees, IQVIA conducts operations in more than 100 countries, and you can learn more at www.iqvia.com. We have a few important announcements before we begin. This webcast is designed to be interactive, and we encourage you to ask questions during the event. [Operator Instructions] And also, our speakers have prepared a few interactive polling questions throughout our presentation. So we thank you in advance for your participation. We will be sharing those results of each of the polls afterward to supplement today's discussion. And now I'd like to introduce today's speakers. We are pleased to be joined today by Kal Chaudhuri, Eric Groves and Michelle Krukas-Hampe. Kal currently leads SOMS & ITD products for IQVIA. His experience is in artificial intelligence and machine learning or AI/ML and new product development and global launch. His specialty is in integrating clinical trial and real-world data sources to discover insights using AI ML platforms and algorithms. These methodologies can be used for disease detection and personalized treatment. Dr. Groves is Board Certified in oncology and internal medicine with 32 years of experience in drug development as Corporate Officer, Senior Manager, Clinician and Researcher. Noteworthy are his accomplishments in clinical and preclinical development, research and general management for agents with oncologic, infectious disease, immunologic, dermatologic and pain therapy applications. And Michelle is a biostatistician with almost 2 decades of experience in clinical trials and real-world data. Michelle has executed patient-reported outcomes, analysis, statistical and clinical trial software development and clinical trial design. She has experience with traditional and adaptive clinical trial design. She has worked on projects with several large and small pharma companies on Phase I through III trials in a number of indications, including rheumatoid arthritis, macular degeneration, pediatric major depressive disorder and oncology. So thank you, everyone, for joining us today. And Kal, would you like to get us started?

Thank you, Lisa. Yes. Let's get started with today's webinar. And in terms of today's agenda, we will look at very hands-on case studies, and we will refer to these case studies and discuss how you can apply both AI/ML and trial design methodologists and best-in-class AI/ML and trial design methodologies for practical trial design and execution. So Lisa already introduced the speakers today. So I will be the first one speaking. Before we get started, perhaps we can start with a poll question and then we can jump into today's presentation.

Excellent. So let's get to that poll question, audience. You should see that on your screen. Is your organization currently using or planning to use AI/ML technologies and genomics for patient segmentation for effectiveness, efficacy, adverse events, et cetera, on real-world data and/or clinical trial data. And your choices are -- and you can choose one. Yes, we have an external vendor doing the analysis. Yes, we have an in-house team doing the analysis. We are currently evaluating if we should do patient segmentation analysis. We're not sure if it will help us or other. So I'm going to ask that again, just so you have some time to answer. We'd like all your participation would be great. Is your organization currently using or planning to use AI/ML technologies and genomics for patient segmentation for either trial efficacy, effectiveness, adverse events, et cetera, on real-world data or clinical trial data. Yes, you have an external vendor. Yes, you have an in-house team. You're evaluating if you should approach patient segmentation analysis. You're not sure it will help or other. So let's take a look at those results.

I think we have enough responders. Maybe we can close the poll questions in like 10 seconds.

Sounds good, Kal. So it looks like the bulk are not sure if it will help us. They do -- some do have an in-house team. Some are evaluating if they should be doing patient segmentation analysis. And the other large bulk is other. So Kal, what do you think about those results?

Yes, this is what usually we see in our experience. So AI/ML and trial design and innovative trial designs are new fields and people are not sure if these new methodologies are going to help them. And that's why we have scheduled this webinar, and that's why we have identified a practical hands-on case study and we will take deeper -- deep dive into it. But we also have a really good participation in terms of yes, we have an in-house team doing the analysis, that's 24%, and that's very significant because this is what we are seeing is trending in the industry. So there are people in different levels of maturity and we see these trends overall. So let's -- with that context, let's jump into the presentation. So this is me, as Lisa introduced me I don't think I can do a better job than Lisa, so we'll leave it at that and jump into the slide. So what we will cover today are 5 steps. I will cover A, B and C and Michelle and Eric are going to cover D and E. So what does A, B and C entail? So when we have a trial, supposed a Phase III trial being designed and Phase III trial being drawn, you already have potentially a Phase II trial already completed. And this is a trial run for the Phase III study. And for the Phase II trial, there are data available, we can use that Phase II trial to the maximum effect and use that Phase II trial to help design the Phase III, and we will take a look at it in a deeper way. So going to the next slide. What is this case study we are going to talk about. We are going to refer to a Phase III design of non-small cell lung cancer. And we will assume that there is a prior Phase II study for this non-small cell lung cancer study, and we will tweak the patient inclusion/exclusion criteria on that. And we will assume that this is a Phase III study, which would be a global study and we will look at how we can tweak and add just a number of patients and time line and there is trade-offs for it, and we will look at the trade-offs for it. So this is the agenda and also the case study context. As I mentioned, I will be covering the first 3. Michelle and Eric are going to cover the next 2. So generally, what happens is when a study is done, for example, the Phase II study we were referring to, people analyze it in a traditional statistical fashion. And in a traditional statistical way, the drawbacks are you have to predefine the subgroups or the patient subgroups you are looking for. And in this representative example, and I have referred this paper down below with a link, and this is very common. You have to find out predefined what subgroup you are looking for? Is it less than 60, greater than 60? What happens if the cut point is not at 60? So then you will not be able to find it. The next thing is if there is a pathological implication on that and you don't know what the cut point is, then you will not find it. You are only going to do a predefined analysis. And that also does a confirmation bias. So with the traditional statistical analysis, you don't find a specific cut point. On top of that, you are not able to do a multivariate analysis. The need for multivariate analysis is if there is an interplay between multiple different patient attributes. For example, there is an interplay between a genetic mutation and a blood biomarker. So those kinds of interplay is not generally uncovered in the traditional statistical analysis. So let's -- we have talked about the problems with traditional statistical analysis. So let's look at the solution for it. And the solution lies in AI/ML-based analysis. And what AI/ML based analysis does, it goes through all the different cut points and evaluate each of the cut point for its predictive value. And if you try to do this manually, it's almost impossible to do that because at each cut point, let's say, there are 50 different cut points that you have to evaluate and that becomes very, very cumbersome. And on top of that, if you do a multivariate analysis, you then have to look at each of the cut point for the other variable and a permutation/combination of the cut points. And very quickly, the combinations become astronomically very high. So that's why we propose that use of AI/ML technologies can help in this regard. So here in this representative or illustrative example, suppose 63 is the age and not 60 is the age. So we can find those cut points and say that, okay, greater than 63 has a high efficacy. Similarly, if there is a particular pathology, and this could be a categorical variable, particular pathology that is predictive, we can then identify that even if it wasn't preidentified earlier. So these variables are not preidentified earlier before the trial started. In addition to that, for multivariate analysis that I was saying there could be multiple different mutations and combination of stage, where the efficacy could be low, and it's important to find the low efficacy of a study. The reason to find out the low efficacy is then going forward with the Phase III study. You can remove those patients from the study going forward. So put those criteria as the exclusion criteria. So now you have a study where you were including the patients who either responded very high or who had responded and excluding the patients who did not respond at all. So by doing so, you are having a future study where you have very high probability of trial success. And excluding the patients who did not respond to treatment not only gives you the widest possible label, it also is ethical because then you are not including the patients who have no chance of not responding to treatment. So these are the benefits of using AI/ML for setting the patient inclusion/exclusion criteria. So now that you have set the patient inclusion/exclusion criteria, what's next? So then you may want to simulate the future study and then figure out whether it's going to be successful, what will be the sample size needed and the other things. So what can be done again using AI/ML algorithms is to use -- is to generate synthetic or in silico data. So in silico data, we can't generate out of thin air. We have to base it on some real patient data. So this real patient data will come from the Phase II study that you have already done. Use this Phase II data to build a model, model of covariant structure, model of interplay between different patient attributes and use this model to create synthetic data for a Phase III study. And it is a good practice to use multiple different models, build multiple different models from the Phase II study and create the synthetic data using these multiple different models so that the advantages and disadvantages of different models can be leveraged. So that way, you are now sure before you start a multimillion-dollar study that your study is going to be successful. So with that, let me go to the next poll question, and feel free to type in your questions on the chat window. So let me go to the next poll question and hand it over to Michelle to take you through the next part of today's presentation.

Excellent. Thanks, Kal. So I think the next survey is up and audience, the question is, does your organization currently use trial design software for clinical trial design, and you can select all that apply. So again, does your organization currently use trial design software for clinical trial design. Yes, we have in-house software. Yes, we use East software Yes, we use nQuery software. Yes, we use ADDPLAN software. Yes, we use IQVIA trial designer software. Yes, we use publicly available software. We are currently evaluating if we should use trial design software or other. So again, audience, you can choose as many of those as are applicable to you. And here we go. So it looks like you're all still answering. Is Michelle going to speak these responses or...

Yes.

Excellent. Okay, Michelle. So the responses are still coming in.

Yes. So let's give it another 10 more seconds.

And again, audience, you can choose as many as you can that apply to you. All right. Michelle, do you want to take it over. It looks like most are evaluating. No, that's not true. There's a division among in-house, East and currently evaluating are in the top 3.

Yes.

Yes.

Yes. And I think that's to be expected, a lot of -- this is a newer area, and a lot of people have been -- a lot of companies have been trying to figure out what the best solution is for them as a company. And so it's very organic market in the in-house software piece and as well as East, which has been around for a very long time. Okay. right. So my name is Michelle Krukas-Hampe. Lisa introduced me earlier. So -- as Kal said, I'm not really going to be able to do better than her. So I'm just going to move on to my presentation. So here, we're going to talk a little bit about study design. I'm not going to focus exclusively on Phase III, but I think we're definitely going to look at a little bit of all the phases. So let's start at the beginning. What is an adaptive design. The FDA defines it as a clinical trial design that allows for prospectively planned modification to one or more aspects of the design based on an accumulation of data from the subjects in the trial. So why would we want to consider a design like this? These designs can reduce study time. They can lead to faster decisions, improved patient safety, lower drug development costs. So one thing to keep in mind, even though there is reduced study time, you do have a longer upfront planning component to it. But these are all benefits that we all look for when we're designing a trial. The faster we can get to market, the safe -- keeping in mind the safety of the patients, being able to make those decisions faster. These are all components that we would like in our studies. So what are some adaptive design methodologies. I pulled these mainly from the FDA guidance. So you could definitely look into that for further details. But you have group sequential designs, you have adoption to sample size. So that would be your sample size or estimation scenarios. And there's a wide variety adaptive enrichment that you can really do keep in mind and change in your designing of the clinical trial. So trial design, objectives and methodologies by trial phase. So adaptive designs are really catering to each phase. So you're not going to use one type of design for every phase, just like you wouldn't do the same type of design in traditional trial design. So in Phase I, you would look into BOIN or 3 Plus 3, these are helpful in terms of evaluating the maximum tolerated dose, minimal effective dose, PK. So this is your Phase I piece. When you're moving into the Phase II -- Phase Ib to II, this is where your MCP-Mod, your patient will come into play. So this would be group sequential enrichment, those kind of things. In Phase IIb to III, you're going to be using group sequential sample size reestimation. So all of these methodologies are unique to where you are in your development. So let's look a little bit at dose escalation. This is where you would be looking in a lot of cases, maximum-tolerated dose. More recently, the FDA has been moving towards the minimally effective dose. So you're going to see that a lot more. And in terms of what we have internally at IQVIA, our software does have ability to do clinical trial design in 3 Plus 3, M Plus N and so on, you'll see the list on this slide. When you're doing a dose escalation design, we can make custom curves in our -- in the IQVIA software, so that's definitely an option to keep in mind. In terms of finding the optimal dose, you're going to see examples of this, especially in oncology. I'm pulling more from oncology in this particular case. So for example, the diagram in the middle would be more of a biospecific, your efficacy, which is the purple line, seems to peak at a certain point. And it really isn't your benefit to go past that point because the toxicity goes up. So you have a reduction in efficacy and increase in toxicity. So you really want to find that particular point. When you're dealing with ADC, like the plot on the right, it's more of an ADC, so you want to make sure you optimize efficacy and minimize toxicity. And so you're trying to find the point where you've reached a certain threshold of efficacy, but you are not putting the patient into a danger of too much toxicity. So you can see that in that green area. So the next aspect to try to keep in mind is when you're in a situation where efficacy size is not well defined. So you have your conservative approach, which would be you design for a large trial and then you have an interim analysis where you can evaluate for efficacy or futility. A more agile approach where you would start with a sample size and then at some point do sample size reestimation. At that point, you may be able to stop if you have very high efficacy. But it's a more agile approach to trial design. And then you have enrichment where you can do a trial and then be able to use software like what Kal had mentioned earlier, SOMS, to be able to identify subpopulations that can benefit further with treatment. So this is going into a little more detail about sample size reestimation. It's -- this is much more in that agile category where you would start with a sample size of a limited amount that you would calculate ahead of time and at the point where you would complete that initial sample size you would be able to -- or you hit an interim analysis on that particular sample, you would be able to evaluate do I have enough sample size. You have a number of ways to approach this. You have the blinded and the unblinded sample size reestimation. Blinded obviously has its own challenges because you need to make sure that operationally, you have a blinded group of people and this information doesn't come out or it could end up having some operational challenges. And you wouldn't stop recruiting during the sample size reestimation, so that you wouldn't have operational bias as a potential issue. And on the bottom, you could see a few methodologies in the trial software that we have here at IQVIA that would be available for use. And then on that note, trial design requires a conversation. It's not just a statistician, it's a team of clinicians and a lot of contributors. And one of our focuses with the trial design software here at IQVIA has been to try to create a format where people could collaborate more easily. So those conversations can stay fluid. So -- and with that, we'll go to poll question #3.

Excellent. Thank you, Michelle. So audience, your question for our next poll. Does your organization currently use any software for patient accrual/recruitment modeling. And again, you can select all that apply. Yes, we have in-house software. Yes, we use an IQVIA software. Yes, we use an Excel tool. Yes, we use publicly available software. No, we have outsourced that function. No, we rely on our past experience. We are currently evaluating if we should use software or other. So let me go to your response as they're coming in. So does your organization currently use any software for patient recruitment/accrual modeling and you can select all that apply. And I can see that you're still answering, so let's give you about 10 more seconds for that. My panel, who is speaking to this one. I'm not sure, sorry about that.

This is Eric Groves. So I'll speak to it. This sounds as though it's kind of current, this is an activity that's frequently outsourced through a CRO who have potentially access to unique data, and they will provide the feasibility assessments for organizations so that, I think, matches option E, we outsource this function. But I think at the time of design, this is somehow an important parameter to put in place. So let's go to the slides here now. And I suppose I'm supposed to introduce myself. I'm Eric Groves. So we seem to not -- you have to release something. So I'm going to talk a little bit about accrual and accrual modeling. So let's take a look at this particular problem. So if you remember from Kal's example, he was talking about Phase III study, and he was talking specifically about what to do to improve the quality of the patients and the value of the patients and the potential for patient benefit from the participants in the study. When you do this, you're obviously treading into new territory. And there is a situation where now you're no longer quite so certain of the effect size that you're going to see. You think that you materially improve that effect size but in fact, you may have wandered a little bit astray. And this is the time when Michelle's comments about sample size reestimation, further efforts to do patient enrichment, all those things come into play and are really essential. So as she said, you can say, oh, I'm going to do it really big. And then we'll prune the study down as with an interim analysis and say, declared that we won. The other way is to start and go the other direction. We're going to say we're going to be optimistic and say we can work with a small study and then perhaps increase this. So integral in all of these different approaches is the issue about how are the patients going to be entered, how can you find them? What are you going to do with them? And Well, you're -- as a designer, you want to be able to work this through. So it's nice to have a CRO to give you feasibility information, but they don't always answer the phone exactly when you call them. So you'd like to be able to do this yourself. And for this reason, our software has provisioned for allowing you to do models of the accrual pattern that you might see, and then you can feed that back into your design. So on the left-hand side of this particular graph, we have a standard set of accrual curves here. In the first one here where I'm moving my arrow, you can see the accrual actually with the median accrual in the purple and then confidence intervals about that and then a little blue swath here represents individual scenarios where someone has done an actual simulation for a single event. And on the below here, you see the efficacy and outcome accumulation over time. And again, there's the median effect and there's confidence intervals here. And these are usually mimicked by doing some sort of Poisson simulation or gamma site distribution. Our IQVIA internal software has provisioned for these things. We can allow you to do time-varying or country-varying recruitments. We can vary the hazard rates. We can vary the hazard rates by subgroups, and then also you need to account for various potential dropouts. This is all in software tables and so on. So let's just go to the next slide here and see. This is an example of how you might start to put in the information for individual patient groups here. You'll be able to enter in the characteristics of the sites. This includes the start-up time, the expected accrual time at various time points, so piecewise accrual rates. And then you can also -- for the overall program, you can put in piecewise hazard rates, if this is a time-to-event start of an analysis. Here, you can see we've picked 80 sites, 7 subgroups. We've sort of hallmarked in those different places. U.S.and so on and South Korea. Each of the sites has a number of centers that are involved in them and so on, and then you can go on figuring the next set of tables, this next information. But this sort of thing allows you to say the sites in one country are more aggressive than others and now we have a higher accrual rate. And in other countries, it will be weaker. If you want to just do this in a very simple fashion, you can just pick one grand site that accounts for everybody and model it that way. But usually, it helps to be a bit more sophisticated and gives you a better sense for what the problems are with your program. So let's just come on to the next slide here. And here is an example of an output from this particular software package. And this is a simulation. It involves at some level, Monte Carlo and other approaches to get to the right answer or the best estimate here. And you can see here in this particular arrangement, we have the accrual and this is the median accrual here. And these outer curves here are the confidence intervals and you can see that actually, although the study is suppose to sort of be completed in terms of its accrual around 13.5 months, actually is quite a variance that you might expect to get here. And so with this sort of thing, you should actually begin to think about how you're going to allocate your resources, what you're going to do, how are you going to tell the senior management that it's late or it's early. All of these things allow you to decide whether you want to have more sites, you want to change the distribution of sites to improve things. You can see about -- think about costs and you can plan your research utilization. Then with all this, there is, of course, the accumulation of your primary endpoint and this is indicated here. The purple curve of the median and again there's confidence intervals on that. And we have indicated here where the interim analysis would be. Here is an interim analysis 2. Here's the interim analysis #1. And as you run the study, you can discover, of course, that things are not exactly as you had originally planned. And so you can adjust the site activation and the site distribution in order to improve the outcome of the trial and to keep it on track, and you can adjust the various aspects of the utilization. So we've sort of put this all together in a grand sort of overall high view of this. So here, we have a design that you might do. So you and -- as the trial designer can take all of these tools and model this. But then, of course, as Michelle has indicated, you can work with the software packages that we have or in your various teams to improve your software -- to improve your design and get it to look like it's going to produce the results that you wish to see. With this information, you can then, of course, go up and begin your trial execution, you can run this internally or you can run this out through a CRO. And way you go, the start-up will happen on time. The contracts are still delayed, you need to reorganize what's going on. You need to replan. That's what the adaptive/accrual evaluation is for. That's when you want to look at your original model here and reconnect it, account for the changes that you've experienced and begin to replan how it is, begin to put in more sites, recruit the sites from the country that you weren't going to do -- use. Now you have to put them in to get them on rolling -- get the whole thing rolling on time. So that's the way we kind of think about how to do all of this. And we find that this approach works very well. It's obviously a team activity but giving capacity to the trial designer to be able to not only think through the original design then to model statistical properties of the design and then finally to model the accrual properties that we design leads to an excellent, best approach design, and that's the way we like to see it done. We're happy to talk with you more about how to work all this together. And I think we have a set of poll questions now as we go here next.

Excellent. Yes. Thank you, Dr. Groves. So audience, as a follow up to what more you need to learn from IQVIA as a follow-up to the webinar, what would you like to know more about, and you can select all that apply again. Is it the SOMS AI/ML-based analysis for better design, patient inclusion and exclusion, synthetic trial data generation, is it a clinical trial design for Phase I/II and/or III trials. Patient -- I can't talk today, sorry, patient accrual/recruitment modeling for Phase I/II/III, adaptive accrual recruitment modeling for in-flight trials. Demo of the software tools discussed in this webinar, including SOMS & ITD. All the above. Or other. So let me go and see how you are responding to our question. I don't see it. I know why, because we're collecting it on the back end. We're not going to talk about it. So audience, just put that information in there for us and then we can roll it into our Q&A.

So thank you again for Michelle, Eric and Kal, for your excellent presentations. [Operator Instructions] For our first question, I'm going to go to Kal. What kind of sample size do you need to perform AI/ML analysis of trial data?

Have we lost Kal?

Did we lose Kal. Okay. So let's -- you want to answer that?

Yes. Let me have a go. So I think the intent here is obviously use a Phase II trial data to build this. You can also use data from other situations to the degree to which you have it available. It may be that you have a CRO that they have data that you can use to supplement things. But if you have a reasonable size Phase II in the order of 50 to 100 patients, I think you can make very good progress with this particular software package. If you're in the cardiovascular space, and studies are much, much larger, then obviously that will improve things. If you're in a situation where the results are anecdotal like the CAR-T study, then I think it becomes more problematic trying do that. So I think let's go to the next question.

Okay. Excellent. So how is it possible to estimate the enrollment rate, patients' months for every center considering particular disease subgroups, responders on the second-line treatment, and you can also use real-world data.

Well, this is the perpetual problem. So where are the secret patients that you want to recruit? And how often are they going to be available? And are they going to really listen to your desire to have them participate in a clinical trial. So we all work this through on the basis of experience and then we try and consult with various feasibility groups to improve our quality of assessments here, and we also look at -- talk to KOLs. So this is a bit of an empiric process in each case. The software allows you to put in the information, your prejudices and you can see what happens. But you can also put in your optimistic prejudices and your pessimistic prejudices and see how things fit together. So that's the approach that we usually use. What we find it's nice to be able to do these calculations yourself without necessarily involving a third-party so you can make optimistic, pessimistic calculations. That's why we'd like to have the accrual capability within our software package, and that also allows us to make estimates about when it is we're going to see the primary endpoint roll in or we talk to the senior management, and you can say, okay, we think this is a 3-year study. We're going to know the results on the interim somewhere around halfway through. Here's our plan. So you can do those for calculations without necessarily involving a third party that's very beneficial. Next question?

Okay. So there's a question here for Michelle, so I'm going to ask that one. Can you compare and contrast between multiple trial designs and pick the best design?

That's a bit of a Pandora's box. Every situation is really unique. And we have a team of statisticians here at IQVIA, who do this all the time. So it's not something that I would be able to get into a lot of detail in a forum like this, but please feel free to reach out, and we can definitely have a conversation to cater to your specific needs.

Excellent. So Kal is back. Kal, we have a question for you. How the patient accrual and recruitment simulation is different from using overall recruitment rates, hazard rates, treatment response rates, et cetera.

I can have a go at this, Kal, if you'd like.

Go for it, Eric.

So the advantage of the simulation is distinct from a simple, straightforward arithmetic is the straightforward arithmetic is a crude simulation. So we have a more sophisticated simulation that works better, that allows you to get some sort of variability in the outcomes and allows you to get some sense of the confidence that you have that the study will be completed at the time you estimate. So by using these simulation approaches, you get confidence about the potential variable durations of the study that makes a big difference in making your planning. So that's where the payoff comes is being able to estimate the variance on it. It's not just always going to end on the 13th of May. It actually may end up in June. It may end up in March. You better be prepared to handle that. Okay, Kal, next question. We've already addressed this one here about Phase II data for the simulations. I think that we probably need to move on.

Okay. So you want to answer. Are you considering recruitment initiatives like patient...

Okay, yes. The slides -- okay, can you -- so what is the question? Here we are. Okay.

Are you -- so I read it verbally, Dr. Groves just said that...

Okay. This is not the case but definitely...

Okay. Are you considering recruitment initiatives like patient servicing tools or only considering a function of sites and countries to recruit patients?

So exactly how you make the estimates of your accrual rate depends upon your experience and the tools that you have available. All companies that do business with IQVIA know that IQVIA uses all the available tools that you can find out to help identify where the patients are and to recruit and all these tools are used for -- by IQVIA. I think other CROs are in a similar position. So I think these are all put together. But the designer needs to think optimistically or pessimistically as they may find best to work out with a good estimate on the accrual.

Excellent. And then you said -- so this question, considering the small sample size, how reliable is it to use Phase II data to identify subgroup cutoff and applying Phase III simulation real trial. Did you say that we already answered that?

Yes. So I think it's -- Kal, I was saying that the data are available, but we -- you need to plan, as Michelle has -- like what to do if your effect size is not exactly as you anticipate. So these predictions are not perfect. And so we try and make the design -- the final adaptive design, able to accommodate the variability that may come when you do these projections. But SOMS does a really good job.

Yes. So I think what they are also asking is the methodology that we use because there are different methodologies and there are pros and cons of different methodologies. So if you are using a deep learning methodology, then you will need a lot of sample sizes to do deep learning on that. What we do is more of a predictive analytics and this is fine-tuned for clinical trial data. And we have a customer implementation within IQVIA of that. And since we have fine-tuned it for clinical trials, especially Phase II clinical trials, we are able to accommodate lower sample size. The typical sample size we look for is between 60 and 75, and we can even find subgroup size, which is about 10. So we have a different customized algorithm, or proprietary customized algorithm that is fine-tuned to analyze clinical trial data. And we don't use deep learning for this. And you are right that if you use deep learning for the sample size from a Phase II trial is not going to be enough. So yes, so that's the answer. And we also do is in order to simulate a future trial using Phase II trial data, we expect a minimum sample size of 10. Again, 10 is the number that we use. Typically, we like 30, both for subgroup as well as for simulation. But if there is 30 subgroup size, which is not available, then we can go as low as 10. So -- and feel free to reach out, and we are happy to have a follow-up conversation and a deeper conversation so that we can also discuss about the methodologies we use, what are the pros and cons of these methodologies and things like that. Back to you, Lisa.

Excellent. Well, thank you. We will wrap up now. I want to thank the audience for attending and participating in today's event.

One more question, Lisa. There is one more question that came up.

Oh, I see. I'm sorry. Thank you so much. We are not wrapping, audience. The question is, would you please comment on how your -- some of your information would apply to registries and natural history studies?

I can take that question. And this is a really good question. People, when they think about subgroup analysis, they don't think about registries and natural history studies. So you have registries and in the registries, you have people on multiple different treatments, and there is a lot of complexity in that. And some people are responding to these treatments. Some people are not responding to this treatment. Who is responding to which treatment and there is a lot of confusing data, but it is possible. It is possible to analyze this data and identify in a competitive effectiveness setting as to who is responding to which treatment. And if it is, for example, in our case study, if it is a lung cancer, are people with certain genetic mutation are responding better to a particular treatment or patients with -- who do not have this genetic mutation are responding better to a particular treatment. And who is not at all responding to a particular treatment. So suppose there are 2 or 3 lines of treatments. And if a particular subgroup is not responding to the first-line treatment, why put this patient segment through a treatment that they are not going to respond to. So we can identify that, and that's a really good question. And the second part of the question is, can you do natural history of disease studies. You see a lot of different variations in oncology or any other disease areas such as ALS. The type of presentation, the severity of disease is different. The progression is different. Some people's progress is faster, some people's progress slower. So if you can segment out who is responding -- who is progressing faster, then you can have a more aggressive treatment for those subgroups. If a patient is not progressing as fast, is not expected to progress as fast, then you can take a more different approach to treatment where the side effects are lower. So these are the different approaches you can take, and you can also segment and subsegment populations for different disease progression, different treatment rates for both registries and for natural history of disease studies. And this is a really good question. People don't think about it on the real -- in the real-world setting, but this is a really good question.

Excellent. Thank you, Kal. So I'm going to wait 2 seconds in case someone else has another question. Thank you, audience. You are very attentive, and we thank you for your participation. And I want to thank Kal and Eric and Michelle for their excellent presentations. I also want to thank our sponsor, IQVIA for making today's webcast possible. Also audience, if you could participate in a brief survey that will appear on your screen after the presentation has ended. We really value your feedback. You'll also receive an e-mail alerting you when this webcast will be available for replay, and we invite you to forward that announcement to your colleagues who may have missed today's live event, and we will see you next time. Thank you again. Bye-bye.

Thank you.

Thank you.