IQVIA Holdings Inc. (IQV) Earnings Call Transcript & Summary

To Validate SaaS or Not to Validate Software as a Service, That is the Question. My name is Andy Studna, Editor of Applied Clinical Trials. And I will be your moderator for today's event. We are pleased to bring you this webcast presented by Applied Clinical Trials and sponsored by IQVIA. I would like to share a statement from our sponsor. IQVIA is a leading global provider of advanced analytics, technology solutions and clinical research services to the life sciences industry. IQVIA creates intelligent connections to deliver powerful insights with speed and agility, enabling customers to accelerate the clinical development and commercialization of innovative medical treatments that improve health care outcomes for patients. With approximately 82,000 employees, IQVIA conducts operations in more than 100 countries. You can learn more at www.iqvia.com. We have a few important announcements before we begin today. This webcast is designed to be interactive. And we encourage you to ask questions during the event. [Operator Instructions] You can enlarge the slide window by clicking on the small icon at the bottom-right corner of the media player. The slides will advance automatically during the event. And if you have any technical problems viewing or hearing this presentation, please click on the question mark help widget in the top right of your presentation window. I would now like to take a moment to introduce today's speaker. We are pleased to be joined today by Lou Pasquale. Lou Pasquale leads the client services organization supporting IQVIA's eTMF Software as a Service application and related technologies. Lou's team performs new client implementations, provides migration and integration services and delivers a variety of professional and managed services for IQVIA's eTMF customers. Lou has over 30 years of experience delivering software solutions and professional services to clients in life sciences and is a frequent speaker at industry conferences on clinical quality, compliance and related technologies. Thank you for joining us today. And Lou, if you would, please get us started.

Super thanks, Andy. So thanks for the introduction, Andy. Today, we're going to talk about SaaS validation. And by the title To Validate or Not Validate, you would think I would sprinkle some Shakespeare through this. But that's not how it turned out anyway. But we'll talk about the aspects of SaaS in validation that present themselves in the last few years. And just so you know, I'm not seeing myself on video. First, we'll talk about the history of Computer System Validation. We call that a walk down memory lane. We'll look at how we are currently as an industry validating SaaS solutions, take a look at where the future might take us and then we'll open it up for some questions. A walk down memory lane. So I started in this industry about 30 years ago. And back then, everything was on-premises, big iron solutions. So I delivered database solutions on Oracle, document management solutions on Documentum. And everything was installed at the client, in this case for me, I was with an integrator, and all of our clients were big pharma. So all these applications were installed locally. And there was a process of testing and driving those to completion to make sure they work, just like any system. Around '94, the 21 CFR 11 was coming out, right? So the FDA had put thought into mostly around e-signatures at the time but computer systems and how treating data and representing that faithfully and accurately after and to be able to consume that data. So in '94, the proposed addition was made to the Federal Register. In '97, a final rule was published. In August of '97 or thereabouts, Part 11 was enacted. And then pretty much right after that, there was mass hysteria, overreaction, dogs and cats living together. It was a whole new twist on system deployment, if you will. So nothing changed about how we built the systems. But how we tested them and how we substantiated our claims that the system did what it was supposed to do, that -- it didn't change so much as it had more guidance. And we may have been doing things like what was in 21 CFR 11. But now we had some guidance around it. So of course, everything went up in the air. And you can see what a long strange trip it's been. I'm sure during that time, a lot of folks were living on vitamin C and cocaine, right, from the Grateful Dead song. Anyway, right around '99, there was a compliance guide that helped us figure out how to interpret 21 CFR 11. A lot of overreaction, a lot of documentation was generated. And some of it good, some of it unnecessary, some of it overreaction to the regulation. And then September 2003, the industry guidance was published by the FDA. You can find these documents at the link below. So what has that done to? If we think about a normal computer system release for an enterprise system that would be on-premises in the old days, there was release planning and build activity, right? So we're going to gather the requirements, state the user requirements, build the functional requirements, turn those into system requirements, a long process to get to building that release of that product. Then we'll go through validation testing, which was new and structured because of 21 CFR 11. During that time, probably all the way back to the beginning of the release planning, there were SOPs being worked on or updated, training materials being developed for the new release or for the new product, a lot of activity there. And then we'd get around the system usage. And then that would go for 12, 18 months or something like that. And then the process would repeat itself potentially for new features and things like that to an on-premise solution. The important thing about the old days, if you will, is that timeline was driven by you, the system owner. So in the case of, at the time, Pfizer and Merck and all, they were customers of ours. And they determined when the next release was out for either new features or to address new regulations or things like that. But they were completely in control of that release cycle, when the new one would start, what would be in it and how long it would take. So if we think about that across the industry for a single solution, in this case, we have five companies doing more or less the same thing on more or less the same kind of system, be it database, document management, clinical trial management systems, something like that, all of these activities happening in place for these on-premises solutions but more or less doing the same thing, Olympian-type efforts. But more realistically, when we think about how big the industry is, there's hundreds of activities just like this, for CTMS, for study startup, for document management, for learning management systems. All of these things came under the umbrella of 21 CFR. And we were doing a lot of these validation activities on these systems. Now during the day as a system integrator, which is what I was doing at the time, it was the heyday for system integrators, right, big project budgets and a lot of work out there. But when you look at it, there was a lot of repetitive activity, right? So let's hold on to that idea for later on. So let's look at where we are with SaaS right now and what that has done for where we are with validation. So what -- first, let's talk about what is a SaaS solution. So I have a pretty good idea with this because I've been involved with IQVIA's eTMF SaaS for -- since its inception. And let's say what that is. Well, that's -- it's a method of software delivery and licensing, which software is accessed via subscription rather than bought and installed on individual computers. Basically, it changed the model, right? In the old days, everything was on-prem or on-premises. You had your own servers. You had your own databases. You had all of your own software. And everything was owned by, in this case, the sponsor in my experience, right? So that required management of the systems, user access management, all of the software development activities and testing activities, even if some of those were outsourced to vendors. For Software as a Service, that changed things a little bit, right? So now the software is installed in the cloud. And a lot of those things that used to be the responsibility of the system owner have now been completely offloaded to the SaaS vendor. So what are some of the characteristics of SaaS? Well, there's a large user installed base. If you look at our eTMF and our hundreds of thousands of users globally, that's a pretty big footprint. And there's other SaaS solutions out there with the same delivery model, right? Global reach, tens or hundreds of thousands of users, everyone using the same version of the application. These SaaS solutions are full-featured, right? They have authoring and tracking and order trails, all of the things that you would have built had you built one of these locally. And some of our customers came from a local, in this case, eTMF -- to our eTMF, a full-featured system. The use of that SaaS software is integral to some higher-level business process, right? The software doesn't necessarily solve a problem, right? In fact, sometimes it makes a problem. But it is used to augment a higher-level business process. So in the case of an eTMF, the managing and tracking of the documents that are collected in the clinical trial, that's what the software does. And then that manages -- or that relates to a higher-level process, which is the conduct of your clinical trial substantiating the activities and then being able to reproduce that trial from the documents that were collected. So that use is at a higher level than just the application itself. Centralized deployment, we talked about that. Welcome to the cloud. When we think about SaaS in this space, so 10 years ago, maybe 12 years ago, had you told me that all of life sciences' major applications would no longer be on-premises and would be in the cloud, I would have told you that you were crazy, right? Because in general, this space has been slower to adapt or adopt certain things. And putting our data out there in the cloud seemed like a daunting thing 10 years ago. Well, here we are now. And all of these major systems are deployed via SaaS. So all that data is in the cloud. The one thing that SaaS has is that you don't upgrade on your own schedule per se, right? If we think back to the earlier process, where you as the system owner did the release planning and figured out what the new features were and what the timeline was going to be and everything. When we get to SaaS, the SaaS vendor handles those activities. Now these aren't in a vacuum, right? Your SaaS vendors reach out to their customers to get input on new features. Obviously, we want to fix defects and things like that. But for the most part, the activities around a software release are driven by the timeline of the SaaS vendor. And then you move with that assessment. And now in some models in SaaS, you can stay behind a release or two. But at some point, you will move forward. And that's not always at a time of your choosing, maybe more so at or near the time of the SaaS vendor's choosing. Let's compare that to COTS, or commercial-off-the-shelf software. So for a COTS solution, it's locally installed like we said. You own the software, you own the hardware and you manage everything. Let's look at the characteristics, large user installed base, very full-featured. If you think of things like Microsoft Word or Microsoft Excel, my favorite tool of choice, very full-featured. In fact, when you look at some of the features in those things, they don't often get used. There's someone in the world who loves the mail merge feature in Microsoft Word. When you find that person, please let me know. We all used it once during graduation announcements and things like that, right? But they are very full-featured applications. Their use is integral to some higher-level business process. So this presentation, right, was done in Microsoft PowerPoint. It's not critical to the conduct of what I'm doing, but it's a tool that supports that. And in this case, it works for me. We have a decentralized deployment model for COTS solutions. So again, that's locally installed on someone's PC or laptop or server and multiple times. So along with that is all of the installation and operational qualifications that go along with that as part of 21 CFR 11 activities, right? If it didn't -- if it wasn't documented, it didn't happen. And with that decentralized deployment model, when we go back to the Herculean efforts, which those of you who were paying attention, Hercules was also an Olympian. But when we think about all those activities for all those big systems, but also for all the desktop-type things, it's the same concept with it is deployed decentrally. The thing that we have with a COTS solution though is you determine when you're going to upgrade. I think we all lived through the Windows 95, Windows NT 4.0 and those end-of-service life things. And a lot of those activities, large organizations did those as late as possible because it was a heavy lift to upgrade to those versions. But within reason, the upgrade time was chosen by the system owner, by the software owner. So if we stack those two side-by-side, what are the differences? And what are the similarities? So both have a large user installed base. Both are very full-featured. Both are integral to some higher-level business process. Here's where the differences come in: decentralized deployment for COTS, centralized deployment for SaaS and the upgrade timeline at your pace or at the vendor's pace. But in a sense, could we not think of SaaS as a COTS solution? So the idea for this topic came to me a few months ago. We had a new client going live with our eTMF. And it was a smaller customer. And they didn't have a large testing and validation team in-house for software deployment. And you see that now with smaller biotechs. But also, you see that now as sponsors and larger customers are no longer building systems in-house, they don't necessarily have a big software testing and validation and Computer System Validation team in-house. So they tend to contract that. And that was the case with this customer. So they brought in a validation consultant. And it was concerning to me, to put it mildly. So we have a really snug timeline for implementing our eTMF, so we can get customers up and running in measuring weeks, not months. And about 3 weeks into the project, I was notified that the customer had brought in an external consultant for Computer System Validation. And I thought, "Oh, there goes the timeline." It was an independent contractor. He does great work. And I've actually established a good relationship with him since then. But I was dreading the first meeting with him because I thought there, now we're going to go from a 6- to 8-week timeline to implement, and we're going to go to something like 12, maybe 26 weeks or something like that. Because our solution is large, very full-featured, lots of user requirements, which equals lots of user acceptance test scripts. And as we got to talking in the first meeting with this validation consultant, he is actually very forward-thinking and was very much in the mindset of risk-based validation. And he basically took the next step and said, "Well, this is -- this SaaS solution, this IQVIA eTMF, has a large installed base. It's been around for a number of years. There's hundreds of thousands of users. Why are we not treating this like a COTS solution?" And I almost fell out of my chair when I heard that. Because that's exactly the opposite of what I thought was going to come out of that conversation. So not to say that the customer didn't do any validation work, but they took a -- this is an off-the-shelf-type solution approach to validation. And if you think about the desktop applications like Microsoft Excel or Word or PowerPoint, maybe in the early days of that mass hysteria, when 21 CFR 11 came out, the thought process may have been, "Do we have to validate those things?" And I can only imagine -- I'm sure someone out there tried to validate Microsoft Word or Microsoft Excel. That's a pretty heavy lift and probably not time well spent. But in the early days of 21 CFR 11, there was a lot of confusion about what should and shouldn't be done. So that -- I really expected that from this validation consultant for our implementation. But he said, "No, just like we don't validate Microsoft Word and we don't validate Microsoft Excel or PowerPoint or even Adobe Acrobat, right, let's bring that thought process into this SaaS implementation of the eTMF." And as it turns out that we held the timeline and the customer is live with a validated system, and it went quite well. But I was concerned there for a minute. And that again, that's where the idea for this topic came from. So side-by-side, the release activity side-by-side. We talked about them on different slides and we're going back to it. But essentially, the same things are happening in an on-prem solution, at least planning and build. For SaaS, the same thing. The difference is you're not doing the risk, they're doing the building, system testing and the release activities. Now they are done maybe not in a vacuum, you will get draft and final release notes. You will -- as a reward for that vendor, you'll have some direct input to the SaaS versions. If you've submitted enhanced request or something through your help -- through their help desk, you'll maybe have some indirect involvement in the release planning. But more or less, the release planning is an activity of the SaaS vendor as opposed to of you, a system owner. In both cases, there is validation testing involved. So we use this recent example as maybe not a good example, but our clients or our customers for our eTMF validate with every release. So they have a number of user acceptance test scripts. And they run through those with each release. And they would be doing that even if it was an on-premise solution, right, just like the old days, maybe writing the UAT scripts and executing them. Maybe the difference with the SaaS solution is you aren't writing the user requirements. You may be adopting what the SaaS provider gives you as far as a user requirement specification and then adapt that to what you need for your user requirements. And you may also be getting user acceptance test scripts from the SaaS vendor. And if you think about this timeline, 18 months versus, let's say, every 3 to 6 months, those UAT scripts really come in handy if your SaaS vendor is providing you. Because you don't have the time for new features that you don't know about to write and preapprove your UAT scripts. So one of the activities that's different here is while you will be doing validation testing in either model with user acceptance test scripts, chances are the SaaS vendor is providing those scripts to you to either use as is or to enhance or simplify or something like that. In both cases, there's going to be updates to SOPs and training, work instructions and things like that. With that on-prem model, you had a lot of time to do that, to do your impact assessment of the new features of the -- or the improved features or something like that. Long time to get those SOPs updated and approved, work instructions related to them, training material developed and then delivered before the new release. In the SaaS model, same activities are happening in much shorter time frame. So in our case, and I imagine all SaaS vendors do something like this in the space, provide you with training materials on the new and updated features. We provide template SOPs on the major system features and how you would represent those processes in an SOP. The difference here is the timeline again. And then you get around the system usage. So when we look at these side-by-side, we can see the similarities in the activities. It's just that the time frame is vastly different. So now let's look at that longitudinally. So this is your future with SaaS validation. Each one of these green, yellow, red sets represents a release. So down here on the on-prem, we've got a release taking about 18 months and then maybe starts right in for another release. So maybe there's some lag after that release before the next release planning happens. Again, on the on-premise solution, you control that. So if you're ready for it to go right back into a new release planning after you go live, then that's what happens. In the SaaS world, that is exactly what happens, right? Releases go out, and then that release cycle begins again. But things happen much more quickly. So if we look at in that same time period, you could go through three full releases of a SaaS product. So if you think back to that validation -- those validation activities, you're doing them more often and you're doing them on a larger feature set possibly than an on-prem solution. And then it repeats again. So in, let's say, 1 and -- maybe 1 1/3 releases in the on-prem world, you might be deep into 5 releases, 4 or 5 releases of a SaaS solution. There's a lot going on there. So another thing that we see with SaaS is with these full-featured applications, and I'll use our IQVIA eTMF as an example, we provide our customers with around 145 user acceptance test scripts. So they test all of the user requirements for the eTMF, all for the major features, for the minor features, all of it. That's a fair amount of scripts. You may not use all the functionality, so you may not be executing all those scripts. But there's -- if you were going to use the full feature set, there's 145 scripts there. Hypothetically, 75 clients, 2 releases a year. If we just accept for the purposes of this discussion that every customer does every UAT script twice a year, that's 21,750 UAT script executions. And those are just the for-the-record executions and they're all doing the same thing. So 75 clients are running 145 of the same or very similar scripts and doing that twice a year. And again, that's just for-the-record executions. You'll probably have UAT dry runs. You'll probably have -- if there's a defect bound or a script there, you'll be executing that script again. So there's a lot of repetition of effort for SaaS validation when you look at the number of customers all on the same version of the application and all running these UAT scripts again and again. There's got to be a better way to do this. I'm going to stop real quick. I can't see the question panel. Did any questions come in? If not, we can move on. So let's look at the future and where things might go. So I used a Grateful Dead reference early on with the long strange trip it's been. And this would be something from maybe Guns N' Roses' Sweet Child O' Mine, where do we go now? I'm not getting a slide transition here. We can treat SaaS as an on-premise solution. And actually, some folks do that. Early on, so let's say, 7 or 8 years ago, as the industry was moving, as life science was moving to SaaS, we've had a lot of friction with our customers wanting to treat the SaaS solution as an on-premise solution, wanted us to deliver artifacts that were part of their internal software development lifecycle, SDLC, that actually we are part of ours. And that's not something that we would share with customers. So early on, a fair number of our larger customers wanted to treat our SaaS solution just like an on-prem solution. So that's not forward-thinking. And it may not be necessarily backwards-thinking, but it's definitely not forward-thinking, right? There's a lot of work associated with that approach, with an on-prem validation approach to a SaaS product. We could move into a more risk-based mindset. And that's actually where we are on the current path, whether it's an on-prem solution or a SaaS solution, in general, the regulatory authorities are more open to a risk-based validation approach. And we as consumers of the system, so our customers, are more inclined to do that because it's less resource-intensive, right? There's -- you're not testing all of the features with every release. You're testing the riskier features and the change features, but maybe you're not testing the unchanged features and things like that. So we are currently in a risk-based mindset. And as we get better at identifying what is a real risk and what isn't and how we document that and substantiate that, then we'll see risk-based validation continue to take a larger role in SaaS, right? And then again, this is the current path. And that's not going to change in the very near term. We talked earlier about how SaaS and COTS are the same. So let's just jump right straight to treat SaaS like COTS. We're not here yet. For some SaaS applications, they may be in this territory already. But maybe this idea is more treat SaaS more like or almost like a COTS. And I think we're close to that. There's a crowdsource validation that happens actually outside this industry. A lot of applications with a large user installed base that aren't as tightly regulated as those that we have here in life sciences use crowdsource testing for -- even during the development cycle, so -- and it's kind of a cool idea. And about 1.5 years ago, one of our customers mentioned that to me, "Why aren't you guys doing crowdsource validation?" And much like 10 years ago, had you asked me if -- if you had said that all of the life sciences' big applications would be deployed via SaaS, I would have told you, you were crazy. 1.5 years ago, when our customers said, "What about crowdsource validation?" I'd say, "Oh, you've got to be crazy. This is -- no one's going to go in for that. It's -- we're just not there yet." It's -- everyone is still too engaged in validating exactly how they use the system and aren't thinking of the larger community. But maybe that's because they're not seeing this aspect of, right? As a software vendor, we're -- we see this, right? We see all of our customers go through at least two releases a year with a fair amount of testing, right? Maybe they're not thinking of things from this perspective, which is all the repetition across not just one application but many. So maybe we are on the cusp of being ready for crowdsource validation. And I'll talk a little bit about that in a few minutes. But there are obvious synergies, right, with everyone on the same or nearly the same version of the application, with everyone on the same or pretty much the same release cadence. Some test vendors, you will skip a release or two. But at some point, you will be prodded to move up to one of the more recent releases. And there's a reason for that, right? That's part of the SaaS deployment model. To help control cost as a SaaS vendor, the fewer versions of an application we have out there, the less our costs are to deliver that service. So if we were to have 10 different versions of our eTMF out there, that would be 10 different installs with 10 different versions of the application and 10 different database structures and things like that. So we as SaaS vendors want to have ideally all of our customers on the same version at the same time or within one-or-so versions of each other. But for the most part, clients are doing these activities at the same time. So there has to be some release options there, right? There has to be some room for some synergies and some economies there. Where could we go after that? When I first started thinking about this, I only had four reasons or four paths on this slide. But thanks to ChatGPT, we all have -- every time we talk about anything, now we have to talk about AI or artificial intelligence and machine learning, so we have a placeholder here because you were going to ask. And actually, we do have some AI bolt-ons -- or not bolt-on, we had a customer who has an AI bolt-on to our eTMF. And we're building that into our eTMF with our next release. So artificial intelligence will be part of our application. But artificial intelligence, I don't think is going to be doing any validation for you. But the questions are going to come, so I'm hopefully prepared to answer those. So this is where I would like to think that we could be. I don't think we're there right away. But I think we're moving in that direction. But what can help with that? So let's take a look at crowdsource validation. Again, with that model of everyone doing the same thing at roughly at the same time, I think it can work. In fact, it does work. And we've done it here at IQVIA. So why will it work? Everyone is on the same basic release of a configurable application. Like these SaaS applications, the deployment model is not to customize those for an individual customer. That's not the SaaS model. Because then you have different software, not just different versions. So our eTMF, highly configurable, all of our clients use the same version of the application. But they just may use some features or not or they may use some features in a slightly different way, right? This space is ideally suited for a configurable model since the [indiscernible] now, reference model for the documents in the eTMF are a pretty good path on what to build the system around, right? So our customers have a highly configurable system. And they may not be using them in exactly the same way, but they're using the same software. So system is configurable, so the test scripts should be able to support that. From the numbers that we saw, 21,750 scripts or something like that, very similar or identical validation testing. A lot of our customers use our provided user acceptance test scripts as they're given. Sometimes they'll simplify them. And in their risk-based approaches, they may choose not to test certain aspects of the system because they don't use them, so they'll test that you can't get those features or these features don't create data. They are simple navigation features. If there's more than one way to navigate, there's -- that's a good risk mitigation approach to testing is maybe skip those simple navigation scripts. The other way that crowdsource validation works is that crowd has to be all in the same room, if you will. So right now, SaaS customers have a -- their own tenet, if you will, that's for testing. And they'll do their validations in that tenet and then they'll upgrade to production. But they're doing testing on their tenet and on their tenet and on their tenet, running all those scripts on their -- on the same version of the software but on slightly different configurations. So for crowdsource validation to work, we want all of those activities to happen in the same tightly controlled environment that all of the participants in the crowd contribute to the consumer. And then the other thing that would help crowdsource validation work is really strong data setup. So there's a couple of components to user acceptance testing or any kind of testing, if you will. There's the tester and how well they follow the test script. So let's just say that the tester knows what they're doing and they follow their test script to the letter. There's the test script itself, is the script correct or if the script itself has errors in it. And then there's the data setup. If the second two are taken care of, if the tester does exactly what's on the script and the script is exactly right, but the data setup is wrong, then you don't have good results, right? And something may look like a defect, and it's just a consequence of bad data setup. So the other thing that would help crowdsource validation take hold is rock-solid data setup, meaning set up by system subject matter experts with -- who also have deep validation experience. What are the benefits to that? Reduced resourcing for release. You can leverage the efforts of others, right? So in a model of -- let's just make the math simple, 4 customers and 100 UAT scripts. If in a complete crowdsource model, where all of the scripts are covered, each participant in that model would execute 25, 1/4 of those UAT scripts. And for that effort, for that investment in the crowdsource bank, they'll be able to get paid back 3x that. So they'll get 75 scripts in return. So what does that mean? Well, these are executed in a crowdsource environment. The objective evidence is the crowdsource environment. The scripts are preapproved and post-approved in a crowdsource environment. And the SOPs, the control access and the use of that system are very tight. So what does that do? It lets you treat that crowdsource activity and those artifacts as something that you would have produced or could have produced had you done it yourself. And that's the -- I guess, the strength of that is it needs to be as good or better than you would produce yourself. Now how could it be better? Well, maybe we weren't going to do all the scripts. But now we can -- we, as a customer, weren't going to do all 100 scripts. But now for the price of 25, the ones that we weren't going to do, we can -- instead of cover those with a risk-based assumption that nothing has changed and everything works in those areas, we now have concrete evidence that those scripts passed and those features worked. So those -- that's the big benefit of the crowdsource model. And that allows you for quicker testing timeline. Now this is no secret. I represent a SaaS vendor, and we want our customers to be able to upgrade quickly and have good results with our system. But we also want to push new releases faster. So if we move from a 6-month to a 3-month release cycle, that basically double -- potentially doubles the validation effort for our customers. Now with smaller releases, less will have changed. So maybe there's more -- more regression testing and less new feature testing. But still, with a quarterly release cycle, that helps us get defects fixed, new features out there and have the product moving forward. But you as a customer of that, this is what you feel. You feel release cycle, release cycle, release cycle. Maybe it feels like you're constantly in a validation cycle for the next release. And that can be a lot to take on. So the crowdsource validation model helps mitigate that a little bit. Now this isn't just an idea. This is actually -- we put this in practice. So for our eTMF with our last release, and it will continue with this next release, we opened up our crowdsource validation, we call it a bank sometimes or the process. So we spent some time thinking about what processes are going to make this, what it needs to be for our customers. We need -- in order for the model to work, the participants in the crowdsource need to be able to trust the results that are done by someone else. And so we put a lot of structure around with SOPs, with very controlled access to the environment. We treat it -- even though it's in a testing infrastructure, we treat it like a production environment with change control and change management and everything. We opened up the crowdsource environment to our customers with our last release. And we had some buy-in. And we hope to have more buy-in with upcoming releases. So the model is out there. It's working for us and our customers. And we're taking a leadership position in this for life sciences. I'm meeting with a colleague of mine from a -- who represents another product, who wants to do the same thing for their products, to adopt a crowdsource validation model. So it is part of the future. It is out there and working for us. And it does doesn't necessarily mean that we are now 100% beating SaaS-like COTS. But it does help us along the validation process to have things happen more quickly. And for how AI will impact validation, stay tuned. So I don't -- so when I think about artificial intelligence, I read something the other day, someone was sick for a year and the doctors couldn't get a good diagnosis. And then artificial intelligence got an accurate diagnosis in a matter of hours or something just by consuming information and making that happen. I think for things in the public domain, that model can work. I don't think artificial intelligence will change drastically how we validate our product or SaaS products in general. I think there will be some changes. But I don't see -- it's not going to be writing UAT scripts for you or executing UAT scripts for you. I think while that is very forward-thinking, I think the crowdsource model is a way to move into the future, if you will, and decrease our validation footprint. But I just don't see artificial intelligence as the save-all or cure-all for our SaaS validation. And now I would like to open it up for questions.

All right. Great. Thank you, Lou, for that presentation. Before we get started on the Q&A session, I would like to remind our audience how they can submit questions. [Operator Instructions] And we will get started with our first question. What happens if my risk-based validation approach misses something?

So good question. And that question applies whether we're talking COTS validation, on-prem solution or SaaS validation. What happens if you miss something? Well, things happen. So your risk-based approach to testing, let's say, there is a release where there's five new features, two features that didn't change at all and two features have changed significantly. If your risk-based approaches were not going to test the features that didn't change, we will regression test the features that changed a little bit and we will test the new features, right? And all of a sudden, there's a defect in one of the features that didn't change and you didn't test it. Well, yes, you missed that. But we don't validate by accident, right, and especially now with the risk-based approach. We provide actually our customers with each release, we give them an impact assessment tool we call a risk analysis tool to let them know what is new and what has changed in the release, so they can make a more informed risk-based decision on the release. Again, back to the old on-prem solution, you had a lot of time in that software development lifecycle to know what was coming out in the release and a long time to then make some determination of the risk-based validation approach you would take. But in a SaaS solution, everything is a lot shorter. So if you miss something, well, was it in your risk plan? Was it in your validation plan? Okay, we -- based on the release notes from the vendor, these features weren't changed. So it was our -- according to our risk profile for this release, we decided not to test those. And the defect came up in those. So I think that's how you would cover that situation. You're going to miss things. With applications this large, if you're not executing all the UAT scripts, if you miss something -- as long as you accounted for it in the risk plan, I think then you're good to go. Your risk rating or your risk analysis may be a little off, and maybe for the next release, you tighten it up a little bit. But you will -- you'll miss a defect, right? But if you think then again to -- let's think to that crowdsource model. Okay, instead of not testing those unchanged features, while those scripts are executed and available with objective evidence in the crowdsource model, I'll consume those. And now I can take that off the table. So maybe I would have then caught that defect, it was caught in the crowdsource UAT execution. And then I take that forward and watch for that defect to get fixed.

Great. Thank you. On to our next question, if when you are using BYOD for a trial, how do you conduct a validation for your EDC systems?

I'm sorry, what was that one more time?

Yes. If or when you are using BYOD for a trial, how do you conduct a validation for your EDC systems?

So that's a bring your own device. So that's a -- I'd like to take that one offline, actually. There's a -- BYOD is a different model, if you will. So I'd like to take that one offline.

Okay. And then moving on to our next question, if or when external auditors question system lifecycle documentation, what do the customers provide?

I'm sorry, one more time, Andy?

If or when external auditors question system lifecycle documentation, what do the customers provide?

Okay. So you as -- so in that case, something you, a customer -- you're, let's say, a life sciences company and you are using a SaaS solution and the auditors want to know SDLC things. Well, just like anything in your clinical trial, there's better qualification, right? That's a whole section of the TMF better qualifications. So when you chose your SaaS vendor, hopefully you put that vendor through a qualification process. So during that audit of your vendor at the time, you review their software development lifecycle. You review everything associated with that. And you have, based on those vendor qualifications, gone forward with this vendor. So that's step one is you qualify the vendor. With each release, your SaaS vendor will provide you with a couple of things, right, at least draft and final release notes. We don't provide all of our internal system test scripts to our customers. We assert that because you audited us and we follow our SDLC that, that's what you take forward. We do issue like a validation certificate. So I think the answer to that question comes more from your vendor and what you can get from them than maybe from me. But the things that I think it's okay to ask for is an assertion that the vendor conducted this release according to their SDLC, right, those final release notes, a list of known defects. And then in the case of us, we provide training materials and SOPs and user acceptance test scripts. Those are not something you would provide to an auditor. But that draft and final release notes, that validation certificate or assertion from your vendor would be step one to answering that question. Then how do you stay on top of those things? Well, periodic vendor requalification audits. Sometimes they're a mail auditor or it's a questionnaire as opposed to an actual auditor. And even so, things have changed over the past couple of years with in-person audits versus virtual audits and things like that for qualification and requalifications. But I think the important thing there is know what your vendor has or maybe conduct a mock inspection to test the limits of what you can get from your vendor and then take that forward as your answer to the inspection.

All right. Thank you. Next question, do all SaaS vendors offer crowdsource validation?

This one does. All the ones that I work for do. I don't know. And we've actually spoken at this -- spoken on this topic a couple of conferences going back a year now. I think the first time we talked about this might have been TMF Summit in London last year or maybe even earlier. So the -- I don't know if that's a question you have to ask your vendor, but we plan to offer it for our SaaS solutions. At IQVIA, the eTMF took a frontrunner position with that, thanks to the question from one of our customers about 18 months ago. But I think the answer is you'll have to find out. And if there isn't, well, then maybe plant that seed or maybe start a community of practice or work with other customers of that same solution and then have a user group, a user group that's not sponsored by the vendor, right? And you may be able to implement that crowdsource model yourself. But I think you'll see it being offered more and more for SaaS offerings in this space as the release cadence quickens.

All right. Thank you. Our next question, is anyone actually doing crowdsource testing in life sciences?

That sounds like one of my canned questions that I provided ahead of time in case there were no questions. I think I answered that during the presentation. Yes, we are doing it. And our second go-round starts with this next release of our product. It's not easy, right? Because we -- I think one thing I didn't mention is there's also training involved, right? So we have our SOPs. We have our controlled environment. We do the data setup. We have the UAT scripts. But to participate in the crowdsource model, you need to be trained. So we keep training records on the customers who are going to participate in the model. So it's -- we are doing it. It's not easy. I think it will get easier. But we see more and more interest of it as we talk to customers about it.

All right. Our next question, can AI help with validation?

Again, I don't think we're there. I think the question might be, how do I validate an AI application? Maybe that's a better question, right? If you think about an artificial intelligence/machine learning-type application, let's say, it's something that's doing document indexing. It's not changing the world. It's just taking what your users do and doing it faster and more consistently and learning from it, right? So it's a force multiplier or a way to reduce costs. So to test what the AI does -- and early in the learning models, there's the whole human-in-the-loop model for AI so that there's actually eyes on the decisions that the engine makes. But essentially, you're checking the effectiveness of the AI model. So it's not using any system or any feature of the system that a person doesn't use. It's just doing it on behalf of them. So in our eTMF, there's a quality control workflow that follows the indexing of a document. So that AI engine kicks in at that point. It indexes a document and sends it off to QC. So the QC loop in there is a measure of how effective that AI engine is. So I think testing the AI is not so much about testing the learning engine. It's just to make sure that the engine is doing the things in the system that is basically replacing the human element of. And this is early rudimentary-type AI from basically a resourcing perspective. The AI in this case is doing the work of people but faster. We can talk about more advanced models with other types of relational-type activities, then I think there's a different approach there. I don't know if you want take some more questions.

Yes, okay. Next question, in crowdsourcing, how long does it typically take to complete the UAT process?

Less than the release cycle of the product. So that's our goal. So -- at least for us. So if we talk about two releases a year, so there are 6 months in between releases, our customers -- we encourage our customers to validate a release in no more than 3 months. So that's our target. But I think with the way that we're managing the crowdsource model, as we accelerate that process, usually the thing that slows user acceptance testing down for new features is understanding how the feature works, running the new scripts but then also doing the data setup for it. So I think because of the way that we use eTMF subject matter experts in our case to do the data setup for UAT, I think the things that fall out of bad data setup, which are the most difficult part of UAT, are removed from the equation, so we can compress that cycle. Ideally, we would like to be able to have our customers participating in the crowdsource to be able to get through that activity within 6 weeks or half the release cycle -- half the allotment of the upgrade cycle.

Okay. And with that, we will wrap up today's webcast. I would like to thank the audience for attending and for participating in today's event. I would also like to thank our sponsor, IQVIA, for making today's webcast possible. We would like to ask everyone in the audience to participate in a brief survey. This survey will appear on your screen after today's presentation has ended. You will receive an e-mail alerting you when this webcast will be available for replay. And we invite you to forward that announcement to your colleagues who may have missed today's live event. We hope to see you all next time. Goodbye.

Thank you all.