Gartner, Inc. (IT) Earnings Call Transcript & Summary

Hello, and welcome to this Gartner Webinar, How CIOs Can Calculate Business Value and Cost for Generative AI Use Cases. This presentation will provide actionable objective insight, expert guidance and solutions that enable faster, smarter decisions and stronger performance on your most critical priorities. I'm Jillian Page, your moderator for this session. And before we get started, I have a few tips to help enhance your webinar experience. [Operator Instructions] To download a copy of the presentation slides and other helpful resources, just navigate to the bottom of the page. [Operator Instructions] This webinar is being recorded. You can watch this presentation again and find more great insights on demand at gartner.com/itwebinars. And now I'd like to welcome Gartner distinguished Vice President Analyst, Rita Sallam. Rita, thank you so much for joining us today. And now I will turn it over to you.

Okay. Hi, everyone. Thanks, Jillian. Welcome to this webinar on calculating business value and cost for generative AI use cases. We'll go over 8 use cases today, but I wanted to start with just conceptually, how we should be thinking about value and cost. So we did some research late last summer. And we asked organizations if they are meeting or exceeding their expectations in various aspects of generative AI. And on this slide, you see the percentage of organizations that said they were meeting their expectations, along the bottom, and the priority of each element on the slide. And so this research provided a snapshot of where organizations stand on their generative AI journey. It also highlights where they're excelling and where they're facing challenges. So we can see that many organizations are doing really well in areas like employee well-being, work impact, even ethical use and risk mitigation. And those are great achievements considering the complexity and novelty and newness of generative AI. So we thought that was really great. And then you see some areas here where they weren't quite meeting expectations in the same way, but they were still pretty good, like adoption of the latest technology innovation or speed to adopt technology or even scalability, which I found a little interesting, but definitely lots of organizations are piloting. So I didn't find many of these too surprising, but still more challenging. I think this last one, though, is really interesting. It really stood out for me who researches value and cost, et cetera, because you can see that the benefits, ROI realization stood out as significantly underperforming expectations. And so somewhere around only 57% of the organizations that we surveyed thought that they were meeting expectations or better at this point. So a bit more concerning, though, as you can see, benefits realization is what was really overwhelming in terms of the highest priority, but over 70% said that it was the most important thing, but many said it was not meeting expectations. So a bit concerning, as you can see. And then, again, what you can see here, further looking at this data, is that there's sort of an inverse negative trend where the least important things seem to be meeting expectations, and the most important trends are not -- things are not. So it's a challenge, it's important. So we'll talk about how you can overcome this in the presentation. We have some more data here as well. We did webinars like this one in January of this year. And you can see here that organizations are still not having a great level of confidence with respect to calculating costs. And then we also asked about risk. Similarly, pretty -- a good percentage of the organizations that participated in the webinar said again, not super high confidence or not a high percentage who have a great handle on assessing risk. And similarly, we go back to the ROI question and again, a higher percentage have no or low confidence in their ability to measure ROI benefits. I hope by the end of this presentation, you'll feel a little more confident. But I wanted to do a similar poll here. Just on the first poll question, do you have high confidence in your ability to calculate costs for AI in the enterprise? If you could just take a couple of minutes and respond: low, medium, high confidence. Again, it's always interesting for us to take a pulse of where organizations are; even 3 months can make a big difference. So hopefully, just click away, and I think we can close the poll, Jillian, let's see what we get. Just take a quick pulse. [Voting]

Yes. So not surprising that the vast majority of you have low to medium confidence, so even a little higher than our poll. Okay. So let's look at measuring ROI and realizing ROI. So it's estimating, measuring and realizing ROI is the next poll. So if you can -- here let me, yes. So do you have high confidence in your ability to estimate, measure and realize value for AI in your enterprise? Again, we'll just take a quick 10 seconds to look at that. In my inquiries, in many of my colleagues' inquiries, this is probably the biggest challenge we see with AI and GenAI in particular, of course, rapidly quickly followed by risk and governance. But this one is probably the top call that we get. So okay, we can close the poll, Jillian. Let's see this one really quick. [Voting]

And again, similar, even higher, as I would expect. So it seems like you have a little bit of a better handle on cost but still not much. So thanks for that. We can go back to the slides now. So as I said, let's take a look now at our framework for helping you think about how you would assess value and cost. And again, we'll focus on 8 of the most common use cases we see our clients implementing. I'll start by just conceptually talking about how you might think about value and cost, and then we'll drill down into the 8 use cases. So conceptually, we find that use cases typically fall into 3 categories, sort of defensive use cases, where you -- they're relatively easy to buy, relatively easy to deploy. You might get incremental task specific or job-specific gains, maybe micro innovations, but because they're often general productivity type of benefits that are often difficult to nail down in hard financial terms, we see these types of use cases, coding assistance, productivity assistance, they generally result in a return on employee, but they still have to be managed to even realize those benefits. The next sort of category of use cases are the ones where you're extending an existing business process or an activity or a decision, you're tying the GenAI use case to that specific process improvement or decision improvement. And here, we see better ability to define and identify value and return on investment still has to be managed. But whereas the defensive use cases are more about productivity improvements that have to be converted into hard financial numbers. Generally speaking, the extend use cases that are tied to a specific process or decision, can be tied to a specific cost reduction or revenue increase. And so a little bit easier to get your hands around. And then the third category of use case that we'll talk about in more detail is those sort of upending types of investments, where you're really trying to change your industry, create a new market potentially, new products and services. You're changing your core processes for a dramatic shift in the basis of competition in your industry. And those tend to have more strategic benefits, more strategic outcomes. And so we say that this is really about return on the future. Typically, you need to have a higher risk tolerance in your organization to invest in these types of use cases are typically more expensive. They take longer -- there's a longer time to benefit. And so that's sort of the framework that we'll use as we think about these use cases. And as I said, typically, when we ask our clients, here's another survey on where the primary business focus is for their GenAI initiatives. You can see it's really cost optimization, but the vast majority is around revenue growth and revenue optimization. And to be honest, that's sometimes not always easy to connect your initiatives to those outcomes, particularly when we're talking about those defensive types of use cases where productivity improvement or time is the biggest benefit because productivity isn't the same as value. If there's one thing I want you to leave this webinar with is that thought. We can create productivity all day long. If you give me an hour a day back, I might take 10 minutes of it and go have a cup of coffee. I might use some of that time to do things that I like to do versus activities that are optimal for the organization. And so we can create a benefit gain in terms of productivity, but we have to convert that time into something that's actually going to benefit the organization. And so how do we make sure and enable people who are using these tools to take the time that they're getting back and use that time to sell more, to improve quality, to work with customers more so that we improve the experience and potentially reduce churn, potentially cross-sell new products or that we use that time to potentially -- potentially the employees is happier. And so they stay with the company and they churn less and we lower our cost to hire. So we have to think about very proactively and intentionally, how do we harvest the time and convert that into something that's actually going to realize a business benefit that can be measured and ideally have a financial outcome potentially longer term. And that's the challenge. And we have this concept, and you'll see it in the calculations when we get to that of productivity leak. Again, just because you give me an hour a day back, that's a theoretical value. I may, in the best of circumstances to switch to another task, there'll be context switching. I may take some time to go collaborate with a colleague, which may be positive, but it still takes away time immediately away from taking that productivity and using it for an optimal use that's going to impact the most important metrics of the organization. There may be task coordination that has to be done. So you'll lose some time that way. And then there's the theory of constraints, just because I optimize one piece of the process, maybe my part, if I don't optimize the rest of it, I'm still not going to get the full benefit of that productivity improvement in one part of the process. So now let's take a look in a little more detail, okay? So we are going to, again, conceptually first now talk about cost. But I wanted to expand the defend, extend, upend concept framework to include the concept that each of these 8 use cases have a specific value and competitive impact profile. And as you move along the continuum from defend to extend, upend, typically, you have a different deployment model. And that deployment model typically is more complex. It involves more enterprise data, there's more upfront development, particularly more complex -- potentially more complex model type of involvement, and so more risk as we move further to the right. So you'll see, as we talk a bit about the cost elements and how that works out. So what I'm going to do right now is sort of visualize the cost components for you. I think one of the biggest mistakes we see our clients make is when they think about cost, they focus on the model costs only. But we know, and this has been, I think, one of the challenges with our organizations that the total cost of ownership is actually far greater for the other components beyond the model. And so I'll talk about what those components are. And so just focusing on the model side of it, unlike previous computing paradigms, we know that the process of training those models, running large language models has high structural costs for vendors selling those models. So even large language models trained with billions of parameters, they continue to require massive computing power because they run billions of calculations on specialized GPU hardware in response to every prompt. But we do know that costs are trending down, pricing models and techniques that lower costs are already evolving as the market matures. Even prior to the DeepSeek innovation, we saw that OpenAI lowered their prices several times. And they've introduced several model versions since the launch of ChatGPT in November of 2022. And we see innovations in engineering, in model techniques, of course, DeepSeek, being one of the most widely talked about and popular. But again, we've seen over time over these past 2 years going on 3 years, innovations that have brought the energy consumption, the compute and then therefore, the pricing, not sure in line with the cost to the vendors, and down. So the opportunities to reduce the cost of GenAI initiatives at scale, they've expanded. And they've also changed, as you'll see in the calculations that I'll share with you, the build versus buy deployment options for cost effectively transitioning from pilot to enterprise deployments. And so let's now just jump to the components. So again, the first piece of total cost is the -- for any of these use cases, any of the defend, extend, upend use cases, is the upfront application development, integration, modifying systems type of cost. Now whether -- even if you're just implementing coding assistance, you still have some upfront integration with your existing systems like Jira, you may have some upfront security and governance work that has to be done. So all the way up to more extensive or complex application development for a product or a custom application. So we have to factor those in. And as you go further up the continuum, you may need more expensive skills, more highly skilled people. So that has to factor in. Now of course, if you are fine-tuning a model or you're pretraining your own custom model, you will have model costs. Fewer of our clients are doing that, although the price of fine-tuning has come way down by itself, and we'll talk about that when we get to that use case. Then you have the cost of compute and your AI technology infrastructure, the tools you need to build out either models or build out the components of your AI application. And so the cloud infrastructure, it can either be absorbed by your SaaS vendor or by your model vendor. But still, you may have to pay cloud infrastructure for things like vector database storage for your embeddings if you are building a custom application. So you have that cost. Generally if you host your own model, then you assume the full cost of inference and compute for the model, if you were taking, let's say, building an application, using an open source -- pretrained open source model that you're hosting yourself either in the cloud, then you would have those cloud infrastructure costs for on-premise, then you would have the hardware cost. Then an often underestimated cost is data. Data -- AI-ready data is necessary for GenAI cost optimization, core data readiness, including semantics, can increase the amount of tokens you need for use case accuracy. So investments in enriching data via semantics can actually reduce the cost -- the token cost for inference and training while achieving higher quality. AI agents will further increase the need for expanding and maturing your data management capability. Now this is one area where I think business folks tend to underestimate because they hear the hype, why can't I just take GenAI and point it at my 12 instances of SAP and miraculously get accurate insights from it. So those of you who are in this and have been -- our data and analytics, leaders or CIOs with experience with data and analytics know that, that's just not the case. Even if you do a simple use case, like analyzing your call transcripts, still you're going to want to strip out unnecessary text like, hey, how are you, to minimize the text that you actually put into a prompt so that you can lower the cost and also get higher accuracy from your queries. But at the more extensive level, you need to apply different types of actions on your data even if -- you may need a data fabric in place to orchestrate the data for different queries. So data management for AI, data engineering for AI, AI-ready data become an important cost as you -- particularly as you move up the continuum to more complex applications. And then you might have user costs. Certainly, if you're buying a SaaS model license, via a SaaS model license, you'll have per user license cost. But another very often underestimated cost is training and business change, change management. If you're giving users tools without helping them think through how this changes how they work, how it changes the processes that you're trying to improve. If you are not teaching people how to use the tools responsibly from a governance perspective, you will not realize the value that you expect. We see this in our surveys over and over again that one of the biggest challenges to realizing value is that users don't know how they use this new capability to actually change how they work. And so this -- it looks like a small bucket. But in this iteration of our model, this is -- we just published this week, the second iteration of the model that you can download as part of the note that is published by the same name. Training and business change become a much more important and wildly underestimated cost for our clients. And then the final piece, one of the other major pieces, is the inference costs, right? So this is where you are leveraging either the model APIs of OpenAI, Anthropic, Google, et cetera, Amazon models, but you are paying by the amount of tokens or text that you put into the prompt and the amount of text that you get out. And that could be impacted by things like the number of iterations users do, the workload of the users, the number of users you have, how many times they use the system every day, how many iterations they do each time they ask a question, plus the sub-API calls that you may need to do to get to the accuracy you need, and that relates back to how much have you invested in adding context to your data via semantics. And we'll talk again a little more about that in detail. Now let's get into some more detail for specific use cases. Okay. So defend use cases, just conceptually, the cost, generally speaking, you will have -- let's take coding assistant as an example. We have a couple of examples that I'll share with you. The minimum you're going to have upfront costs for integrating the application into your existing systems, you will not have model costs, you will likely not have compute costs because that's incorporated into your SaaS license, minimal data, although for a couple of the use cases, you'll see that we do have some data costs in there. But you'll have the SaaS license and you should be thinking about the training and change management even for something like Copilot. Most of our clients are just giving Copilot out and hoping for the best. The ones that are -- that they tend to have a hard time getting their hands around what the value is. Those -- that value needs to be actively managed and again, harvested to realize the value and to demonstrate the value. So here are some sample numbers we have actually estimated, done some scenarios for these sample use cases. The first one you can see is coding assistance. The next one are these business productivity assistance, and the third one in the defend category, again, these are the most common we see, are marketing content creation use cases. And we've made some assumptions about number of users. We made some low-end, high-end assumptions for cost -- the different cost elements, including the upfront pilot, rollout, deployment, development, integration, training as well as the recurring costs, which include licenses as well as the ongoing maintenance of the initial application. And you can see some of these numbers for coding assistance, for business productivity, for marketing content creation, basically, the recurring costs include the SaaS application as well as some percentage for the ongoing maintenance of the application. Relatively low. You'll see those numbers go up higher as you're actually developing custom applications, and you'll need to continue to update the application, to extend the application, to maintain the application. Now let's talk about value, okay? Value is, again, for these use cases, harder than for the extend use cases. So what we were able to do pretty easily is calculate the theoretical value of coding systems, right? We can make an estimate about the percent of time our developers spend writing code. We can -- we know their average salary or their loaded salary, we can make some estimates even. I mean there's plenty of examples out there of companies that have used coding assistance. They're reporting different productivity numbers. We generally see much lower than what's being reported out there. So I and the model that I have and in this model in this calculation, I use a much lower percent of productivity, between 7% and 15% productivity, but that does vary widely. And again, that's a theoretical value. You will have productivity leak like we described earlier. Just because I give somebody an hour a day back, doesn't mean they're going to use the full hour for productive use. Now again, this is a productivity use case. So how will developers use their time? That's what you have to think about proactively and manage to. So for example, I have clients that say, "You know what, we're going to not reduce headcount." If you do reduce headcount, that's an easier direct link to financial outcomes, but most of our clients for developer productivity are not doing that. They're saying, okay, we're going to keep our developers, and we're going to just let them work less, as one possible way to reap the value. So that is return on employee, but also potentially can be linked to better developer retention, lower cost to hire, increased productivity because you're onboarding people -- fewer people. Or you could say, you know what, we want to improve our product. We want to get more story points per sprint, more committed code out there that's of high quality, that's going to impact the product, and we're going to be able to monetize those new features. So we may be able to drive new revenue. Or even if we're not going to be able to monetize it, it's going to improve the competitiveness of our product, we're going to improve our win rates, we are going to improve our customer retention. So you have to build that story to ultimately link that productivity improvement to hard business outcomes and then measure those hard business outcomes, whatever they may be. So these are sample KPIs. You have to think about what's optimal for your organization and proactively manage that. It may mean, it will mean helping also users rethink how they work. It may mean changing performance incentives for incenting people to actually do what you want than to do with the new found time. And maybe you want them to work down the COBOL code so that you can reduce your technical debt and the cost of technical debt. That has to be thought through early in the process during proof of concept and then manage through rollout potentially adjusted as you move to scale. Similarly, with business productivity assistance, you have to help your users think about how they're going to use, how you want them. What is optimal for the organization to use the time. Marketing content creation, you could also say cost avoidance. So I hear that a lot, too, where our clients will say, "Well, we'll do more with the same number of people." And that's a great one if your CFO buys into that. Often the CFO doesn't because that doesn't have a direct impact on financial statements. It's a future financial impact. But if you can sell cost avoidance, go for it. But it's usually one that doesn't get a lot of traction with the CFO. In terms of marketing content creation, similar idea, you can talk about quality of output, you can talk about more content per marketer and potentially reduce headcount. We are seeing other kind of direct savings that can be used with this benefit in terms of agency fees with in-sourcing, reduction in translation services and others. So this one may have some more direct line of sight to business benefits, but you have to make that connection. So now let's look at the extend use cases. So extend use cases could have a couple of different deployment option approaches. It could be you're buying the GenAI-enabled capabilities within your enterprise applications like Salesforce, Workday, SAP, Microsoft, or you're building a custom application, let's say, you're a mortgage company and you're putting a conversational interface in front of all the documents you need to underwrite a loan, and you are using a large language model like OpenAI, embedding the APIs into your application to be able to underwrite a loan much quicker, say, going from 2 months to 2 weeks. That would be sort of you're using a standard commercial model, you're using the APIs, you're embedding that in a custom application that you build. So extend use cases really are about taking an existing process and extending them with GenAI. Now you can see here that -- most organizations, by the way, start out with these buy options, right? And a lot of times, why is that? Because they're often inexpensive to pilot, but they are often more expensive to scale. SaaS per user licensing across -- multiplied across tens of thousands of enterprise users if you're a big organization in many cases, also coupled with usage-based pricing, a usage-based pricing component to approximate token consumption, many of these enterprise vendors have both a SaaS component as well as a token sort of surrogate tied either directly token-based pricing or something that approximates token consumption. So it can make the license cost pretty significant and unpredictable at scale. When you build a custom GenAI application, leveraging the APIs from a general-purpose GenAI model vendor for process or decision-specific use cases, you may also be leveraging your enterprise data. And we know that new large language models enter the market every day. It's like dog years every day for large language model introduction. And so organizations with the skills to execute have a range of options, but where the uniqueness comes is from your enterprise data. So in these types of use cases, you are getting more involved in your enterprise data, you are leveraging your enterprise data more and the uniqueness of the data that you have, you are going to need to work on your data more. We have an example of PageGroup. They've talked to us on our D-Suite podcast broadcast. They're a large job recruiter in Europe, operating, I think, in 100 different countries. They wanted to build pretty straightforward GenAI applications to reduce the time it takes to write a job advertisement description, from 90 days (sic) [ minutes ] to 5 minutes, amazing, right? Well, what they'll tell you is, however, is that GenAI, they were able to do that. They went from 90 minutes to 5 minutes, amazing productivity. And of course, they can use that time to do lots of things like sell more jobs or prepare their candidates better so that they are likely to have higher close rates, et cetera. So great use case. But they'll tell you that they couldn't have gotten from 90 days -- 90 minutes to 5 minutes without having invested for the past 7 years in their enterprise data fabric, because that's what's enabled them to pull data from 4 different systems that hadn't been previously integrated and pre-populate the job descriptions, that's what got them a big portion of that productivity improvement, and then GenAI helped to generate the text. So the investment in the data is nontrivial. And particularly as you move up the food chain, -- and so definitely think of that, extend GenAI use cases will require a more -- or higher total investment than defend use cases. And again, the cost can be more unpredictable, but so you have to assess cost to determine whether they are offsetting the financial benefits from productivity or other future gains that you can tie directly to revenue and cost reduction. And again, you still have your training, you still have potentially an inference cost. Now if you decide to build, let's say, build a custom application and you do that using, let's say, an open source model, that you now host yourself in the cloud or on-premises, these inference costs, you can see hosted build versus buy, these inference costs now shift to your cloud infrastructure, in addition to the cost that you will have for your vector database storage, for your embeddings that you will have, even if you are leveraging the commercial large -- general-purpose large language model APIs in your application. So things to think about. And again, we'll look at that in a little more detail because I do have the big build versus buy calculations. So here, we're just looking at customer support, as an example, build -- buy versus build. You can see that just -- I'm not going to talk about the numbers in detail, but some of the assumptions here, you have to make some assumptions about how many agents, what's the workload of the agent, how many calls do they do per day, what is the token amount per call in and out on average. So we've made some assumptions here, and you can see that buy versus build, the upfront cost for the build are higher. And potentially, therefore, the ongoing maintenance is higher because it's an application that you continue to enhance and build and maintain. But even so with that, the ongoing inference costs and recurring costs, because with the buy option, you've got the SaaS plus the API consumption calls, whereas with the build, you are just paying your inference and cloud infrastructure costs. There is a breakeven point with the assumptions that we have here. Your assumptions will vary, your breakeven point will be different, but you have to think about if you have the skills to do the build and if this is not just a commodity use case where -- if this is a strategic versus commodity use case, you have to think about, is there a point at which maybe I start with a buy, but then I shift to a build at scale. Of course, build versus buy decision framework should include multiple factors, besides costs, but including risk and compliance, time to market, et cetera. And of course, the skills that you have and whether you want to allocate those scarce highs because you're going to need more expensive skills here, too, for the build. Do I want to allocate those scarce resources to something that may be a commodity even if the cost equation favors a build. So those are the things you need to think about. Let's see. So let's move on to sales creation, again, similar idea where you're going to have a number of salespeople. You're going to make some assumptions about their use, same. The standard document search and summary use case, this is really where you're going to build a custom application for some search and summary type of use case, whether it's, again, job recruiting or a mortgage or something, just general productivity, which is harder to justify, but you're going to try to think about in terms of the cost, you're going to have to look -- think about the workload. Again, how many times do you -- how many users, how many times do those users will they typically use the application, how many iterations per use, and then how is your data, how many sub-API calls, context prompt calls do you need to do to get to the accuracy you need. And in both of these -- all of these cases, you're going to have to spend more on data management and on the AI infrastructure, data engineering and the technical infrastructure, the data fabric that you're going to need to support these use cases. On the value side, of course, these tend to be a little -- at least the first 2 tend to be a little easier to justify because you can measure the results programmatically. For customer support here, we do see cost reduction, particularly in terms of headcount. We'll do the agent use case in a second where we're shifting human headcount to agent headcount basically. But you can see, you can measure programmatically within the systems, time to resolution, response time, customer sat scores. And potentially, you have better agent retention, you have productivity of junior staff, number of escalations potentially can go down, maybe they go up if you have GenAI, but that is the KPI. And then what we do see, too, which is an interesting side benefit, is the call center, the level of detailed call center analytics that you can now do with GenAI goes way up. Personalized sales. Now sales is an interesting one. Again, the theoretical value is what you need to convert. So just because you give salespeople more time back, they basically have to use that time to sell more and then close more. So there's surrounding processes back to the theory of constraints, that if you don't have high-quality leads for them, now that they have more time to sell, if they're not given the right territories, they may not be able to convert that extra time fully into new revenue and then how much does that then percolate down into margin based on your margin rate, but you should be able to measure things like deal size and percent cross-sell, upsell size, close rate, et cetera. And then the document search and summary, value will highly depend on what is the use case. Are you able to tie it to a specific process, activity or decision and then you measure those benefits. Now I'm going to talk quickly about agents because that's the whole range right now. And agents do have a specific cost profile, right? But they do -- the agent pricing that you're seeing for the build -- the buy options really do approximate token consumption. And so in our model, the way that we've calculated again a build versus buy, is that we're looking at, one, again, the same type of usage patterns, how many calls per agent per day. But then we're looking at, depending upon the license model, some licensed by message, some message by GenAI versus non-GenAI message. So you have to get to a cost per agent message. And then you have to be able to estimate approximately how many messages tie to this kind of call volume. That's what we've done on the low and high end. And again, here, too, you see that building upfront costs are going to be higher. You're going to maybe build your agents in Python or some other programming language rather than using the out-of-the-box tools, low-code, no-code tools for building agents or just leveraging out-of-the-box agents from a buy option like Salesforce or Microsoft or ServiceNow. They actually have an interesting performance-based pricing model that's based on response rate. So you have a number of options to consider here. We've modeled some of them, low end, high end, and these are sort of the results, low and high end. But again, what you're trying to do here is figure out the cost of a human agent supplemented by GenAI with a SaaS license and potentially a token license versus an AI agent whose cost is based on messages and the workload that they would have. And so you can see generally the cost profile, again, for our assumptions here. Again, it's more favorable for building after 500 users for these specific assumptions, right? And then, of course, the value is a little bit different than a customer support just, of course, customer support application because we're shifting -- we're replacing a certain percentage of human agents with AI agents. That's a benefit, that's a cost reduction benefit, but we're assuming the same productivity of human agents, et cetera, the same type of KPI impacts. So now let's look quickly at the custom extending. So you could also, what we see, is many organizations will build a custom application using a large -- a general-purpose large language model. And maybe they haven't done the investment in the data that they needed to do or even if they've done so, they're not getting to the accuracy they need or they're in a regulated industry, and they need to do better. So sometimes you'll do an extend application, existing process, but you'll decide to take a pretrained maybe small open source model and fine-tune it. The cost of fine-tuning has come way down due to techniques and just fine-tuned parameter optimization. And commercial large language model vendors offer fine-tuning options. But again, you need to assess. We see some clients doing that once they experiment via buy, but then they're not getting the accuracy they need or the amount of tokens that they need to put into the prompt to get to the accuracy they need are so exorbitant, that it just makes sense for them to move forward if they have the skills to do a fine-tuned option. And so we see that. Now the upend use cases, fine-tuned or custom model, you potentially have more of these costs, right? You've got -- if you're doing a custom model, you've got not just model tuning, you've got model pretraining. You're assuming all of the cloud infrastructure costs, the inference cost shifts essentially to compute. And that's basically what we do. And then you can see the costs, of course, are much higher. You have upfront application development costs. Based on the size of the models that you're fine-tuning, you're going to have -- we based it on a small model. We've got a small model option, a large language, a large fine-tuned model option and then we've got a custom large model option. But you can see, the costs go up substantially upfront. You need higher trained, more expensive developer and data engineering, and data science resources. Potentially, these applications take longer, and you have higher ongoing inference costs that relate -- that you're assuming based on compute. But again, the assumptions for calculating that are different. They're based on embeddings. They're based on user concurrency and a number of other detailed factors that we include in our model. We don't go in great detail, by the way. In our model that, again, you can access through the research that has been published, we try to give you just the levers that you would need to pull and what are the major factors that drive cost and value. It's not an incredibly detailed down to the penny type of analysis. These are just broad brush, high level -- order of magnitude types of cost and value estimates, and you can refine them based on your own assumptions. Now what's interesting about the upend ones is that if you notice, the KPIs that we use here are more strategic. They are about market share. They're about percent revenue. They are about fundamental aspects, let's say, of a process and customer lifetime value. So when you are investing in upending types of use cases, you have to also, as an organization, recognize that these are more strategic investments. The value is going to be very use case specific, but there are going to be more strategic types of impacts, longer-term realization of value. And if that's your AI ambition, then these investments are great. So again, one really important, let's say, position, is that as GenAI models are increasingly commoditized, again, they're coming out every day, realizing competitive advantage and full benefits are really driven by how effectively you use your unique data, how effectively you redesign work, process so that expense on change management and training, and how you manage your change and risk. Those are really where the rubber meets the road for realizing the full value and even managing the cost to get to the ROI that you hope for. So again, to summarize, you have to align your investments as a portfolio, how you invest in the percentage of investment that you make in defend, extend, upend is really going to be based on your AI ambition as an organization. So that needs to be done, that kind of ambition has to be defined by your leadership. Do they want to disrupt their industry? Some industries investing in upending investments like pharma, that's actually a competitive requirement because most of the major players are investing in things like drug discovery and clinical trials processes. So a lot of this is going to be very industry-specific, but you have to have an AI ambition as an organization within your industry as your leadership has to determine that, and your portfolio should reflect that ambition. Again, how you realize value is critical, so you have to strategically plan to actively harvest whether you're harvest value, whether you are investing in defend, extend, upend types of use cases. And again, what you need to think about is what's the right portfolio for you to realize your AI ambition and how are you going to actively manage the cost and the value to realize the ROI and ultimately optimize the impact for your organization. So again, just to reiterate, right, measure and manage total cost, not just the cost of models and value, and do that from proof of concept. We see a lot of times, organizations have done the proof of concept, they've thrown out these tools to their users and they're like, now we're being asked to justify the value to move on. No, that has to be done early on and how you're going to do that, whether you're looking at defend, extend or upend use cases. Don't underestimate the need to invest in AI-ready data and governance and risk management and really, really, really don't underestimate the need to invest in and manage business change. That will be the only way that you're going to be able to convert all of these potential theoretical benefits into actual business outcomes and value. So I'll now turn it back, let's say, to Jillian, and we will take some questions. Let me take a look. Thanks for listening. Let's see. Jillian, do you want to ask the questions? Or should I just pick them? How do you want to do it?

You can pick them on your own, Rita. But while you're looking through them, I can go through some additional resources for the audience. I'll give you a minute there. So we've got a lot of questions and comments coming in. [Operator Instructions] And before we get to the Q&A, we have some additional resources to take you deeper into the topic. Gartner's CIO Leadership Forum provides you with inspiration, strategic ideas and tactical actions to address the challenges you'll face in 2025 and beyond. Visit the link to learn more about this and other upcoming Gartner Conferences. Additionally, we invite you to download our e-book, Get AI Ready: What IT Leaders Need to Know and Do. Ready your enterprise to capture AI opportunities and bolster your cybersecurity, data and AI policies and principles. Also, Gartner recently released a new framework that will help you set the right pace for your AI race, so you can achieve the desired business, technology and behavioral outcomes for your organization. Get the guidance you need to achieve your AI goals at scale. You can download the research by clicking the button on this slide. And if you want to stay in the know, you can connect with us on LinkedIn and X, formally known as Twitter, so you get the latest Gartner insights on the go. And if you want more information on how Gartner can help you achieve your mission-critical priorities, you can contact us via the methods on this slide. Okay. Rita, over to you to address the questions.

Okay. Thanks, Jillian. Okay. So let me just go down the line. So is there a breakdown of what functions, processes, organizations are targeting and seeing value in cost reduction? Again, it depends on the variables that I've highlighted here. If you can, in theory, tie GenAI to a specific process or function, and as a result of that, there is a direct cost reduction or revenue increase, then you're good, but you have to still actively manage the realization of that cost reduction or revenue increase. I'll give you another example that I didn't talk about during the call. Have another client, they're in logistics. So they're managing shipping packages and containers across country borders. A big cost for them is customs brokers, right? But what they were able to do is create a generative AI application sitting in front of all of the customs regulation documentation globally. And so they're able to query this application along -- it's a bit of an agent type of use case, and then generate the documentation that's needed for each package to cross the country borders that the package needs to cross. So by doing that, direct cost reduction of the customs brokers, so very clear, right? And so the key is to find those use cases, if you're looking at the extend use cases, again, if that's your AI ambition, is to really target those and maybe prioritize those that have those direct cost reduction or revenue increase increases. As I was saying, when we were going over the extend use cases, we are seeing the customer support use case commonly associated with headcount reduction in our clients. So that one is pretty clear. And yes, and particularly as you shift to AI agents, but even without that, we've seen the headcount reduction benefit as being one. But again, like the customs broker, there could be other cost reductions that can happen, but you have to find those use cases. Let's see. I think there's another one, a lot of case studies. Let's see. For GenAI, it's hard to justify if only a single use case is considered, but a lot of use cases can only benefit by scaling the GenAI architecture for several use cases in the future. How do you justify the initial cost? I mean that is a trick, right? That is the trick. How do you justify foundational investments in AI-ready data? How do you justify foundational investments in AI governance? I think one of the best examples I've seen in our -- in one of our clients, in particular, is rebranding, Don't even talk about foundation. These are accelerators. These are necessary elements to realize the value that you expect from AI, particularly as you want to expand and scale across other use cases. It's not an easy sell. I'll be honest with you. But you have to start articulating the value of these foundational investments, not as foundations, but as necessary accelerants to value. You can't have enterprise value from a single or multiple investments in AI without making the foundational investments for -- in parallel at a minimum. And so there you go. Let's see, let's see. According to Gartner Research, 60% of AI projects that run without AI-ready data will be abandoned by next year. Could you -- well, it's actually without AI-ready data, without governance and without being able to demonstrate the value, that was the prediction. And basically, the reason why is because if your data is not ready for AI, either you're not going to get to the accuracy that you need or you're going to have higher risk and higher -- you're going to have higher hallucinations. And so you're not going to then get the benefit from those use cases or it's just going to cost you more, and the cost is going to exceed the potential value that you expect. So AI-ready data is super important. There are a number of tools. I mean, we talk about the data fab, multimodal data fabric architecture basically where you can -- and data engineering for AI capabilities. It's not just the tools. I mean, tools are one piece of it to support this, but it's also practices, it's skills, it's how you're adjusting your data engineering capabilities to support the specific AI use case that you need. GenAI will have different requirements than forecasting or prediction or fraud detection or types of use cases. We do see knowledge graphs playing a key role in AI-ready data because knowledge graphs provide contacts. They provide semantics that help to improve the accuracy of GenAI and again, reduce, we believe that the amount of contextual prompting that is necessary to get to the accuracy that you need. So the spreadsheet that I've been mentioning is part of the research that has been published on gartner.com. It was published yesterday, and the calculations that were used for all of the numbers that I showed today are in that spreadsheet. So you can access that on g.com. I think we might send the name of that research afterwards, Jillian, or if not, you can reach out. Let's see. So this is a good one. Any -- are there any use cases in the defend, extend, upend groups that have shown negative ROI? And these are avoidable. And that's a really good one. It's a really great question. I mean the ones obviously that we focused on are the ones that -- these 8 are the ones that have shown positive ROI given the right circumstances that we have been talking about. So any of these can show a negative ROI. We've seen coding assistant show a negative ROI for clients if they're not implemented properly with the right guardrails, with the right change management. So really, any of these use cases will have a negative ROI if they're not managed properly. I think that's the final message that I hope you leave with, is that it's pretty much the case for any technology, right? If we are just -- but more so with GenAI because GenAI is fundamentally about changing how people work. If we are not changing how people work, if we're not doing business change or business transformation and managing the cost side of this thing all along the way, which means that you have to be sort of value-centric from the beginning, every single thing you invest in could potentially and will likely have either a very small or negative ROI. So that is the key. You have to factor in total cost, realizing that things are going to take longer and be more expensive than you initially expect, and you have to manage the value, recognizing that the value -- the actual realized value is probably going to be less than you expect because there's going to be productivity -- because we haven't fully incorporated all of the change and everybody is not adopting things optimally. So really, any of these use cases can show negative ROI, even the best of them. So great question on what about the cost on the environment? Yes, impact on energy, hardware, that will have to be compensated by the company from a carbon net-zero perspective. So absolutely, that is such a great question. I think I have been sort of relating the cost and the compute -- the compute cost of these applications with energy consumption because really that's the major component of that cost. But that is a huge consideration for some organizations more than others, in some regions more than others. But I relate this to just improvements overall -- engineering improvements in model deployment, model development and then, of course, hardware improvements. Hopefully, we'll see -- we'll continue to see innovations in both. I believe we will, that will reduce the cost and therefore, the environmental impact of these large language models. So thank you so much for all these great questions and for listening. We just have 1 minute, so I want to turn it back over to Jillian to close this out. And wishing you a great day, evening, wherever you are. And yes, hopefully, we talk again soon.

Fantastic. Thank you so much, Rita, for sharing your insights and addressing all those great questions, and thank you to the audience for joining. And just a quick reminder, once again, you can download the presentation slides and other resources in the bottom of the page. [Operator Instructions] All right. With that, I'd like to again thank our presenter, Rita Sallam. And of course, thank you for listening. Have a great day. Goodbye.