Accenture plc ($ACN)

Hello, and welcome to today's Carnegie Mellon University Software Engineering Institute's webcast, Rethinking and Maturing AI Adoption. My name is Ipek Ozkaya, and I'm the Technical Director of AI Native Software Engineering at the SEI. And I've had the incredible pleasure of leading this project focused on AI adoption maturity with our team at the SEI and the incredible team at Accenture. We want to make today's conversation as attractive as possible. So please feel free to put your questions into the YouTube chat area. And we've already received close to 200 questions. There is no way we'll be able to get through any of them in completeness, but we'll try to get to them as much as possible afterwards. It is no surprise today that businesses are -- across all sectors are redefining themselves and going through a structural shift through AI solutions. And they are trying to redefine their operational relevance, their operational workflows as well as get ahead of the businesses through ROI. Software-driven organizations are also going through the same challenge. In fact, the software as a discipline is being redefined through AI, looking into efficiency, productivity and of course, some of the risks that come with it. And clearly, all the organizations that deliver us the frontier models, OpenAI, Google, Microsoft and Anthropic are developing improved capabilities around the cloud, and we're receiving these capabilities around a lot faster. If we look into 2 years ago, the early generative AI models could barely solve some of the cybersecurity tasks. But today, we know the Mythos and GPT 5.5 could actually execute some of the complicated multistage attacks on vulnerable networks. So this is a very challenging act to get ahead. And lasting AI value and ROI is at the top of every executive, every engineer and any worker or across any of the sectors. And intentionally to finance systematically manage AI practice maturity is not now a luxury. It's a key differentiator for organizations to get ahead. However, doing more AI does not define maturity. Scaling AI to every single engineer in your organization does not define maturity. So that's actually where the challenge is in today's conversation. True AI maturity is measured by how you are able to deploy with engineering rigor, how you're able to develop resilient and transport the capabilities, how do you bring together governance approaches where it's needed and be able to evolve the technologies and catch up with the evolving technologies. So of course, it was not an easy task to develop an AI maturity model amidst one of the most fastest changing technologies, and that presented two challenges for our collective teams. One challenge was how do you provide structure in a rapidly changing environment. And then the other structure is to really focus on maturity rather than compliance and checking the box. And we'll try to get to some of those and how we address them. Our approach has been in [ price ], focused on the data and the reality and running some of the early adopter programs. So the results we are going to share with you, and you also have the open model available to you, that reflects the collective insights of the research, academics, industry and our private partners throughout different organizations. So today, I am joined by my colleagues, who I have the incredible pleasure to have shared the journey. And I won't be able to do justice to their background, so I'm going to allow them to introduce themselves. So thanks a lot for joining me today. And to get us started, could each of you please briefly introduce yourself and share the lens through which you view AI adoption and AI maturity? And I'd like to start with RP from Accenture.

Thank you very much. It's been a pleasure working with Software Engineering Institute and CMU while we build the AI maturity model. My name is Rajendra Prasad, I go as RP. I'm the global technology lead for Accenture. One of the key business challenges when I speak to a lot of leaders in the industry today is how do we measure the business benefits of AI and how we scale AI implementation across the enterprise that can deliver the promised business outcomes. For that, one of the key aspect is to kind of approach the implementation of and the benefits in most measurable, structured disciplined, methodical way. And the maturity model that we work together will help us to accomplish that thereby delivering business leaders business benefits and then scaling the implementation. Thank you for that.

Thank you. Anita?

Okay. Great. Well, first, it's great to be here with my colleagues from Accenture and SEI. It's been very exciting to work on the AI adoption maturity model with our 2 organizations. As far as introductions, my entire professional career has been about defining, researching and advancing modern software engineering practice. And of course, now that includes AI. I currently direct the software engineering portfolio of work at the Software Engineering Institute at Carnegie Mellon University, where I lead efforts to make software a strategic advantage, especially for national security and national defense. We do that by advancing trustworthy and resilient software-intensive systems. The lens that I view AI adoption and AI maturity is to build on the many lessons that we learned when we worked with [ Watts Humphrey ] on the capability maturity model for software. The CMM and CMMI, CMM for software and CMMI, was a structured framework to help organizations assess their software development practices, also improve their software development processes and then most importantly, improve project outcomes. At a time when there is increasing hype and pressure to take advantage of AI for business transformation and for cost improvement, we can use the AI adoption maturity model in a very similar way to assess AI capabilities and build a road map to strengthen the practices to achieve AI goals.

Thanks a lot. And we also have some voices from the trenches. Tony, could you introduce yourself for us?

My name is Tony Leraris. I'm the Accenture CIO. And I'd also just like to thank you for letting me be part of the conversation and being part of the work that's been done in the past few months. It's been a real pleasure. For me, I'm responsible for enabling the 750,000 people of Accenture with the right technology and innovations to be able to serve their clients as well as the technology that we need to run our business. And when I look at AI, I tend to look at it from a very practical lens. And so for me, it's how do we scale that across an enterprise of this size in a way that is sustainable and responsible. It is easy to deploy generative AI technologies. It is hard to operationalize it at scale. And so what I'm looking for is how do we do this in a sustainable, responsible way to generate value for our company and for our clients.

Thank you. John?

First, I'd like to say thank you so much for our Accenture Partners. My name is John Haller. I've been at the SEI for about 10 years in total, doing applied research around operational resilience, right? How can organizations sustain their critical services, what's the balance between protective controls and sustainment controls? And now, what are the AI implications of that? I also spent about 8 years in a large financial institution, helping with information security and the join between information security and resilience and operational resilience. So when I think about AI adoption, I'm really looking at it from the perspective of technology risk management and how that impacts the other risk areas and also the cybersecurity perspectives, which we'll talk a little bit more about.

Thank you. Let's go to Majd.

Thank you, Ipek. It's a pleasure to be with you from Accenture's beautiful New York City offices overlooking Manhattan. My name is Majd Sakr. I'm the Chief Learning and Research Officer at Accenture. That means I worry about our workforce and their capability to make sure Accenture remains competitive. I'm in the same role at Accenture [ Learn Vantage ], which is the external facing workforce training arm of Accenture, where we train millions of our enterprise clients, those workforces. I'm also a computer science faculty member at Carnegie Mellon University. I'm delighted to be with my SEI colleagues here, where I've been for the last 20 years. And I'm part of a large research lab that runs a lot of studies on workforce training. From my perspective, Ipek, I spend -- from my Accenture roles, I spent multi-days with the C-suite and with the board talking about AI and workforce training and the evolution transformation that needs to happen. And a lot of the questions that we get is, are we doing this right? And where should our big bets go? And what are the right transformations that we need to think about and we need to strategize around? And so there's a big need for a scientifically proven AI maturity model that helps guide the enterprise and all of these leaders to recognize where they are, where they're headed that is anchored in a lot of fundamental research that enables them to feel confident in terms of the outcomes that are going to come out of that and then the next steps that they can take so that they can move their organization in the right direction.

Thank you for that. And Kaveh, please.

Ipek, thank you for having me here. I have spent my entire professional life in healthcare. I serve as Senior Adviser to Accenture for the healthcare sector. I practice medicine. I spent early part of my career as an operator and delivery system and funders. But the last 2 decades, I've been focused on the intersection of technology and healthcare delivery, how do you make healthcare affordable and accessible and effective. I have been very concerned about a particular issue that we see globally in all rich countries, which is that the demand for healthcare is rising because of an aging population while the workforce available to serve that demand is actually in absolute decline. And so the mismatch is putting every developed country in the same place, which is they're going to run out of caregivers. And there were no good answers. And I became interested in AI primarily because I saw no other path for us to extend human capacity. If we couldn't have a technology that could take some of the tasks away from humans, give them time back so they can take care of other people. And generative AI, in particular, because of its ability to take over language skills, was really the first technology we had seen. What was interesting to me about that and why this maturity model is so interesting is because the benefits of society did not -- do not come strictly from the existence of technology. It's necessary, but not sufficient. The societal benefits actually come from how the work has changed. And what's important about this maturity model is how it emphasizes that maturity is not just data and tech. That's 2 of 8 elements, but there are other elements, to include work and workforce-related elements, that are just as critical to AI maturity. And I think that is a really critical understanding that needs -- that organizations need to have.

Thank you all for that. And for our audience, if you realize it's by design that we have different perspectives because AI today is what in terms of industries, in terms of disciplines, and it's all around in many of the work we do. So let's then go through -- as we were going through our data collection, we've come across a lot of misconceptions of AI adoption and maturity. So that's what I would like to ask to my colleagues here. If you have to identify the single biggest misconception organizations have about AI adoption maturity today, what would it be? And maybe Anita, if you would get a start [indiscernible] we see at the SEI?

Well, I'm going to build on all the points that you just made because I think they're right on. I think a big misconception and one that I think derails many AI programs are just the things that you talked about. People say AI maturity is primarily about tech. Okay? It's specifically about having better models, more data, all things like that. This belief really can lead organizations to investing money in some of the wrong places, millions in fancy, large language models or ML Ops platforms before they really clarify what business problems actually need AI. Secondly, hiring top-tier data scientists while ignoring that the frontline staff really lack the training or incentives to be able to use AI outputs. Another issue, building models that work in notebook, so working in the small, but really fail in production because operational workflows, data pipelines and/or governance aren't quite ready. And one more, confusing experimentation. So maybe 1 successful use case works. But with maturity, we're talking about something else, right? We're seeing it sustained, scalable, accountable value delivery. So these are at least some of the misconceptions that build on what Kaveh had to say.

And RP, you see organizations of all sizes and all sectors. So what are some of the misconceptions that you observed?

I think the fundamental -- I do say that more than misconception, the belief that the technology can transform in a large enterprise where people are the most critical aspect of the change is where we need to balance both. The change management, all the aspects that Anita talked about are very critical for enterprise. How does the change management and a maturity framework can help enterprises, the critical success factor for managing change is quantify and show the benefits to the people that the implementation of technology, in this case, AI, be it systems, be it software, is delivering the promised business outcomes. So the measurement aspect, change management aspect and managing that very, very [ thinly ] is the key fundamental premises on which enterprises has to operate as we move forward. That's kind of what I get when I speak to my leaders in the field.

True. And I guess 1 of the misconceptions related to that, as we saw through is like bringing tools does not mean AI adoption if you're not measuring and if you're not building that change. And Kaveh, you mentioned it from the perspective of healthcare. So what are some of the misconceptions in one industry sector, maybe?

Because healthcare is probably a sector where you're not going to be having technology doing whole replacement, it's just going to be partial. What you're actually going to do is have to remodel the jobs. And to emphasize that, there's 3 layers that we're seeing organizations in healthcare have to deal with. Because AI maturity is not about does the AI work, it's, does the organization get a benefit. So first layer is individuals need skills because technology is a coworker. You're now spreading the tasks out between you and technology and you need certain skills. The second one is at a more organizational level. I'll give you a hypothetical, and we model this. If I have 10 nurses in a hospital medical surgical floor doing the same thing today, the future state for the same workload is 6 nurses plus technology reallocating the tasks. And those 6 nurses aren't all doing the same thing. Some are doing highly specialized things that come and go as needed, and some are doing the same thing all the time. So as a manager, instead of having 10 people showing up all doing the same thing, I have 3 people showing up doing the same thing, 3 people showing up doing specialized thing, plus technology. And as a leader, my work is reorchestrating all of that. That is a different job for the people and for the manager. And then the last one is going from here to there is an operating problem. A CEO of a very large health system -- and I had this conversation. And he completely accepted the premise of the nature of the problem. And his statement to me was, "I don't know who to give this problem to. It doesn't fit in our operating model today. It's not the job of HR, it's not the job of IT, and none of my operators actually know how to take us through this journey." So we're seeing all those levels in play if you are going to get the benefit of the technology.

And it's so true because as we were developing the model, one of the areas that we focus significantly on is workflow reengineering because it's really rethinking workflows with AI, and that's actually a reflection of maturity. John, we've seen a lot of those. So what are some of the misconceptions that you and I have observed?

Well, I've heard organizations talk about reinventing cybersecurity and risk processes, right, as if something completely new is needed to safely manage AI adoption, right? And I think there's some nuances and caveats there. I mean many of the things or most of the things that your organization is probably already doing are very applicable to managing AI technology and managing those risks, which we'll get into a little bit more detail about that. So the cybersecurity practices you have now are the good basis for safely adopting AI. Notably, this conversation is not specifically about Mythos and AI sort of in the attack, which we could probably have a separate webinar about, right? But start with the cybersecurity practices you already have. And then when you think about risk, right, it's not a separate risk function. It needs to be baked into the risk activities that you already do. From an enterprise risk perspective, it's really about this idea of cross-domain risk, right? Like if you have a large enterprise risk function and you already have certain domains, technology, information security reputation, like those conversations between risk areas are really important. They've always been important for technology, but now they're really important, and they support the kind of things that Kaveh is talking about, how do you really bring all the stakeholders on board? How do you understand maybe what impacts and other areas might be?

I think it's very important because we're not AI does not mean you throw away all the good of the discipline of any of the engineering software and cyber. And that is, in fact, a misconception that a lot of organizations go through. Majd, I find your dual role very interesting, and you see things probably from different perspectives. So anything else to add?

Sure. I'll mention two quick misconceptions. One is this laser focus of AI as an efficiency and productivity opportunity rather than seeing it as -- yes, productivity and efficiency alongside growth and potential reinvention of what you do. Products and services are all now up for being disrupted, being thought of differently and how do you offer these things has to be thought through. The other one is something that was already mentioned, which is if you get the tools and you train people on the tools, then you should see like where is the productivity or where is the innovation. And we recognize that there are many different things that need to happen along AI literacy building and then hands-on training. And then you have to start going to where Tony was speaking earlier on how do you integrate and operationalize AI in the day job of the individual. So you have to start thinking about role-based training, and you have to start thinking about domain, whether it's healthcare or financial services. That's where you need to take the training. And then you will start to see that people will integrate AI in their day-to-day work.

So we -- I'm following some of the audience comments, as we like there's an audience comment that summarizes the conversation as skill sets, tool sets and mindset. And I think that's a very nice way to bring it to summary. So Tony, I'd like to go to you next because as the CIO of a very large operation on the global scale, you have a perspective from the trenches that probably changes by the minute. So what keeps you awake at night [indiscernible]?

Yes. I mean the -- the challenge is not for me how to give people access to AI. What I really think about is how do you scale this and operationalize it. So what am I concerned about? I'm concerned about governance. As John mentioned, I think a lot about risk. And then I want to make sure that we have accountability for the outcomes that we should be driving from AI. And it's not just productivity, as Majd mentioned. And so for me, I really do think about how do we operationalize that. And I think that for me, AI maturity is not just how we deploy this. It's how do we do it responsibly, sustainably and end up with measurable business outcomes. If we do that, like that's what I'm worried about is how do we do those things, not how do we give people the technology.

Some of your responses actually are very relevant to some of the questions we got. So here, I'm going to pull an audience question that we got during the registration. And John, this has all your name written on it. The question for goal is, what are the cybersecurity challenges when adopting AI?

No, I think it depends on where you're at in your journey. And there are certainly risks, and there are certainly cybersecurity challenges, right? But if you're starting out and you're in the exploration or experimentation stage, you need to have the basic blocking and tackling of cybersecurity, right? Because there is there is certainly a data protection risk here, right? If you say go forward and experiment, the proper controls need to be in place so that your data is essentially not going out the door and being used to train models and things like that. So that basic blocking and tackling needs to be in place. And then as you're adopting more and more AI and your organization has a dependence on kind of the stability and the AI services and the trustworthiness, there are some risks that we already manage in cybersecurity, but that become a little bit more important, right? Like the integrity of data, the trustworthiness of data, controlling access and things like prompt injection and so forth. When you think about kind of towards the future ready and particularly agentic workflows, right, then we get more into how do you manage all those agents and how do you manage their access and how do you manage their entitlements, right? And this is something that a lot of organizations that I've seen, due to varying degrees like service account management, for example, but it just becomes particularly relevant as you go towards agentic. But the message really is do the blocking and tackling, think about the additional risk as you implement and put those controls in place. And then as you go to agentic, there are plenty of tools and resources out there to help people. The other thing I should mention with respect to the model, the model is the super set of practices for adopting AI. And there is cybersecurity content in the model and the specific kind of slice that is adoption-specific. But for anyone who's wondering it's not everything cybersecurity for AI, right? For example, you're not going to see specific content about vulnerability management. Why is that? Because you need to manage your vulnerabilities anyway, right? So it's the things you need to focus on that we think are very -- there's practices and processes that we think are very relevant to adopting AI.

And that's actually an important point because you need to have the practices to start with and how you're augmenting for AI and what are the AI-specific ones rather than just reinventing the wheel and some of the aspects. Unless there's anything else to add, I think it's time to start focusing on talking about the model and aspects of it. But I'd like to start with asking RP and Majd because what happened is about 1.5 years ago, RP and Majd came to the SEI with the challenge of developing a maturity model for AI adoption, so given the reality of the [ state ]. So why -- what triggered that, RP, if you would like to give us a little bit of historic background to why we started?

It's a very important and very interesting question. Thank you for asking. I think Majd will add to it. One of the key discussion point at the time when we thought about implementing AI, AI and the technology is changing at a rapid pace. The speed at which it changes, I call it as 1 hour is equal to 1 day, 1 day is equal to a month, 1 month is equal to a year of speed are you changing. So the speed at which AI technology is changing, enterprises needs a methodology, a mechanism, a process discipline to kind of go from point A to point B. I come from what Anita mentioned is what's [ Hamre School ], where if you want to go from point A to point B, you first need to know where you are on the map. This is what's famous [ court ], right? So we wanted to understand how do we get this methodology and maturity spectrum designed and built. Then who is better than SEI because you guys have the research, the experience and the expertise to look at systems, software engineering, multiple disciplines like security, configuration management that required project management, measurement area where Anita's expertise in there. So if you look at all of them, including the continuous evolution of the technology, we thought we should work along with the SEI. And thank you for the partnership. And that's the trigger, giving industry and enterprises model and the mechanism and the methodology to drive and implement the rapid pace of change of technology in a very structured way. That's kind of the context.

And that was actually a fun challenge for us to tackle. Majd?

Well, I mean, we've gone through many inflection points, and every one of them requires us to rethink how we do things and how we're going to approach to solve certain problems. And what are things that we need to challenge ourselves on so that we can do them better? And one of these things that AI is challenging is how does the enterprise rethink itself, what it offers and all of these things. And when you look at what was out there in terms of AI maturity model, unfortunately, there was a lot of initial efforts that were not -- that were narrowly focused, and they were not evidenced in research. And so we wanted to head in the direction where we can substantiate the work that we're going to put out there for the world to benefit from. And so we were driven by a lot of the agendas and strategies that were set by organizations and governments across the globe. So it was important for us that the model will be a global AI model, not necessarily North America-centric or something like that. And we wanted to make sure that it tackles all the different enterprises and organizations. And to double down on what RP said, coming to the SEI was a no-brainer. The global credibility that you hold and the history that -- of what you have accomplished in the past and how much you have benefited the world in improving their adoption of technology and software development gives a lot of credibility to us now taking all of that and enriching it with a lot of what Accenture brings to the table, which is process reinvention, how do we think about the workforce, how do we train the workforce. So it seems like a perfect partnership. And so far, it has been. So that's how the story came together.

Well, thanks a lot for that background. So let's share with the audience, a little summary of the maturity model, AI adoption maturity model that we've developed. And we've also gone through a number of early adopter programs. But before I get started and talking about the model with Anita and John, I would also like to pull two questions from the audience that came earlier on because this is important and we're going to address them. And one of them is from Leonardo, how much data is behind this framework and what companies have been surveyed, given how novel AI adoption is? Incredibly valid and very fair question. And the other one is for Marco, who asks, most maturity models assume a fixed end stage you climb toward. What model capability is improving faster than most organizations can execute a single phase of their road map? How do you build a maturity model that doesn't go stale? And where do you draw the line between sequencing adoption deliberately versus just staying adaptive because the ground keeps moving? Excellent questions. And in fact, we did not take those questions lightly. We ask those questions to ourselves from day 1 because there has been quite a number of successful initiatives in maturity modeling as an instrument. As we sit across these two organizations, SEI has been very fortunate and has changed some of those techniques, and we still use some of those like CMNC. So the way we approach this was how do you rethink AI adoption and maturity. That was the challenge that we ask ourselves. Our #1 activity was to look around. Because if there were already successful models, why not use them? So we did a very extensive literature search as well as survey and executive interviews in terms of looking in the challenges and what instruments they use. Of course, there are quite a number of techniques out there that look into AI readiness and maturities, some of those [ maleate ]. And within the next couple of weeks, we'll actually release that study. There are 136 examples. Some of them are only research efforts. Some of them are organizational techniques that are mostly internally focused, but there are other publicly available models as well. What we did not observe was the evidence of how they were developed or how they were used. So with that, and we also looked into a number of organizations as well as industry -- large-scale industry in terms of where their challenges are. So the way we rethought the AI adoption is, first of all, how do you focus on AI relevant capability? Because at the end of the day, AI is software. As the Software Engineering Institute, we very strongly believe that. And because AI is software, if you're not disciplined in your software, guess what, your AI adoption journey will fail. Same can be said for cybersecurity. So we try to focus on that delta that is focused on AI and AI adoption maturity. And how -- where will you provide value. Given that the model that we've developed is focused on AI-relevant capabilities, because a lot of organizations will [ die select ] to move fast. And there are a number of components that we inserted into the structure of the model that will allow us to adapt. One of them, we call context attributes, which will allow organizations to adjust where they focus as they are looking into understanding their maturity. And these context attributes will also allow us to develop industry-specific or sector-specific or particular workflow-specific aspects of the maturity. And the other aspect is the goal is not to define and identify a level. The goal is to really get to a road map. So there is the structure of the model at the end, which allows us to recommend where you would like to start where quick wins could be or where you really need to invest to get ahead of the curve. So that was the background that we developed it. And all of that data will -- we have, and we're going to share it through other means as we go through the next couple of weeks and months. So -- but of course, any maturity is a progression, and we did think through the progression as well. And Anita, will you please talk us through some of that levels as well?

Sure. Do you want to go to the next slide?

Sorry. There you go.

Okay. There you go. Thank you. So first, today, many organizations are pursuing AI adoption, but it's with a really vague AI everywhere mindset rather than a clearly defined way or a clearly defined strategy for approaching AI adoption. So when you look at this diagram, the AI adoption maturity model is really intended to help organizations determine the level of AI maturity to pursue and then also help them to guide management of its implementation road map. So when you think about maturity and you see these maturity levels, it's determined by the presence of intentional, repeatable, actionable and governed approaches. There need to be clear objectives standardized practices, and Ipek, you've talked about it, measurable outcomes and continuous improvement. Yesterday, I think Manish Sharma, he's the Chief Strategy and Services Officer at Accenture, he had made a post in LinkedIn. And he said that this is about how we create value differently. And that's what we're trying to do here is create value by applying AI, but getting to a very different set of outcomes. I think, Majd, you said yesterday, it should be a part of the new DNA of the organization. So that's another way to think about this. But since we talked a little bit about CMM or CMMI, this is unlike CMMI in that it is not about organizational certification. So the work that we've done here doesn't go into certification purposes.

And John, would you like to talk to some of the dimensions and the capabilities that we have?

Sure, maybe we'll go to the next slide.

Yes, we'll go to the next slide.

Just to kind of segue off what Anita said. It's not a certification. And what I've seen a little bit of my professional life is sometimes when you put a maturity model in front of technology executives who are in front of, I'll just say, type A people who want to achieve, they want to get to that level. They want to get to green. Well, it's not about getting to green. It's about identifying your gaps and areas for improvement and sort of challenging yourself, right? RP referred to it as like a catalyst guidance and catalyst in the forward to the document here. And that's really important. It should be that catalyst for just for where do you want to go and what do you need to get there. So with respect to the different dimensions, they're really divided into organizational change management and then engineering life cycle. I'm certainly not going to drill into the slide, but it's all of the areas considered essential for adopting AI, from workflow reengineering, to strategy, including how do you identify technology partners and people that can help your organization, to cybersecurity and basically to architectural aspects of adopting AI. One of the things that you don't see on the slide but that's in here as well is what we call maturity indicators. And for anyone who's familiar with CMM or CMMI and the generic practices, this is the modern or, I'll say, tailored version of maturity indicators for the model. How do you know these things will actually persist and you're really adopting that culture? Do you fund them correctly? Do you measure them correctly? Do you think about the risks of what you're doing well or not doing well in a certain area, right? So again, we could spend a lot of time looking at each of these. But this is all of the areas in the model that are essential for adopting AI and figuring out where you need to go.

And the model is publicly available, and we're happy to answer questions online as well.

Ipek, I wanted to build on some -- I think it was a question that you received from someone named Marco. Also, we are working really hard to keep this as a dynamic model. So as we are working with early adopter organizations and we have lessons learned, we are incorporating those lessons learned back into the model. So Version 1 of this is available, but we are going to keep it dynamic and keep it current by constantly refreshing with the lessons learned and the early adopter experiences that we have.

And then the question about climbing towards a fixed state is actually a very good question because one of the constant debates we had as we were both going through our early adopter program as well as developing some of the practices was how do you measure maturity and what does measuring maturity mean. And we actually have a dimension called maturity indicators. And John, maybe if you could say a couple of words about the maturity indicators as well? Because that's essential because the goal is not to climb to a state, but the goal is to be able to understand how those maturity indicators guide you through the process.

Sure. So they really revolve around accountability for the practices that are in place, measurement of the practices and then how you're actually resourcing those practices. So at lower levels, you're basically assigning accountable individuals, you're making sure that your policies and procedures and all of those things are fully in place to support adoption. And then this will sound familiar to people who are familiar with maturity models generally, right? As you go up the scale, you're thinking about the risks of gaps you may have in certain areas, and you're actually building that into your risk management process, right? It's not just we have some risks and we get together. It's we have risk, we formally recognize those. We figure out how important they are. As you go further up to scale, you're really assigning metrics, key risk indicators, key performance indicators and making sure that people have kind of the same picture. And then as you go further up to scale into future ready, this is where your organization is really striving to be a leader in the space to influence the community, right? I would say, this is where you're really kind of, I would say, spreading your wings, it's built into your business. So we use all of these -- we use words like repeatable and predictable. But at the higher maturity levels, it's not -- you don't really have to think about adopting AI. It is how you do business, right? But the -- I mean with that, that doesn't mean the model is only for organizations that want to get to that level. The model is for organizations who are trying to improve and who are trying to get on the path and get on the journey.

That's very true, and it's also driven by what you would like AI to accomplish. Kaveh says that all the time. It's not what AI can do, it's what AI should do, and that really defines the guidance. For this next question, RP, I'm going to put you on the spot because AI is a very technology-driven endeavor, and we're bringing a maturity modeling approach. So given some of your experience from the past, why is a maturity model approach is the right approach to this problem that organizations are facing with the fear of missing out versus the resources spend and the value to get out of it?

Oh, that's a lot of things in there. For example, one of the topic we discussed in the last 30 minutes is AI is a software and systems engineering. I always say, all the rules of software engineering and systems engineering applies still. I think if you go with that premises, and the way John mentioned in the model, we have -- what are the two typical aspects of implementation of any technology? One is the engineering aspect. Another one is people aspect, which is the change management aspect. As you go through the maturity adoption of implementation of the technology, your risk management, your change management, your expectations and business alignment continues to evolve, unchanged dynamically. That's the word that Anita was speaking, right? We need to get the dynamic inputs of implementation, both from the chain management aspect and engineering aspect of the technology that we implement. And that's the critical reason why a maturity framework, I call it as a guidebook to move forward for enterprises, to look at something industry-relevant, business aligned and that covers all the aspects of the software engineering and people aspect to drive the implementation and adoption in a true sense. That's why the aspect of maturity models is critical for me.

And we've definitely seen the value as we went through some of our applications. So I'll take another question from the questions that came previously because it's very relevant to how we thought through the -- both the development as well as the implementation of the model. And the question from Richard is, using your AI maturity model, how did you balance quick wins against long-term government staff capability building and sustainable institutional change? Is there a road map that can guarantee successful implementation while sustaining momentum? First of all, nothing can guarantee outcome. If you're not a disciplined organization, that's a people problem. With that, John?

Well, I mean, so with respect to quick wins. The question, I think, is quick wins versus how do we know this is going to last? How do we know this is going to be sustainable, right? So within the model and the approach is the idea of context attributes, right, that, first of all, context attributes to me really means what are the inherent risks and what are the important things to the organization. Right? If you're in a highly regulated organization, I spent a lot of time in finance, right, your approach to these areas might be a little bit different, right? So that, along with, frankly, an assessment, helps you identify your quick wins or the things that you need to do now or in the next 6 months, right? The long term is really where we get into like the maturity indicators and kind of going up to scale. How do you know that these practices are really going to be around in a year or 18 months and that you're actually going to be able to, we kind of say, emerging property, right? But that idea that you can just -- that you can do these things and you have to think a little bit less about it. That's what I would say about it.

For our purposes, quick win is not necessarily just check the boxes for the next practice. It's really prioritizing in terms of that road mapping approach. And it actually is very true. There's an audience comment, which I will share with my colleagues. AI is advancing rapidly, suggesting that models are also rapidly modernizing. In this scenario, the company has adopted AI. But if it doesn't keep evolving, it tends to [ allude ] competitiveness over time. Absolutely. And this is, in fact, some of the areas the capability is, in fact, focused on how you evolve and how you think about evolution, modernization and to be able to keep up with the space. So it's not a state in time, but it's really how you have practices to be able to continue to reinvent, as Majd says, as well with the evolving technology. So I would like to make sure we spend some time talking about our early adopter experience. And I'm going to turn to Tony here because Accenture Global IT graciously stepped up, and we had to convince them that it was worth their time and allowed us to do our very first early adopter pilot with their organization. So Tony, why did Accenture Global IP step up and try the AI maturity assessment out? And what was the journey and what has happened since then?

Yes. So I think for many -- just like many other enterprises, we knew that generative AI technology was going to be part of our landscape. We know and knew the kinds of outcomes that we expected to get to. But like for many organizations, you have to figure out what path you're going to take to get there. For us, the opportunity to participate in the research-based assessments seemed like an incredible opportunity to help us assess where we are and where we need to go. We found some of the strengths that we have in terms of like technical capabilities. We also identified some areas that we needed to focus on, like governance, value measurement, workforce management, workforce transformation. And coming out of there, I think we are able to use the assessment based on the model to create a road map for us so that we know where we're going to go. And we're using that to help guide our future now, especially with like our workforce transformation and some of the work that we're doing. And for me, it's pretty obvious. I talked to many other CIOs in my role. They're all facing sort of similar things, but I feel like we have a head start on that because of this assessment we took and because of the things that laid out and where we wanted to focus going forward. And that is driving our road map.

Thank you. So before I turn to Kaveh, I'm going to read the question so that there's a backdrop to this because I think this will resonate with as well. But it's -- so the question is, how do you distinguish between an organization that's exceptionally mature at automating all processes versus one that's mature at using AI to enable fundamentally new and establishing stakeholders? And I think you see this a lot within the healthcare and some of those that you actually look at that.

Great question. It actually highlights in many ways, what effectively the maturity model highlights, if you go from less mature and more mature. Because in the beginning, you're largely probing, experimenting, learning, dropping AI into existing processes and seeking optimization. When you get to more mature, the organization has made a commitment to change what it does fundamentally to get a benefit that doesn't exist today. And for that, you have to make a strategic decision that, that's actually an objective. And what's interesting in healthcare sector specifically is that healthcare organizations, particularly, let's -- I'm going to focus on delivery system rather than insurance, which is a little bit more financial in its orientation. At the end of the day, delivery system wants to do what it's always done. It wants to provide safe care, effective care, affordable care. It doesn't necessarily want to be in a different business. And so some of the commitments to change that you'll often see an organization -- so let's -- an analogy might be if you're an automotive company and you want an autonomous vehicle, you're not getting there without AI because it doesn't exist. There is no healthcare analog to that right now. But the nature of that commitment doesn't actually exist. And therefore, the kinds of organizational changes that you need to do aren't really an imperative. So what you'll see is a lot of organizations will use the maturity model, but once they realize what are the attributes of the more mature, they have to make a decision as to whether that really relates to where they're at. And I think it's important to not take the position that it's axiomatic that you must change. But to the extent that you have that as an outcome, you are going to have to change things that are very fundamental to your organization. Another way to think of it is, if -- the quick wins thing is interesting. The way I think about it is most healthcare organizations. What they say by a quick win is I'm going to drop the technology, and 80% of my benefit will be tech and 20% will be changing processes. That's quick. But most of the things that happen to have in healthcare is 20% tech and 80% work. There is no quick win in that kind of a change. And that, I think, is really a sector-specific, problem-specific decision that has to be made.

And it's also -- I think we've observed in some of the specific assessments, the assessment, in a way, helps them make that hard choice, right, that...

It illustrates to them the choices that must be made, and these are strategic choices.

Yes. Exactly. And John, any other points to a from other early adopter experiences that we've had?

I thought I would talk a little bit about scoping, and we experienced this during another early adopter experience. Right? And we could talk for a long time about scoping. But it's important to think about the problem you're trying to solve and how you're going to apply the model, right? A lot of times, enterprise leaders want to apply a tool like this at the enterprise level. After all, they're responsible for the enterprise. There are lots of different ways that can be used. You can do organizational unit. Of course, you can do an enterprise assessment. You can think about what is your maturity relative to a specific service that you're trying to support with AI. One of our early adopter experiences was in a safety critical industry. And it was not enterprise. We essentially assessed and appraised 2 organizational units, both of which were adopting AI for the purpose of software development. And one of the -- some of the findings where we noticed differences in their understanding of risk and also their understanding of the corporate process around risk, they were doing a great job, but they were managing risk in slightly different ways that I think was kind of a surprise to the people who were involved. So it was useful.

Some of the things that we also observed through these experiences is if you are already using AI for your product development and have that from some of the traditional machine learning AI, you actually adopt a little easier because you know some of the uncertainty aspects. And that really speaks to existing discipline in software, cybersecurity and of the other aspects which we've seen in some of the early adopter programs. So I'd like to take a couple of questions that we had come in because some of them are very interesting and also speak to the measurement and the challenge aspect. So this one is probably at the top of everybody's mind. How is token cost management factored into maturity? So any takers? Maybe I'll start with Tony because that's probably what keeps you awake at night that you don't share with us.

I was waiting for this question. I think that this is a great example of how you have to have maturity across your enterprise and as you move token economics overall and token cost management. And I think as you move from this world of experimentation to sort of visibility and control, you're going to have to have incredible discipline to understand the right places to implement this in your organization. You're going to have to have the discipline to be able to really do value measurement. And ultimately, this discipline is going to be required in order to build this into your overall budget, to your overall business strategy, how you price your services to what your costs are. And this ultimately is what you're going to use to calculate sort of the trade-off between cost and value for what's right for your business. And so my viewpoint in general is the discipline that you're going to get from having a strong model is what you're going to need to be able to do token economics throughout your enterprise to find the right value and the right amount of AI to use.

And John, maybe if we could answer from the perspective of the model, where we do focus on hard questions like that, that might help our audience?

Right. So just to be clear, you're not going to see a section in the model that says token costs. You're not going to see a section in the model that says maturity is less than a certain dollar amount. But what you will see when you read the model is there are a handful of practices and capability areas that very clearly address token cost and how to manage token cost. We're looking at the ROI behind automating workflows, the monitoring and some of the technology infrastructure practices. There's a slice or a down select, or for people who are very familiar with models, model scoping, right, that gets you to the question of token cost. There's not a section that says token cost, but it's certainly in there.

Absolutely. And it's a combination of where you're focusing and how you're actually evolving these. So Tony, there is interest from our audience in terms of understanding why Accenture IT has stepped up. So I'll read the question, and maybe you could -- some of it might be a repetition, but let's make sure we address it. So the question is -- right, here is the question. Seems Accenture IT did an assessment against the model. Yes, they did. And what was the objective of the assessment? And was it done at the start and our current state to determine progress and how and where the results used and useful? You did somehow talk to this, but maybe if you could...

Yes. I mean, I can confirm, we absolutely did that early stages. Our goals were both to assess ourselves and to give feedback and because we think that there's real value. I did comment that we think it helped us understand what some of our strengths are. We do think it helped us understand some areas where we need to focus. And so I want to make sure I get the question because I feel like I'm repeating myself. But we absolutely used it to lay out our road map for where we want to go because we think that it's hard to assess yourself, right? And it's hard to innovate from -- within your own company. This kind of assessment was very valuable for us to figure out and lay out the road map for where we want to go.

And also to emphasize there were dual goals there. One was to use this as an opportunity to see how the global IT was faring in their adoption journey, but it was also very important for the model team for the SEI and Accenture team to really see whether the model works. That was, in fact, one of the biggest goal, and we actually call it Pilot 0 because we got some feedback, for example, there are repetitive areas. Do we really need to spend as much time in those areas? And we folded that feedback into the model before we released the fabric, and there were -- and before we tried the actual early adopter assessments with actual organization. So there was that big goal in addition to the assessment goal as well. So it had multiple goals there.

And we will continue to do that.

And we will continue to do that because the technology evolves organizations evolve and our understanding of what AI adoption means evolve. In fact, when we started this journey, agentic AI was not what is agentic AI today. That's like -- and this is only we're talking about 12 months. So we have a question, and Majd, maybe this is something that you may want to chime in on. The question is, is the company's maturity in adopting AI linked to the number of models or the complexity of the problem being addressed also? I think this is a very good question. What do you say?

Can you say that one more time?

So the question is, is the company's maturity in adopting AI linked to the number of models or the complexity of the problem being addressed?

Well, it definitely has to do with what the organization is trying to accomplish and what are the processes that enable the organization to get there. There are many things that the model gets at. And one of them is what is your -- what is your strategy towards AI adoption and AI transformation. Another one is do we have the right governance, do we have the right data, platforms and access to the models. Doesn't have to do with the number of models that you're currently running internally. Of course, when you are managing a large number of models, then there's complexity in the economics and the budgets that you need to be able to roll them out. And you need to maintain them and you need to make sure that you have a constant flow of the data so that you can update these models and make sure that their accuracy is still relevant to you. One key component that maybe we didn't touch on is the workforce aspect of this. And the other one is the workflow aspect of this. So in an organization, you're probably visiting all of your processes and saying, how do we reinvent these processes end to end. And based on how we reinvent these processes end to end, the work changes. And now as the work changes, what kind of workforce do I need to be able to deliver on this work. And then -- so when the work evolves, based on all of the changes that you're thinking about, then you have to say, what are the roles inside my organization that I need to have? And then how do I train people for the roles that they are in? And as we've all been saying, evolution is ongoing. So this is not a one and done. This is something that will be constantly happening. So the power of the model is in assessing where you are as an organization. What are the strategic bets that you want to make because they are low-hanging fruit for your organization to make and to develop the capability to say, we know how to do this. More importantly, after you've done some early wins, you can start to have some medium- and long-term plans as well. Those are things that you have to think about and plan for. And you have to think about these things from the core thinking of the model so that you do them in a good way. So another thing that we don't want to forget, and I'm sure we're going to revisit this at closing, is this is an ongoing process for any organization as they're adopting AI. And so the model will help you recognize where you are, but this is an ongoing process that you will continue to assess, you'll continue to evaluate and you'll continue to evolve your processes, your governance, your people, just to make sure that you remain relevant and competitive in this market.

And for the number of models perspective, the word model is overloaded here. Number of generative AI models that you have within your organization are yours. This is, in fact, lock-in hedge betting that a lot of organizations are going to because their capabilities evolve so rapidly, but that is really depending on how you're actually mapping to your goals, as you said. So that's not necessarily the determinant. I would like to get to as many audience questions as possible, and we have a number of them coming. But there is one that we also received early on from Stefan. And the question is, I work with Department of War programs. And many want more AI capability. Who doesn't want more AI capability? Especially large language models, but struggle to bring them into the environment where they can keep them AI gapped and isolated. Do you see this problem getting solved sooner than later? Any additional thoughts in this topic will be appreciated. John?

Well, I think the -- I mean, there are architectures where you can do that, involving cloud tenants and so forth to operate AI in -- it's not fundamentally a different problem in terms of architecture and knowing where your data is going. So there are resources out there to solve this problem. And the DoD DOW is developing those capabilities. So we're headed there. I think there are some various administrative hurdles and things like that. But this is a solvable problem, I think. Both -- certainly in industry and certainly in the defense establishment as well.

Yes.

And we're already seeing that the DoW, they are building their own models. And they're able to deploy them, and these things are portable. And as John said, you can host them on your internal cloud infrastructure that is [ air gap ]. So definitely these things are evolving pretty rapidly and the DoW is leading the way there.

And in fact, we see in our engagements a lot of the organizations are trying to bring some of them within controlled environments to the extent possible to help with their software engineering and software development capabilities to provide them to their software developers, which is a usage scenario. They are still not looking at from the perspective of how does AI change the work, how does AI change some of the mission aspects, but we have initiatives within the Software Engineering Institute looking into those as well. So I think we'll see an acceleration as you mentioned, John, but there are different scenarios that are already happening.

Ipek, we're seeing a lot of programs apply things in -- well, like in dual environments. They'll use the traditional ways of doing things, but also start applying some of the new AI models. So we're trying -- we're seeing a lot of the dual ways of accomplishing the missions.

True. There's a share of that. I'll take audience -- another audience question. We have a number of interesting questions coming in. So the question is, what signs are expected that AI is indeed driving evolution within the company? It's a good one. Any want to take?

When you don't think about AI.

It's just the way that you do your business.

The way that you do your business. Any other?

I think if we can -- when an enterprise is seeing the positive trajectory of the business growth -- and like any technology in the past, if technology is industrialized, institutionalized in an environment, and this is the way people do it every day, like what Majd said. It is the way of working. And thereby, the exponential growth of business, when you see that, that means your technology is in action and the maturity has come in. And the spectrum of maturity changes, and that's the way of thinking. That's how I see it.

One other dimension is that there are either strategy or tactics that simply wouldn't be possible without AI. You couldn't remove it anymore. In many cases, when you ask that question to organizations before they -- in the early stages of maturity, you ask them if there's any part of their strategy that cannot be accomplished without AI. They'll often say, no, anything can be done. A, I would make it better, but I don't need it. When you're on the other side of that, there are things the business won't exist without AI. It's a fundamental requirement of the decision that you've made, and I think that would be another way to know.

Yes. And a progression that is also shared in literature is whether you're augmenting, adapting and autonomously changing. With augmenting, you might be just increasing already existing automation but using AI, but as Kaveh mentioned and automation, existing automation is just with AI. With adapt, you maybe have a little bit more reliance, but with autonomous when this doesn't mean everything needs to be autonomous, but you actually have really changed the process steps and the floor to it. So that's another way that you could use as a...

Ipek, there's another thought there that if you have many AI pilots and many early wins, that means your organization is AI ready. And that is a misconception that we also have to address, right? So the fact that you have several champions or several AI projects that are currently running does not mean that the organization as a whole can achieve this natural progression of an idea all the way to production in the process that RP just described. So we have to make sure that we differentiate between these two and not force ourselves to start to say that we are AI ready and we are an AI-first organization just because look at the number of projects we have going. You really have to think about all of the different dimensions of the model that Anita and John described.

And this is actually a very important point, Majd. And thanks for bringing that up because we do see and we did see a lot in terms of when we ask, for example, how do you measure success. Or is the number of people using whatever is your favorite AI-based tool. Or we have a number of use cases that are with AI, which is an aspect of the experimentation, but it is not adoption or it's not necessarily evolution. We have a lot of very good questions. So there is 1 that we received early on, which I would like to focus that from there. And the question is, what section of the maturity model do you see as being highly important, but most widely missing in enterprises today? So I'll look at Kaveh, maybe?

No, I think the work in the workforce. I think that's -- because it's the hardest and it's often viewed as -- this is still viewed as a technology problem. At least that's my observation.

But Ipek, to build on Kaveh's point. It's a cultural problem also. So when you talk about workforce and people, you're talking about changing the culture as well.

So if I may amplify what was just said there. All organizations are going to have access to the best AI out there. So if you think by just enabling that capability inside your organization, you're going to be able to offer differentiated products and services, you're missing the point. So it is the people that are going to help. So it's their ideas, their creativity, their approach, their strategy. So back to where we started and how you open, Ipek, it's what can your people do with this capability. You are integrating that capability within your organization, but then how do you unlock that what your people can do with it, and that's going to be what shows up in the market against your competitors.

Absolutely.

I would also the workflow and people aspect, absolutely the most important part of the model and the differentiator. Purely from a technology management perspective, a lot of the model content around ecosystem, right? How do you actually integrate with your legacy environment and scale is sort of in the upper part of the model or the "higher maturity," very important from how do you really scale this in the long term. I mean, without a doubt, it's people first. But there are technology management pieces of the model that I think will be very beneficial.

And in fact, if you're not building the right partnerships, you're not going to be able to succeed. And that is how the AI ecosystem is evolving, whether we like it or not.

I'd like to channel Tony here a little bit and say governance requires attention. It sometimes is seen as a burden, but it's really a major unlock if you have the right governance mechanisms in place. Tony, any thoughts on that?

Yes. I mean I agree, Majd. I mean what was resonating with me when you made those comments is like the easiest thing nowadays is just giving people the technology. And when you're really thinking about what's going to help enterprises being successful. I mean, you start with like did we achieve the outcomes that we wanted to achieve. And then you go back and you can look at -- did the technology help us to get there. And if you don't enable the workforce and you don't do the things you guys were commenting on, you're just not going to get there. So you start with the outcomes, think about the workforce, how you're going to enable people, what's going to be differentiating, completely agree. And if you don't underpin that with governance, I don't think you could be successful.

Can I amplify 1 issue? That -- I think people don't realize until they're in the middle of it. The technology arm has a technology and has a data piece. And actually, we find a lot of the organizations, their data is not AI-ready at all. And so I'll go healthcare specifically, we have all kinds of standards around transport and other attributes. But the problem with data for AI is that it needs to be accessible in a semantically and contextually useful way. And it is a myth that language models can solve that problem. And we're discovering that right now, and it's becoming a huge source of frustration. There was a theory that you could take dirty data, throw an LLM at it, and it will work. That is absolutely not true. And because it's not true, all the data is sitting around not being useful, and you have to go back and think about this in a completely different way. And because the semantic standards that you need will never be solved by forcing the creators to create it in a standard way, there's a lot of heavy lifting that goes between what you have and what you need to actually ingest into any form of AI.

So there is an interesting question which selfishly, I would like to take because this is something that John and I and my colleagues have had endless conversations. The question is, there's local optimization, business units, individuals, and organizational optimization. From an organizational performance perspective, how does the AI maturity model measure this? Which is a scoping question. Right, John?

What's -- so it can do both.

Well, yes.

But there's a lot to be said for having a defined scope, right? And really thinking about -- I mean, let's say, you're a multinational or a nationwide company. Yes, you could do an enterprise assessment, and that can be very valuable. And frankly, all of these activities cost money at some level. So there's a certain sense to that. But there's also a sense that says, I really want to have an exemplar within the organization. I really think this is the area where I should focus. So there's 2 ways to approach it. But I would -- I mean if you're in a large organization, I would really counsel people to think about that sort of model speak subcomponent piece of the organization that you really want to focus on. Because you can get things that are more actionable sometimes when you really focus on what you're trying to achieve.

And the context attributes aspect of the model also help you understand whether you're focusing on the business unit or if you need to focus on the enterprise, how do you need to approach that? That might require you to do the assessment with -- especially if you're a large global organization, you might need to do that with multiple business units to be able to get at enterprise level understanding, but the way we've set the model up is you could actually get an enterprise level understanding, especially of your gaps as easily as well. And I invite the person who asked the question to look at the booklet that we shared of the model and look into some of the experiences that we'll be sharing as we go on. So we cannot talk about AI adoption without talking about humans. So the next question is something that we've received during the registration. So most everything we hear nowadays suggests humans are expandable and will become eventually AI's waste product. On the surface, AI CEOs are starting to backpedal on this messaging, but there's some truth to it. How does CMU SEI see this whole thing going? Where are the opportunities for human in the loop? We call it human in the lead. I guess that's Anita? I have to ask that to you.

That's a lot to unpack, but I did see RP cringe when you asked that question. Yes, I mean, I guess, the SEI's work really emphasizes human-centered AI engineering, not human replacement. And for example, humans aren't just approving AI outputs, they're codesigning guardrails, detecting drift, recalibrating systems that are in production. So there's a lot of different roles there that work in tandem. SEI's research and decision explainability helps human spot bias, hallucination or mission drift before harm can occur. So that's also an important area of research. We've also talked about humans in the loop. It's -- they're feedback loops, and we can use that as training data. So it's not just oversight. It's continuous learning. Humans label edge cases, correct model assumptions and signal when the context has shifted. So again, there's a lot of different and important roles that work in tandem. We also have some very active work active learning pipelines where there are domain experts that curate high-impact samples to retrain the models themselves. So making the loop efficient, not just human in the loop as a check box, but really making it an efficient part of the process. So for us, some of our research is if you're building or deploying AI, you've got a design for fallback. How does the human take over if the AI fails or stalls or is ambiguous. That's sort of one area. Measuring human in the loop efficacy, not just is there a human, but how effective is their input. Is it real time, can they improve the models as well. And then the other thing that I thought about was upscaling teams in AI literacy. So operators need to understand model limitations, not just to code the models, but to question and even interrogate them. So there's a lot of really important roles that have to work together. So the future isn't humans versus AI, it's humans plus AI.

Right. Majd?

One thing to add here is that if we're not careful with this overpush of just use AI as is, we are also pushing human beings at some point to relinquishing cognitive capability when they're utilizing AI. And while working with our CHRO last week here, we were thinking about how do you -- as you're training your workforce and as you're evolving your workforce -- RP, you were there -- we were discussing on how do you retain cognitive ownership as well. So as you are training your workforce to leverage this capability inside the organization, you have to be principled and intentional about retaining the cognitive ability of your workforce and what value they bring into the discussion that they're participating and when they're leveraging AI. So a little bit of a caution tail here that this reckless abandon of quick to use the tools and then the wrong incentives to amplify the use of the tools without thinking about where do we want humans, as Anita mentioned. And we want to make sure that they retain their cognitive ability and don't start just providing an average of this is what AI produced, and that's the best that we can produce. To maintain agency over all of that is critical as we do this.

Right. So there's a joke in healthcare right now that says all -- any doctor that can be replaced by AI should be replaced. And that's a real statement about...

I like that.

You're not really worth your salt if you can be replaced by AI because you're not doing enough important work anyway. I think about that through the lens of low complex cognitive test, high complex cognitive tests. And in healthcare delivery, specifically, there's also physical test. To take a job, AI can only do parts of those things. And in certain sectors like healthcare, we have a raw shortage of high complex cognitive capabilities. I need to liberate as much of that as possible by taking low cognitive abilities and moving them somewhere so I can give somebody more time. That, by the way, is rate limited because a person can't work at peak cognitive capacity all the time. We have another issue in healthcare, in particular, which is who's responsible for the liability associated with an error. So think about the question about a patient having a direct interaction with a bot for a diagnosis. If it's wrong, whose problem is it? The patient? Is the patient going to have the liability for that? Is the tech company? Guarantee you, neither of them are. As a society, we have no liability construct that doesn't include a learn-it-intermediary. And that will be a big journey that will have nothing to do with the AI companies for us to ever get to that stage.

And it's also important that tools are increasingly having the AI capable -- the features. So that doesn't mean all of a sudden tools are taking over as well. You really need to understand it from the human's responsibility perspective. And at the end of the day, AI is a tool that actually is an exciting tool. Well, we're fast coming to the end of the time, this really flew by. So I would like to give our -- my guess, an opportunity to close, and I very much appreciate all the exciting conversation, we could go on and on. So as organizations move from experimentation to enterprise scale AI adoption, which is still a journey, not a destination that anyone really have effectively reached, what do you believe will become the most critical differentiator for long-term success? And what is one recommendation you would have for our listeners? And Kaveh, why don't we get started with you?

Yes. Actually, I think 3 of them. One of them is you have to get the why before the how. That's a strategic question because that drives everything. The second one we're seeing is the CEO has to personally be the loudest voice with the most leaning in effort. It's not a task you hand to anybody else. I've seen this repeatedly. If it's struggling and the CEO doesn't be -- doesn't wait, lean in and put their own personal time and effort, that's a big one. And then the third one is recognizing that the -- that you don't gain the organizational benefit without changing the work. You can prove the AI works, but you don't get the benefit until you change the work.

John?

A CEO or a leader who is unafraid to actually ask the why question. And then this is a really old-fashioned answer, but human teamwork, right? The leaders in the organization really having honest conversations and teaming together, if you're very siloed, that's not going to -- bad idea, right? It's the human teamwork that's really going to drive the vision forward for the organization.

Majd?

I would say that we have to think about AI as a capability that we are building inside the organization. And then that has to be coupled with a culture of experimentation. And what does the culture of experimentation mean given that Tony is in the audience is that we have to make sure that the token economy takes into account these kinds of things. And the culture of experimentation is budgeting for time and money for throwaways so that you can explore and experiment into what actually is going to stick and move the needle in the right direction. The last thing I'll say is this is never a one and done. This is an ongoing process. So the organization needs to be in that mindset that we will continuously update ourselves and we'll continuously disrupt ourselves so that we can continue to lead in terms of what we can accomplish.

Tony?

Yes. I think for me, it's organizations that are going to have a structured maturity approach as they figure out how to bring AI through the organization. And you have to have balance in that structure. You need to have experimentation. You need to understand your costs, you need to think about your security. And you need to then think about the outcome, the business outcomes that you're going to try to achieve. And a structured approach will allow you to balance all of those things. We read a lot that experimentation is very important, but you have to move past experimentation at some point. So how do you balance all 4 of those things with a structured approach as I think the organizations that master that are the ones that are going to be successful.

RP?

I think the model is out there, like you said. And there are 2 critical components that we discuss today. One is organizational competency and then engineering competency. That kind of summarizes how an enterprise need to move forward in adopting AI. I think it is simple, scale and sustainable. We talked a lot about scalability. I think it is 1 step after scalability is how do I continuously sustain the business value and the AI implementation that I put in. So I would like to see all the enterprises move from scale to sustaining the business value by leveraging organizational competencies and engineering competencies. I also want to thank SEI team for partnering this with building the model, bringing our expertise. I think as we move forward, there's a lot to learn. We can continue to help enterprises to get trained on the model, get more expertise in the model. We can help enterprises to baseline their maturity processes and working along with the SEI to build the road map. So there's a lot we can do. We can contribute to the industry as we move forward, and we will continue to get the feedback from the field and strengthen the model and organizational and engineering competencies, thereby we can scale and then such day. Thank you.

Thank you. And Anita, bring us home.

Okay. So I agree with all of mym colleagues. But I think I -- when you had talked about misconceptions very early on, I'll go back very quickly to that, that, I think, long-term success has to be around organizational alignment around it on AI as a strategic capability, not only a tech initiative. But everyone here talked about the learning organization. So not only do organizations have to be learning about how to adopt AI effectively, efficiently and with the AI adoption maturity model as a road map, but I think even for us as the general community to share data, to share experiences, so we can be a learning AI adoption maturity model community. So we can feed that back in to the road map and make it current and based on what actually works out in the community.

And in fact, the successes of the past in using maturity approaches have been defining practice in a consistent way across organizations globally, and that's our hope is with this AI adoption, which is model up. Well, thank you to all of you. And it's only through collective experiences like these and collaborations between industry, government and academia will actually be able to solve this AI problem because the technology moves fast. It's exciting and it's scary all at the same time. And our goal with the AI adoption maturity model is to help serve that purpose and shape some of the broader practices to come along. We -- while this is the first phase, established the maturity model, which you are able to read today, the second phase will include building the empirical data and the basis for different kinds of use cases. We will continue to share our experiences using the model as well as some of the learnings, and we'll fold it back into the evolution of the model. And we'll expect these experiences to provide critical insights to all of us as we understand what AI adoption really means. There were questions there, which we couldn't get to, for example. Well, there are tools out there that change by the minute. How do you give guidance to tool usage? We've talked about whether scaling a particular tool really means maturity. So all of these questions are valid questions and our goal in developing the practices and the capability area as well that you would actually find answers as you're developing your AI strategy as you go through the AI adoption experience and define your organizational or team or business unit goals. And we extend our deepest gratitude to everyone who contributed. There are the faces you see, but there are teams behind us, and we appreciate everybody's contribution. But also, we appreciate the contribution of our early adopters who trusted us with their problems, challenges and their data and everyone who talked to us and who shared their experiences and data with us along the way. And we're looking forward to working together with Accenture and all the partners and our collaborators. And we'll be sharing our findings and the data, and we welcome your feedback as well. Anita, John, Majd, Kaveh, RP and Tony, thank you. I know I've spent those 90 minutes with the busiest people I know. So really, really appreciate it, and I know our audience appreciate it as well. So at the end of the webinar, we ask you that you complete our survey and the link will be posted on the YouTube chat area now. And we appreciate your feedback now or later. You can send all your questions to [email protected], and you'll be able to get the links to the references that we made throughout the conversation. Thank you for spending your last 90 minutes with us, and looking forward to the conversation as we all understand what AI can do for our organizations for our workflows and make our lives better without introducing the risks. Thank you, everyone.