Moody's Corporation (MCO) Earnings Call Transcript & Summary

All right. Well, hello, and welcome, everyone. Very excited to be hosting this webinar today. This is due to the significant demand we've had from investors to see the research assistant and see a demo. So I'm very excited to be joined by Steve Tulenko, the President of MA and members of his team. The agenda for today will be that Steve will say a few words, providing an update on our strategy for GenAI. We'll then have a product demo of the research assistant. And then we will open it up to Q&A. If you could please put any of your questions into the channel, the track channel, we will make sure that we do our best to cover as many as possible. But with that, Steve, happy to hand over to you.

Good morning, everyone, in New York. Good afternoon in Europe. Here in Asia, good evening. I hope everybody is doing well. It's the holiday season, so what better way to spend, enjoy our festivities, then talk about research assistance. Shivani is right, we've had a lot of excitement. And in our meetings with investors over the last several weeks, we are getting many, many questions and inquiries as to how does this thing work? How did you do this? And for that matter, can you show it to us. So we thought we would do exactly that. So you've got an informal session here. This is really a demonstration, very similar to what we might do for customers to give you a feel for how we -- what we've created here, maybe a little bit of a sense for how we've built [indiscernible], what kind of work we do in terms of product development and where this all came from. I thought it would be helpful just to frame this with a little bit of perspective. We've talked a lot about innovation as it relates to generative AI and machine learning and AI generally -- [Technical Difficulty] last year, but maybe start off with an acknowledgment that the transformer model, which is sort of this famous model that started the GPT and large language model -- excitement in 2023 really got it started in 2017. The famous paper was released in 2017. We started doing work in experiments with that technology right around that time. We actually created a tool that analyzes the sentiment of text, especially as it related to new stories to understand whether there were credit implications and whether they were positive or negative in those new stories. And so we've got a few years of experience in terms of working through natural language processing and deriving sentiment from text, we actually were experimenting with these transform models a couple of years ago. And that set us up, I think, in good stead for us to embrace what I think has been a generational opportunity for us and for us, knowledge workers and for all of you knowledge workers out there to leverage this new generative AI capability. And really, in February or March, we started experimenting in earnest with the advent of -- you all are probably aware that chat GPT was released, I think it was November 30, 2022. The GPT-3, version 3 was really impressive. And we certainly had a strong conviction that this was going to be very helpful for us going forward. We already see tremendous improvements with 3.5 and 4.0 and then all of the other models that many of you have probably experimented with as well. We are excited to engage with many of the other tools that are out there from an experimental purpose and explore the different benefits that are available through the other forms of large language models that are out there. We created a group in March. We called it the Generative Intelligence group, might have been late March, and that was the team where we consolidated a lot of our experiments and our, I'll call it, innovation with purpose initiatives and try to create a center of excellence that we could then share best practice throughout the company. They created a copilot capability with us or for us so that we could work with this technology internally, not just in Moody's Analytics, but also in the Rating Agency. And it was through our partnership with Microsoft that we were able to roll out that co-pilot literally firm-wide through the Teams application. So literally, every person in Moody's, all 14,000 of us, can now think of themselves as an innovator, leveraging GenAI capabilities. You may remember in an earnings call, in the summertime, we announced that we were going to be building a product and releasing a product. We moved through the preview stage in the third quarter, gathering feedback. We worked with scores of customers and many, many demonstrations. We engaged with a couple of dozen -- actually, quarter or 50 in a much more, I think, a serious way, gathering feedback, learning where we -- where they thought we would be able to add value and identifying those areas where we really needed to make tweaks in order for this most -- sorry, for this minimum viable product to be ready for a corporate or commercial release, which we have done as of the first week of December. So we've gone through our previous stage, made the changes, iterated basically constantly for 6 months or so and are now at a point where we're making this available to our customer base as part of our commercial launch here in the first week of December. So maybe I should add, there are several of these projects going on. We're finding this GenAI capability to be tremendously valuable for everything from our lending solutions to our regulatory capital solutions, to our commercial real estate efforts and beyond. And we have several projects underway to release additional assistant or GenAI capabilities to support our customer base, already. Some of those are in preview now. And some of those will be released in the first quarter as we move through 2024. So I will not take any more time, but instead hand over the call to Cristina Pieretti and Sylvia Beck. Cristina leads our effort related to our website and to our service that you undoubtedly have heard of called CreditView, and Sylvia is a member of the product team. And they'll walk us through a demonstration to give you a sense for how this works and what we can do with this initial launch of our GenAI capabilities here at Moody's. So Cristina, take it away.

Thank you, Steve. I just realized my background, I think it's the only way. So I'll fix that during the call. But -- so it's a pleasure to be with all of you today. As Steve mentioned, we are very excited with the launch of Research Assistant. We've had great, great feedback, both from the investor community and from the customers we've shown it. I've been very engaged in customer meetings over the last month. And I've been 16 years in Moody's, and I've never seen the reactions I've seen from the customers and the engagement when we show the product. So I hope that you are as excited as -- or I'm sure that you will be as excited when Sylvia goes through the product. This is part of the CreditView franchise. It's an add-on. It works -- it's, I think, a great way of leveraging the Moody's content because it puts together GenAI technology that as you all know, it's extremely powerful with a proprietary content from Moody's. Basically, Research Data and Analytics that come from the Rating Agency. Before I pass it to Sylvia, I want to highlight a couple of things. It is real time because of the way we've developed it. So we're not fine-tuning a model, but using RAC technology, and Sylvia will talk a little bit more about that. So that allows us to -- since we don't have to train a model, we can pull the moment the Rating Agency publishes a piece of research, it's immediately available in the Research Assistant. The second thing I would like to highlight is the trust that we've been able to create among the prospective user community because as you see in the demo, we always provide a citation on the answers. And we've made a lot of work making sure that the research assistant only answers the questions for which we have Moody's information available. So we spend a lot of time making sure that if it doesn't know the answer, because Moody's doesn't have information on that topic, it will stay, I don't know as opposed to going to the outer world and looking for an answer. And that has been incredibly valued by our customers. I think with that, I'll pass it to Sylvia, who is going to ask some problems to the -- some problems to the Research Assistant, and then we will make more comments about how it works.

Thank you for that intro, Steve and Cristina. Hello, everyone. My name is Sylvia Beck, and I work in the product team within Moody's. I'm very happy to kind of present to you guys today Moody's Research Assistant, which, as we've all been mentioning, it's an AI-powered assistant that enables users to quickly access information from across all of CreditView to assist you with identifying opportunities, mitigating risk or even something as simple as helping you find relevant research, all using the power of generative AI. The Assistant offers insight at micro, macro institutional levels, and it can also help you very quickly synthesize vast amounts of information all for your analysis. As we kind of mentioned a little bit before, just for some background, it is built on the OpenAI GPT-4 model, which it is in a controlled environment inside Microsoft's back end, which nobody else has access to. And what that really means is that our usage and our clients' usage is very tightly protected in our infrastructure. So the questions and the answers provided through Research Assistant are tightly permissioned and governed within our systems. What we have done on top of ChatGPT is we have implemented our own Retrieval-augmented generation, you've kind of heard it referenced as RAG for short, of our own logic on top of the large language model. And RAGs are really used for information retrieval. So what it does in essence is it helps guide the Research Assistant, look it across the content of data that we have given access to and helps to determine where should it look to find the information to the users prompts. So at the end of the day, while all that stuff really means is that Moody's Research Assistant, unlike other assistance or chatbot you see out in the market today is the only one that has access to Moody's proprietary data as Cristina mentioned -- and that's ratings, research and financial data. [ Undo ] the tool, we kind of enable you to utilize that content in your analysis. So with that preamble, let me move on to the demos. I'm sure you guys are very interested in seeing the tool in real time. All right. So the Moody's Research Assistant start screen is laid out very simply. It's probably very similar to what you saw for ChatGPT, where we have a list of sample questions that the users can select from to get started, a list of the capabilities but also what are the limitations. So users know exactly how to ground their questions and what can they really engage with the assistant about. And I'm sure many of you are very curious. So what does the Research Assistant actually do. So I will start off with really just a common workflow for a lot of our users. I will ask it to write me an investment report on Apple, include the company description, provide key ratings drivers, top 3 peers with their ratings drivers, sector, outlook and key indicators. And you can be very specific on what kind of sections you might want it to include in your prompt. So now you can see here the Research Assistant working in real time. When you submit your prompt, what its first trying to do is understand your intent. So what are you really asking me a question about? Is it financial information? Do you want me to look through our research, et cetera. So as it determines your intent, it will then go and call multiple different APIs to pull in the information for what you're asking about. So something like this, which many of our customers are already doing when they come to CreditView today, which normally takes them anywhere between like 2 to 4 hours because you have to find the relevant research, read the research, synthesize it, look at financial data, determine who are the peers, et cetera. So something that normally took you anywhere between 2 to 4 hours, you can see it being streamed here in real time in a matter of seconds or a few minutes. And that's really the huge benefit, I think, of Research Assistant. I think as Steve was kind of mentioning, the time savings is a really big quality that our users are very happy with. And on average, what they've been telling us is that somewhere between 25% to 30% of the time that they normally spend trying to do this is being saved. And so if you think about for the end user, what we can do with that time that you're saving, it's pretty much a gold mine for them. Another thing I want to point out here, as Cristina was mentioning, when it comes to GenAI, most people are very concerned that things are just being made up. So how are we kind of solving for this? What we did, and as you can see, when I scroll up and show you the answer, there are many kind of footnotes here. And what those footnotes kind of denotes are the citations. And the citations are linked to things in CreditView. It's our research. It's where we're actually going for the source information to then consolidate and provide as part of Research Assistant. So you can actually open up any of these links and see exactly where that sourced information is, whether it's a landing page such as this one or the latest credit opinion. And it's actually a really great thing for -- giving you that visibility as to where the information is actually coming from. The second quality that I do want to point out here is that it will also provide you some additional questions that you can ask it. So it knows that you asked a question about Topic A. So it will also provide you some other questions about the same topic that you can ask it. And this helps you with your analytical journey, and it keeps the conversation going within Research Assistant. And the last thing here, which I think is, call it the best feature in my opinion, is the feedback loop. So for every response at the Research Assistant will give you, you can actually give feedback back to us, letting us know whether or not the response was good or bad and for what reasons. So I'll give you an example here, where I could say it was very good. I could say it was very helpful or maybe it was accurate, but not very helpful. If that was bad, here are a couple of other options that you could choose from. Maybe the citation wasn't very good, et cetera. But you also have the option of entering additional feedback here. And we regularly will review this and it helps us to better understand how the Research Assistant is actually performing. And through the feedback, we will look about -- look for ways to improve Research Assistant and also the RAG model that we have developed. Oh, actually, I forgot because you were developing on such a quick speed. You can also now very easily copy your response. To say this is like a rough draft of the investment report that you are writing and you want to add some supplemental information, you can easily copy this, go into something like the word document, paste it. You can see it's very nicely format. It has all the hyperlink, the citations, et cetera. And you can just very easily use this drop to a credit -- I'm sorry, of our investment report and then add any additional content as you see fit, which I think is very impressive. But the tool can do more than just summarize and put together research documents for your perusal. You can also ask it to create content for you, such as a table or a chart. So let me show you an example for something like that. Great chart of [indiscernible] over the last features.

So Sylvia, this might even be interesting to equity investors, not just classic credit type that might work with CreditView. You can imagine leveraging Moody's content to address questions and additional use cases.

Yes, very much so. So again, here, RA's determining intent and is trying to stream the response back to them real time. It's going to look a little weird at first because it's trying to create adjacent structure for the large language model to then read and interpret. But I promise, when it's finished streaming the responses back to you, it will be a chart.

Yes. And in fact, that was a deliberate decision we made because, well, it's crafting the response, there's a wait time, right? So we want to -- want to convey to the users that we are working on the answer as opposed to having a well that its -- to have them waiting with no interaction.

Yes. So you can see here, as I promised, it did render a nice graphic. But if you wanted this in a different format, such as a column chart or a pie chart or even a table, the Research Assistant can do that as well. You just have to be very explicit about that in your prompt. Let's try a different type of question. I'll say, what are the most recent rating actions in the bankings? Answer me in French.

You can imagine we have many CreditView customers that are in large financial centers. English is often very common. But as we expand into other geographies, language and local language can be more important. I think Japan, I think Brazil, certain parts of France, for example.

Yes, Spain, Latin America, a lot of customers with -- banking customers that are either based on these territories or have subsidiaries where the first language is Spanish. A couple of things, while this is working that I wanted to note. And you might have -- we can show it again after we finish with the questions, but the Research Assistant is placed side by side from the search box, right? So it's very, very easy to find and very easy to interact. You don't have to go to somewhere else. And you might say, well, what is that important? I'm going to tell you a little bit of an anecdote from a customer meeting I had last week. So it was -- basically, we were talking about CreditView, right? It was one of the big global banks. And the user, which is a core-core user of our product set, we talked about the new enhancement about Research Assistant, and the first reaction was, well, I don't really that -- I don't really use that technology. My kids use more of that, but I'm not sure I would use it at work, right? And we said, well, wait until you see a demo. So first, tier have to go anywhere. It was right there by the search box. And then it's completely conversational, right? You don't really have to learn about anything to use it. By the end of the meeting, she was like this could be a huge, huge efficiency play for our analysts. And I want to use that opportunity to recover a little bit the stats that Sylvia mentioned. When you think about the workflow of something that uses CreditView, but as Steve mentioned, it could not only be a CreditView user, this expands the scope of the profile of users significantly. You do data collection, right? You want to answer a question around that investment, a lending decision or entering some sort of commercial relationship with a third party. You're going to -- first, you have to collect the data. And this collects the data -- Sylvia mentioning in seconds, right? You don't have to go through 3 different pages or 3 different places or more to collect information on the company, the sector, the peers, but it's right there. We've observed, and when I say we've observed, it's from the trials that we had during the preview phase with customers from the usage of the tool from MIS analysts and also from industry publications that the savings from that collection phase are around 80%. Then you do analysis. We've observed that in the analysis, the saving because of things like Sylvia said, because it does the pure comparison, because it does charge, because the render saving the language you can, it can be around 50%. And overall, in the whole decision process, it saves between 25% and 30%. So I hope that gives you an idea of the value that we see in the tool and that the market is seeing in the tool.

Yes. I think those are really great points, Cristina. And I just want to kind of show you an example here that the Research Assistant can accommodate more than just the English language. You can ask it to answer you in your native language. You can also ask the question itself in your native language. I would have run this in French, but unfortunately, I have not taken French in high schools, but very rusty. So maybe I'll just close out with just show casings. We've kind of said the Research Assistant is a very trusted source. It only looks at the scope of content that we have given an access to, to be able to answer questions on. So what would happen if I actually asked a question that we don't maybe mention our research or have data on, and I think it's a topic we are all probably very interested in. What's the current stock price of Moody's?

So maybe while this is happening, let's just remind people, the RAG, the Retrieval-augmented generation, the prompting and the engineering we've done here is to try and determine the intent of the question and then collect the information that might be relevant to help address that question and generate text associated with that question. So here, you've got a good case of something where we actually don't have information on stock prices in this data center in this corpus. We have not -- we're not able to present that to the LLM, and there may be information in the LLM's training that would potentially create a hallucination, but we've engineered this to try and prevent that from happening. So in addition to this being up-to-date with real-time content for Moody's, it's also designed to prevent or at least mitigate the risk associated with the hallucination. So you've got a response here as Sylvia has indicated, to give you a sense for. We really don't know that one. Maybe I want to take a look at these links, but this is not my strong search.

Yes, exactly.

Yes, go ahead.

Oh, I'm sorry. I just -- it will very explicitly say to you that I cannot answer this question because it doesn't have a mention anywhere in our research. So I just wanted to kind of maybe make a few closing remarks before I hand it back, is the power of generative AI really stretches beyond my imagination, I think your imagination as well. We're really just on that cusp of discovering how much this can really retool the way that people are using our information today and how can we become more sticky in our users' journeys. So I just want to thank you for your time and letting me demo the Research Assistant. And I hope this makes you as excited as I am about the possibilities that this could unlock. And with that, I'll pass it back to Stephen and Cristina.

Thanks, Sylvia. That's great.

I just wanted to -- sorry, Steve, I wanted to make one more comment. Sylvia, can you show your screen again for a second. So I wanted to go over the citations again because I think the citations, we talked about the trust that they bring to the user experience. But I think there's another dynamic going on. And it's the ability -- it's the ability to highlight content that you might not be aware of. Or when you look at -- if you scroll down from the citations and you look at the additional questions that is going to suggest -- are we -- do we have them here? The additional questions. It allows the user to get -- to explore topics that might be relevant and were in the radar. And the other thing, it also has with it a cross-sell dynamic, right? Because we might be highlighting content that are not part of the current user subscription. So you see a lot of things at play with the product.

So Cristina, what is next, just maybe you can probably pull this off the screen, Sylvia, but maybe you can talk, Cristina, about what's next for the product in the coming months.

Yes. So what we think -- and when we think about what's next, we have 1 million ideas, right? And we have 1 million ideas first because the technology is extremely powerful, as Sylvia mentioned, and also because we're getting a lot of feedback, both internally from the rating analysts and also from customers that have been using it. We like to think about what's next in 2 dimensions, right? One is the content dimension. So what other content could we be surfacing through this. And this can be -- some of the rating sectors are not included. So for example, it doesn't -- it has research for structured finance, but it doesn't have all the ratings for structured finance. So that's an area that we're definitely going to explore. We could also think about other content sets across Moody's Analytics, right? Our economics data, our default data, our commercial real estate data, to give you some examples. And then the other dimension we like to think about is in terms of features. So what other things could it do? What other functionality could it -- could we refine the way it writes an investment memo? Could we allow the users to customize more of that investment memo? The ability to export, for example, Sylvia, this was just launched, the ability to copy, but could we allow the ability to explore or save some sort of query. So that's basically to that [indiscernible] for exploring when we talk about what's next. But what I can tell you is we're doing releases on a daily basis. So as the user of the tool, you can see changes almost on a daily basis, which is extremely exciting for us.

Two other comments I'll make there, Cristina. One is, you hit the point about other content. And if you think about the vast data estate at Moody's, right, we can obviously talk a lot about credit. We can talk a lot about climate and physical risk associated with weather and climate, wildfire, et cetera. We can talk a lot about financial crime. We have tools that enable us to process news literally all over the world, and this is a number that we've been talking about a lot lately. We process almost 1 million some days, 1 million stories a day understanding the sentiment that's in those new stories and then giving us a chance or an understanding of whether that sentiment is positive or negative. We could start to weave some of these other capabilities available in the data estate here about us, sometimes proprietary, sometimes they're just historical databases that we have uniquely combined and make them available here. The other thing I think that's very exciting is, as we roll out capabilities like this in our product array, you can imagine some great cross-selling opportunities with things like our credit lens product where we have literally hundreds of thousands of users across thousands of banks where they are interested in doing credit work. And with the Research Assistant and also with some of the work we choose to expand our coverage across unrated names, you can imagine how that will really help the lenders of this world make themselves more productive and be more efficient and maybe even create content that is more insightful. So we're really excited about the cross-selling opportunities as well. So Shivani, should we -- are we dealing with questions?

Yes. We've got some questions come through. So I think the first one is just for folks trying to understand that this is an add-on to the CreditView products. So I think people are trying to understand the scale of CreditView and just how much revenue does it generate?

Cristina, do you want to take that?

Yes, yes. So yes, this is being sold as an add-on to our CreditView product. I think it's safe to say that CreditView product generates north of $0.5 billion -- north of $500 million.

Then we have a question on the impact on your cost structure. What is the cost of compute?

Yes, Cristina, another one for you guys.

Yes. Okay. So I would prephase the answer with its early stages, right? And it's early stages. We just rolled out the product. I think Steve mentioned this in his remarks, but this is the fastest product that we've ever developed here. And so we have to be careful. I think we are very early in the journey. That said, I would say, for the prompts you saw, they are around $0.10 in terms of cost, something like that?

So the tokens per second metric is what I think people in this space will be talking about. What you're seeing here is the cost of tokens per second in a prompt like you saw in this demonstration might be on the order of pennies to a dime, something like that. As things get a little bit more complicated, you can imagine creating that chart was a little bit more expensive than creating a summary of the rating changes. As things get a little bit more complicated, we introduced more content and the engineering we do to make sense of the intention behind the prompt, right? That may become a little bit more expensive because we may be grabbing more data, more content and more capabilities that we bring to the table and then trying to synthesize those in a way that is more effective. It's one of the great benefits, it will probably be a little bit more costly. I don't think the costs are material to us at this point. If you were to think of this as a cost of goods sold item, we are, I would say, is not dramatically affecting our decision-making at the moment.

And that leads on, Steve. There have been quite a few requests for more information on the economics of the Research Assistant and how are we pricing this product. So I don't know if I hand it over to you or Cristina for this one.

Either one. Well, I mean, maybe the first thing to do is just talk about the return, the ROI thought here. We talk about the surveys we've gotten from the customers. I think Cristina said, between 25% and 30% of their time saved or maybe more importantly, they can be more productive to the tune of 30% more productive. That is, I think, an estimate that we have heard, echoed in our rating agency test group, the group of people we work with internally to validate the results we're generating and give us a sense for the quality of the work that the Research Assistant is producing. And we're also hearing similar estimates from third parties. There's consulting firms on public studies and universities -- that the public studies that indicate that 30% number feels about right right now. So you can do the math, right, depending on how many people you have doing investment work or credit work or research in your operation. If you have 100 and you're talking about, I don't know, $250,000 or $300,000 a year all in, including their office and including their salary and their bonus and so on. You can imagine 30% savings of 30% more productive adds up pretty quickly. So on an organization that might be 100 people, we might say -- or contribute $5 million or $6 million, maybe even more in terms of value. And the commensurate number for us in terms of pricing is probably about 20% of that number. So an organization of 100 in size, we might seek an incremental fee that might be about 20% of the return that they'd be seeing in terms of making the investment in us here. It amounts to, at the end of the day, a 30% premium for the module that you add on to CreditView.

Thank you. So we've got some questions about how do you handle security and compliance concern.

Cristina, do you want to talk about the environment? How we set it up? Or shall I?

You can do it or I can do -- or we can pass it to Sylvia, if you want.

Sylvia, can you just talk about the environment that's set up here? What protections do we have in place? And why can customers take -- have confidence that this is an environment be fitting of Moody's.

So as I mentioned before, it's built on ChatGPT-4, and it's built in Microsoft's backend. And so you could think about as confidential a lot -- our company emails are, that's how safe this data is. The people that have access to the questions and answers at Research Assistant is providing, also the feedback logs are very tightly maintained within our team. And so we can assure you that the restrictions or the placing around the information and visibility into it is quite strong.

I would add a couple of things from looking at the chat as well. The data sits with us. So the data is [indiscernible] data, and it's in our environment. And we -- what we send to the -- what we used to the LLM4 is we try to determine intent, and we then craft the response using the LLM. But all the data and all the processes are inside the current moodys.com/creditviewinfrastructure, which has very high levels of security.

So we have all the benefits of an Azure environment that is basically our private instance of the OpenAI model. We've constructed and architected this, by the way, so that we could use other LLMs. We find that the OpenAI LLM today is the one that's most effective for us. We are experimenting with several others and are ready to adjust if it makes sense. And I think over the long term, there will be developments with other LLMs where they have special skills and special capabilities that we'll find interesting and our customers will find interesting. So we're architecting this to be able to leverage any of these. When you're in Azure, you've got all the benefits of working with Microsoft and you've got all of the new -- the Moody's controls and processes that we normally would have for any of our products. So this is protected and secure just like you would expect from a company like Moody's.

Yes. A couple of comments I would make there, Steve. One is we spent basically -- during the last 2 months, we spent a lot of time in making sure that this was as robust and as protected as possible. So we could have kept adding more content. We could have kept adding more features, but we made the deliberate decision to make sure that this was a trusted, robust secure products. So, yes -- so I wanted to emphasize that. The other comment I wanted to -- Steve mentioned how we're kind of, I would call it, or you could think about it as LLM agnostic. And this is something that I'm extremely proud that in general, when you think about the moodys.com technology platform, the way we've built it, and we've been in a journey over the last 2 years rebuilding the whole platform is to make sure that it's kind of tech agnostic. So whenever new technology comes to the market, the LLM via anything else, we can always switch to it without a major revamp.

Steve, we've had a question of will Research Assistant be rolled out to other kind of noncredit view franchises?

Yes, something like it, I think we'll make available especially to people that are doing research on companies. You can imagine the lending community that I mentioned before is a good example of that. And we may -- the architecture here is to leverage these capabilities as APIs. They may be packaged in the form of Research Assistant or they may be packaged in a way that might be more commensurate with the way a lender operates their workflow, but the components would be reused and made available in that context. You can imagine doing this also with insurance underwriting, for example. I think if you're underwriting a policy related to a specific company or a property owned by that company, you might want to do some homework on that name or their peer group, maybe get an idea in terms of relative pricing. These same APIs we can leverage in some of our workload tools in the insurance sector. There's many examples of those -- those kinds of applications that I think will follow here. The reason we're starting here is because we know the research business very well. Cristina, Sylvia, me -- I mean literally, I've been selling research since 1990. Cristina has been here for way longer than -- a long time, have been very valuable all along. Fantastic contributor, especially when it comes to research in the last few years, right? And we're all very familiar with, I'll call it, the customer sentiment dynamics, maybe the shape of the demand curve here, and we wanted to start with something that we all know and kind of can intuit as well as analyzed. So this is why we started with this franchise.

We've had a question, and I know we've been in the market since the 1st of December, but we're being asked, has the rollout impacted the contract renewal period within 4Q? Is it just too early to say?

Yes. It's probably too early to say what impact we have, right? What we can say is we did it very intentionally with the December date because there are -- we have a lot of customers that renew in December. December and January are both big months for us. We wanted to force or create a vector of force in the customer conversations to introduce it to many and as quickly as we could. So all of our sales reps are trained up and aware of and able to talk about the benefits here. All of them can demo and we practiced talking to customers to make sure they can see how valuable this can be. So we're looking forward to evaluating, I call it, the impact on the business over the coming weeks. But it's a little early to expect that buying behavior would change on the spot.

I would add to that, maybe that you can definitely see the impact on the quality of the conversations, right? Every time we engage with the customer, I would say we ran out of time steadily because there's so much interest in asking questions, engaging with the product and understanding the value. So definitely see the potential, but as Steve mentioned, too early to tell.

And we've had kind of a tangential question, which is about how does this initiative -- how does this initiative, and I'm reading it out. So articulates with the AI partnership you announced with Google Cloud. So...

Yes, sure. So we have a tremendous relationship with Microsoft. We have, I think, made some nice contributions for them. They've made some fantastic contributions for us. They've given us some -- an environment to work with out of the box that we didn't have to really worry about. That was fantastic. Their platform, the Azure platform is great. The engineering help that they've offered along the way has also been very, very useful. And I would say the people that we work with there have been great. With respect to Google, we've had some really good experience also. The project we're doing there that I think is most remarkable is the idea of leveraging some of Moody's expertise and understanding financial statements and applying that in the context of a large language model to see if we, together, can -- I'll use these words a little loosely, but fine-tune and potentially train smaller LLMs in order to read financial statements and glean insights from them. So we're doing some experiments together with an eye toward understanding financial statements. You can imagine, just like many of you on this call would do, what do these ratios imply? And what does that footnote mean in light of those ratios? And how should I interpret that in terms of the implication. So we're trying -- you can imagine, there's lots of people at Moody's that do financial statement analysis around here, and we're trying to leverage that expertise to help deploy LLMs in a context where we can actually read financial statements and gain insight from them very quickly.

Steve, this goes back to comments that you gave at the start of the call and just to clarify, we're being asked what are the Moody's Analytics solutions present would most naturally benefit from GenAI augmentation? And then a linked question that we've had as well as, do we expect to launch one or more additional products in 2024. So they're tied together, I think.

Yes. I kind of hit that a little bit at the beginning. I would say the current capabilities most effectively leverage text and natural language. And benefit from understanding the intent behind natural language and then pulling the relevant content to generate or augment the generation of text to help you do analysis. The LLM capability is fantastic in this way. The idea of doing match within LLM is something that is at least at early stages. And it's not quite the same and not quite as powerful as what we're seeing with the language at the moment. But reading data tables and leveraging models that we have around here, and we have hundreds of models that are used industrial strength models to do calculations on risk and to evaluate opportunities, these are things that we're very excited to experiment with and develop in order to make these tools like Research Assistant even more capable and more powerful. We'll be rolling out things especially that benefit from the use of large language models with respect to text, soon, sooner and then data and engaging other calculation engines a little later in the product life cycle. At the beginning of this year, we'll be talking certainly with our banking customers. We've got a couple of different projects there where we think we can add a lot of value in lending and in managing and understanding regulatory capital adequacy and capital calculations for regulators as well as some work in commercial real estate. There's some other areas where we've got investments made. What Cristina and Sylvia are doing is creating a franchise for us in many respects, earning us the right to sit at the table to support people in a way that is, I think, profoundly impactful and adding more content to this capability is also something that we expect we'll be doing through the course of 2024.

And Steve, we're coming up to time. So just wanted to open it up and ask any final comments or remarks you'd like to make just to close this out.

Yes. Okay. So best for me, right? So I mean, I'll say -- I've said this maybe before on a call, I really feel -- we are very, very excited here as knowledge workers and as people who help knowledge workers do their jobs even better. We believe we've got a fantastic opportunity to leverage these new technologies and maybe add value in a way we've never been able to do before. The opportunity is tremendous. We've got a fantastic data estate, some of which we are uniquely able to provide, much of which we are one of the best providers of and we're really looking forward to leveraging that to add value for people in this generational opportunity. This feels like the beginning of a very, very interesting ride. And we're very excited to see what comes to this and how we can help people in the future.

Well, thank you, Steve, Sylvia and Cristina. I really appreciate you making the time for our investor and analyst community. For those of you listening in, there will be a recording available and it will be posted on the IR site shortly. So thank you, everyone, for your time and your participation and all the best for the holidays. Thank you.

Happy holidays, everybody. Thanks very much.

Thanks all.

Thanks, Sylvia. Thanks, Cristina. Thanks, Shivani.