NVIDIA Corporation (NVDA) Earnings Call Transcript & Summary

Hi, everyone. My name is Phoebe Lee, part of the product marketing team for AI software at NVIDIA. I'd like to thank you for joining us today's webinar on Fast-Track to Generative AI with NVIDIA. Generative AI has become the groundbreaking technology to transform businesses. Today, we are covering the latest NVIDIA AI advancements and solutions that will help organizations to get started with generative AI. I'd like to take some time to introduce our speakers today: Anne Hecht, Senior Director, Enterprise Product at NVIDIA; as well as Tony Paikeday, Senior Director, AI Systems at NVIDIA. Before we begin, let's go over some housekeeping items. More information can be found in the upper-right corner of the window. All windows open on the screen are resizable and movable. [Operator Instructions] A copy of today's slide deck and additional help materials are available in the resources list. We encourage you to download any resources or bookmark any links that you may find useful. So here's what we'll be covering today: how enterprises are using generative AI, what are the best practices for model customization and when ready, what does it take to put generative AI into production and how to get started. Now I would like to turn it over to Anne to begin the presentation.

Thanks for joining us today, and thank you, Phoebe, for kicking us off. Generative AI is being used across every industry. McKinsey estimates the impact will be an annual increase of $4.4 trillion to the global economy. In this slide, I summarize some of the industry-specific use cases for generative AI, which fall into 3 general categories: augmenting employee productivity, improving customer experience and automating processes. We're mostly seeing that enterprises in their early stages of generative AI are focused on employee augmentation, improving their productivity with chatbots and copilots. While the generative AI opportunity is great, there are challenges faced by the enterprise. For example, enterprises need models that understand their business and domain-specific terminology, which often community models lack. And while enterprises have huge data sets that can be used to train and customize these models, they also need to protect that data and the privacy of their customers. And finally, enterprises need to be cautious and protective of their intellectual property and proprietary data. The diagrams on this slide illustrate like a blueprint how enterprises are building and deploying generative AI applications. On the far left are community models. This is where many enterprises start their generative AI journey because it's much more efficient than trying to build your own LLM model. NVIDIA has taken the additional step of optimizing many of these popular community models for best performance as well as contributed a series of models that NVIDIA has developed. We call these enterprise-ready optimized models the NVIDIA AI foundation models. And because they've been optimized, these models can be readily deployed into production anywhere on NVIDIA AI software using techniques like retrieval augmented generation. This is shown by the diagram on the far right. However, many enterprises will also want to customize the models first. And for that, we recommend what we refer to as a highly specialized AI foundry, which has the compute power DGX Cloud, specialized skills for customization provided by the NeMo framework and, of course, the foundation models. Throughout this webinar, we'll refer back to this diagram as we explain each component of the blueprint and the process of developing generative AI for the enterprise. In this next section, we're going to double-click on the components of the AI foundry. As I mentioned in the earlier slide, community models are fine for many use cases, but enterprises need a better approach than off-the-shelf models. They need enterprise-ready models that are built using trustworthy data and to understand the company's unique vocabulary, [ NAP ] and that don't hallucinate while also providing appropriate responses. And below is a great example to use a community model and ask the same question versus an optimized model for your enterprise, you're going to hear different answers. So the question is, how much storage do I have in my Google Drive? And of course, for a consumer that's limited, well in enterprise that might be unlimited and so a very different answer. NVIDIA has built and pushed the Nemotron-3 family of models. These are enterprise ready, trained with responsibly sourced data and provide results similar to larger models while also lowering inference costs. They support over 50 spoken and 35 coding languages. The Nemotron-3 models can be used for chat and Q&A applications across a broad range of industries including health care, telecom and financial services. NeMo -- or NVIDIA also publishes optimized community models. Llama 2, for example, has 2x higher inference performance after optimization. Other optimized community models from NVIDIA include Llama 2, Mistral, Stable Diffusion, [indiscernible] and Code Llama. The availability of these models is from NGC catalog on nvidia.com as well as the Azure AI model catalog and Hugging Face. So now let's take a look at NVIDIA NeMo. This is an end-to-end cloud-native solution used to build, customize and deploy generative AI models and is a critical component to the AI foundry. Starting from the left and working to the right, NeMo includes data creation capabilities to effectively extract, deduplicate and filter information from massive amounts of unstructured data. This sets the stage for your training process to utilize high-quality relevant data at scale. Next, NeMo efficiently uses GPU resources and memory across tens of thousands of nodes leveraging distributed training with advanced parallel techniques for optimized training. By dividing the model and training data, NeMo enables maximum throughput and a significantly minimizing time. As I said earlier, most enterprises will only customize a model that's already been created. And they will avoid trying to create their own LLM. And using NeMo customization, you can add functional skills to the foundational model, focusing it on specific domains and implementing things like guardrails to prevent inappropriate responses. You also have the option to continuously refine your models with techniques such as reinforcement learning supplemented by human feedback. NeMo also enables you to deploy at scale. It integrates seamlessly with NVIDIA Triton Inference Server to accelerate the inference process, delivering cutting-edge accuracy, low latency and high throughput. NeMo allows for secure, efficient, large-scale deployments using techniques like retrieval augmented generation and applying guardrails that align with safety and security requirements. This all comes with comprehensive support because it's part of the NVIDIA AI enterprise suite, which we're going to cover later in the presentation. So in summary, NVIDIA NeMo framework is a game changer for enterprise developers looking to leverage generative AI and is a critical part of your AI foundry. Next, Tony is going to join us to go over the features of the DGX Cloud and how it plays a critical role in the AI foundry providing the essential compute power.

Thanks, Anne. So now let me put the lens on infrastructure and specifically what one might consider the engine that underpins our concept of an AI foundry. But first, if you look at the average day in life of an AI developer, you might be surprised to find that they spend a lot of time on things that have nothing to do with data science or innovation. I mean just look at the developer we put on this circle here. We had to erase her face because she's so angry, she's wasting time on stuff that has nothing to do with how she's paid. So if you put aside the fact that many don't have a large multi-node computing infrastructure ready and waiting for their jobs, they're probably, in fact, spending most of their time on engineering their software stack to run on a particular service, adjusting and optimizing their model to try and get faster training runs. Maybe they're contending with their peers to get access to whatever resources are actually available or maybe they're just babysitting their training runs, restarting failed one over and over again. We recently polled enterprises asking them which of these factors impeded their developers. It should be no surprise that their people are expending a lot of effort on software engineering and losing time waiting for their jobs to run. From a composite of discussions with AI practitioners, we found a common picture of where effort is lost or wasted. It's not uncommon for the average developer to spend over 30 days of work on stuff that has nothing to do with the creative exploration of innovative models, i.e., the data science. In the average enterprise, most will spend weeks waiting on compute resources to be provisioned. They'll often do a ton of software engineering work along with troubleshooting to get their stack to run without error, which includes a lot of time wrestling with open source and scouring community forms. And many will have to do extra work to adapt their models to execute on whatever service has been provisioned for them. Now once they actually get access to their resources, they'll spend a lot of time continually supervising, troubleshooting and restarting failed training jobs. This productivity drain gets pretty expensive. It can amount to over $1 million in wasted OpEx for a team of just 30 enterprise developers in one month alone. This is why we created NVIDIA DGX Cloud, a completely new kind of service optimized for the unique demands of modern enterprise AI and specifically gen AI. It's architected for multi-node training with capacity that scales. No more worrying about whether your service region will have the cluster size you need for that all-important model, never mind the correct GPUs that you need within that cluster. It's powered by NVIDIA AI enterprise that Anne has highlighted, powers the gen AI life cycle from develop to deploy. And it comes with our containers that are already optimized for maximum performance. DGX Cloud also comes with a direct line to our own AI experts. And it's offered with a single price that covers everything you need. And because it closes the developer productivity gap that we've been talking about incurred with traditional infrastructure, it yields a dramatically faster ROI for your AI endeavors, maximizing developer productivity along with infrastructure efficiency and effectiveness. So now let's reexamine the case of developer productivity. With DGX Cloud, we have multi-node compute clusters that are already provisioned, ready to go, waiting for your developers to use them. This is very different than marketing that you have multi-node and then building multi-node when an order actually comes in. DGX Cloud is purpose-built for scalable multi-node training. If you've worked with NVIDIA AI enterprise before, well, then you already know that our containers are optimized for maximum performance, and your team won't incur additional engineering work to make their models run on DGX Cloud. It also includes pretrained models that save time on model optimization. And unlike traditional services, DGX Cloud includes a direct relationship with a dedicated team, including AI experts, a technical account manager and a customer service manager, all of whom's mission is to keep your team productive, including helping you troubleshoot the issues that impact the performance of your model training runs. DGX Cloud gives your developer their ability to use their code as is without having to do a lot of prep work ahead of training. And with advanced telemetry and resource management capabilities, you always know every job is maximizing its use of available resources to deliver the fastest time solution possible. And we're not doing this alone. A lot of folks tend to assume this is a competitive play with cloud service providers. Quite the opposite. We've chosen the leading cloud partners shown here who've invested in our stack and our accelerated computing infrastructure, and we're using them to host this service. So our featured cloud partners are just as excited about DGX Cloud as we are. So to recap, DGX Cloud is a completely new AI training as a service offer from NVIDIA, built on DGX-based architecture and built in multi-node scale that's ready to go. It includes high-performance storage and its network with high-performance, low-latency NVIDIA networking fabric. It's powered by best-of-breed software for AI training workflow, and it includes NVIDIA AI enterprise for pretrained models, optimized frameworks and accelerated data science libraries. You get a direct line to our AI experts, and all of this comes for a single no-surprise price. Now with all of that, let's turn our sights on the deployment side of the AI foundry. And for that, I want to turn it back to Anne. Anne, over to you.

Thanks, Tony. In this next section, I'm going to review best practices for generative AI in production using the NVIDIA AI enterprise software platform, the essential run time for the NVIDIA AI foundation models. We created NVIDIA AI enterprise software offering to address the challenges of the enterprise developer and IT team. Generative AI requires access to reliable, efficient, accelerated compute to be productive during the full AI pipeline from data prep to customization and deployment. But maintaining accessible and optimized infrastructure can be challenging. Generative AI is also continually evolving, which is why enterprise developers need to stay current on the latest development skills and techniques. But with over 65,000 public generative AI projects on GitHub, which is a 250% increase from 2022, this is a resource-intensive task. And lastly, transitioning AI projects from pilot to production continues to be a challenge for many enterprises with about half of all AI projects failing to make it into production. NVIDIA AI enterprise addresses these challenges. It's an enterprise-grade software platform for your AI development and run time and supports the full AI pipeline. Accelerated computing increases productivity while lowering TCO because it enables developers to do more with less. These efficiencies also contribute to more sustainable computing. NVIDIA AI enterprise provides optimized AI frameworks and tools to save cost on time to production. This includes NeMo, which we reviewed earlier for generative AI, RAPIDS Accelerator for Apache Spark to speed data processing up to 5x and frameworks for accelerated inference, which improved performance by up to 146x compared to CPU-only. NVIDIA AI enterprise is built on curated, open-sourced NVIDIA AI software to provide enterprise developers access to the latest AI development techniques. It also provides enterprise-grade security, stability, manageability and support. This saves the developer time and ensures a more reliable run time for production. And finally, NVIDIA AI enterprise software runs everywhere on a wide variety of video GPUs. Current and past generations are supported. And because NVIDIA AI enterprise is available everywhere, the cloud, the data center, the workstation and on the edge and on every major cloud in over 400 certified servers, enterprises have the flexibility of where they build and run their AI. This diagram shows the full breadth of the AI software included in the NVIDIA AI enterprise software platform. You'll see it includes AI development and deployment tools to support the full AI pipeline, application frameworks for specialized workloads such as NeMo for LLM and Riva for speech AI, just to name a few, and infrastructure management software such as cluster management to streamline deployments at scale. Through the NVIDIA AI enterprise subscription, enterprises have access to 3 branches, providing them flexibility to choose a branch that is best for their development and deployment without compromising on security. Monthly feature releases are ideal for developers who want top the tree, latest code and don't need API stability. Production branches ensure security and API stability for 9 months and release every 6 months so IT can more easily plan updates. And long-term supported branches are available for highly regulated industries. I mentioned earlier that NVIDIA AI enterprise is the run time for AI models and ensures performance and lowers cost of inference. On this slide, I'm showing an example of the performance and cost savings using the segment anything model, which is a model that's been trained to identify any object in an image even one it hasn't seen before. We chose to segment the skyline of Taipei. We optimized the model with the TensorRT software included in NeMo and NVIDIA enterprise and are running it on an L4 GPU. You can see that the throughput is 24x higher with acceleration. We were able to process 32 images per minute as opposed to only 1.5 images per minute on the CPU-only infrastructure. And since this run time is more efficient, the model can be run at 5% of the cost as opposed to running it at CPU-only. While GPUs are more expensive per unit than CPU, this shows the performance gain of the GPU lowers the overall total cost of ownership while also providing, of course, a much better experience for the user. We've covered a lot on this webinar. So let's do a quick review of the blueprint I shared earlier, which shows how enterprises can customize and deploy generative AI applications. First, we recommend enterprises start with NVIDIA AI foundation models, which include community and NVIDIA-built models that are optimized for performance and enterprise requirements. These models can be customized in the AI foundry, which includes NVIDIA NeMo tools to streamline the process and DGX Cloud for accelerating compute power. Since these models are optimized with the NVIDIA AI enterprise run time, they can also be deployed anywhere and augmented with enterprise data using techniques like retrieval-augmented generation. There are just a few of the enterprises building generative AI applications for their customers on the NVIDIA AI foundry, which runs on the reliable run time of NVIDIA AI enterprise. Amdocs is a leading provider of software and services to communications and media companies and is optimizing models for the Amdocs AM AIZ framework to speed adoption of generative AI applications and services to telcos globally. Dropbox is building generative AI capabilities into products such as Dropbox Dash, a universal search that connects apps, tools and content in a single search bar to help customers find what they need and Dropbox AI, a tool that allows customers to ask questions and get summaries on large files across their entire Dropbox. SAP plans to use the service and optimize retrievable augmented generation workflow with NVIDIA DGX Cloud and NVIDIA AI enterprise software running on Azure to help customize and deploy its dual offering, a new natural language generation AI copilot. And ServiceNow is developing custom large language models trained on data specifically for its ServiceNow platform, a cloud-based workflow automation platform that enables enterprise organizations to improve their operational efficiencies by streamlining and automating routine work tasks. I hope this content was valuable and will help you start your AI journey with us. I'm going to turn it back over to Phoebe, and we'll be available with Tony for your questions.

Thank you, Anne and Tony, for a great presentation. Before we move on to Q&A, I'd like to introduce some upcoming events from NVIDIA. We have a digital event webinar on improved spear phishing detection with AI on January 30. So if your organization are building a powerful solution to defend against cyber threats, this will be a great webinar to join. You can find a link on the slide to either watch the live event or watch the recording. NVIDIA GTC, our annual flagship event to connect developer, researchers, industry and business experts in terms of what's next in AI and accelerated computing. This year, we're bringing back the event in person hosted in San Jose, California starting March 18. Please visit the NVIDIA GTC website to learn about a great agenda as well as how to register.

Now let's move on to Q&A. All right. Now we're officially in the Q&A section. And thanks again, Tony, Anne for a great presentation. We have seen a lot of questions coming in, and we're trying our best to address all of them, but if there are some that we couldn't get to, we'll find a way to get the information to you afterward. So one question. I see this -- actually multiple questions related to the same thing. It's asking about the DGX availability in cloud. So Tony, would you like to take that question?

Yes, sure. It's a great question. And yes, we did see a lot of interest. We're very excited that the world's leading clouds are our partners. As mentioned in the presentation, we're not doing this alone, and we're not doing it in competition. They are all our partners in delivering DGX Cloud to customers. So last year, just to be clear, we launched DGX Cloud on both Oracle Cloud and Azure. And since then, we've seen dramatic escalating demand for the service. Additionally, we announced partnerships with GCP at Google Cloud Next that happened last year as well as with AWS at AWS Reinvent last fall. So to summarize, today, you can get DGX Cloud. It's generally available on OCI and Azure. And in the future, we'll be adding both GCP and AWS.

Thank you, Tony. And I also see some questions, some comments from the audience saying that they're new to generative AI, and they're more industry-focused use cases and how NVIDIA can help them get started. Anne -- and Tony, would you like to help provide some insights or recommendations for the audience that are looking into getting started with gen AI?

How about if I take that one, and then, Tony, you can go. And by the way, thanks, everybody, for joining us this morning or this afternoon, wherever you are. I really appreciate your participation and your attendance. So we have a number of areas where you can go and experiment with generative AI and learn more. We have a LaunchPad program that actually has learning labs that are freely available with tutorial guides, which you can walk through about AI and generative AI, very specific use cases. And I think we can put that URL probably in a chat or if you just search on NVIDIA LaunchPad, you'll find the program. And then the other area we have is the URL I shared in my last slides that take you to the foundation's area. And we have a whole -- essentially it's like a playground where you can actually do a prompt response with our foundation models, access to API. You can experiment with the models posted on our NVIDIA NGC environment, where we're hosting the models. So you kind of do some experimentation before you decide to use them in production. So both places are great place to start, learn more. And there's documentation in both areas as well so you can learn more.

Yes. I love what you've described, Anne. And as you heard from Anne, what she's described is a great way to engage in productive experimentation, learning about what's out there and figuring out which are like pragmatic use cases that apply to your environment. And as that journey progresses for you, if you kind of identify what is probably hopefully low-hanging fruit and high ROI for you in terms of use cases you can deploy in your environment, I'd encourage you to work with our partners, NVIDIA partner network that includes global systems integrators and our value-added resellers who have a deep bench of expertise. They can help you from the early whiteboarding and solutioning, kind of envisioning what this application looks like through the entire life cycle of deploying in production. So obviously, leverage their help and as well as ours to get your deployed application.

Sounds great. Thanks, Tony and Anne. So before we move forward with more question and answer, I would love to understand our audience today, who joined from all around the world. So if you can see poll questions right now on your screen, please help answering. What stage are you in adopting generative AI? So there are multiple options. The first one is discovery phase, for investigating, exploring. The second is you are in early development, starting taking things either from open source or from other depositories trying some AI frameworks or if you are in a prelaunch or POC stage that you have -- you know what you want to build, and you have things in place you want to put in a test or are you in the production right now actually get ready to deploy the models and see the results? So greatly appreciate it if you take some time to let us understand where you're at right now. And we can circle back on the results later on if we have time. All right. With that, we're going to move on to more questions. I see questions asking about NeMo. So does it cost anything to get started with NeMo and how does it look like for the smaller enterprises being able to use the enterprise software offering? So Anne, would you like to take this question?

Yes, sure. Happy to, Phoebe. So again, those places that I recommended, LaunchPad and then our NVIDIA AI foundation site on NGC has -- will give you access to NeMo and to NeMo labs, and you can access obviously our foundation models. And you can also on NGC access the NeMo framework and experiment with that. That's all containerized for you on NGC. And then when you move into production of the models and you're running it on our NeMo Inference that's when licensing is required for that run time. But there's a lot you can do, obviously, to access and use NeMo prior to moving to production, right, to evaluate, build out and learn more about how to use NeMo. I highly encourage you to go to LaunchPad. We have a whole NeMo lab. And actually, one of those labs also is focused on helping enterprises deploy using retrieval augmented generation using NeMo, which is actually, like I mentioned in the presentation, the most common way enterprises are deploying generative AI right now. They're actually a lot of times not moving to customization, first, that they're actually going to POCs using retrievable, evaluating accuracy of the model. And sometimes they're able to go into production and not do the customization and the fine tuning. It just depends on the use case of the customer.

Thank you, Anne. Next question, I think, is DGX Cloud-related. Are there any pipeline available for the deployment of models trained in DGX Cloud Jetson hardware?

Yes. I think the way to look at it is DGX Cloud, if you think of the pipeline, is the first step in that process, let's say, after you've curated data and you've basically engineered your data for a training run. So think of DGX Cloud is where you go then to execute a training run especially if you need multi-node capacity with ultra-high bandwidth, low-latency network fabric, and they're connecting your instances. So if you need the fastest time solution on a training run for a very complex model, DGX Cloud is a great place to go there. You can also customize an existing model that you may have on DGX Cloud. After that, you'll want to then take that model into the downstream parts of your pipeline. And that might be for deploying it, for instance, in production or inference within the context of a RAG. But think of DGX Cloud within that pipeline as supporting the training part of it.

Thank you, Tony. And I see some questions regarding the industry use cases because the presentation that we have today is more with general approach to generative AI. So does NVIDIA have use cases or industry-specific solutions that help people get started with generative AI. Anne, Tony, would you like to both chime in for this one?

Maybe, Anne, do you want...

Yes, sorry, I had to unmute you guys because I was like typing. I don't want to come across on audio. Sorry, everybody. How many of us have had that problem in our lives? This goes with video conferencing. Yes. So the question is whether or not we can -- now I'm all distracted. What was the question?

The question is does NVIDIA have any solution for use case-specific or industry-specific so for -- as a quick way for people to get started?

Yes, yes. So I think -- so what I hear customers doing and actually a lot of the customers who I profiled at the end of that -- at the end of the deck before we started the Q&A are actually in this space where a lot of enterprises are looking at generative AI as an opportunity to augment productivity of their employees. And I think that's really important to understand. I mean we're still in generative AI's early stages, right? It's super exciting, the potential of this technology. But deploying it when we talk to customers, they're really interested in how do we help our employees be more productive, more efficient, get more access to data faster, help minimize the -- like redundant tasks so that they can do more high-value work. So with that in mind, you see a lot of focus on chatbots, copilots, so use cases where we're augmenting a process through generative AI that already exists. And a good example of that is like ServiceNow, who has added these copilots to their offering to their enterprise customers who obviously use ServiceNow to support their teams and provide support. And so on in order for those enterprise customers to be more efficient, they provide some copilots and efficiencies on their ServiceNow platform. SAP is doing the same thing. So I'd say that's where most companies are starting. The second area is to provide more like customer recommendations and a better customer experience, either through support or, again, through recommendations, understanding that customer better and providing and anticipating what their next question, what their next purchase will be and then recommending as a result of that products that they should consider because they bought product A, maybe you want product B, and then answering some anticipated questions that they would potentially have. So again, it's all about improving customer experience. And then finally, the last one is actually doing automation, but I think the first 2 productivity, chatbot use cases and then customer experience are where most enterprises are starting.

Thank you, Anne. Now we jump back to DGX for a couple of questions. The question is how is DGX Cloud integrated with the cloud service provider?

Yes. That's a great question. In fact, I saw that a number of times in the chat. So think of it like this way. We work directly with the cloud service provider's engineering team. So our engineering team and theirs, they jointly design a special enclave of infrastructure that is hosted within the cloud service provider. So we're not simply, let's say, reselling the existing CSPs instances and calling that DGX. We're literally working with their engineering team. We've basically purpose built an enclave of infrastructure that meets our design requirements. It's very modeled -- very much modeled after how we design our own internal infrastructure at NVIDIA that powers our own R&D and now replicated that within each one of our CSPs partners. So when we sell DGX Cloud, we're selling essentially that carve-out, that enclave of infrastructure instances within that are being hosted by our valued CSP partners.

Thank you, Tony. And another question. Why would I go with DGX Cloud instead of the native GPU instances that they offer?