NVIDIA Corporation (NVDA) Earnings Call Transcript & Summary

Hi, everyone. Thanks for joining us today for our live webinar, Drive Innovation and Speed Up Scientific Workloads. Before we begin, we wanted to cover a few housekeeping items. At the bottom of your screen, you can find various widgets for use in your event. Once open on the screen, they are resizable and moveable. If you have any questions during the webcast, you can submit them through the Q&A window. We'll try to answer these at the end of the event and in between each speaker. A copy of today's slide deck and additional help materials are available in the resource list. We encourage you to download any resource or bookmark any links that you may find useful. Here's some tips that can help make this event as best as it can be. To maximize the quality of this audio stream, please close any open applications aside from your browser window. Also a good old-fashioned browser refresh can cure many ills. And so if you hear audio sputters or the slides seem to be lagging, give that a try. You can also try opening this event in a different browser. If you encounter any other technical issues today, please let us know in the Q&A box, and we'll help you troubleshoot. Now with further ado, we'll turn the event over to our speakers to begin the presentation.

Thank you very much. This is Earl Joseph with Hyperion Research. I want to welcome everyone to joining us. These are exciting times with lots of new technologies, new interesting ways to construct computers, make them more performance and productive. And today, we're going to be talking quite a bit about DPUs in the industry. For those of you who don't know us, Hyperion Research does a lot of market and technology studies. We also operate the HPC user forum, and we welcome everyone to join us on that. It's where we, 4 times a year, get vendors, buyers, users altogether to talk about best practices and new technologies. And with that, I'd like to give you just a quick overview of what we see going on in the market today and kind of our forecast for the next 5 years. So for the 2 charts you see here, the first one is the on-premises purchases of servers, storage, middleware applications and services, and we see the HPC market growing very robustly from just over $32 billion in 2022 to over $40 billion by 2026. So a very healthy growth on the on-prem part of the business. The lower chart is showing us the growth in cloud computing. And what I mean by cloud computing, this is the money that people spend to run their HPC, AI, quantum computing problems in the cloud itself. And we're seeing extremely robust growth here of over 17% a year over the next 5-year period. When you add these 2 together, the market is actually going to be greater than $50 billion in 2026, which is just amazing. But there are many other segments in the market that are even growing at higher rates, in particular AI, with machine learning, deep learning, those workloads are growing at over 22% right now. So some really strong growth. So here are our top 10 predictions that we have for the coming year. We just finished these. As I just mentioned, the strong growth. There's a lot of issues around the supply chain sector and government policies and so on. But today, we're going to talk about prediction 7 and 8. And those 2 predictions are #7 is that we expect systems architectures to actually split into multiple types of systems. And we're going to have some systems that are highly optimized for traditional computational work. And we're going to see ones that are going to be much more optimized for memory and data-intensive work, and then those that for AI machine learning and deep learning. So we think the world of the future data center will not be to have one singular system, but they have multiple systems that can address these different architectures. And we see these divergent requirements really driving the architectural focal points and the requirement for interconnects, improved storage systems and getting in and out of the storage system, moving data around, especially with the growth in model sizes and the growth with machine learning and deep learning are just going to explode in the marketplace. And so let me start with prediction #7 here. As far as the architectural changes, it's a continued prediction from last year, but we think there, users and the systems designed for HPC users are going to factor in new requirements. There is a lot of new workloads. I mentioned the AI ones, big data. But in addition to the growth there, the amount of workloads, application areas are just phenomenal. We're seeing some areas where within 60 days, a new concept or idea hits the market, and it has, last I heard, 15 million users on one case. It's a push for a faster time to solution. If you're doing things more in real-time or near real-time applications, we're seeing those applications coming on board, too. And a lot of interesting new areas of research. And for the system decisions, we only go to purchase a machine or a vendor goes to build them, there's going to be a much larger and diverse set of building blocks. We're seeing all types of new processors, GPUs. And then you see new types of applications of processors and DPUs and different parts of a system. And as I mentioned earlier, we think we'll see multiple smaller systems. But when I say smaller, these are still large systems. It's just not one singular one. And then we're also seeing the use of the cloud fitting into almost all sites, overall architecture and operations in the future. And then in addition to this, these heterogeneous systems will incorporate data-focused areas versus processing intensive designs, a lot of node different configurations. For example, the most extreme I've heard recently is with Amazon. Amazon right now, if you want to run the HPC job in the Amazon cloud, you have over 450 different processing node configuration to choose from. So you almost need an AI system to figure out which is the most optimal way to do that. And then infrastructure accelerators like DPUs to offload some of the processes, some processing accelerators into the storage and as far as moving the traffic round are going to have just a much more impact and you're going to see them much more frequently. And then we're also going to see some smaller systems targeted to very specific application. I mentioned here, AI and big data, I'm also mentioning within that realm, a very specific area, as far as the application area. So we think a lot of data centers will have the split of the large system maybe into 2 processor memory one or a third one and then a number of small ones that are very application-specific. And with that, I think I'll hand it to Mark.

Thanks, Earl. I'll touch on kind of related to the last prediction, but the -- as far as different architectural perspective, we are definitely seeing a much increased focus on storage and interconnect, in particular, as a new architectural focal point for any number of reasons: performance, both the bandwidth and latency aspects to -- with the ultimate goal of improving time to results of applications and simulations. And with the emergence of the newer modern workloads, data-intensive workloads, complementing and even maybe even more so dominating systems and the traditional compute-intensive workloads, the interconnects and their ability to be able to handle a much more diverse I/O workload profile is becoming even more critical and will be much more increased focal point for architects in defining their systems. I wanted to touch on just a couple of points from our annual site surveys that are related in this area. We've been looking at part of the architecture, interconnect, networking architecture, differences in the -- in whether there are separate independent networks, node-to-node networks, node-to-storage networks, and converged networks. I wanted to provide a snapshot of what's happening of the most recent site survey on converged networks. And it's showing that the -- very strong leadership of InfiniBand interconnect with still a large adoption of Ethernet as well as the interconnect technology. What is interesting is a fairly strong beachhead adoption of the 400 gigabit line rate and aggregate rate for the interconnect is being adopted and emerging, which is quite interesting how it appears to be being quickly adopted. Apologies for that. We're also looking -- and we've seen GPUs become emerging, and we're also interested in how people's current adoption rate is and future adoption rate is of other offload type of accelerator architectures, including DPUs. We see today that there is somewhat of an adoption of about half of the sites surveyed are adopting some other form of acceleration, whether DPU, APU, other types of processing units. But we see moving forward into the next currently planned procurement that not only is just the alternative acceleration types of architecture is going to be adopted, but the adoption of DPUs themselves, our interest and adoption is growing quite rapidly as well. That closes out our intro and market overview section. I do want to invite people to submit questions in the Q&A within the platform, and we'll be answering them as we go. We will also be -- as we go, as they come in as well as at the end of each presenter, we'll field questions and then we will close the session with a more roundtable Q&A as well. And with that, we'll move on and invite Gilad Shainer from NVIDIA to join in with his presentation on NVIDIA's HPC networking platform. Gilad?

Yes. Thank you. Thank you very much. So what makes NVIDIA unique? What makes NVIDIA unique? And I would say that NVIDIA is essentially a computing company. A computing company in the way that we build the hardware elements, the silicon devices for CPUs and GPUs and switch networks and NICs and DPUs. The system software that runs on top of that, the software development kits that enable our customers to write their applications on top of our platform and be assured, by the way, that future generations of the devices that we build will carry what they already written and a further improved performance and capabilities. We created platforms on top, and then we create applications that runs on the entire system. So when you send everything in a full stack approach, we built computing platforms, computing systems. We're using those computing systems in order to understand the next generation of applications, the next generations of performance bottlenecks. And with that, we're building the devices that will take us to the next generations and so forth. On the networking side, the NVIDIA HPC networking platform includes the Quantum-2 InfiniBand Switch, a 400-gigabit per second switch platform that introduces in-network computing. And in network computing is one of the key elements in the network to actually analyze data within the network closer to where the data is and that can accelerate applications in a very dramatic way. The ConnectX SmartNIC with 400 gigabit per second interface, PCI Express gen 5 in different kind of accelerations around data movement. BlueField-3 DPU, which I'll talk a little bit in more details about it, which runs at 400 gigabit per second, PCI Express gen 5 and including in-network computing engines for storage, for infrastructure, for security, for simulation such as NPI for AI workloads and so forth. Our Spectrum-4 Internet switch, we're creating Ethernet platform for high-performance computing Internet platform for AI. Our customers that utilize or use Internet for their infrastructure, for AI infrastructure, will be able to leverage from Spectrum-4 to build their communication network or communication fabric. And the ability to manage to orchestrate, to monitor, to look on telemetry information and so forth from the fabric from the network from the system itself. So building the full stack, looking on the full stack, looking on the applications of learning today, the ones that we want to run. We identified several performance bottlenecks that impact workloads, impact our workloads, which we aimed or plan to solve with, first, in-network computing; and second, with the DPUs. The first one is be able to achieve overlapping, overlapping between compute and communication. Running simulations, multiprocess simulations includes compute cycles and includes communication cycles and be able to hide communication cycles behind computation cycles, be able to achieve asynchronous progress or overlap in between compute and communication is something that was desired for many, many years. It's kind of the holy grail of HPC. And finally, with DPUs, we are able to achieve something like that, and I'll show you examples later on. Load imbalance, one of the key performance bottlenecks. When we're doing multiprocess simulations, it's not that every GPU, every CPU finish the work at the same time, and then you can do multiprocess communications or multi-process synchronization. It's not a fact. And therefore, a slow CPU, for example, can slow down the entire system. By utilizing the DPUs for managing the communications, by managing NPI operations, for example, we can finish those operations even though CPUs might be still busy. And we'll then be able to reduce the effect of loading imbalance and improve application performance by that. Jitter is another area. The infrastructure workloads running on a CPU create jitter, and the jitter affect performance of the system. It can be jitter because of storage processes, infrastructure management processes and so forth. And that's another area where the DPU was designed to. The DPU was designed to run the infrastructure processing to separate the infrastructure demand from the application domain, to run storage, to manage storage access. And by moving the infrastructure processing outside of the host to write on the DPU that enabled to reduce the jitter or almost eliminate the jitter effect and enable to have CPUs and GPU fully dedicated for running HPC simulations to AI workloads. Last and not least, multi-job performance. More users running HPC, we are hosting more jobs or running more jobs on a supercomputer. We are hosting more users on HPC cloud. We need to make sure that one job, one user cannot impact another user, another job performance. And this is a problem that exists today. And with the combination of e-network computing telemetry that run on our platform as well as DPUs, we can use reinforcement learning capabilities in order to isolate jobs performance from another job performance and provide the best performance to a job or process regardless of what other jobs are running in the same system. So the DPU is one of the key elements that enable us to overcome those bottlenecks and dramatically improve application performance or the full HPC or AI system performance. The DPU combines the entire ConnectX inside. So it does include the entire NIC functionality inside. BlueField-3, our latest DPU includes the ConnectX 7 inside. So it's a 400 gigabit per second device with PCI Express gen 5. And it includes 2 areas of computing programmable elements, 16 anchors within the device and a data path accelerator. Data path accelerate can do quick things in real time on the data, and then the anchors can run or host multiple processes or elements of services that can enable to migrate the infrastructure processing to the DPU to deal with storage, to accelerate simulations, and accelerate AI applications. And that's what we do with the DPU. The DPU runs fully RDMA, so you can touch the host memory, you can get to the host memory completely bypassing the CPU or GPU or accessing GPU memory, bypassing the GPU in that sense of or bypassing the CPU. And it can access remote host memory as well, bypassing the remote CPU. So the DPU can actually move data locally and remotely and bypassing the CPU in that sense and do it in a very effective way. And by doing that, to completely manage NPI operations, for example, or help with NCCL operations. And with that, as I mentioned, to overcome the bottleneck that I described before. So in-network computing is the key. And BlueField is the key for overcoming the performance bottlenecks that I mentioned. In-network competing on the InfiniBand side exists also on the switch. The Quantum-2 switch includes data reductions, small message reductions and large message reductions. And those large message reduction can support virtually unlimited concurrent large message reduction risk that runs on the network that improved reduction operational performance and in turn, of course, increased performance of AI training, deep learning and NPI applications. BlueField, I mentioned, includes the programmable units around the security around storage, around NPI, around NCCL. And BlueField-3 compares to previous generation, improve the compute capacity of it by around 3 to 4x, improve the memory speed that it has, and it's become an important factor for building the new generations of HPC or AI platforms. So here are a couple of examples on what we do on BlueField related to NPI, related to simulations. Here I took in, for example, a multiprocess operation, one of the most complicated one, ones that actually it's more sensitive to anything that happens on the system and impact performance dramatically of different workloads. Here is an example of Alltoallv. So Alltoall with vector. On the left side, you can see the performance benefit of moving that operation to the DPU. And on the right side, you can see a nonblocking in Alltoallv performance and the impact of moving that operation to the DPU. You can see that with the blocking operations, we're almost reducing that time or improving that performance by 2x and for the nonblock operations, it's even more than 2x. Not less important, we're able to achieve full overlapping, full overlapping between compute and communications, which means that we can completely hide communications behind computations and of bringing the Holy Grail of HPC to HPC finally. What does it do to applications? It's increased data center or HPC center performance in a very nice way. So at the beginning, we started with FFTs when we start the work on the DPUs and demonstrate more than 1.2x on FFTs, and we'll continue to improve that performance, continue to optimize FFTs. We did an example on a wire code, demonstrating more than 1.2x performance and recently working with Max Planck Institute on Octopus, which is a physics and chemistry code, demonstrating also more than 1.2x performance improvement. Those improvements are due to the ability to move NPI to the DPU and then the DPU will be able to manage and execute those NPI operations. By having that, we're going to accelerate almost every NPI application out there. Almost every scientific computing applications will be accelerated by around 20% because of using the DPU. So those are a few examples at the beginning, and there will be more examples coming out shortly from us and from our partners. Now what does it mean more than 1.2x performance improvement? It's not just the performance itself. It's bringing BlueField into the server and you're using BlueField instead of the NIC for the connectivity and for the infrastructure processing and NPI accelerations. We're also increasing the performance to TCO, performance to dollar by almost 1.2x. And also improving performance per watt. And we know that power has become one of the limiting factors, one of the elements that determining what we can build. And of course, we want to reduce power consumption in many data centers of power limited. So bringing the DPU into the AI and HP systems also improve the performance per power, improve the performance per watt by almost 1.2x. Those are great elements here. It saves millions of dollars for building -- of building the HPC in data centers. And for the BlueField is something that we'll see become the de facto element in every server, CPU servers or GPUs. So with that, if there is any quick questions, happy to take. And if not, I know that there's other presenters that -- who are going to present cool things today.

Thanks, Gilad. There is a question that has come in, and it may have been answered after it came in, but I'll go ahead and share the question that came in. How do you envision using in-network computing to improve simulations in HPC?

Yes. So in-network computing means that we are moving data algorithm that can be better executed on the network instead of doing that on a CPU and a GPU. Reduction -- data reduction is one example, and data reductions improves simulations and deep learning applications in 20%, 30% and 40%, which is amazing. So this is one area. The in-network computing capabilities on the DPU, the programmable in-network computing on the DPUs enable us not just to improve security and improve storage access, but also NPI. And you saw the results of doing -- running NPI on the DPU instead of doing that on the host and achieving more than 1.2x performance improvement. So that will help to improve performance for data centers that obviously limited in expense. And of course, power care data centers, DPUs is a must for them.

And quickly, there was a secondary question to that. Is in-network computing an alternative for synchronization? Or is it better using a DPU? And if so, can SHARP be used for that?

It's not one versus the other. It's actually -- you're using all of those. So SHARP is running on the switch and does the data reduction there. The DPU uses the switch for doing the data reduction. So data reduction elements are managed by the DPU. The DPU is using the hardware elements, the in-network computing elements on the switch in order to run data reduction operations such as reduced and unreduced and [ bigger ] and broadcast. Other operations such as AllGather and all-to-all, those are running completely on the DPU in that sense. So once you combine the network competing on a switch and in-network computing on the DPU, you get amazing results.

Okay. Great. Thanks, Gilad. If any more questions come in for Gilad, we can submit them and hold them for the Q&A at the end. At this point, let's move on to our next speaker, Dr. Jithin Jose from Microsoft, who's going to talk about InfiniBand network computing technologies at Azure.

Thank you, Mark. Thank you. So in this talk, I want to mainly focus on the NDR and some of the performance systems with NDR on Azure. So let me please start with the different breakthroughs, I mean, over the last 2 or 3 years. So I mean, in 2019, like we started the very modest 20k core MPI job, it was for the cloud. Later -- I mean we offer like HDR 200 gigabits per second IP, and we try to scale out more like [ ADK, MK ] jobs again, first, for cloud. And later like we built multiple different SKUs with HDR, like focusing on the traditional HPC applications and also AI plus of applications. And then eventually like -- I mean, we, according to the top 10 supercomputing rank and again like top first for cloud also and many more in the top 4 key ones, right, for the different types of workload. And I'm happy to share that these are actually numbers on a public cloud. It's not like any private or any dedicated cluster that we have. So any more like should be able to like go try them and then reproduce these numbers, right? And again, like end of last year, see, like we announced in the RFQ, I mean that is the HBv4 AMD Genoa SKU. So I mean in this talk, I want to focus a little bit more on that, like how we design this SKU, what are the motivations behind the different design choices and also some of the preliminary performance systems, right? So yes, so this is -- I mean, the general architecture of our scale, like HBv4 or HXv1 Genoa-based AMD SKU with NDR-400, of course, right? So as you can see, right, so we are -- we picked the 96-core based solution here. So I mean 96 cores per socket and we set the number of new marker socket as 2. So 48 cores for NUMA and then 4 of them, right? So -- and we selectively -- we selected a few cores from the different NUMA nodes and dedicated those on the host, right? So the Hyper-V, the host stages, which coordinate all the rollouts, the performance monitoring or monitoring different telemetry, et cetera, those are completely done on that, right? And so there is a complete isolation between hypervisor and between VM, right? And the VMs actually get the full -- a full control of those cores, right? And of course, NDR is actually exposed over SR-IOV. So all the features that -- I mean, that NDR is offering. So different features that Gilad mentioned, like we can call it the performance results, right, but all of them are visible to the end users, right? And until we can come up with like different designs based on the different workloads, like Earl mentioned in the very first slide, right, many of the architectures are now being like focused based on workload characteristics, right? So just to support from that, right? So we have 2 different configurations for this SKU, right? So one like 700 GPU memory configuration and another one, the higher recomputation because certain plus of workloads, as you know, right, like they need a higher memory per core requirement. So that's the main situation. And again, another dimension here is like we can selectively pick core that are from the different CCDs. So you can have a full 176-core VM, and you can also have a like a 24-core VM, right? So you're selecting the different cores. So moving ahead, so this is the -- I mean, let me particularly introduce a different IP fleet at Azure, right? And all of these VMs have fully on VM per house. So it's not like you're sharing in the house between different VMs, right? So you have a full control, right? And the main reason is that for many of the IT workloads or multi workloads you want to scale out, right? And if you want to do that, like why not start from the node itself, right? So that's the main motivation. And I mean active overview of the different VM family, like we have HPC or traditional HPC-based work cores. Those are just by the H series. And then we have the M Series, which is for the AI or training/utilization skills, right? So those are the NCBs. And here, I mean in this program, particularly focusing on the HBv4 or HXv1 SKU, which is the NDR based SKU. And on the right, I mean, have quickly noted on a few of the IV features, right? I mean, I have mentioned, I mean, a few of them in the previous section. But I mean, I just want to reiterate that since I mean the VMs actually exposed through SR-IOV, I mean there is no feature. I mean we are able to like expose all the features to the VM also, right? And let me quickly jump in through some of the performance numbers in the interest of time. So basically, this is particularly the NDR performance results. These are at the MPI level. So you can see right for -- I mean, for the MPI core latencies, we are able to keep -- I mean, in the 1.5, less than 2 microsecond range for different transports. And if you look at the bandwidth single node. So directional bandwidth or bidirectional bandwidth, right? We are able to hit close to the peak theoretical bandwidth, right? And like these numbers are pretty much comparable to the results also. And this is another view of the performance, right? So in this case, so we -- typical NDR cluster like we did like an all fair performance, right? So just to simulate and there will be multiple users and all of them will be like doing operations, right? So I mean, in this case, all -- I mean, like, say, for example, 2 different nodes and say that will be like the N square different pairs for N minus 1 by 2. So about 10k different pairs. And then we -- like N by 2 different pairs communicating at the same time, right? So you can see there are 2 different bands for the latency. So I mean, these 2 different bands are because of -- so we have a topology of a 2-layer factory, right? So basically, the [ night ] systems can -- between any 2 nodes can be like 2 nodes or 4 nodes. So this matches exactly with that, right? So the lower volumes of 2 cores and then the other one is 4 cores, right? So the maximum latency that -- I mean, any -- between any 2 nodes is in this range and then this is the best case, right? And then on the right-hand side, we have the MPI bandwidth or bi-directional bandwidth distribution, right? So here also, you see the [ topology ], right? So these are strictly for bandwidth intensive workloads right? So with respect to you, like where -- how you picked the nodes, right, you were able to hit the peak bandwidth, right? And we quickly move on to some of the chart performance. So this is again NDR performance on a public cluster. So like Gilad mentioned about the SHARP V3 and which has the 2 different protocols, the LLT and the SAT protocols, right? So you can see for the different scale, 16 nodes or 32 nodes or all the way to like 128 nodes. Yes, you can see clear benefit with the SHARP report in fidelity on small messages and SAT on the large messages, right? And then as you scale, like you can see that things are actually getting much, much bigger, right? So really nice results with the SHARP NDR. And then -- yes, so this is another, I mean, that's by price. I mean there can be perceptions that the cloud, like the performance can be variable at different times, right? So in this situation like this is the 128 node [indiscernible] in 2 different input considerations, one with a relatively small size, like 8k [indiscernible] node. And then here is like a 20x input model node. So yes, in both the cases, I mean like these are different runs on the same cluster. So we are seeing important -- we are seeing a steady performance like plus probably a 2 percentage. Yes. So -- and yes, so with that, let me conclude, it's time also. So I mean I just want to follow up a few points. We are actually taking a generic HPC approach rather than having and building different types of SKUs, right? And it doesn't really matter like how many different bond predictions you have, like we want to really focus on the HPC workloads, right? And then we build SKUs targeting the different HPC workloads, right? So yes, for example, the different types of SKUs, I mean, based on the AI, based on HPC. So the different [ actions ] there, right? And I mean we also focus a lot on the time to market initiatives. For example, like even in the case of the HBv4 or Genoa-based SKU. So when AMD announced the Genoa-based, like we had a cluster ready that can take customer workflows, right? So like the other previous point on like time to solution, right? So that's the motivation we have with those kinds of initiatives. And again, like partnering with some of the customers, for example, OpenAI, right? So like [indiscernible] or like their training models, right? So they have like key customers on that. Yes. So yes, that's all. And we had a lot more different scaling results, et cetera, like showing really nice scaling patterns. In the interest of time, I'm skipping those. But with that, let me conclude. And if there are questions, maybe we can take it now or maybe towards the end.

Thank you, Jithin. I appreciate that. Again, I invite attendees to post and submit questions in the Q&A window. I'll start with one. Jithin, how does Azure InfiniBand performance compare to bare metal systems?