One Stop Systems, Inc. (OSS) Earnings Call Transcript & Summary

Welcome to Trends in High Performance AI at the Edge webinar. This is Jim Ison at One Stop Systems. We have Charu Chaubal of NVIDIA and moderating the questions at the end is Tim Miller from One Stop. The agenda today for Trends in High Performance AI at the Edge, we're going to go through edge system solutions with OSS' AI on the Fly concepts, then look at NVIDIA for AI as Charu will go more into the A100 and NVIDIA accelerated computing platform, introduced the Data Center in the Sky platform, that's NVIDIA and OSS solution for the edge. And then take your question and answers. [Operator Instructions] So according to Insight Partners in a recently released market survey that the global rugged server market is valued at $2.4 billion in 2019 and projected to reach $3.8 billion by 2027. So a very good growth rate, and that's why these trends are very timely. OSS AI on the Fly is what we're going to talk about first to bring edge solutions that transfer raw data into actionable intelligence. So just a quick review of what AI on the Fly is. It's basically One Stop’s way of describing bringing the full speed, power and performance of different HPC elements out of the data center where they typically reside out into the field. And really, that's done with no compromises. So the position of, say, AI on the Fly as compared to an office workstation PC is more on the rugged side that you would see in the rugged embedded market. But where the embedded market will use low power chips or compromise on performance in order to meet the rugged environments. AI on the FLY tends to be more on the higher performance side, where the data center products that you want to use the full power of can be placed. So how is that different than AI at the edge? Whether people have heard an AI at the edge. Well, that typically refers to IoT products, anything from low -- a camera that may allow a construction worker into a construction site and that kind of AI. And all the way up to a very high-performance product like the NVIDIA drive, that is made specifically usually for AI inferencing, so a single purpose. Where AI on the Fly is different, and it really takes all the elements that you would see in the data center, not necessarily an embedded type product and put it out on the edge. So you can do high-speed data collection, AI training, AI inferencing and retraining all at the edge. So AI on the Fly is really a path from the sensors, which is on the left of this diagram, anything from video data, medical data, radars, things like that, be able to ingest it using high-speed data acquisition, store it for model creation and for training data sets and then compute with training, inferencing and even retraining, all that can be done at the edge. And all that brings you to the actionable intelligence part of AI. So on the data acquisition side, we really need high speed and high number of IO on the ingest. So that's the trend that's being seen and all-flash arrays tend to be that product that can handle those things. You need a lot of high-speed sensor inputs and that could be from some of our partners like Delphi that make very high-speed FPGA type data acquisition boards. Also have to have multiple power input choices because not every edge application has standard data center power, so DC power inputs and wide-ranging inputs is really important. It needs to be ruggedized for the environment when it's put out on the edge, have integrated software like data collection software. And of course, as the theme of this is, it'd be very high performance. So it could be a semi rugged platform like on the left or a more rugged platform that can be deployed even in aircraft on the right. And all this could be done with the OSS DRC data recording software or a product from somebody like Dynetics, who is a partner of OSS, that does a very purpose-built edge data recording systems. On the data storage side, now that you have the data ingested and now you need to [ massage it ], get it ready for models, get it ready for training and inferencing. Again, very low latency, high throughput is the trend in the market for the edge. All-flash arrays all still work in that marketplace. And it needs to be flexible, whether it be storage area network, network attached storage or just a bunch of flash connected to a single system. So very much has to be in a scalable form factor. NVMe over fabric fits that go very well. And in some cases, they need to be very highly available, so you don't have no downtime. So duplicates, hot spares, things like that in the storage is very important as well as the typical rate. So some products along that lines, the all-flash array on the left that can be fitted with the ion accelerator storage, which is SAN and NAS software. Or on the right, you have a SB2000, which is a semi rugged platform. What these 2 platforms have in common, even though some might be PCIe drives and some might be U.2s or EDSFF that the canister concept for transportability, for ease of moving data, if you need to sneakernet something from one place to another from the field to a data center is easily done through the drive pack concept, which are in both of these solutions made for the edge. After you have the data and it's stored, there's a step of model creation, and that does typically in the office, not necessarily at the edge with something like DSPro, in this case, a workstation that has all the AI frameworks preinstalled, pre-trained models preinstalled and as part of the NVIDIA ready program. So it's really turnkey and fully supported by both OSS and NVIDIA. Then we get to training. So that's where the big scale out solutions come into place. So when you're doing GPU accelerated training, it could be GPUs or FPGAS, but you need scale out. Performance per watt when you're at the edge is pretty important because you need to get the most out of the system when you're out at the edge rather than when you're in a data center and might have some lots to spare. Again, ruggedized to the environment and software and management are pretty important. So products like the GPU accelerated servers from OSS, the GPAU, being in current generation, and we're going to be announcing the new generation here later in the broadcast. And even in Rackscale solutions that can be done in traditional or mobile data centers, whether it be standard scale out type solutions or even composable infrastructure, all available for these type of applications. And then when you get to the inferencing side, again, what we're talking about here, there's so many products in the simple inference that One Stop is really focused on the high end inference, the scale-out inference where you can have a single server and many inferencing engines, even have some ARM solutions that are coming in the future. Again, in having the tunable performance, meaning that if you have one site that has a lot of power, you can use the full power of the system. But with the same system that might have a little more constraints, you can actually use the tuning capabilities of the -- or throttling capabilities of the NVIDIA products and the BOSS products to rightsize the solution, be a perfect fit for that environment. Some of those could be edge GPU accelerators that could take 4 of the A100 PCI Express or even scale-out inferencing platform like the 3U short. They can have up to 32 T4s in a platform like that. So all those are made for a lot of scale at inference. If you had thousands of cameras at an airport and you're trying to take those in through encoding a decoding, something like the scale-out GPU inferencing platform on the right could do that. So now I'll turn it over to Charu for NVIDIA for AI.

All right. Thanks a lot. So going to the next slide. All right. I'm going to be talking to you about the NVIDIA accelerated computing platform for AI. Now it might go without saying, but AI has been transforming every industry. It's helping to generate better outcomes in all these different areas. So just some examples here. So for example, at Ohio State, it's being -- AI is being used to analyze CT scans and identify critical cases for prioritization, which results in over an 80% accuracy in identifying stroke victims quickly. And when doctors are alerted immediately, they're better able to provide health care -- the critical treatment right away, which minimizes long-term damage. In the area of infrastructure, another story, Kansas City, Missouri, is using AI to analyze photos of its road network and identify where they can perform preventative maintenance before it escalates into an emergency, and that saves a lot, 50% on emergency road repair costs. Another example is in the IoT space with GE digital, their training sensor, AI to analyze sensor data and then be able to more proactively identify potential failures on their turbines. So by having this in place, they're able to save more than $6 million a year by having a reduced risk of outage. So just some examples of how AI is transforming industries. So what is driving this transformation? A couple of things. One, of course, is the large amount of data that's being collected, the volumes of data are exploding. And this is due to 2 things. One is the digital transformation of business. So as everything in business world becomes digitized that becomes a data stream. And secondly, as alluded to earlier, that's really the Internet of Things. When you have lots of sensors and cameras and other remote devices that also generates a large stream of data. And really, the way to think about this is that all this data is just raw material that's just waiting to be refined. You can get tremendous insight and benefit out of this data if you just know what to do with it. On the other side, there's compute. So there's really been this kind of, we call it, Cambrian explosion of algorithms in the last 5 to 10 years. So it's just incredible, like the kinds of things that are able -- people are able to do now. So things that were considered just like maybe science fiction, even not too long ago, is not only possible but really a competitive advantage. And if you don't take advantage of that as a company, you actually do risk losing out to your competitors. So the real need for GPUs has become more and more evident over the years and really Moore's Law has ended. The doubling of performance over a certain number of years, whether it's 2 years or 1.5 years or wherever it is. That's really just has stopped in the area of CPUs, which are built on a really a single-threaded type of architecture. On the other hand, GPUs really came in at the right time and provide the new approach, this kind of a parallelizable approach. So they've really taken up the stand [ mental ]. And now they're the de facto standard for AI, not only just AI, but also more and more so in data analytics and HPC. So as an example, you can see here some performance data, and it's really one of the obvious areas is in AI training. So you can see the speed up time of a system using 8 A100 GPUs as compared with 8 CPUs is nearly 30x. And furthermore, the latest AI models are becoming so large that they don't even really [ decide impossible -- decide infeasible ] to solve them with CPUs only. GPUs are not really just an option, they're a necessity for this large-scale training. And then in the area of inference, this is an area where the benefits provided by NVIDIA technology is becoming more and more evident. So as you can see, not only do GPUs provide higher throughput as compared with CPU-only solutions. An example, it shows 6x greater throughput for a typical example for our existing T4 GPU. But also, there's tremendous efficiency in terms of energy there. So energy efficiency is almost 18x greater. And furthermore, there are certain areas where latency of the inference is becoming very critical, such as when you have real conversations with smart speakers. And so GPUs are really able provide that real-time response time that's needed for that kind of application. So what NVIDIA really has today is an entire platform for accelerated computing. So it really starts with our GPUs, which has been our technology since day 1. And then also now we have technology from our -- from -- now that Mellanox is part of NVIDIA, the Mellanox SmartNIC and really the whole fabric that they provide. So we have both compute side and the data side covered in terms of being hardware acceleration. Then we work with our partners, such as One Stop Systems, and install these devices into industry-leading computers and servers. And so that way you have this hardware platform for accelerated computing. But where NVIDIA really has put in a lot of effort is around the software. So we have a lot of software for people who are developing AI applications. So we have the acceleration libraries and toolkits. We're providing pre-trained models and workflows and even industry application frameworks that make it easier to develop these applications, but it's not just for developers. For the people that are operating these systems, we have a lot of technology as well. So we've invested a lot in Kubernetes as an operations framework. We’ve created services to help you really optimally deploy AI models in production. And we also have software for doing -- managing large fleets of devices at the edge. And finally, for the IT admin. We've worked -- we've been working over the years to make it really a form that IT can support well. So we've had support for virtualization with the leading hypervisor platforms for a long time. We have in-depth monitoring and management interfaces and then integration with leading IT platforms, such as VMware and Red Hat. So all this together makes the best platform for doing accelerated computing, and this is really becoming like the reference design for the modern data center. So diving a bit further, the NVIDIA AI developer platform is a complete end-to-end AI software platform. So it really consists of several parts. So for data analytics and training, NVIDIA maintains a base set of libraries and SDKs that coupled tightly with the compute and data hardware accelerators to yield the best performance. And then on top of these, there's a large set of SDKs and other frameworks that accelerate a wide range of algorithms. Now in the area of inference, inference is actually a pretty complex task, but NVIDIA solutions throughout the workflow to help businesses achieve results more quickly. So starting out, we provide pre-trained models in a variety of different areas. And each one of these is generated from many hundreds of hours of supercomputing time using our own in-house supercomputer. The NVIDIA transfer learning toolkit lets customers take these pre-trained models and refine them for their own usage. Like, for example, you can take a general language model and then refine it for a more technical jargon. Once you have this model, the TensorRT is a graph optimizing compiler that optimize these models for accuracy throughput, response time and memory size. And then finally, Triton inference server is a micro service that can help DevOps bring these models production more easily. And finally, to help developers create these AI applications more easily. NVIDIA has created a number of application frameworks for different areas such as Clara for health care, Drive for autonomous vehicles, Jarvis for conversational AI and so forth. So -- and all of this software that I've mentioned is available for free from NGC, which is our catalog of software, which is really meant to help developers and data scientists build and run all their applications quickly and easily. As I mentioned, we -- NVIDIA has also invested quite a bit into a whole software stack for the operations side of things. So we have a whole stack for operating AI infrastructure, both the data center as well as at the edge. As I mentioned, the NGC catalog is a critical part of this being the hub for GPU accelerated applications, SDKs and tools that provides secure and deployment pipelines across different platforms and simplifies the building of these solutions. Kubernetes is a critical technology. It lets NVIDIA infrastructure be easily provisioned with the right infrastructure software, such as drivers and things like that. And so this enables these applications being managed by industry-leading IT platforms such as OpenShift. As mentioned earlier, but NVIDIA fleet command, allows IT departments to securely remotely manage a large-scale fleet of deployed systems. And with it, admins can add or delete applications, update system software over the air and monitor the health of devices spread across fast distances from a single control plane. So we -- the Ampere architecture was released earlier this year, and it represents the greatest generational leap in GPU performance. And it's the basis of the A100 GPU as well as the recently announced A40 GPU. So it brings a whole host of new features to accelerate all workloads. So the third-generation of Tensor Cores provides up to 20x greater AI performance without any co changes and up to 2.5x greater P64 performance for HPC applications in the case of A100. Now the second-generation RT Cores, which are available in the A40 provide up to 2x greater throughput for rendering operations. So NVLink and NVSwitch are 2 key things that enable multiple GPUs to work together. And so this third-generation of these technologies enable 2x more bandwidth with their data movement. So by using these technologies, you can really combine the capabilities such as the memory of multiple GPUs into like one large GPU. A new feature called sparsity acceleration takes advantage of sparsity in AI models and provide up to 2x greater performance. And then finally, the new multi-instance GPU feature, which is available in the A100, lets you actually divide a single A100 GPU into 7 independent instances, each with their own streaming multiprocessor, their own cast, their own memory, hardware isolated. And you can do this on the fly, and this enables optimal usage for multiple purposes as needed. You can split it up when needed and combine it back together as needed. So you can see here, in the recent MLPerf 0.7 training benchmark, the NVIDIA A100 set all chip performance record. So in a variety of different types of AI applications from image applications -- image classification, object detection, recommendation, et cetera, you’ll see that it did quite a bit better than any other competitive system for -- even for the -- and in some cases, the competitor wasn't even present. And you could also see how much better it does than the prior generation of V100. So really shows how we're continuing to show -- drive forward in this area. Also was -- even more recently was the results posted from the MLPerf 0.7 inference data center benchmarks. And once again, you can see how NVIDIA really is taking the lead on inference now. So doing inference with GPUs is becoming anywhere between 6 and 8x higher first in certain cases than CPU -- than the prior generation. And in some cases, then 200x faster than doing it on a CPU. So you can see that in terms of inference, both back to just where is real-time GPU really becoming the best choice. All right. So what is the value that we bring? So this is really showing the whole overall value of our accelerated computing platform. So we have a single unified architecture, and it's able to accelerate modern applications across many types of workloads, like data analytics, deep learning, visualization and more. And this architecture is available in a diverse set of form factors, anything from edge to virtual workstations to enterprise data centers to even the more technical like high-end supercomputing type of data centers. And as I mentioned, we have a diverse set of software frameworks that will let you make use of this architecture for a variety of different applications. NVIDIA is continuously driving full stack innovation in both hardware and software. So with this, we're continually getting better and better performance with also maintaining this lower TCO. So you can see that as compared with where we were 4 years ago, we've managed to create an overall 9x better performance. If you look at a collection of top applications in these various areas, such as simulation, AI, data analytics. We have a large and growing application base. So our developer community is just exploding in terms of its number of people. We're at more than 2 million people as of today, whereas we're at only 200,000 5 years ago. And this has a tangible benefit. So the number of accelerated applications. So the applications that have GPU acceleration built in has gone over 700, and these are the more commercially available type of applications. This is, of course, in addition to all the applications that are built in-house using the frameworks that we provide. So by adopting this platform, you can really maximize the utility of your investment because you can -- it's going to be ready for running not only the applications you have today, but also tomorrow's applications. As I mentioned, we have a comprehensive platform ecosystem. We integrate with the leading platform providers, whether it's in areas of like data center management, like VMware, Red Hat or data -- other data platform or even more of the [ visitation ] side with Autodesk. So this really provides enterprise great orchestration and management so that these are not systems that are hard to manage. They can be managed using standard operations. And finally, we have this -- we're working on, and we have tested and certified designs to really make it easy for you to adopt this technology. So working with partners such as One Stop System, you can be assured that you're getting a system that's really optimized for your needs. So the last thing I'll show you is a view of our data center GPU portfolio. So these are [ certainly ] shipping products. So the NVIDIA T4 provides a balanced set of capabilities and is able to run a diverse set of workloads in both the data center and the edge. Now if you want to go up and become more specialized, then the A40 is the best choice for demanding graphics workloads, with large graphics memory and fast ray tracing other capabilities. And the A100 is really what you choose for the highest compute performance, so in the areas of data analytics, AI and HPC right. Now I'll turn it back over to my co-presenter from One Stop Systems.

Thanks, Charu. Just trying to pass the baton, and here we go. So we heard about some of the trends in AI on -- that we call AI on the Fly, the NVIDIA solutions that are around that. And now we'd like to talk about Data Center in the Sky platform, which is the culmination of really what Charu and I have been talking about. On the NVIDIA side, the DGX products and performance that Charu has showed you for the DGX NVIDIA platform, they make available to OEMs like ourselves as part of this HGX A100. So it is the GPU section of the DGX that we can take and make available to platforms that are needed more in edge applications. So that HGX A100 has the 8 SXM A100 GPUs that can be partitioned with the multi-instance GPU from one large GPU of all 8 of these, down to 56 smaller GPUS, 7 on each of the GPUs in the system, all interconnected by the NVLink 3.0 mesh of somewhere neighborhood of 600 gigabytes per second of interconnect, supporting 8-way systems, we're going to talk about here, or even 16-way systems. With all the uplinks to the NICs, SmartNICs that Charu had referred to. And really an optimized building block for OEMs like One Stop to do things. One of those products would be made for the data center, the Ampere 8 product. This one is a NGC ready platform that has all the optimized AI frameworks preinstalled, turnkey system with the full support, but it is made to the data center. But what if you needed that HGX solution, that large GPU scale-out solution for training or inferencing in these type of applications, driving vehicles, the aircraft to helicopter or even on the road in a media and entertainment venue. Those type of applications, where maybe they have access to the cloud is too slow for what needs to be accomplished. In an aircraft like the submarine hunter, where you have to identify, monitor and process and act on your own while you're there because the down link to the data center is way too slow over wireless. And similarly, in autonomous vehicles, a lot of decisions need to be made on the fly in tenths of a second to hundredths of the second. And even the 5G network, although very fast is still too slow. So the intelligence really needs to be at the edge. So there's some key challenges that Ampere 8 for the data center would have and that's in depth, weight, power, temperature range and ruggedness are all concerns for that product. So through extensive thermal analysis that OSS has using our tools in the engineering to optimize airflow around this be in the HGX section of the product that we're showing, we can optimize that airflow and the fan size is needed or something might be a little bit a higher temperature application. Similarly, on the structural side, using finite element analysis that we can do structural load analysis for ruggedness and shock and vibration of edge type applications on something like the HGX A100 that would be experienced at the edge. In addition, power is different at the edge. You might have -- we have to have 2 types of power to the GPUS, 54 volts to handle the GPUS, 12 volts for the rest of the CPUs and I/O boards and things like that. So you need a wide range of power inputs from 100 to 240 volt AC from 50 all over to 400 hertz on the AC input and even 48 volts up to 270 volt DC on the input side to be able to power something like the HGX A100. So that's where the Data Center in the Sky platform comes in. When you need the Data Center in the Sky in the vehicle, in a mobile shelter, OSS has solutions that are optimized for doing AI at the edge. So the Data Center in the Sky challenges that we talked about, we’d start with the HGX A100, take the motherboard, storage, networking and edge-friendly configurations. It could be Intel, it could be AMD, it could be ARM. Pack is in for an enclosure, keeping the set of NVIDIA ecosystem software and frameworks that Charu had explained earlier, really keeping that access and capabilities for the developers and the users to fit the application and use those innovations in power, cooling and ruggedization to come up with the Data Center in the Sky, the GPU accelerated server rugged that we're showing here. So it is fully mill tested for shock and vibration. 24 inches depth tended to go taller because the 24-inch depth will fit in the rack in different applications, a more wide range of applications than, say, something data center depth in the 30 to 35 inches deep as these things keep getting deeper and deeper in data centers. And think about things like variable power and noise. So if you have a different edge requirement, you could lower the power or you can bring it up to full power. Nonhazardous wiring and even low center of gravity, which is good for things like mobile shelters. So the design includes a tiered system. So have compute, networking and power zones that can optimize the cooling. So have a PCI express switch fabric tier, use the PCI express from the HGX out to the SmartNIC, so you can get the data into the system and out of the system through the switch fabric. Have the server tier. So that can be, like I said, AMD, Intel or ARM processors, to give you the CPU management that you need. HGX A100 sits in a tier with optimized cooling, and power at the bottom, again, keeping that low center of gravity, which helps in mobile type applications. So the rugged design includes aluminum frame and skin, cross bars along the entire depth to keep the product safe. And then additional features like rounded corners, milled fillets to reduce the weight and stainless steel mounting points where the most, say, dagger pins or rackmounting ears where those take the most abuse in an edge type application to be able to protect all the high-performance products inside. And it's really made for the edge. You can see the lower depth, lower weight, the variable power that even starts at a lower power point and has shock and vibration certifications, connectors that can be positive locking and things like that. It's really solving those edge challenges. Now in addition to the Data Center in the Sky product, we have many other edge AI solutions using the A100, including liquid cooled designs, customized form factors or maybe that might be better for a autonomous vehicle, so to say, instead of that larger system. So we have customized solutions where you can customize the form factor. This one showing up to 4 GPUs on a whole plate style design that could be liquid cooled in the trunk of a car. This is being deployed today, not with the plastic heating, but with a real metal heating. And is fully PCI express 4.0 to support the A100 products. So that ends the trends in high performance computing full agenda. You could use the question drop down box in your screen, and I'm going to turn it over to Tim and unmute him, so he can ask Charu and I have some questions.

Okay. Well, thanks, Jim. Yes. So we do have a couple questions. Let me start, I'll paraphrase. I guess to start, Charu, the concept of the multi-instant GPU with the A100. Could you maybe explain that a little bit more in terms of use cases, how we might see that being utilized by folks, maybe especially on edge applications?

Yes. So it's a really nice feature because it really lets you get, I might say, 2 GPUs for the price of one. So what we've observed is that customers, although they [ get ] power of a very large GPU like A100, they like the large memory size and the power of the capabilities, they indicated to us that they don't need that all the time. On the other hand, they still have the need for running smaller jobs. In this case, a great example is inference. So what you can do with MIG potentially is to say that I'm going to use it as one large GPU for doing larger jobs, maybe large-scale training, things like that. But then when it comes to inference, I really don't need a large GPU, but I have -- I need to process a lot of data in a rapid amount of time. And so what you can do is you can effectively get 7x the throughput by just having it split into up to 7 instances. And so inference is a great use case for using MIG, where you can have one system with 1 or 2 A100 GPUs. And then you can split it up into a large number of -- up to 7 or 7x as many -- the number 7 times the number GPUs you have and then have that many inference streams going. So I think we see that inference is going to be one of the key use cases for using this MIG capability. Another use case would be, of course, like you have developers that are developed model. They need just a small slice GPU for the model development. And then when they're done, they want to actually try it out on a larger GPU. So then you can -- instead of then giving them 1/7 of a GPU, you can reduce it -- you can make it -- split into 2 slices, which is equivalent of like having 3 of them in 1 slice or 4 in another, for example. And then you can have them work on that. And finally, when you're ready to do the actual large-scale training, you would then recombine it to one large GPU. So there's a lot of ways that we think that MIG will be useful and make it a really versatile product.

Okay. Great. Thank you. Jim, this one for you. We've heard a lot about ARM processors and NVIDIA GPU computing. Does OSS have any plans in this area?

Yes, that's a good question. I know I just touched on it briefly. But so as far as OSS is concerned, some of our platforms are all-in-one solutions, like the Data Center in the Sky platform. But we also have expansion products for A100s Gen 4 accelerators of all types. Those are really server agnostic because you're using a host plus adapter board from a server. It could be ARM. It could be Intel or it could be AMD and connect over to a large-scale A100 solution like our -- we have 4U Pro product and a 4UV product. Those all do that. But on the all-in one systems, we plan to -- and all that's capable -- possible because NVIDIA has support for CUDA for ARM and all of those ecosystem things that we're talking about. So that really opens up ARM, which has a much better performance for a power performance curve than, say, an x86 does. So we do plan to take advantage of that in our all-in-one platforms like we do today in the expansion platforms. So look for future products in that.

Okay. Good. That actually looks like it addresses the questions that have come in. So thank you, and Jim, I'll hand it back to you.

Yes. Charu, thank you. And this is Jim Ison. Thank you for attending the webinar, everyone, and appreciate your attendance.