NVIDIA Corporation (NVDA) Earnings Call Transcript & Summary

Hello, everyone. Thanks for joining. This is Isaac ROS Webinar Series. My name is [indiscernible], a senior robotic software engineer at NVIDIA. Today, we're going to talk about ROS 2 benchmark, which is a benchmark framework for measuring performance of robotics applications. It was released in April this year with Isaac ROS DP3. We're really excited to introduce this open source framework to you, to the ROS community. [Operator Instructions] Okay, agenda. First, I will introduce a little more details about how ROS 2 benchmark works. Next, I will introduce r2b datasets, which is a datasets consisting of 8 rosbags that are useful for benchmarking. This was also released along with ROS 2 benchmark on April 6, 2023, with Isaac ROS DP3 release. Then I will run some demos to show you how you can get ROS 2 benchmark up and running fairly quickly. We also talk about how we extended ROS 2 benchmark to support our hardware accelerated Isaac ROS nodes, which will be good to know if you want to run benchmarks with our Isaac ROS nodes. Last, I'll walk you through a template for how you can create your own ROS 2 benchmarks group today. So let's talk about motivation. Why do we want to introduce a framework for benchmarking? We know that robots are real-time systems and nowadays robots are complex and complicated in terms of their graphs, applications and competition platforms to perform various crucial tasks like perception, planning and control. Those graphs and nodes need to perform their work deterministically with no one there to see. Yet it is especially hard to analyze when you have heterogeneous computations on various hardwork like CPUs, GPUs and even dedicated special-purpose fixed function hardware accelerators. So as a robot developer, something that can allow us test and benchmark the limit of our graph in nodes and most importantly, share and compared performance results is in need. So this is why we introduced ROS 2 benchmark. The benchmark framework with simple architecture that enables testing with shareable data sources and reproducible results. To achieve this, we use rosbag as the data source in our framework, and we made so that this framework is self-contained. So it doesn't depend on any specific packages. As a result, it is very easy to create and run your own custom benchmark with very little dependency needed. In this framework, the main elements are shown as green blocks in the diagram. They are benchmark nodes provided by the ROS 2 benchmark framework. Next, I will introduce their roles in that framework and how the benchmark can proceed. First, benchmark controller. It is the entry point of the benchmark program. It is actually a python class that you can extend to create a custom-based Python script that can be started by using large test to run a benchmark. It is in charge of orchestrating the entire benchmark flows based on some configurations. And the configurations can either be provided as a YAML file to be loaded at the start-up time or be written in your Python script programmatically. During run time, the benchmark controller is also in charge of invoking some prettify performance calculators to compute performance results and generate reports in the end. The report will be stored as a JSON file, so you can take the file and process the measurement data further. We introduced was included in the JSON file in later slides. Next is dataloader node. It is responsible for loading the provided rosbag. And just like rosbag play, you can select which time range of the messages in the rosbag you want to load by specifying time range in the benchmark configurations, which we will introduce later. And what it does is to read messages from the rosbag and publish it to its corresponding topics at a fixed rate. Then with proper topic remapping, the messages are expected to be received by the playback node. By doing this, you only prefer those that are going to be used later. During benchmarking, the playback node will play those messages in the way you can figure. This eliminates the overhead of loading messages from the rosbag file during benchmarking. Of course, it will take up some memory space. So if the memory space is limited, you can also configure the number of messages you want to buffer and make it reuse the same set of buffered messages during benchmark. For example, let's say, you can see that to buffer 10 messages and that they published 1,000 messages for benchmarking your [ graft ]. So either way, the timestamps when each message is published, will be recorded and shared with the benchmark controller to calculate performance after. So we're not competing any performance during benchmarking. Last, you can start as many monitor nodes as you want to take measurement for the topics of your interest. Each monitor will record the messages arrival timestamps for the topic you select. Again, the collected arrival timestamps are shared with benchmark controller after each benchmark run to calculate the performance for a topic. It's also possible to assign different performance calculators or even create your own custom performance calculator for any monitor node. These are all configurable from the benchmark configurations. Now let's delve into more about the benchmark flow and how those benchmark nodes complicate with each other. We can run launch test with your benchmark script. It will first load the default building configuration YAML file. You can either provide another file to overwrite a subset of the configurations of your interest, or like I mentioned earlier, you can programmatically revise that configurations in the script. The configurations should also include a path to the rosbag file you want to use in the benchmark. And the controller will use the set data service to request the dataloaded node to read the rosbag file. During handling this service request, the dataloader node will create all the publishers for the topics available from the rosbag file. Once the data loader node finishes creating the publishers, the benchmark controller will ask the playback node to start recording and ask the dataloader node to start loading messages from the rosbag. If you want to load the messages from a certain time range in the rosbag, this is where the relevant configuration will be effective. The dataloader node will publish all the message to the playback note at a fixed rate. And like I mentioned earlier, you can configure the buffer size in the playback node. Once we get a response from the playback node notifying that all the expected messages are buffered, then the benchmark controller will ask the dataloader node to stop loading. Then we will start the main benchmark process. The benchmark controller first asks all the monitor nodes to start monitoring, then it asks the playback node to start playing messages here. Here ROS2 benchmark framework supports 2 playback modes, which we will introduce shortly, and you can select which playback node you want to use via the configurations. In most common case, you can configure the duration of the benchmark for measuring performance. And once the playback now finishes playing messages, the base BAR controller will collect start and end timestamps from both, a playback node and monitor nodes, and they ask the corresponding performance calculators to compute the performance outcome. Now let's talk about the 2 message playback modes that ROS 2 benchmark supports. They're timeline mode and [ loopy ] mode. The timeline mode as it name says, it plays messages based on the timeline or timestamps when the messages are in the rosbag are recorded. The timeline information is passed from a dataloader node to the payback node. In this mode, messages are only played once. And again, the benchmark controller still collects the start and arrival timestamps from the playback node in the monitor node to compete the performance outcome. You can treat it as rosbag play, and it's integrated into the framework nicely. To use this mode, all you have to do is to set the benchmark mode configuration to timeline. The [ loopy ] mode, on the other hand, is more interesting as it's designed to test the limit of your graph. In the benchmark stage, it will try to probe and search the [indiscernible] sustainable frame rate of your graph for the given duration and then run a few more benchmark grounds to measure the overall statistical performance outcome. As I mentioned earlier, in most cases, you want the playback node to buffer just a little messages and then let it look through those buffered messages for the given benchmark duration, which is why you will see that there are a lot more relevant configurations you can adjust in this mode. Now let's talk about the algorithm. In the probe stage, it would run several rounds of tests and use banner research algorithm to narrow the [indiscernible] sustainable frame rate to within 10 hertz based on the [indiscernible]. After that, it was switch to a linear search mode to further narrow the [indiscernible] sustainable frame rate at the resolution you desire. And by the way, all these numbers I mentioned, they are configurable. So you can change them per benchmark based on your needs. And who determines the [indiscernible] sustainable rate, it will advance to the next step. The benchmark then will run 5 iterations or any number of iterations you configured. Performance results are measured in each iteration separately. In the end, we remove 2 iteration measurements that have the best and the worst performance in terms of the [indiscernible] and calculate the average results. This mode can also be configured to run extra benchmark iterations for some fixed rate. By default, it will run 10 30/60 FPS after the main benchmark for the [indiscernible] sustainable rate. Again, you can add any fixed rate you want to test in the configurations, and the result will be included in the final report. On the right, it is an example of the performance management result from a benchmark iteration. You can see that it shows the test duration in the iteration, the mainframe rate, the number of missed frames, [indiscernible] and CPU utilization. Again, you can also write your own performance calculator to measure and compute for any of your custom metrics. This table is the final report, you will see in your [indiscernible] output. It contains the main benchmark performance conclusion and also the measurement result for the benchmark iteration with fixed frame rates. You can also add some user-defined key value pairs. It can be added either in your custom configuration file or programmatically in your launch script. It is particularly useful if you want to note down some conditions that you set in your custom benchmark program. Last, some [ meta ] data that depicts the platform core system on the test, including CPU architecture OS version, the path to the benchmark rosbag and also rosbag hash value, so that you can verify if it is actually using the same rosbag file for benchmarking if you're trying to reproduce some previous benchmarks. And all this information will be exported to a JSON file, so it can be processed by some utility scripts or shared with other people. In the exported JSON file, or the configurations that are used in this benchmark will also be [ stream filed ] and stored as a YAML stream. This is useful when you go back and check the performance results, and you can see exactly what the configuration is used to generate this report and potentially recreate the benchmark with the same configurations. Before I dive into some demonstrations, let me introduce a little bit about the r2b datasets 2023 that will release along with DP3 April 6, this year. This data set is essentially a collection of sequences starting rosbag. It is free to download and free to use. And you can go to the link I showed here or just search r2b datasets from our NVIDIA NGC website, which is a place where you can download a lot of useful resources hosted by NVIDIA, like models and the assets. These assets consist of 8 sequences with recordings from [indiscernible]. Most rosbags contain decent data sources we capture from our dedicated coding system that produces LiDAR, realsense cameras and IMU data from HAC D455 realsense camera and XT32 with highly precise time synchronization. Other details are given in the r2b datasets page on NGC. Now let's get into some demo. I'll run through the quick start and show you how easy you can set up your environment and get the working ROS 2 benchmark running. These steps are the same as what you can get on our ROS 2 benchmark page on GitHub. Some highlights here before I show you my terminal. We'll be using the official docker container that has ROS 2 Humble preinstalled. We also download one of the r2b datasets from NGC website and then we'll be benchmarking in third-party Aprotech node. Okay. This is terminal. A reminder here, this terminal I'm showing is a prerecorded demonstration, because it may take a while to download and build packages before we can run a benchmark. So I think it is a good idea for us to record this video on my console and illustrate how it is playing. In this demo, I'm using a Jetson Orin board. And of course, you can use any boards and platforms you have on your hand that can run ROS 2. The ROS 2 benchmark is hardware independent. So you should work with any board of your choice. The first, let's download the official [ docker ] image and start a container with ROS 2 Humble preinstalled. This should be small, so it shouldn't take too long. Okay. Here we go. So next, we will set some environment variable that makes our life easier. The ROS 2 benchmark override asset route variable is to specify location of some resources that are going to be accessed by the benchmark scripts. And if you have your own data sets or the models that you want to use in your benchmark scripts, you can either use the default path or just override a path like what was executed here. And then we are going to update and install some basic tools. If you have them installed already, of course, you can skip this step. Again, now we will create a folder for hosting our packages, then we will clone 2 packages in this demo. One is the ROS 2 benchmark package itself and another one is a third-party Aprotech ROS node packaging, which is only needed for demonstration. Then we will run [ ROS 2 ] install make sure that everything we need to run, the packages are installed in our ROS environment. So here, it takes a little long, so I will fast forward a little bit here. Okay. It's done here. And then what's next is vital. Here, we are cloning and building the image pipeline package in this container by ourselves, because the one in Humble has an issue where the input and output topics in the resize node are conflicting, which make it unusable. Fortunately, this is fixed in [ rolling. ] So we are patching the image pipeline we cloned here with the fix. So we can have a functional resize node. And this is required by ROS 2 benchmark because in most examples we show, if you remember that I mentioned earlier, we used a resize node to downsize the images loaded from rosbags for benchmarking with a desired image resolution. And then once we clone that, we patch that. We will build the image pipeline again. Okay, once it's done, we pretty much have our code and environment ready. Now it's time to download the rosbag, we will be using in our demo benchmark from r2b datasets from NGC. We create a folder and then download the rosbags. We can do so by running double [indiscernible]. And of course, you can also go to the NGC website to download the entire datasets with 8 rosbags altogether as a compressed file, if you want. We are almost there. We can now build our ROS 2 benchmark package along with the demo package. These 2 packages take less than 1 minute to build on Orin, not too bad as there are not too many dependencies. After the build is finished, don't forget the friendly source [indiscernible]. Then we can start the benchmark for the Aprotech node. We have an existing [indiscernible] script created for Aprotech. So we just run launch tests that command with the script. In front of the lock, you can see that there are 3 benchmark nodes started as the first step controller will compute the hash value of the rosbag, so it later can be stored in the exported report. Once it's done, the dataloader node is informed and create a lot of publishers for other topics available from the rosbag. Now the benchmark begins the probe stage where it tries to find a [indiscernible] sustainable rate for Aprotech node under test. And this is done by [indiscernible] the playback nodes and messages are the fixed rate and observe the frame drop rate. It goes higher or lower, it depends on the results, which are showing in the terminal. And from the beginning, the test frame rate is usually high. So you can see that from the lot, the node complains about the messages not being synchronized as there are so many frames dropped due to not being able to handle that many frames at high frequency. And as the probe frame rate goes down, you can see that the synchronization issue is getting minor and minor to the point where it no longer complains about the frame drop. And eventually, the controller fights rate that has really little frame drop rate, which, by the way, is comfortable depending on your system requirements. After the [indiscernible] sustainable rate is determined, it will start to run 5 benchmark iterations with this rate. So in the end, we can remove the best and worst management and take average of the rest to conclude the performance results. And after 5 iterations, it will start benchmarking with a predefined fixed rate here even though by default, it is set to test 10, 30, 60 FPS, but since the [indiscernible] sustainable rate is around 56, which is smaller than 60, so it will keep benchmarking at 60 FPS. It's only around for 10 and 30. Finally, we get a summary table from all these measurements. This is the same as the table I introduced earlier. It also tells you where you can find the exported version of the program's final report in the JSON file. Okay, let's get back to the slides. We also use ROS 2 benchmark to measure performance against our hardware accelerated nodes as our Isaac ROS nodes use type adaptation to pass messages with data stored in [indiscernible] memory. The playback note and monitor node classes are extended to support our custom message types. Those extended nodes are implemented in a different repository named Isaac ROS benchmark. If you need some examples for supporting your own custom type with type adaptation, then that would be a good place to go. And because the framework is so modulized, we don't have to touch the benchmark controller or the whole architecture, but only to alter the playback node and monitor node with our dedicated nodes that support our message types. And based on this framework, we've been running more than 150 benchmarks [ nightly ] across 7 different platform configurations to track if there's any performance change in our node as we develop new features. With this framework, it's really easy to add new benchmark tests to run daily. So you are informed of any performance improvement or dictation every day. The raw Isaac ROS benchmark will be using our Isaac ROS custom [ docker ] container. This container has pretty much everything pre-installed you need to run the hardware accelerated Isaac ROS nodes. So you don't have to worry about the compatibility and going through the hassle to set up your environment. After you are in the container, steps are pretty much the same as the plain ROS 2 container. You pull down the r2b datasets with the packages and around benchmark. Okay, here comes to our last topic in today's presentation. I'm going to talk about how you can create your own benchmark with using ROS 2 benchmark. Of course, the best way is to take any of the existing base benchmark examples from the ROS 2 benchmark repository and replace the graphs on the test and configurations with your own design. Here, I'm going to show you at a high level what are embedded in the benchmark script that extends the benchmark controller. In this slide, it shows the template of the benchmark script. It has 2 parts. The top part is the graph declarations where we will put the benchmark nodes as well as your graph under test. The bottom part is where we will add any custom configuration changes. And you can actually see that it is quite similar to a standard launch test template with slight differences in customization for ROS 2 benchmark framework. Now let's take a closer look at each of the parts. First, in the launch setup method, you can add your graph of nodes to be benchmarked here by using composable nodes. There, you'll want to revise the mapping in the dataloader node, so it can reflect the topics loaded from rosbag. Optionally, you can also insert some nodes as preprocessors to handle messages before they send to the playback node for buffering. For example, in most benchmark examples, we added a resize node as the preprocessors to downsize the images loaded from our r2b datasets rosbags, which have 10 [indiscernible] in resolution. Then you have to make sure that a playback note and monitor node are connected to your graph correctly. Also, you need to specify the data type for each topic in both, the playback note and monitor node. This allows the playback node and monitor node to create correct publishers and subscribers for sending and receiving messages of the types you are concerned with. Next, don't forget to add other declared notes in the composable node container. Last, in the part where we extend the ROS 2 benchmark class, you can add any custom configurations. The part you added here has higher priority than the configuration YAML file [indiscernible] and also higher than the default configurations. So whatever the configurations you put here, they will override the existing configurations. And that's it. After you have the script, then you will just run the benchmark script like how you run launch test. Then you will see the output like I just showed you in the demo. I hope these are helpful. Next, we will have our Q&A session, and we will be answering those questions that are not answered to the presentation. And please give me a few moments, while I'm checking some questions. Actually, before I start, let me promote this. Okay. So here's a way for those who have used our -- will be using Isaac ROS, please let us know how you use it and what we can do better. By having some feedback from you, we will be able to plan our future releases better. And also if you have any questions in your web page, you can actually submit your questions, the QA widget. If you have any other questions not related to this webinar, please also feel free to go to GitHub page and create tickets. We do check questions there, and we do our best to address your problems.

Okay. So we already have some good questions, and we responded a few of them. Now, we also have a few that I think are interesting to answer here. The first question: Can I use the benchmark with ROS Foxy? Okay. So since we implemented for Humble, the ROS 2 benchmark is actually fairly independent. So I think if you take this package to Foxy and try to build it, it's highly likely that it can work, but if it doesn't work, it's still fine. The main packages we rely on are ROS 2 bag and patch image proc. For the patch image proc, if you don't use resize package, the resize node, then you'd have to patch that. So I think you can try that, and that's now the results. But I think the implementation is fairly simple. So that may work.

There was a question about why do we need to search for the best frame rate? And can you explain why we do this before taking measurements?

Okay. Yes. So the design of us ROS 2 benchmark is to try to find the [indiscernible] sustainable frame rate output for your graph. So that's why we have this better research and the research stage, where we try to probe the [indiscernible] sustainable rates. Of course, you can try to set as high as possible and then see what's the result. However, there are also frame drops that can impact the performance measurement. So that's why we try to narrow it down to the exact frame rate. So we test with that frame rates. Then you can see that there's no frame drop and see that whether your graphs are stable or not, because sometimes it may have [indiscernible], you don't notice when you have many frame dropped. So I think to find that exactly sustainable frame rates is important to us. And also, if you want to test the fixed rate, then with the [indiscernible] option, you can add the fixed rate in the looking mode, then it will test the fixed rates at the end.

And [ CY ], could you explain why someone might want to create their own benchmark? Or what would be the reason for doing that?

Okay. So if you implement the graph, of course, you want it to be useful, right? You may want to run it on your robot. And we know that robots are like resource constraint or [indiscernible] systems. So you will want to know how it performs like its execution time and also frame rate. And when you have multiple nodes in your robots, you also want to know their performance individually. So you know what's the -- where the bottleneck is. And then you may want to combine some of them and try to run it and benchmark it and know what's the influence, because we know that if you benchmark one node and then come by them with a graph, the [indiscernible] may not be the same. So using this ROS 2 benchmark, you can actually test all of them before you put them on your robot. So you know better what's the resource, what's the CPU utilization. We are looking at whether it is sufficient to run on your robot is essentially to create an optimal environment to test your graphs, so that you know better about your graph and hence, you can decide your robots better.

And then maybe one final question. Why do we use launch test and not ROS 2 launch?