Keysight Technologies, Inc. (KEYS) Earnings Call Transcript & Summary

All right. Welcome to the first ever SONic Plugfest hosted by Keysight and Aviz Network. We have a pretty pack agenda for you guys today. We'll start with some quick introduction, talk about why we are here and how we started the SONic Plugfest AV events. We'll go over what we have tested what we have done -- what we have collected for the results. And then we have some Q&A sessions with our panel of experts. First of all, let me introduce myself and the presenters. I'm Dean Lee Senior Director of Keysight Cloud Solutions team. I have been in the test and measurement business, building the most sophisticated test system and methodology for advanced router and switches for the past 30 years. I still remember the big good old X25 package switches, that is how far back I can be. The next in our panel is Manod Ghose, who has over 10-year experience as Keysight Product Manager responsible for the introduction of many solutions in testing hyperscale data center. We also have Vishal Shukla, founder of Aviz Networks. Vishal is an industry veteran with solid experience in switching ASIC open Source NOS, system design and an author of several SDN publications. Finally, one of our SONic Customer Advisory Board members, Justin van Schaik will also present today. Justin has over 20 years experience in network architecture and design, specialized in low latency, high-volume networks and hyperscale data center ecosystem. Before we begin today's program, I'd like to give a big thanks to Aviz Networks' team with their expertise and contributions. We couldn't have done this without them. Why we are here: a key side with our network test business, the folks formally known as Ixia. We talk to a lot of key players in the industry, including switch ASIC vendors NEM/ODM vendors and data center operators. We see there is a lot of excitement and momentum behind SONic, software for open networking in the cloud and is growing fast. Already SONic has brought significant disruption to data center networking with its disaggregated architecture. According to the bullish prediction by Corner, by 2025, 40% of organizations that operate large data center networks with more than 200 switches will run SONic in production environment. With an open source NOS and the flexibility of choosing, ASIC [indiscernible] apps and services and support, network operators are looking for inserted business. And technical decision around migrating to SONic. Meanwhile, the SONic community is focusing on feature development and testing, deploy ability and system readiness for use in production networks and for the most part, being led to the hyperscaler with their huge investment in test engineering. As key contributors to SONic, Aviz and Keysight are aware of many of other challenges and know where the gaps are. We organize this Plugfest for network operators looking at SONic, with the goal of answering some of their questions and providing much needed insight into readiness and deployability. The first thing we did for Plugfest is to form the CAB, the Customer Advisory Board. And Justin from WWT will talk about the CAB next.

This is Justin van Schaik from World Wide Technology. SONic has provided data center network operators around the world the opportunity to enjoy the same innovation and flexibility, the top hyperscalers like Azure, Tencent and Alibaba enjoy. As Dean mentioned earlier, there are many challenges when it comes to deployment and system readiness. With the formation of the Plugfest Customer Advisory Board, enterprise network operators, system integrators, and cloud service providers finally have performed share, debate and review the most effective testing methodologies and to validate the maturity of SONic based data centers before deployment. The Customer Advisory board has been meeting regularly. On behalf of CAB members, we are very pleased that Keysight and Aviz have completed the first ever public test event, unifying SONic users, vendors and test tools to rigorously validate what makes for successful SONic deployments. The test methodology and tools are repeatable and commercially supportable, and the industry can leverage them to accelerate SONic deployments. We hope to see similar Plugfest events in the future covering the expanding features and capabilities of [indiscernible] SONic. With that said, let me introduce the next presenter, Mano.

Thanks, Justin and Dean for the insights. Let me jump right into our next group of participants who came forward to support this Plugfest with hardware and support to run through the testing. Keysight and Aviz were the neutral players in this Plugfest. Our motivation with the Plugfest is not to compare one vendor against the other but we do want the community to be able to see how far SONic in general has come and also to understand where the remaining gaps are. This being the first edition of the Plugfest, we wanted to keep the scope limited to a few vendors. This made it manageable to conduct this focused event despite the limited accessibility and social distancing due to the pandemic. We were fortunate enough to have vendors in this Plugfest representing all different types of ASIC makers, ODMS, OEMs and NEMs. This gave us an excellent cross section, helping with insights on the maturity of the platform deeper than just vendor a versus vendor comparison. By the way, a big shout out to Edgecore, Intel and Media, [ Chemeo ], Innovium and Delta, without their participation and hardware and software, there would not have been a Plugfest. Keysight is a test and measurement company. One of the areas we focus on is testing and validation of network applications and security. With a 25-plus years history of industry-leading testing solution our focus is helping our customers accurately understand how their products are performing so they can ensure needed quality and accelerate time to market with bleeding edge solutions. Embracing the disaggregated and software-centric approach of modern-day networking, our focus has been also on how to enable the open source communities and share our test expertise to drive adoption of SONic and other open source technologies like DENT, ORAN and others. Aviz Networks' vision is to enable commercial adoption of open source networking operation systems like SONic and DENT by providing enablement services, multi-vendor support with next-generation predictive tools and ultimately, using these [indiscernible] as platforms to provide containerized software-defined solutions to address various network challenges. Keysight enables have partnered to develop vendor-neutral test solutions, providing a level playing field, so one can better validate SONic across vendors. We then reviewed the test content with the CAB members and incorporated their feedback into it to match the requirements of the SONic end users. As mutual players in the SONic community, the Plugfest core team of Keysight and Aviz supplied the lab infastructure across two lab locations, the Test Suite forward to run the Plugfest and subsequently share findings with the community. In our vision to accelerate adoption of SONic. We have been in touch with many players in the SONic marketplace. That includes ASIC vendors, [indiscernible] as well as end users of SONic like hyperscalers, cloud service providers and enterprises. In those insightful conversations, we have come across a common theme. There is a need to standardize the testing. The community provides testing infrastructure for feature on [ duct ] testing. However, for successful deployments, there is always a need for real-life system tests to simulate complex networks and fail-over scenarios. Once we have the testing standardized into a solution called the Keysight Ixia Open NOS Validation Suite for SONic we opened our labs and ran through those tests with a variety of vendors. This helped us better understand the current situation with SONic and also made it clear that there was a need for this Plugfest. The Plugfest is an attempt to bring the community together and put SONic under the lenses of both the end user of SONic and the vendors in the ecosystem. So without further ado, let's jump into what we tested in the Plugfest. SONic is a layered architecture that is the SAI, SDK, management, configuration and then comes to control plane and data plane. As we built our standardized test packages, what we call the Ixia Open NOS Validation Suite, we wanted to cover these areas as comprehensively as possible. All these tests are built with system testing in mind, these are multi device tests including T1 and T0 typologies with up to four switches and many traffic generator ports in the test bed. This can be broken down into various logical typologies used in different test cases. The entire test [ two ] was driven by our flagship test management application [ Ixia ] suite store, which gives an easy interface with test playlist, yaml based configuration files, real-time test logs and HTML reports for comprehensive end-to-end testing. When we planned out testing, we wanted to cover three areas. Essential features, scalability and operational readiness. We started with the basic feature checks as part of the first package, the essential. In this first section, we covered a [ bung ] of platform tests to make sure the device we are testing is SONic ready. We validated platform features like [ SSA ] [indiscernible] authentication, LACP, et cetera. Then we did a basic check on the led to three functionality of the switch, like [ Vlance ], IP Forwarding, [ Harp ] and BGP. Moving on to the scale test section. Since SONic is really intended for hyperscalers and those with large and advanced data centers, it is only reasonable that one of the key test areas would be scalability. In this section, we scale each of the features to various limits to see how it does in those scenarios. Some of the areas that are tested at scale are BGP, [ CMP ], [ FDB ] routing, among others. Vishal will discuss some interesting findings from these tests when we get him to discussing the results. Finally, we had to look at the operational readiness and deployability of SONic, is SONic, ready for production using our data center? The most important KPI is that data center operators would look for is the time to failure and recovery. We are in a high availability world and network components needs to keep up with it. Some of the well-known failure scenarios that were exercised were linked failures, including packet loss and its effect on the device under test. So in all, we covered over a 100 test cases across six tenders which gave us a significant pool of test results to drill down on areas of SONic. We included both stock community SONic as well as specific vendor distros in the testing. We wanted to give SONic the best possible opportunity to shine. With that said, the features that we tested covered only SONic made features. Nothing specific to our vendor. This helped provide results parity across these systems under test. With the test of [ typologies ] in place the next step was to build the test beds in our labs and run through the Plugfest tests. As part of the Plugfest, we also discussed our findings with individual vendors and detailed the results analysis with them. Again, I really appreciate the time and effort the vendors spend with us in providing assistance with the execution and debugging. Okay. That brings me to the section we are all legally waiting for. Let me hand it over to Vishal, who will discuss the findings from PlugFest.

Well, thank you, Mano. Thank you, Dean. Thank you, Justin, and thankyou everyone, for joining the Plugfest -- so the webinar on Plugfest. So essentially, before I go into the results and talk about what the findings are and how we have summarized the Plugfest. I want to go into the details of that how we did the scoring on to that report. And the report will be public in a few days, so you'll be able to see what it is and how it is. But -- but this is a good platform where I can actually explain what we did. So essentially, as you can imagine, scoring was very, very complex for us to kind of see that how can we score some software, which is open source, having a lot of traction, and then you have multiple vendors. Different vendors have different ASIC solutions out there. And then basically, the way they support is different? So how do you score these solutions? So what we looked at is that we should look at from the consumption point of view. So there are customers, there are vendors and there are enablers, right? These are the three entities which are attached to SONic. So keeping that in mind, we basically came up with three different pillars of a scoring. The first one is a high level evaluation, which Mano just explained that we touched upon SONic features, we touch upon SONic scale and then we touched upon SONic operations, right? So the first pillar of evaluation was on that. The second pillar of evaluation was on the component level in valuation. So let's say, when customers start deploying SONic, they look at okay, which platform I'm going to use? Is the management function which I require is there or not? If there is a layer to deployment? And does it have layer to functions? And what are the scale of that layer to layer three functions have those available or not? What is the scale for that? And then the day two operations. This is more aligned with the customers who look for, okay, what they want to deploy, the component level. And then there is a third pillar, which is essentially goes into a double [ quick ] version of the component level where we talk about the detailed section of the category. So let's say, in platform, we have a bunch of test cases for protocol: a, bunch of test cases for protocol: b, similarly, in the management side layer-two, layer-three and day two [indiscernible]. So this is not essentially the number of test cases is essentially the group of protocol specific test cases. And in some cases, it is the convergence test and fail-over test and things like that, which is mainly listed in day two operation those 15 tests, right? So this is how we kind of made the pillars. And what we did after that we had six vendors are taking part in this block. So we took vendor one, and we said, okay, what is the score for 3 categories. Now score is essentially okay how the number of test cases passed as on a number of test cases failed. Similarly, for the second one, which has 5 subcategories. And then similarly, for the third one, which are 34 of categories, right? Similarly, we gathered the data from vendor one, vendor two to vendor six, right? After that, we took the minimum score out of the vendors. We took the maximum scores out of the vendor. And then basically, we average it out. Across all the vendors. The reason why we have done this way because this Plugfest is about SONic readiness. It is not about that vendor A is better, better than vendor B or C or things like that. That's not what we are trying to achieve here. What we are trying to achieve here is that SONic as an open source network operating system, how ready it is across multiple vendors and how many options out there are for customers across multiple vendors, across multiple options which customer has. So you'll see that I'll go into the next slides and kind of go into the same methodology, wherein we will have the high level evaluation table, which will talk about the maximum we found across all the vendors, minimum we found across all the vendors. And then, of course, the average out. And then similarly for the component level evaluation. However, the protocol level evaluation, that particular table is available for the customer advisory board only. And customer advisory board, I mean, every customer is actually free to join because this is more like a discussion about what they are looking for from SONic. And of course, they have given a lot of inputs on our test scope, as Mano mentioned. So that's the report will be given to them. And again, there also, it will not be mentioning about customer A versus -- sorry, vendor A versus vendor B. It will be more about average out and talking about how ready SONic is. Having said that, moving on to the next slide, under that high level growth. So you see these table and this chart is essentially cutting across six different vendors. So when we talk about essential, it's essentially the SONic feature packet, which is essentially talks about all the features, which SONic has today. And of course, there were some gaps which has been discussed in CAB, and we haven't put for that. So the average aggregated maximum was 97%, minimum was 91%, and average was 94%. Similarly, when we scaled it up, Layer 2 and Layer 3 put together, we saw the maximum test pass from A vendor in 93, the minimum was 76, and the average is around 80. Our operations, as you can see, there was some vendors who did very well and then some did not very well. And that aggregation is around 86%. Now as you can imagine, these are three categories which cut across all the protocols cut across all the scale operations and things like that, right? So this is good, but not that interesting. So I'm going to move to the second interesting slide, which is essentially going on to the component level. So this is where we kind of cut the pie into a different view altogether, which is essentially, we started with the SONic platform. Now SONic platform has probably nothing to do with the protocols. It has nothing to do with the week and everything. It is just a platform readiness, how good the platform was working and things like that. So you see -- I mean, it's a big rainbow in there. We have 64% to 100% and the average is around 89%. Then the management is essentially your SSH, your ping and your access ability of the switch via out-of-band management. And that you can see is pretty solid. Layer to function, as you can see, I mean, I was seeing there was one question with someone, and of course, we'll answer that question. Did you test STP? So essentially, as we can -- as we all know, right, SONic -- the DNA of SONic comes from the cloud networking, which is essentially ECMP based BGP fabric, right? And Layer 2, I think, is catching up right now. So you see that clearly in our test results. Is that the minimum is around 70%, which is one of the least on the protocol side, and that kind of shows that how it is catching up on the Layer 2 side, right, on the function at scale level booth. Then we have Layer 3 function, which is pretty solid as compared to Layer 2. As you can see, I mean, and 92% and 100%, the average is 93% which shows that all the work which has been done in community around BGP and ECMP and all the Layer 3 features, which makes that -- the next-generation cloud networks is actually showing up here. All the good work has been done in the community. You can see it here. And then comes the day two operation. I'll talk a little bit more about day two operation. What it means? Well, what it means is that when you have a fabric running, right, which is fine, data plane is fine, control plan is fine. And then you start doing some fail-overs, right? I mean, you start doing the link fail-overs, simulating the trans-receiver failure or you start doing the switch fail-overs or you start doing the control plane fail-overs and fill back in addition to fail-over, and then you start looking at the timings, then you start looking at the CPU usage, then you start looking at the memory usage. And you kind of realized that, okay, is this fabric is good enough from the SLA point of views, which are typically expected from the black box solutions, right? And that's where you see that, again, there was a rainbow from 73% to 100%. Wherein there is some work needed, I mean, according like Layer 2 here, right? So this is what we found out onto the component levels. Right now we do have details on the detailed protocol side, which will be given out in the CAB meeting, which is going to happen in a week or so. Right? And these results will also be available on to the reports, which -- the public report, which going to be coming out in a week or so. So now the results are one right, this is a mathematical data. You can look at it offline and everything, but I want to go into the key takeaways from the Plugfest. What exactly what we were able to do and what exactly we are looking for, right? So the first thing, right? I want to start with the Plugfest pact, right? So the very first thing and most important thing, which we were able to do for SONic is to be able to form a Customer Advisory Board. Dean talked about it, Mano talked about it. Justin talked about it. But I also want to say that this is one of the finest outcome of Plugfest that now we have a platform wherein, the customers who want to deploy SONic can come and talk about the features. They can talk about the quality. They can talk about the deployments and things like that. So it's essentially a group, which can -- anybody can join, right? So far, we have 18-plus SONic customers, prospective customers, existing customers and the customers who are looking for something to do in future with SONic. All of them are there, right? It's a mix of it, right? We have -- in this Plugfest, we have gotten 64 inputs on our tests scope from these customers, right? I mean -- and the inputs were like, "Hey, can you test this? Can you test that? Can you treat this test? Can you increase this scalability of that test? All sort of inputs, right? And we were not able to include everything in the time crunch and with the resources which we have, but we did added 10-plus quality inputs, and then the remaining one is essentially going into the next Plugfest vendor, we will do that. And we plan to do it very soon, right? And then the effort. I also want to thank the team here, the effort. So I just want to talk about how much effort was went into this Plugfest, right? So we basically ran 1,000-plus test cases in an automated fashion, of course, because why 1,000 pluses because there were multiple runs on the same platforms. And then there were six platforms, right, reported, right? And these issues, which we have and in our database, a bulk database has been given to the vendors out there to look upon it, fix it and things like that. We did that in essentially six different test bags and three labs. I mean there was a Calabasas, Keysight lab. There was a [ Fentorpara ] Keysight lab, and then there was a Fremont average lab. In these 3 different labs, we had six test beds and then we had 20 bugs out of those 50, which has been fixed by vendor so far. And of remaining 30 we are working. And I'm sure that with this Plugfest, there was a quality addition which has been made into the SONic across all the vendors. So this is essentially the fact about Plugfest what we have done. Now what are the takeaway? If I'm a customer, right, what I'm getting out of it. So the #1 thing is that the ecosystem readiness, you have to look at that, that the ecosystem is ready. This whole idea of Plugfest started in March time frame, we started reaching out to the customers. We started reaching out to the vendors. And you see within the period of 3 months, we were able to have this good six vendor and 18 customer coming onto a platform and doing this Plugfest. So one -- the first takeout I want to give away from this blood fest is that ecosystem of SONic is ready. You have ample amount of choices from vendors. You have ample amount of functionalities which is available in SONic, and there is ample amount of customers who wants to try SONic, right? So that's number one. Multiple options are available. You want a 1-gig option, 10-gig option, 40, 100, 400 all sort of options are available, even though this particular Plugfest was focused on 100 gig. And there was a slight addition of a 400 gig switch on the spine level. But the test which we have done kind of shows that there are multiple options which are available vendors and ASICs and switches, right? Layer 3 feature scale and management works good. So what it means is that if you have a fabric, which is BGP, ECMP and you have a basic management requirement. SONic is ready for that today. [indiscernible] right? Now I'll talk about the protocol gaps, and as you can see in my slide, the last point, which is the great takeaway from this is the consistency of SONic. Now you see the Plugfest test the way we had designed it is based on the community feature set, right? We didn't tweak our test for vendor A and then vendor B and then vendor C. No. The tweaking was not our business year. It was just one test expecting the feature to behave the way it is designed in the community. And I can tell you that you can imagine like in 2 months, we were able to do these 1,000 test cases with 6 vendors if this consistency would not have been there, I don't think so that we would have been able to pull it across, right? So that's the most important part is that it's not just you have multiple options available. It is also that the SONic behaves consistent, more or less, right, around, I would say, 90%- ish, right, consistent in all across all the vendors, right, that we have been able to also ascertain by this plug festival, right? So that's one of the great thing. I mean, if I'm looking for myself as a customer, a multi-vendor SONic deployment, then one of the things I'll worry about that, okay, will SONic work this way on vendor A? I mean, will it work same ways and vendor B or not? So here's the answer, right? Now what needs attention, right? So first of all, layer to scale. So we do have -- the features were available in layer 2, whatever is in community, but scale didn't work very good. And part of the reason is that the DNA of SONic , like I talked about, that essentially the design for layer 3, but now it is catching up for enterprise campus and you have the layer 2 features. The failover improvement is another section. I mean you do warm reboot, cold reboot, link failover and essentially fail the BGP control plane then the failover time and then the failback time, how much time it takes, and that needs improvement for sure. So that's another improvement area. And then the protocol gap. That's, I think, is a deployment inhibitor in many -- for many customers. At Layer 3, we identified, and we talked to some of the CAB numbers about it. So EVPN, VXLAN and BGP are numbered is top of the list on to the layer 3 side and spanning tree and MLAG, Multi Chassis LAG is another one, which is on to the layer 2 side. So this is what needs attention from all the vendors and community. And of course, OCP papers are going to be submitted and everything. We -- if you've seen our CAB, we have Microsoft in there. And we're going to be doing a readout about this in the main community to talk about the findings of the Plugfest, right? So of course, this will not just be in this webinar or just in this slide or just in this report. This will be taken to the community, and we'll talk about it, right? Now what is the next step, right? I mean what do we propose for the next step? So when I talk about next step, I see 2 dimensions there. First, from what customers can do, the second one, what vendors can do. So I would say for customers from the networking industry based on my experience, if you want to transition from a particular kind of switch or NOS to a second one, is essentially, you have to start at least 6 or 8 months in advance, right? You need to kind of understand the system, you need to kind of understand the behavior. So if you're even thinking 1% to going towards SONic , I would say that you need to actually start on it on a can POC thing. When I say can POC is essentially okay, talk to some of the SONic solution providers. There are many now. Who can actually help you to have a guided POC. So you can actually tell you, show you and get your hands dirty onto that, how and how good or how bad the SONic will behave for your kind of requirement. Get a support partner. Now each and every vendor switch and ASIC vendor out there have some sort of a support available. The -- be it native support, be it heterogeneous support, whatever support, there are many vendors out there you should get a support partner who can help you onto this journey of SONic. Now what you should look for is also the plug-and-play apps. And when I say plug-and-play apps is that the whole change the paradigm shift which SONic has done for networking, is that it has opened up the thing for the enablers. Remember, I talked 2 slides back that there are 3 kind of entities in the SONic era is one is the vendor and one is the customer and the third one is the enabler. Think of us right now, Plugfest is the enablement part of it, right? So what we are doing, essentially, we are enabling not only the vendors, but also the customers to make the migration easy, right? I mean, think about coming from a traditional legacy switch, and then going into the SONic. I mean when I talk to customers, they say, well, I mean, my legacy switch works fine and this migration of SONic, I need to have an army of engineers to hire for SONic, if I have to go to SONic. Well, not really. And why is because SONic is open source, you have a third entity in the network industry now, which is all of enablers, which can work with all the vendors and which can help you to enable you where the migration efforts goes to minimal, right? And that can actually put you on to the SONic path. So number one thing on that is the -- is something which can consistently test. So there are apps available. This Plugfest was done by using that such one test tool than there are fabric managers, which can give you the look and feel of exactly how Cisco or the traditional switch looks like and everything. Then you have intelligence suites, which are AI and machine learning driven support applications, which can actually not only give you the flavor of support, which we are getting from the top large network management tool, but also can complement it with the support options, which are coming out there. So a lot of options there. I would say if you're looking for SONic, start with the POC and then go from there, right? Now for vendors, right? So vendors, I would say that the most important thing for vendors is focus on the system quality, the end-to-end quality, community has done a great job because -- and you see that in our feature report is that the feature works fine. Right? I mean, if you just open it up, open the box, put the SONic bid, the feature will work fine. That's the testament for the community work which has been done, right? But then when you start pushing the boundaries, right? Just take the switch and push the boundary in different dimensions. You mix the dimension, start pushing the boundary or you basically start looking at the fan and the heat and all temperature sensors and all those kind of things, that's where it starts breaking up here and there. Right? So the vendors have to fix that system quality. They have to do the failover improvements, right? I mean, not like 2 BGP neighbor and 5 routes and do the [ BG ] thing and it will work up fine. That's not how our production network looks like, right? You need to push it at least to 10,000 routes and then do the improvement from that point of view and a protocol coverage. Protocol coverage, when I say it here, it means the gaps, fixing of the gaps, EVPN, BGP and numbers, spanning tree and things like that, right? So this is essentially -- is the most important thing. And I know community is working on that, which will be taken care from there. So -- and we are there to help and everything. So with that, I'll stop here for the questions and everything, and start on the Q&A.

All right. We do have several questions in Q&A section. So let me just start it from the bottom and a [indiscernible] expert, feel free to jump in and answer. This is one question. The SONic hardware compatibility lease includes on VoQ hardware. [indiscernible] what books does SONic provide to allocate, control and monitor this resource. This sounds like more like a design questions. And you can also look at it from a test perspective. So maybe, I would say, Vishal is probably more on your expertise. If you have any advice and comment on this one.

Yes, yes. So definitely. So basically, in this particular Plugfest, we focused ourselves on to the Trident and Tomahawk and spectrum and Tofino and Terralink ASIC. We didn't go to Jericho and Qumran ASIC, that was not part of our Plugfest, but I agree with you. When you talk about SONic on any platform, you are essentially not only testing the feature set, which software is giving you, but also how would ASIC behaves. Right? So this Qumran and Jericho and the VoQs for the hardware was not part of this particular Plugfest. But we are aware of it. And essentially, as part of this particular package, when I say that this commercial package of test and everything there are tests which can actually test the buffers, how fast the buffer fills up and how fast the buffers drains out. But those are designed mainly for, like I said, in Trident, Tomahawks and spectrums and Tofinos and Terralink kind of ASICs, not for Qumran and Jericho. And the reason for that is the major uptick which is happening on to the SONic right now is on those ASICs. As we see that Qumrans and Jerichos of the world will start doing -- we see the uptick on that particular ASICs we will definitely extend the tests onto that one. But right now, your question is very, very well valid, but it's not -- we don't cover those things right now in our coverage. Justin, you want to add something? So?

So I think the one comment I would say is our test methodology we use in this Plugfest is mostly based on black box testing. However, we do have a lot of input to retreat CPU, memory usage to check on specific counter register a part of the test. But the general test methodology could pretty much apply to any type of ASIC design. And so it's focused on system-level performance and scale and resiliency, right? So I think we just didn't have a chance to test these additional ASIC yet. So maybe the next Plugfest. Let me go to the next questions. It's good to see that your committee is tackling customer feature requests. So who is taking on that work? Isn't that based on SONic team? I think I can answer on this one. Yes, I think the SONic community is pretty much the group of people that actually going to make it happen. So like Vishal mentioned earlier, what we are doing here is we gather the people with the same interest right, the SONic customer end users and pretty much structure the test based on that particular interest. So we have discovered area for improvement gas and even the test methodology itself could be much more improved for proof of concept, for deployment, for operational readiness. So all that, yes, there's going to be a lot of input. What we're doing now is actually in 3 channels. We are working closely with our cap members to deliver the results and communicate the finding. So this will be one level deeper than what we have presented publicly today. So you guys all feel welcome to join our CAB. I will post the link there for you to sign up and join. And the second channel is with the vendors, with our 6 vendors, we have done this thorough testing and the test result is key sign of this intellectual property. However, we share that detailed results with each vendor, and that's how they end up fixing and fixing improve enhancement is up to vendor to publish and to contribute those enhancements and fixes back to the SONic community. Right? So that's the part they will take the lead on that. And the third channel is we're going to have both the Aviz and Keysight are active contributor to SONic community. So we're going to take our finding and read out to the appropriate SONic working group. And that's going to be our contributions when it comes to testing, we will be very actively contribute the test methodology and even the test automation code to enhance the community test coverage. And one other good example is the test we have implemented discovered, especially on resiliency, testing [indiscernible]. Those are very effective tests, and we did discover some interesting issues, right? All right. I kind of answer the next question. Are you planning to publish these 64 quality inputs? Yes, that will be pretty much how we going to communicate back to the SONic community. All right, let me go to the next questions. Can vendors run an automated test suite before joining these Plugfests. Mano, do you want to take on that?

Sure, Dean, thanks. So yes, so these test suites are all commercially available test suites from Keysight and Aviz. So we -- anybody can take them and run them on in their labs. This is a fully web-based application that I talked about briefly in my presentation as well. And these are like 3 sets of test cases like we have revalidated the features, the scale and the day 2 operations. So all these test cases are baked into a solution called the Ixia open NOS verification suite for SONic and that's something that anybody can take and run in their labs.

All right. I wanted to actually go back to a question we already answered, but I think it's good to share with the entire audience here. And this one is the U.S. Sonic in enterprise campus data center? I think Justin has already answered that, but I think it's good to share with the audience here, Justin?

Okay. So yes, we are seeing a lot of interest in enterprise and campus deployments for SONic, largely the evaluation stages right now. I think as the feature sets mature more, it's going to push out towards the edge. For the most part, it seems to be sitting at a core level [ we're spying ], which depends on mature baked BGP and works pretty well. As they go towards the edge a little bit more, we're going into the new community projects. So short answer is, yes, and we expect to see more still at the initial stages now.

I would also add because of the open source flexibility of SONic. I mean we did see publically target, we recently announced their design and implementation for their retail edge networking, which may use's SONic. And they pretty much took the initiative to add these missing part of a feature like STP. I think that was a question asked earlier, we answered in the Q&A section and something like POE, Power Over Ethernet. And my [ LLDT ] [indiscernible]. I think these are significant enterprise features, especially in L 2. It's not supported by SONic community today because in open source nature of SONic and a company like target can implement these features for their very specific needs. And according to that, [indiscernible] these features target planning to contribute back to the community. So that's really good for the industry to accelerate Sonic's deployment, specifically in the enterprise retail networking. All right. So I think we still have some time. This goes through the next questions. So did you use the management framework container for configuring, meaning [indiscernible] API, COI through management framework container? I think this is more on the test automation side. So Vishal?...

So right now, in these tools, we have used the SONic CLIs. We have not used the management framework and the reason is because the SONic CLI has been consistent across multiple switch vendors, multiple ASIC vendors. So that's why we have used the SONic's CLI. And it worked pretty good for us, right? So we'll see, I mean, as the management framework gets more runway, then after that, we will start actually testing that as well, adding some of the test cases for that as well to make sure that, that framework is also working fine 100%. So for this particular Plugfest, we just use SONic CLI.

All right. Let me get to the next question. Where can we find a detailed test report? Mono do you want to give everybody a quick update on that?

Sure. So yes, so the final report would be shared publicly. So first of all, we'll have a press release next week with all the details, where to find the report and all that. It will be in there with summary of the findings as well as part of signing up for this webinar, we'll send out a follow-up e-mail where we'll have the link for this public report, which is a white paper in that email. So please look out for that e-mail or please reach out to us. We all have to share all that information that is being discussed here.

All right. And the next question is going to be a tricky one. How can we get the list of bugs found in your test? Unfortunately, the answer is, we are not in the position to share that detailed report issues, but we did report back to the vendor. It's really up to a vendor how they want to handle that, right? I think I would expect this is something they -- for sure, they are fixing it. And also reporting contributed back to the SONic open source community and the [indiscernible] branches, right? So Vishal do you want to any comment on that?

Yes. I mean, so same as what you said, Dean. I mean this is something there were 50 -- close to more than 50 issues, which we have given to vendors, vendors are working to fix it -- fix those issues. And then it's up to them, they want to kind of report it into the community or they want to fix it internally. Because as in the beginning of the, we said, the motto of this Plugfest event is to kind of have the platform where we can make the SONic adoption faster, right? And the way to work with it is you work with vendors all of them, right, and make sure that customers have a lot of choices. And the part of it is to make -- work with vendor closely to get bug fixed. I mean this -- which vendor has what bug and all those things that -- I mean, that's not the intent of this particular Plugfest.

Right. I got one more question here. Can vendors get the test we access for free? Or we need to be existing customer of Keysight Ixia? So let me answer this one quickly. This automated test we used in this Plugfest is a software product running Keysight Ixia, hardware right? So there's a requirement, you need to have this particular Keysight Ixia hardware. And these test are licensed product. And so it's very different than open source community test bed, most of you are familiar today. And for a very obvious reason, right, we are providing support and services and turnkey solutions and very focused on system-level and deployment test. So this is an excellent complementary solutions in addition to the community test bed for developers and test engineer, right? So I think the short answer is, I think you can contact us or contact your sales Ixia Keysight's salesperson. They can arrange demo or the behind unit for proof-of-concept testing, right? So it is available and we are here to help vendor accelerate and enhance the test coverage. So with that said...

There's one more question.

Yes. I think there's similar questions, which key side product we need to have and use Keysight and you've already had key that product already. I think it will be good. Just reach out to us. We can give a lot more detail. I don't want to advertise our product yet, but it's pretty much -- if you have a Ixia 100 gig test system, this will be just add on software license, right? And we can also provide test as a service for a customer who doesn't have Ixia hardware. What we have done in this Plugfest could be delivered as test, as a service, right? So that's some flexibility here. Actually, I do have a question I think it's actually the question from the team here. And it's probably good for Justin to answer that because I do see this kind of question and concern quite a lot. What do we see as an impediment for SONic getting adopted by enterprise and fintech? Justin, maybe this is a -- you have more knowledge from all of us here. What do you think?

Sure. Easiest question to answer out there. So first of all, let's take a step back from SONic just refer to the open networking community, which include everything from DENT, [indiscernible] that sort of thing. So at World Wide, we actually see 3 different impediments towards adoption of the open network solutions in general. One is the historical act of support model, which, of course, if you're on your own, it's cheaper, but you will eventually be broken to be able -- unable to fix it. [ Obvious ], NVIDIA, Dell, various others stepping into this mix. So I think that historical problems gradually shifting out of the way, depending on the solution that the customer is going for. The second one is, I think, largely a matter of industry education. There's a misunderstanding of the scope of requirements to deliver a proper open network solution. Lab testing and [indiscernible] reverse logistics required to deploy these things, especially at scale are difficult for a lot of organizations to completely absorb on their own. It's a quote a very old phrase. It takes a village. Essentially SONic delivery requires partnerships between various vendors who specialize in delivering those kind of functions. Not to say that most of our customers aren't savvy enough to deliver them on their own, but at scale, it becomes a matter of specialization. The third -- and the customers who try to take it completely on their own, frequently have a rethought within 6 months and start trying to bring in partners to outsource various aspects of it. The third one is really just resistance by the big boys, Cisco, Juniper, Arista. They can do close ranks around competition. And we'll often some to drop price to buy the business. I mean honestly, everybody wins at that point. But those are the 3 things that generally are the headwinds that we encounter when we're trying to implement open network solutions across the industries.

Thank you, Justin. That's a very good analysis and observations. The one thing I would add to that is we also see a challenge in the area of testing, group of content and especially for the enterprise and tier 2 network operator, I think might like we mentioned earlier, the tier 1 hyperscaler, they have invested enormous amount of test engineering, right? So they have a pretty good coverage in terms of testing for success deployment. But when it comes to the general enterprise, I think in the past, they rely a lot on these major NEM supplier, right, to implement a lot of testing and services. And now with something like open source, a lot of those will have to kind of fall back on the network operator themselves. And like Justin mentioned, I think the key in network is those services will probably need to be handled by somebody else, my system integrator or in any other technology company by Aviz Networks that could accelerate the deployment and make the validation of design, performance in a much simpler and integrated as part of the overall services to enable the deployments, right? So we see that as a challenge when enterprise want to deploy, adopt Sonic. That is the area, probably need some work there.

Can I add a clarification point into it? One aspect of that, it's not just the tier of twos that have that problem. The tier 1 s also have that same challenge. And it's not a lack of sophistication in their testing methodologies. As they tend to typically run on 18-month to 24-month production cycles to go from new product to [indiscernible] enhancements through new releases to [indiscernible]. It's a 24-month cycle typically. And what we find is that when you're sitting there with something like SONic and there is a security advisory. There is a bug, there is something. This are running a 2-week cycle, not a 2-year cycle. So it's just a matter of agility of resource devoted to it.

Yes, very good point. Thanks, Justin. All right. I think with that, I think we're on the top of the hour, and I think we don't have any more questions, we answer all in. And finally, I think this is going to be the conclusion of Plugfest webinar. And once again, I'd like to thank Aviz Network for their partnership. And thanks, Justin, represent WWT to enhance our expertise in the panel in Q&A. And also like to thank Customer Advisory Board members with their strong feedback and guidance and most importantly, we have 6 vendors joining this. And with their support and their switches and software that really help the entire industry, understand the maturity and the deployment readiness of SONic. And as you can see, the result on today is very positive and with some area to improve, but this open source community has done an excellent job and I think we anticipate is going to be a much more successful adoption moving forward. So thanks a lot, guys. Thanks for joining, and have a good day. Talk to you soon.

Thank you, everyone.

Bye.