Intel Corporation (INTC) Earnings Call Transcript & Summary

Hi, everyone, and welcome back to our monthly Intel generative AI webinar series. I'm Ed Groden, I'm in the AI sales enabling team here at Intel and I'm going to be your moderator for today's webinar. A couple of things before we get started. So the presentation is going to last about 40 minutes, and we're going to take questions afterwards. So please submit your questions. We'll try to get to as many of those as we can after the presentation. Also, the PowerPoint slides are going to be made available, so look for those in a follow-up e-mail. So the topic for today is emerging frontiers in generative AI, Agentic AI workflows and multimodal large language models. So I'd like to introduce our 2 expert presenters. So first, we have Dillon Laird. He's a founding machine learning engineer at Landing AI, where he leads efforts to bring multimodal models and agents to customer use cases. So his team is interested in making it easier for users to solve their vision-based problems, and you're going to see that in today's webinar. And then we also have Vasudev Lal, and he's the principal AI research science Manager at Intel Labs where he leads the multimodal cognitive AI team. So his team is focused on pushing the state-of-the-art in multimodal AI by using Intel Gaudi based clusters in the Intel Developer cloud. So without any further ado, let's turn it over to Dillon to get the webinar started. So Dillon, take it away.

Yes. My name is Dillon, and today, I just kind of wanted to give a quick overview of the different types of models that we're going to be talking about today. So on the left and to this is one that everyone is familiar with, a large language model. This is often times you have some sort of text input and text output. And then Vasudev is going to be talking about large multimodal models. And so those are models that take different types of inputs, typically image in text, but it could be other modalities as well. And then we'll also be talking about agent frameworks. And agent frameworks combine multiple LLMs or LMMs with sometimes outside tools to take in text image or other things and do kind of even more complicated tasks. And with that, I'm going to hand it to Vasudev.

Thank you for that excellent primer on the distinction between large language models LLMs, large multimodal models LMMs and AI agents. So here, we'll talk a little bit about large multimodal models. Now many of you might know that large multimodal models are already deployed in production and 2 popular proprietary examples are GPT-4V and Gemini Pro. In our team at Intel Labs -- our cognitive AI team at Intel Labs, we've been very interested in a class of large multimodal models that are known as the LLaVA architecture. LLaVA stands for large language and vision assistant. And these models were first introduced last year in the paper that cited below, which was presented in Europe's last year. The basic idea behind LLaVA is actually pretty simple. You take an LLM, you attach an image and coder with the LLM, which is typically a vision transformer. You add some projection layers between the vision transformer and the LLM and you do typically 2 steps of training on image text data. The resulting model is called a multimodal LMM or a large multimodal model, as shown on the slide. And as Dillon pointed out, what these models can do is that they can take as input both image and text and produce text. So for example, here, we give an image to the model, and we asked this question, what is unusual about this photo. And the model is able to provide a correct answer that the person is ironing underwater, which typically happens on land. And I would like to point out that a lot of these LLaVA architectures are very competitive with proprietary large multimodal models. And it's fairly simple to change out components in this LLaVA architecture. For example, the LLM can be changed out to the leading Opensource LLM, the vision encoder. The image encoder can be changed out to a variety of image encoders within this particular architecture. So in our team, we've been very interested in also understanding how these models work, how do they ground the output text and the image. So for doing this, we've made a tool that we call LVLM interpret. There's a link on this slide, which is linked to our project page. There's a link to a paper and screen cast for this tool, and we'll be presenting this paper at CVPR next month. We will also be opensourcing this tool within this month. So in this example, we can see that there is an image and the model has produced this text about the image, an elderly person is sitting on a bench. And what we can do with this tool, is that we can understand by probing the internals of the model that as this multimodal LLM is generating the phrase elderly person, the attention heads are appropriately focusing on the right region in the image. And this is how these vision language models are able to ground the output text in the input image. So -- as soon as we made this tool so that we could understand how LLaVA might be able to ground the output text in the input image, our next question was that could we actually use this to detect when LLaVA might hallucinate, when LLaVA might be inaccurate. And this is exactly what we show in this animation here, which is taken from a screen cast on our project website. We take this image, which is -- this image of a truck, and we asked this question that is the door open? And as you can see, the model actually answers it correctly this time. But if we analyze the internals of the model, we see that the model is actually focusing on the token open, which is part of the question and not even looking at the image. So consequently, if we alter the question now. So the question is, is the door of the truck closed? The model will actually say that the door of the truck is closed because again, if you look at the internals of the model, the model is focusing on the word closed and the question. It's not even looking at the image. So by using this tool, we can detect what part of the image is the model focusing on when it's producing individual words. And we can detect when the model might not even be grounding it's text output and the image itself, which can be a sign of when the model will be hallucinating and will not be accurate. So now that we've learned a little bit about large multimodal models. Dillon, would you like to show us what an AI agent can do?

Awesome. Yes, definitely. Thank you for that introduction there. So I'm going to share my screen here. Can everyone see this? Now I'll go to the next slide here. So let's say, you've got a camera that's watching your pool and you want to check to see if your kids near the pool. So you might have a view like this and that's your child and you want to maybe write some sort of program to automatically detect if your child is near the pool. So this is a use case that we could use vision agents for. So I've written a little program here. And I'm going to create a vision agent. And I'm going to ask it to analyze the video and figure out if there's a person getting very close to the pool, output a JSON list of timestamps, height of the person in pixels and corresponding tag danger if the person who is within 60 centimeters of the pool, safe otherwise. And then maybe for a simple conversion, let's say, 1 pixel is about 0.75 centimeters. So I'm going to run this, and let me close this window here. And what's going on is this question is going to be proposed to the agent framework. And the first thing it's going to do is analyze the question and come up with a plan of subtasks to accomplish the -- whatever the user asked in this case. And so we're using kind of GPT for -- as our LLM to do this particular task. And interestingly, oftentimes the planning task actually takes a while for GPT-4 to finish. I'm not exactly sure why sometimes it may actually be using an agent framework itself behind the scenes, but in a few moments here we should kind of start to see the output of the agent framework and kind of the different things that it's doing to accomplish this task. So -- yes, let's give it a few more moments here. And you'll see it kind of output the task. And what it will do is it will make kind of -- yes, here we go, a dependency graph of which task it has to do and which task it has to do before accomplishing other tasks. So the first thing it's going to do is take that video and extract frames from it. So we have some frame extraction tool. And then it's going to call a tool grounded-SAM, I mean it's going to use this to segment out the person and the pool. So this is a model that we posted [Audio Gap]. And you can see here that it's picked the tool name [Audio Gap] and then after that it's going to calculate the distance between the person and the pool. So we can figure out how far apart is the child from the pool. We're going to get some statistics on those boxes so we can actually figure out how many pixels tall the boxes are because in this case, we're outputting sort of relative coordinate so instead of 50 pixels, it would be 0.1 percentage of the image here. So right now, it's calling our internal tool grounded-SAM. This might take a little bit because it's got [indiscernible] on each of the frames. So we've extracted 3 frames here at 0, 2 and 4 seconds, and we've gotten a high confidence score for the person, maybe a little bit lower confidence score for the pool in each of the 3 frames. And you can see the coordinates there. Those are X1, Y1, X2, Y2 coordinates. And then after this, it's -- it should try to look at the distance. So we'll see if it gets the right tool to calculate the distance between the person and the pool? And you can see here, so each of these steps, it's actually using a different agent to do them. So the first step, it was using an agent to plan. And now what it's doing is it's going through each of those sub tasks and it's using an agent to find the tool. It's using an agent to get the parameters and then it's actually calling the tool. And then sometimes, it will actually reflect on the tool output. So maybe the tool fails and it needed to pass a different set of parameters there. Cool. So now we've gotten some distances in pixels. And we also -- we asked it to calculate the height of the person. And because those box coordinates are relative, so 26%, it needs to get the statistics from those. So -- okay, so now we've got -- the person is anywhere from 118 to 150 pixels, probably as they move closer or further away from the camera. And then we're going to -- let's see, do our conversion. So this was the pixels to centimeters conversion. And we're using this on the previous calculation of distance between the pool and the person here. So -- yes. So at the first frame, they were maybe 200 centimeters from the pool and then they got closer to 60 and 9 centimeters to the pool. And that makes sense. If you saw in the video, the child is sort of running from the edge of the lawn towards the pool. So we kind of expect to see distance numbers like that. And I think the final one here, it should try to calculate the JSON output. So it's going to check if the -- these are within 60 centimeters of the threshold we gave it. And this here might be an example where it actually didn't need to use that particular tool, so sometimes reflection kicks in and it starts to reflect on some of these tool users and say, no, that wasn't right or this was right. So it's still a little bit noisy and then how it accomplishes tasks, but yes, I think this is the final JSON output that we would expect here. So at 0 seconds, the child is safe. And then at 2 to 4 seconds, they're closer to the pool. So they're in what we call danger. And at the end of this, what's actually going to happen is if it -- we will actually plot the boxes and the masks on the image. And then we'll send those to GPT-4V and now we'll actually look at the plan, so it's going to see these images and look at the plan and the final output and make a judgment call about whether or not that model did the correct thing. So in this case, it reflected and I thought it did the correct thing. But if it didn't, it would have sort of replanned and tried to do the problem differently. So we can see here, this is probably the fourth second. I think this is the last frame. The child is definitely near the pool. Here, they're near the pool, it looks like there was maybe a false detection there as well. And here, they're sort of safely away from the pool. So that's just an example of one of the things that you can use vision agents for. And I'm going to go back to the slides here. And let me move my screen down here. Yes. So just to kind of give like maybe more of an interesting sort of look into what agents are and how are they different from LLMs. LLMs are really good at kind of these like zero-shot problems, right? So you ask it a question and you get one answer. But for more difficult problems like, let's say, you wanted to write program. Oftentimes, you need to do more things to do that correctly, right? So when I write a program, right, I'll usually write an initial version and then I'll maybe test it out, and it often fails the first time. And so I need to iterate on that, right? I need to see what the failure is, I need to think about it, reflect and I need to update my program. LLMs, by themselves are kind of very limited in how they can sort of search through this problem space, right? So agents are kind of a way to expand the search capabilities of LLM to handle more complicated tasks like self-reflection, planning and tool usage. And right now, we sort of manually program in these workflows. But you could see future work where we sort of try to embed these more interesting search capabilities into the model themselves. One graph I'd like to look at, that's maybe a more practical view of agents is often times like a latency versus performance graph. So you usually see this with like CNN or other models like that. But on the low latency side, you have LLMs, maybe you're running in less than 5 seconds, maybe it's even faster with some of the latest advancements, but you're getting some sort of text in, text out. LLMs because you're taking in more modalities such as images, it might run a little bit slower, but you're going to be able to do more types of tasks and probably use that extra modality to squeeze out more performance. Agent frameworks, on the other hand, are going to be much slower, maybe 30 seconds, maybe even 5 or 10 minutes but you're going to be able to do more complicated tasks and get sort of higher performance out of those. So each one sort of has their own use cases where they're useful, but this is maybe one way to look at it. And this is a slide from Andrew that he -- presentation he gave recently at Sequoia. So this is a particular coding benchmark called human eval. So this is a bunch of coding questions. And you can see here, this may be slightly old now, but GPT-4 was a zero shot, trying to finish a coding question, was getting about 67% performance. But with some of these agent frameworks like agent coder, or LDB Reflexion, you can boost that performance to 95%. So that might be like a practical reason why you might want to use these agent frameworks. And so now I'm going to go over some of the different components that make up agent frameworks, right? So that demo, we just saw what exactly is going on inside the agent framework? And what are the different agents doing. Maybe the most common one is planning, right? So when you're first given a task, you usually want to decompose that task into smaller subtasks. So that you can have individual LLM solve each of those sub tasks, right? So if I wanted to detect if the child is near the pool and I give it a video, what it's going to do, is come up with this sort of [indiscernible] of subtask, right, where the sort of subtask might be convert the video to frames, then detect the child, detect the pool and then calculate the distance between the child and the pool. And you can do sort of more advanced takes on this where maybe as it's going through the plan, one of the nodes fails and you want it to self-reflect and sort of replan dynamically. There's all sorts of interesting things you can do with this. And some papers to read that are kind of related to this. Our chain-of-thought is a very, very good paper that was -- came out a few years ago. And HuggingGPT is a really cool agent framework that sort of uses planning, plus a few other things to utilize hugging face models. And so what might this look like? You can think of this dependency graph with each of these nodes indicating some subtasks typically associated with the tool usage, right, so you can do detecting the child and the pool maybe in parallel, but definitely measuring the distance depends on doing those 2 tasks first. And then finally, can output the child is nearby the pool. Another important component is tool usage. So tools can help the model sort of solve tasks that it would otherwise struggle with, right? So I don't know, maybe the LLM can do a particular task, but oftentimes, it's better to get a specific tool to use it. If you were to ask me to label a bunch of people over 1,000 images, I wouldn't want to do that. That would be very expensive. We have object detection models that can do that much more efficiently. And that's similar with using an agent framework on an LMM, right? You wouldn't want to run an LMM on 1,000 images to label people if you could have a much smaller, cheaper model to do that. Tools can also be more specialized to tasks. So maybe the LMM doesn't know a lot about medical images, but you have a particular tool that does, so they can be incredibly useful for agent frameworks. And sometimes, you can see tools usage sort of decomposed into 3 steps, where you have a tool retrieval, right? So maybe I have a huge list of tools and I want to retrieve maybe the top 5 and then tool selection and parameter selection. So once I have that top 5, I need to pick a particular tool and then I need to find what parameters I need to pass to that tool for that particular subtask. And then finally, execution, you actually run the tool and it may fail. I may need to retry that. Easy tool is a nice paper on how to use tools for LMM-based agents. And Gorilla is another great paper that talks about sort of scaling up tool usage to really large numbers of tools. And what does that look like in our framework. So these are some of the things that you might see. We might have a list of tools with name, description and usage where usage describes like a scenario like detect the apples and then the prompt that -- or the arguments you would pass the tool for that. So you pass it prompt equals apples. And then I have to choose the tool and then you have a parameter chooser. So okay, now I know I'm going to call Grounding DINO, what is the prompt argument that I need to pass it. And then finally, you execute the tool. Third component here is reflection. I think this is one of the more interesting components or sort of techniques that agent frameworks can use. So oftentimes, you can think of it in an abstract way of having an actor and the actor does something to the outside world, right? So it calls a tool or it executes a Python program. And then from there, if you execute or run the tool, it may succeed or it may fail. And if it fails, you can send that feedback back to a self-reflection module, which can look at what you passed it and the failure message and come up with some maybe new plan to try to do this task again. And that reflection can then passed as context back to the actor to redo the same task so that it doesn't make the same mistake and you could maybe think of this as some sort of memory or some sort of pseudolearning via this sort of context memory. So this is a really, really cool area. And what might that look like in the actual agent framework. Typically, the reflection agent has a prompt like you are an advanced region agent that can improve based on self-reflection and you give something like the user question, the tool that has available to it -- maybe it shows the wrong tool and you wanted to choose something else. What was the tool call and the resolve and what was the final answer it got. Either that worked. And so it's going to go on, or maybe it didn't work and it says, "No, you should use Grounding DINO and you should have used the prompt adult, right? So it will retry the task there. And then finally, kind of combining all these things into one, we have multi-agent collaboration, right? So the figure on the right side there is from data interpreter, which is part of the MetaGPT paper and repository. But you can see you're planning tool recommendation, verification, reflection, kind of all combined into one sort of multi-agent framework. And this is where it gets really interesting. So you can sort of mix and match different workflows and different sort of techniques to accomplish whatever type of task that you want to do. So data interpreters are really cool where they can do sort of data science tasks like train a model and download a data set. And there are some other papers as well such as communicative agents for software development, where you have different agents take on different roles like CEO and CTO where one does planning and one does designing and one does coding and one does testing. And how you put together those workflows can help you better or worse accomplish different types of tasks. And then finally, we have vision agents. So this is kind of the one that we built. And we really tried to build an agent framework that's geared towards solving vision-based tasks. So we have a whole bunch of tools such as object detection, segmentation, visual question and answering, different video tools, all geared towards solving vision-based tasks. And maybe kind of the one interesting thing that we do with our vision agent is we actually use GPT-4V for the self-reflection parts. So you can see on the right there, maybe you use Grounding DINO to detect person who want to send that image to GPT-4V to reflect on it and it will see all those boxes and those scores and it might say something like, "Oh, we'll try using a higher threshold to remove some of the false positives there." And then it can reflect and redo the task to improve it. And something kind of we're looking at in the future is to utilize programming to also do these tasks to give it a little bit more of a general capability there to do more things. There's some really cool papers in this area. Control LLM is maybe more similar to the vision agent that I demoed earlier by using tools and searching over the sort of graph of tools to try to figure out how to accomplish a task. And ViperGPT is a really cool paper where they use Python to try to accomplish these vision-based tasks. Yes. So if you have some time, you can check it on GitHub. We're at landing-AI/vision agent, and we also have a discord where you can come and ask any questions you want or maybe propose anything you want us to fix or do for the next iteration. And with that, I'm going to pass it back to Vasudev here.

All right. Thank you. Thank you, Dillon for that excellent summary, And to me, it was really fascinating to learn about the capabilities like planning, self-reflection and tool usage that these Agentic AI workflows are capable of. So I want to switch gears a little bit and talk about bias detection and mitigation in large multimodal models. So as these multimodal models get deployed into production. It is important to develop techniques that can help probe biases in these models and also be able to demonstrate that these biases can be mitigated. So another paper from our cognitive AI team at Intel Labs at next month CVPR, we demonstrate a technique to create counterfactual examples for probing intersectional social biases. We use a over-generate and filter approach to produce over 5 million synthetic images that we then filter down to about 170,000 images to measure, race, gender and physical trade biases in large multimodal models. We also show how by training the foundational pretrained models on the synthetic data set that we created, we are able to mitigate biases that exist in these pretrained models. So I want to show some examples of a synthetically generated counter factual data set that lets us probe these intersectional social bias attributes. So this figure shows some examples of our synthetic data set. If you focus on one row, you will see that the set of images in the same row have the same aesthetics in the image. People are wearing the same attire. There is the same background in the image, and that is held constant. So the aesthetics of the image has held constant, but we are able to vary the social attribute in these images. And these social attributes that we vary are perceived gender, perceived race or physical attributes. So for example, in the first row, it's all images of construction workers wearing the same attire, same background, but we are wearing the gender and the perceived race of the person. Same with the second example in the second row, where all of these images depict doctors with the same -- similar entire, similar background, but we are changing the perceived race or the perceived gender. So this kind of synthetic data set where we control the aesthetic quality of the image, but we vary a social attribute, gives us a very controlled way of probing the model's response to gender and race separately, but then also start studying the intersectional behavior of these models for a combination of race and gender, combination of race and physical attribute and so on. So here's a short overview of how we generate this data. We've broken it down into these 3 steps on this slide. So if you look at the first step. So the first step is we construct captions that will be used to generate the synthetic images that we are interested in. So in this work, we focus on images that depict people with various occupations and social attributes, especially we consider 3 types of social attributes. The gender, the perceived race and the physical attribute. So for example, we could create this caption using templates like photo of an Asian female doctor. In step 2, after we have generated the captions, we use the fusion models to produce images for each caption. We also employ a technique, that's called Cross Attention Control, so that within a set of counterfactual images, we are able to aesthetically maintain the same image aesthetics. What Cost Attention Control does is that when a diffusion model creates an image, it's able to store the Cross Attention Activations and use it in a subsequent generation by the model. And this preserves a lot of the aesthetic quality in the image. But through a language prompts, we are able to change the social attribute that we want to modify in the image. So once step 2 is done, we employ this overgenerate and filtering approach, where in step 2, we generate a really large data set, about 5 million synthetic images. But then in step 3, we employ 3 stages of filtering. So in the first stage of filtering, we do image text similarity. So we ensure that within a set of counterfactual images, the images are visually similar. And we also make sure that those images are matching the caption. In the second stage of filtering, we use a filter, not safe for work filter, to filter out images that we don't want to keep in our data set. And in the third step of filtering, we ensure that the social attribute that we are interested in is detectable in the image. So once we create this data set, then we can probe the retrieval of -- the retrieval across these images for a prompt, that's neutral -- so a prompt that's neutral to race and gender. So here, I show a case study about the retrieval proportions for the phrase doctor. As we apply it to the images -- the counterfactual images created for the profession of doctors. So in the left 2 plots, we can see the marginal distribution of how these pretrained vision language models will proportionately retrieve images related to perceived race and perceived gender. And in the right most plot, we show the intersectional study, where if it's a combination of race and gender, then how do these foundational models retrieve the images for a neutral prompt of doctor. So there are some interesting aspects that we find on models. I'll point out a few. So if you focus on the middle plot, you will see for models like Slip and Open Clip, they have -- they'll predominantly retrieve images of male doctors. But then if you look at the race, gender distribution plot for the same 2 models, Slip and Open Clip, you'll see that Slip tends to retrieve -- it prefers to retrieve Indian male doctors, while Open Clip prefers to retrieve Asian male doctors. If you go back to the middle plot, you can see that for the Clip model, it has a preference for -- it has a slight preference for retrieving female doctors. But if you go to the right plot, you will see that in the intersectional study, there are some groups which are missing in the retrieval for clip. For example, black female doctors. So this shows that there is an interplay of how race and gender together interact in the way these models might have some biases baked into them. And this is why it's important to study these intersectional biases in these models. Now like I mentioned, we initially create a 5 million data set -- a data set of 5 million synthetic images. And for this, we need a fairly large compute power. And so for all the work here, we used Intel core 2-based clusters and the Intel Developer cloud. Typically, each node here has 8 Gaudi 2 cards, so 8 Gaudi 2 AI Accelerators. One 8 of such nodes give us 1 MegaPod, which is actually shown on the image on the left, you might even be able to read the sign on top of the block, that says Gaudi MegaPod. And for the synthetic data set that was created for this work, we used 4 such MegaPods that amount to about 256 MegaPod 2 AI Accelerator cards. Now what's interesting is that once we create the synthetic data set, not only can we use this data set to probe biases in these pretrained models, we also show how we can mitigate biases in these models by training the foundational models on the synthetic data set. So in this chart, you can see pairs of these bars, where the left bar indicates a skew metric of how biased the model was before we fine tune it on our data set, so the pretrained model and the right bar shows how much we are able to -- what is the bias in the model after we train the model on our synthetic data set. So in every possible case, we see a reduction in this bias metric. And overall, we see that about 20% of the bias in these foundational vision language models can be mitigated by training these models on the synthetic data set that we've generated. Now as -- so we really hope that this study inspires further studies of biases and models and how to mitigate these biases and multimodal models. And to aid this, we've open sourced our filter data set on hugging phase at this particular link. And we really hope that this way developers can test their models for biases, even use this data set to try to mitigate biases. And in general, use a methodology to create a synthetic data set that can probe foundational models for particular biases that they might have. So with that, I really hope that the material that we presented here, that Dillon and I presented here, helped gain some further understanding of large multimodal models of AI agents, of how biases can be probed in these multimodal models and how biases can also be reduced in these multimodal models. So with that, we are now at the end of our prepared content, and I would like to invite Ed back here so that we can take some audience questions.

All right. Cool. Yes, great stuff, guys. Yes. So quite a few questions came in. So we have some time to get to them. So let's start with Dillon, and which framework did you use for the vision agent? I'm going to combine a couple. And are we actually giving frame-by-frame video to GPT-4 vision?

Yes, that's a good question. So the framework that we use is our own framework, and you can find it at the landing AI/vision agent at GitHub Repo. It's based off of easy tool HuggingGPT framework but with quite a lot of modifications. And then the next question, I think was, are we just sending the frames to GPT-4V? No. So what it's doing is it's using its sort of repository of tools and then it's picking from those tools to analyze those frames. So in that particular case, it was extracting the frames and then it was calling a tool called grounded-SAM, which is a tool that takes in a text prompt and then tries to segment that text prompt on the image. So if you type pool it will like segment the pool and the image. So that was the tool that it was using there. It could -- if it wanted to call GPT-4V to analyze the tool, but I think that defeats the purpose a little bit because this may -- workflow may be something you want to execute many times over and over again. And GPT-4V would be a pretty expensive tool to use there.

Okay. Cool. Dillon, why don't we stay with you for one more. And so is it prompt based only or is there an option to input the embedding so that the use case specific questions can be asked too. So sort of like a RAG implementation?

Yes. So for the current version, you can only prompt it. So you can do a couple of things. You can give it a text prompt, right, which is what I showed, you can also -- and obviously, you can give it like a target image or video to analyze. You can also give it a visual prompt. So I didn't demo this, but I could give it like another image and a mask or a box on that other image. And I could say, drawn a box around this item. Maybe it's like an industrial screw or something kind of where you're looking. Can you find that item? So it can use visual prompting as well. In the future, we are working on a version where sort of in the last slide there, I talked about maybe memories, where you could give it, let's say, a list of memories of pay for this task to use this tool, and it would actually go in and use a rig-like system to retrieve those memories per task. So you can stay tuned that you should be able to see that maybe in the next couple of weeks.

Okay. Great. Why don't we switch over to Vasudev. So you showed an example of how hallucinations could be detected for the LLaVA models. So are there ways to reduce these hallucinations as well?

Yes, yes. So in fact, this is what we are currently working on in our team. Since we have this method of detecting when the LLaVA model is grounding its output and an input image, we are trying to further strengthen that mechanism. So right now, when these models are trained, you sort of just give the data, the image text data, and you try to do the language modeling task, that's influenced by the image. But there is a concerted effort to make the models response be grounded in that image. So now that we can -- we have this mechanism that we can use for understanding how the model's text output is grounded in that image, we are trying to strengthen that mechanism. So we are trying to modify some aspect of the loss functions that they can be stronger grounding of the output text inside the image. And we feel that by doing that, we would be able to reduce the hallucination. We would be able to get to a point where these models can ground their output better in the input image. And I also think that a lot of what Dillon talked about, the AI agents and that framework could also be used in conjunction with these LLaVA-like models to teach them how to reduce hallucination. So that's also something that we are very interested in.

Okay. Great. And the next one is, so were there any conclusions you drew -- sorry, were there any conclusions you drew about like common situations where the models tended to hallucinate? Or was there anything that you found that was surprising as part of work?

Yes, I think a lot of failures that we see for LLaVA-like models actually come from the vision and coder itself. We've done a lot of failure analysis, and the thing is that the image is still very costly sample by a vision transformer and then send to downstream language models. And this is where I feel that having more tools to understand the image, like having a segmentation tool, having -- or having the capability of a model to sample certain parts of the image more finely could really help. And for this, I think we will need like Agentic AI frameworks, not just one single pass through the model and get a final answer. Yes.

Okay. Great. So Dillon, back to you. So can you tell us more about some real business use cases you're currently working on for vision agents?