International Business Machines Corporation (IBM) Earnings Call Transcript & Summary

Hello, everyone. Welcome to Part 3 of Becoming AI Developer in Four Weeks. We appreciate all of you joining us today. Some of you have attended the first one and second one, and then this is the third one. And this session will be led by Eric Martens today. Before we get started, this webinar will be available on demand and you can receive the link in the next couple of days. You will also find the slide on your console, and please open that and find the first PDF that -- it says today's presentation. And it is also very interactive, so you can use the Q&A panel to answer -- to ask questions at any time. Eric might incorporate this as part of his presentation or we might answer it later on, depending on the volume of questions and applicability of that questions to the presentation. So if you look at the console again, the portion you see is the -- today's presentation. Second one, you see how we recently was named 2020 Gartner Peer Insights Customers' Choice. That is for the Watson Studio and other IBM portfolio. So feel free to look at that. It gives you some insights into how Watson Studio was developed with the distinguished engineer who was the -- who is the lead architect. And you also see a couple of additional webinars that you can either sign up for the future session and also on-demand one as well. And a couple of other things. So we also have some of the early access program and the beta program that's running. You'll find it in the Watson Studio Desktop and Machine Learning 2.0 beta as well as time-series and other AutoAI feature. So feel free to sign up for those, and there's also a variety of assets that you can find. So with that, I wanted to turn it over to Eric to go through his part of the presentation. Eric?

All right. Thank you, Julianna, and thank you to everybody for taking the time today. I apologize in advance. All the delivery trucks have shown up at once, and my dog is not happy about that. So I apologize for any barking you're going to hear in the background. Today is going to be a little bit different. You might be wondering why the last 2 weeks have kind of focused strictly on development. Today is just -- what we're talking about today is going to have some development in it, but it's a little bit of a shift because we are talking about OpenScale, which is a part of Cloud Pak for Data and part of Watson Studio and monitoring our machine learning for different -- monitoring our models for different things like unfair bias or explainability. And it has to do more with trusting your artificial intelligence, trusting your machine learning models to make sure that they're making the right decisions. And the reason we're doing that is everybody wants AI, right? It's the cool thing. It's the buzzword. It sounds great. It's like, "Oh, we're infusing our business with artificial intelligence. We're using machine learning." Certainly, if you've heard IBM talk about machine learning, if you've ever seen this before, you've probably seen this slide. This is a study that was done. 85% see -- of businesses view AI as a strategic opportunity, whether that's automating some processes, reducing costs, providing better value to customers, that kind of thing. Everybody sees the value in AI. But at the same time, a lot of companies are struggling to integrate that AI. And when we -- IBM looked at that, and we said, "Why? If everybody can clearly see the value proposition, the value that AI can bring to your business, why aren't more companies more heavily using it in their day-to-day operations?" And the reason -- one of the main reasons we found for that is that business stakeholders just don't trust it. So a big part of being an AI developer is, obviously, you need to know the skills. You need to know the code. You need to have strong abilities when it comes to the data science. And -- but you also need to make sure that whoever you are working for, whatever you're trying to sell, that they trust the outcomes of the models that you are building, and you can -- you want to make sure that you can demonstrate that your models are making decisions that are based on real-world logical things and not just some weird artifact in your training data. You also want to make sure that your models are making fair and ethical decisions. As you start from training data in these models, that training data may contain unfair biases against certain groups, particularly the older the training data is, it may be causing your models to be unfairly biased because your models are just a collection, right, of that training data. And the operations you perform are -- so if you were starting from biased training data, you're going to end up with an unfairly biased model. So that's kind of where there's a lot of different pieces of the Watson Studio product, OpenScale, comes into monitoring those for -- monitoring your models for bias, being able to explain the decisions those models are making so that you can really trust your machine learning. There are some open-source projects that IBM has contributed to. I highly encourage you to go and check these out. The Adversarial Robustness 360 Toolkit is not something we're going to talk about today, but it's talking about people who are attempting to attack machine learning models and to get it [ damaged ], force them to generate their own kind of outcomes. It's a fascinating field. It's something that I would really like to learn more about. So that's the first one on this list. Then we've got the Fairness 360 Toolkit and the AI Explainability 360 Toolkit. These are both open-source, free toolkits. They're out there on GitHub. They've got code. They've got data sets. They've got demos. I'd highly encourage anybody to go out and take a look at some of that, try and get some of that sample code up and running to kind of understand what's going on with fairness and explainability. The other part, obviously, that you can use to do that is OpenScale, which is a part of Watson Studio, part of Cloud Pak for Data, part of IBM's machine learning offer. So these toolkits are going to be more geared around finding these issues with fairness, with explainability as you're building your models, and then OpenScale's monitoring capabilities will be once you deployed your model and you're continuing to monitor it. All right. So while OpenScale is, again, part of Watson Studio and it is doing monitoring for -- making sure that your models are compliant, if there's a lot of fields, they actually have regulations around machine learning to make sure that they are maintaining certain standards of bias. People have the ability to request explanations of why machine learning models, if I applied for a loan and the machine learning model turns it down, then I have the right to go out and say, "Okay, I need you to explain to me why your machine learning model made that decision that my loan was risky and should -- my application was risky and should extend it." So OpenScale can provide that kind of information for you. And yes, like a lot of aspects of Watson Studio, OpenScale can monitor those models kind of wherever they are, whether they're on Microsoft Azure or Amazon SageMaker or even custom model service engines. There's ways for OpenScale to monitor that. Obviously, we feel like Watson Studio is the best platform for machine learning, and then you guys -- everybody should be using that, but we understand that there are other companies out there. Okay. So the rest of the materials, I'm not going to cover in depth on this. I kind of want to run through the exercise a little bit and set up the model for monitoring. So I'm just going to very briefly cover OpenScale as it's doing bias detection. When we talk about bias and machine learning, it's very easy -- it's easiest to talk about that in terms of societal bias, so things like gender, like ethnicity, like age, all those things that you think of when you hear unfair bias. But from a data science perspective, bias can be any feature of your model, any input to the model, we could have unfair bias against. So for example, if our -- if we run a mortgage company, and we've just started offering online mortgage loans and we are starting to shift more to an online loan application, particularly with the pandemic and less face-to-face, we're trying to shift our operations more to online applications, online reviews, that kind of thing. If that's newer and we don't have that as well represented in our training data, then that might be some source of unfair bias against people who are applying for those mortgages online. So OpenScale is going to help us detect those unfair biases, whether biases that are not grounded in necessarily reality but maybe artifacts of the training data or just outcomes of the model that we don't anticipate. So again, there's a lot of information out there about how we are identifying and mitigating these fairness issues. I'm not going to go too in depth in that today just for time, but there's a lot of information out there for you to do that. The second factor that we want to talk about is explainability. Why is your model making the decisions that it's making? What factors? What about my loan application caused the AI model to reject it? And we -- OpenScale is using a couple of different open-source algorithms. One's called the LIME algorithm. The other is an IBM-developed technology where we've open sourced a good chunk of it called contrastive explanations. So we are not only able to explain why the model made a prediction that it did, but we're able to say, "Okay, what do we need to change about this prediction to get a different one?" So that's the feature of OpenScale that we're going to talk about. And so anybody familiar with the LIME prediction knows that we're doing data perturbation, that we're going to make thousands and thousands of little tweaks to the data. So as a 41-year-old male applying for a loan, maybe I'm applying for a $10,000 loan that I'm going to repay in 36 months. What the algorithm will do in order to provide that explanation is it will say, "Okay, well, a 41-year-old male, what if you were a 50-year-old male? What if you were a 41-year-old female? What if your loan application is for -- instead of $10,000, it was for $15,000? What if the term was shorter?" It makes thousands and thousands of perturbations of that data, sends it to the model and sees how those changes affect the model's prediction and confidence in that prediction. And then it aggregates all that information together. And we can tell why the model is making the decision that it is. We get a future importance. And it's important to note that we can do that not just for decision trees and random forest classifiers and some of the more simple models that we've got out there, but also neural networks modeling, very complex models with hundreds and hundreds of features, that same method, nesting algorithm will work regardless of what our model looks like. So there's a little more information about LIME and contrastive explanations. And then the last part that we want to talk about, when we're talking about monitoring our models and making sure that they're making the right -- making good decisions, it's dealing with drift. And drift is what happens to your model when the data that's coming into your model starts changing. I'm sure that you guys can think of a lot of scenarios right off the top of your head, caused by recent events, that might cause some drift in some models. Let's say, I have a telecom company and/or a mobile phone company and we've got a machine learning model that's trying to predict whether customers are going to continue using our service or cancel our services. And one of the factors in that model is usage rate and how long it takes to get somebody on the phone when you have a customer service issue. Well, with the pandemic that we've had recently and everybody working from home, I'm sure everybody's usage rate has skyrocketed. And then my help desk staff is probably working from home. Maybe he doesn't have -- know how to have as much help as a staff. So the wait times for people who are calling customer service, those have increased as well. So what we've got is drift. We've got data points coming in. There are people sending predictions to our model with data that is outside of what our training data was, and that can cause drops in model accuracy as the model attempts to deal with that. And so OpenScale is going to attempt to monitor and diagnose some of those issues in real-time as they're happening. So it's a very valuable piece of information and something that's good to think about as an AI developer. It's good to know how your model deals with drift, how it deals with changing conditions and how that can affect you. I can see your -- drift detection in OpenScale is excellent, good source of information. Okay. With that, I don't see -- flip over to Q&A. Looks like I don't see anything coming up in Q&A yet. So what I'm going to do now is I'm going to go ahead and share my screen and just kind of -- I put the exercise out on GitHub with some step-by-step instructions. I'm just going to kind of run through that notebook, very quickly talking about at a high level what the notebook is doing, and then we'll go through actually the OpenScale configuration and set that up and talk about why we're doing what we're doing with the model. So let me go ahead and share my screen. Okay. So this is the notebook. And hopefully, everybody got a chance to run this. I did my best to comment it and make it as easy as possible to run. The only information that we're entering is the service credentials, which, if you've been through the earlier parts of the course, you guys are -- you folks are excellent in using service credentials in notebooks right now because you got to do it several times. But -- so what this notebook is doing is for OpenScale to monitor something, we need a model, right? We need to have something for us to monitor. And so this notebook is building that model and then deploying it as a web service, kind of similar to what we did with the AutoAI and building that model in Jacques' sessions last week. So it's a fairly straightforward notebook to run. I did catch an error at the very bottom where I had to sweep in there with that important time variable. Hopefully, people were able to get past that and run that notebook. But -- so that's kind of what we're doing in the notebook. And we're using the Watson Studio and the Watson machine learning libraries with the Python client. Otherwise, this is just a very standard Jupyter Notebook that you can run on any Jupyter platform to build this model. The only parts that are specific to Watson Studio are where we're stating it and then deploying it in our environment. So we're using pandas, and this is just to download the data. And the use case is that one I mentioned earlier, where we are a mortgage company and we are trying to determine if mortgage applications are likely to default or not. Should we extend this person a mortgage loan, somebody who's trying to buy a house? And so we've got some demographic information, how long they have been at their current address. We do some exploration of that data here. And then this next section, we're just going through and building our model. And this is all fairly straightforward. We're using scikit-learn, and we're building a random forest classifier. For the people who did last week's, you know that you can use AutoAI to do a lot of this stuff. OpenScale doesn't currently support monitoring AutoAI models in cloud. That feature is coming very soon. So you can't build an AutoAI model to run it on OpenScale yet, although you will be able to, I would say, within the next month or so, but just know that, that's there. What I did do is I used AutoAI to kind of rapidly prototype my model and say, "Okay, what should I use, random forest classifier and then any future transformations that are in there?" I used it to get a good idea of what my model should maybe look like, and then I went ahead and build that in code, which is this one step here. So we're just doing our train/test split. And we've very quickly built our model. We've got a model score of 72%, which is a great precision score. And then we're going to go ahead and save this model to our Watson Machine Learning service that is here, and then the model is deployed as a web service. So once it's deployed as a web service, I can actually query it using [ the rest of ] demands. I can go find it in the Watson Studio and send request to it via JSON or filling out a form, pretty similar to what we did in the last week with our AutoAI model that we built. Okay. So then what I want to do is now that I have this model that is out and deployed and in production and ready for use is I want to set OpenScale up to monitor. And the way OpenScale does that is we are -- it's the concept of a subscription. We're subscribing to the model. We want to get all of the data that's going into the model and then all the data that's coming out of the model, and we want to save those 2 things. And we're saving that in a data mart. And that's what this code is doing here, is it's using market credentials to subscribe to the model, and it's saying, "Okay, we want to monitor this model. We need all the information that's coming in and all the information that's going out so that we can then take measurements and do our metrics reporting on this model." So we have this concept of a data mart, which is just a database with a bunch of tables in it where we're saving information about our model. And then we are going to do [ bindings ] so that we can talk about the machine learning that we are going to subscribe to the model, which is right here. And then we are providing OpenScale with our training data. So this piece right here is so OpenScale can give some statistics about training data for drift, like we talked about earlier, if -- we need that training data in order to compare the data that's coming in with the training data so that we know if we're seeing any model drift. So this is just a lot of -- this code looks very technical, but you can actually do everything I'm doing here with the OpenScale UI. It's just -- for simplicity's sake, I had it all -- I did it all in the notebook, so it just made it easier. If you really want to dig down into it, you can read the OpenScale documentation and you can take a look at it. All that it is really doing is telling OpenScale, "Hey, I have a machine learning provider that has lots of machine learning that contains a model, and I would like to subscribe to that model." And that's all it's really doing. And once we've done that, we will sign on. And now let's go take a look at this in the OpenScale dashboard. And again, this is part of Watson Studio. It's part of Cloud Pak for Data. If you have that, you can get it to OpenScale. And OpenScale is kind of the second piece of IBM's attempt to -- I guess, citizen data scientist is how I'd further describe it, where not necessarily experts in data science but people -- smart people who are interested in the field, maybe business users, maybe experts in other subject areas and are -- want to get started with machine learning and -- or understand some of the metrics of machine learning, we want to give them the tools to understand those things. And we want to automate some of the tasks that our extra data scientists are doing. So that's where AutoAI comes in with the rapid prototyping building the models. So OpenScale comes in where we're trying to show these metrics around unfair bias, around explainability, and show them in a way that you don't need to be a PhD data scientist to understand it. So I'm going to switch my screen here so that we can see OpenScale. You see I've got -- this is the -- if you provisioned OpenScale and you go out to it, it asks you, "Hey, do you want me to run auto setup?" And if you say, "Yes, run auto setup," then it puts the German Credit Risk model in here for you with a bunch of metrics that it's been in. You guys are welcome to go out and take a look at that. It's got a lot of data in here around, I don't know, a bias for a bank. But this is the one that we built in our notebook right here. So -- excuse me. I can click into this one, and I'm just going to run through configuring it for the monitoring that we want it to do. So we're going to configure the monitor. So we have this model and we have our subscription. OpenScale is monitoring the model, but now we want to set up the monitors so that we can set some parameters around what we are monitoring. Okay. And this should already be all done so that I can go over here to my fairness monitor. And this is just me stepping through the screen shots. All of this stuff is in the instructions in the GitHub repo. So if you fall behind or if you're trying to follow along, know that I've got instructions, screen shots, it's all in my GitHub repo. You can run through this on your own time. I'm just going to kind of talk about why we're doing what we're doing in the screens. So this OpenScale is going to tell us if our model is going to be unfairly biased. And we want to monitor for people who are applying online for that same scenario I talked about earlier, who maybe didn't have a lot of online applications in the past or if we did, we didn't accept them at a high rate because they usually came from people who were just kind of applying for mortgages and weren't -- like they weren't serious homebuyers, for whatever reason it may be. And now as we try and shift our business more to an online model, we want to make sure that our model is not unfairly biased against people who are applying for these mortgages online. So we need to specify, hey, what purpose [indiscernible] the outcome, so that OpenScale can tell us if it's good or bad. In this case, open -- the model is trying to predict, hey, is this person going to default on that loan? So we want to say -- if we predict they'll default, then that's an unfavorable score. So yes is favorable, no is -- or sorry, no is favorable and yes is unfavorable. So we're just specifying that here. Sample size is how many records do we want OpenScale to take into account each measurement. 1,000 is a good minimum sample size. It just kind of depends on the traffic of our model. However, I'm going to set this sample to 100, just so that we see the results quicker. Note that you'll -- know that as you're seeing results come into your model, you're going to get a much -- less like a wave and more like a heartbeat almost, as you're going to get a lot of higher variance. Okay. And now we want to select the fields that we can monitor. OpenScale can actually try and recommend some of these. If it sees that some of them are related to societal stuff like gender or age or those kinds of things, we don't have any of those features in this model. So what we want to do is we want to stay [ online ] and monitor for the people who apply online. So I'm staying to monitor that feature to make sure that we're not unfairly biased against it. And now we are specifying the monitored groups. So in this case, it's a yes or no question. But if it was an age, you can imagine it would be a range. You could say, "I want to monitor 18 to 25 year olds and make sure we're not discriminating against them. And I also want to make sure we're not unfairly biased against people 65 and over." So this dialogue here would ask me to provide a range that I can monitor. In this case, we want to monitor the people who did apply online and we want to compare them to our reference group, the people who applied in person, face-to-face. And then it's going to ask us to set a fairness alert threshold. So you may remember back on that tree where we looked at earlier with the dashboard that was showing alerts for the German Credit model. And this is if the fairness score drops below this rate, that's when we want to see the alert. And this could be a business best practice, it could be a business goal or it could be provided for you by a regulatory body. I think there are regulations around AI and what the minimum amount of bias they have to maintain. 80% is a good number for that. So I'm going to leave it at 80%. But just know that this could be set for you by a regulatory body or it could be something that your business users set a score they want to achieve. So that's the threshold, 80% score is what we're going to set. And if it drops below that, then we're going to see an alert. Okay, and we are done with setting up fairness monitoring. As you can see, everything I did just now can be done via code in that Python notebook. There's also a REST API you can do it. But again, much like with AutoAI, we're trying to provide an easy way for people who are maybe not expert coders, not great at Python notebooks, to come in and specify the stuff and see how it works using a basic…

Hello, Eric, there's a question if this particular notebook is available?

Which, the notebook that I ran through?

Yes.

Yes. It's on the -- if you take a look at the GitHub repository that stores all of the information for all of this, it's been -- my part is part 3. And there's a notebooks folder that's got 2 notebooks in it. This is one of those 2 notebooks. And if you follow the instructions that are in the README for that part 3, it's a step-by-step that will take you through everything you need to do to fill out and run that notebook.