Customer Stories: AI in Financial Services (Cloud Next '19)

[MUSIC PLAYING] AAKASH BORDIA: All right welcome to day three of Next The number three is really important today because you will see a lot of threes in our session here So it reminds me of an old joke or old saying, there are three kinds of people, one who can count and one who cannot So it takes a little time to hit you, but we’ll get your gray matter going for this talk So I was talking to some of you before the start of this talk, and seems like there’s three types of us here in the room, one who had a late, late night yesterday and had a great time at the concert, please raise your hand if you’re one of those All right, awesome Thank you for making the effort to get off your bed and get in here, filling the room And then there’s the second kind who were disciplined and just went to bed on time or about time If you’re one of them, raise their hands All right, thank you for your discipline and being here And then there’s the third kind who are not sure if they slept or when they slept If you are that kind, if you are that brave heart, please raise your hand Oh there’s two or three, I see four or five, thank you Thank you for whatever you did to get here We really appreciate it So welcome to the first session of the last day at Next AI and financial services My name is Aakash Bordia, I’m on the customary engineering team with Google Cloud If you are a prospect, or a new customer, or an existing customer of ours, you are likely working with someone on my team or a partner of ours So our session here is a story session, which means we have two customers and a partner that will come up here in approximately 15 minutes sections and share their journey to cloud AI Which means they’ll both talk about their problems, their use cases, and their solutions So in these three stories, you’ll hear our partners and customers talk about how they are leveraging the cloud AI stack, be it the platform which covers our development environment and our robust infrastructure or be it building blocks, which could be our predefined models or the AutoML models that you can train Or we could also talk about services and solutions, which is very important from a strategy and culture perspective at a financial institution So this should excite all you builders and all you leaders in the audience because we are going to go top down the stack Since we are likely going to be running out of time towards the end, I encourage you all to use the Dory through your mobile phone and post your questions If we have time towards the end we’ll do live Q&A in the room Otherwise the group of speakers will huddle outside the room and you can come and ask us questions and discuss Awesome Having said that, let’s dive right into it We’ll start with some fantastic motivation by David Furlong from the National Bank of Canada on how to set up your AI program So this is a more business track of this session where Dave talks about how do you influence culture to execute on a successful AI program in a large organization such as yours We’ll also take it a notch deeper with Daan Gonning of RaboBank and Lee Boonstra who is my colleague on the customer engineering team who come up and talk about customer excellence using our machine learning APIs And finally, Jonathan Jeng and Druva Reddy from our partner from our premier partner Pluto7 will come up and talk about how they have worked with our PSO professional services team to work on solving some of their tough for risk management problems that financial institutions face So let’s start at the very top of the stack In fact it’s a level higher than shown here because it encompasses the whole stack Very important to a successful cloud AI program is your strategy and the culture in your organization They are both very critical to setting up a successful AI program and execution in your organization And so to talk about this I would want to welcome Dave Furlong, SVP for AI and innovation at National Bank of Canada up on the stage Dave [APPLAUSE] DAVID FURLONG: When I was younger I could jump, but that’s behind me now, I think Yeah, I can try I’ve been asked to add a few things to the presentation this morning to give you a little bit of context about National Bank There are six large banks in Canada We are one of them We’re 160 years old, we manage about $500 billion in assets, and have about $300 billion on the book

We’ve been involved in AI in some capacity for quite a while Now, we didn’t know it when we were doing it, but we were We started deploying particle filters in 2007 to replicate hedge fund performance, and then we moved on into algo trading that was strict algos, and then we actually started to apply AI to the algos to improve them, all with supervised learning and supervision by a human So we’ve been at this game for a long time, but that’s the capital markets arm of the business I won’t talk about that I’ll talk to our guys about what we’re doing in retail and commercial, which tends to be a slower pace organization in any bank than the capital market side, and it presents different challenges And you’ll see when my colleagues come up they’ll talk a bit about some of the challenges they’ve faced, and it’s not just technical So here we go So what does it take to bring a 160-year-old company into AI? How many people in the room work for an old line company– company older than 20 years? Perfect I have friends So I’ll talk a little bit about our strategy and how we go after things, but the really important part about what we’re going to talk about is the lessons learned, or as I call it, two years of self introspection or self discovery So from a strategy perspective, it’s pretty straightforward I can characterize it very simply One, we want to improve revenue opportunities for us, for our company Two, we want to make our experience for our clients and for our staff better because that leads to retention on both cases, really good in our organizations And third, we want to increase our operational efficiency or remove costs from the ecosystem So for the next six months, here’s our top tier priorities in the retail commercial side One, build out a production ecosystem We already have a smaller production ecosystem, what we need is something that is infinitely scalable and infinitely expandable, allows us to use open source, allows us to use platform as a service where appropriate, and to deploy containerized solutions Second thing we need to do is enhance our decision models If you think about banking this way, you make a decision to market to someone, then you check for fraud, then you underwrite them, then you manage them over a lifecycle, and god forbid they go into collections, but you manage that as well So those decision models can be adjusted and synchronized using AI leading to a much better outcome The next one we want to undertake is in operations– to apply AI, natural language processing, intelligent routing to how we manage all documentation and customer interaction from a servicing perspective And the final one, and this is really important, sometimes chat bots or dialog engines are referred to pejoratively I can promise you they’re not because what happens is in an organization where your front line staff has turnover you need to codify the knowledge and put it in front of them and the client Because you’re empowering both your staff and the client So the better your dialogue ecosystem is the better the experience is for both So lessons learned, or as I call it, David’s mistakes Two years ago we had put in place an AI team and treated it as if it were advanced analytics How many people in the room have an advanced analytics capability and are just building AI now? OK, so don’t do what I did And what I did was I went out and hired a bunch of AI scientists, put them together, and said, now AI stuff That is not how it works In advanced analytics ecosystem, what happens is someone comes in with a business request, they model it, and then they hand the output of the model to somebody else who creates business rules, or procedure, et cetera AI embodies all of that So as a result of that initial decision, of course, what we didn’t do is we didn’t fund end to end software development AI is software It is what it is We also then as a consequence of that didn’t hire data engineers, software engineers, people who are familiar with the AI stack And of course, then the other thing we did was we continued to focus on building data repositories So we kept building data warehouses instead of focusing on data inflow or data instream Because if you really want to be effective in the next generation of banking you have to catch the stream and change the stream with AI And then finally because of all of those things, we anticipated at first that our technology stack could be internal Turns out that that wasn’t efficient either because we need infinite scalability up and down as required We need open source We need different security models and security paradigms than we’ve used historically So about 12 months ago we stepped back and reconfigured things, and it worked pretty well

So what we did from an operating model perspective is we took a business owner with a business problem, gave them an AI scientist, gave them a software engineer, data engineer, verification engineer, and said, solve a business problem So they started going end to end because for those of you who’ve worked near telecom, telecom has always had what’s called the last model problem So does AI The second thing we did was we started to fund end to end software builds So we changed our funding model And of course, with that you need to change your talent profiles So you need to hire cloud engineers, full stack developers, software engineers who are familiar with AI, people who know how to do monitoring of live AI models because they change And then data, we began the process of changing so we could intercept data flows I have a very good colleague here in the room who runs our fraud ecosystem And he’ll look at me often and say, David, if I catch the fraud after, the money’s already left Catch it in stream, deny the transaction And then finally, if you want to do that I’ll go back to you need an open, scalable, robust infrastructure And one of the things that we found with Google as a partner is they can provide it all So we use GKE to containerize solutions for us and put them out there so we can buy a third party specialized solution, deploy it Or we use GCP for platform as a service type behaviors– AutoML, et cetera And the breadth of the solution set is what’s allowing us to attain velocity, scalability as we need it So with that, I’ll conclude because there’s a couple of things I want to leave you with One, AI can absolutely help you accelerate your objectives No ands, ifs, or buts But you do need to change the way you work I went down this path and hopefully if anybody comes out of here today understanding the mistakes I made so you don’t make them, it’s time well spent And then finally, if you’re going to go after this change management is critical Because if you change something that affects staff, clients, et cetera, you need to manage all of the change around that Do not underestimate its importance They’re critical, critical items And with that, I’ll conclude and give time to my colleagues who are going to get up and talk to you about practical examples of everything I just spoke about So let me turn it back over to Aakash [APPLAUSE] AAKASH BORDIA: Thanks, Dave That was motivational, inspirational on how to go about setting up a cloud AI program in the real world in a real financial organization such as yours So now let’s take it a notch deeper Now let’s talk about how could the builders in the house use some building blocks from our AI stack to build some customer engagement improvements using technologies such as Dialogflow? And to do that, I would like to invite Daan Gonning from RaboBank on the stage Please welcome Dan DAAN GONNING: Thanks Heyo, thanks for having me on stage today It’s really awesome to be here and to showcase our Google Assistant project First of all, let’s see how this works Almost How does it work? SPEAKER: The green one DAAN GONNING: Doesn’t work SPEAKER: That big green one DAAN GONNING: Yeah, oh, thanks Little green one, cool First of all, my name Daan Gonning I am freelancing and currently on an assignment at RaboBank where I am leading a program about digital interaction For over one year now we are working together with Google to build our Google Assistant action, or as we call it, the Rabo Assistant Basically what we want to do is bank in which a voice make it just as simple and just as easy as banking with one of our mobile applications But why? RaboBank is doing this– Yeah, let me rephrase In the Netherlands, RaboBank is a pretty traditional bank Definitely no first mover, but still we developed a Google Assistant action, and still we are the only bank in the world that has connected clients account such as balance and transaction information through the Google Assistant environment And we do this for one simple reason, because our customer is changing A couple of years ago banking was pretty simple

We managed the whole flow of data We had a Rabo customer which which has a Rabo bank account, banks in a Rabo mobile banking app, and when in trouble he reached out to the Rabo service desk Like I mentioned, we all had the whole flow of data We call this the happy flow of data You have a great service or a good product which attracts customers Those customers generate data, and with the data you improve the product, and that goes on, and on, and on But now our customer does a lot of interactions with third parties People are tired of using mobile apps, and with open banking combining, there are a lot of combinations of new services and new channels People are switching more and more to third party conversational channels like WhatsApp for Business, Google Assistant, Facebook Messenger And that means the happy flow of data as you see it here is being disturbed We don’t have all the data anymore, we don’t have the full control And that is the reason RaboBank is in this conversational domain We want to learn what it means to be on a third party channel We want to learn if our customer likes banking with voice And how do we make sure when we have no freedom in design and no freedom in user experience, how do we make sure our customer sees the Rabo Google Assistant action is the official one, for example? All kinds of questions we are doing in this program, all to validate and just to experiment with new technology And that in combination with the graph behind me, it says it all We need to be present on the voice domain just to test It could be the next big thing And when it’s not– I doubt it– when it’s not, we just learned a lot That’s for sure So that’s the why, but then the how And that one is pretty easy to say but fairly complex because we are a bank with a lot of processes, a lot of rules, a lot of everything– just a bank And a couple of those lessons I want to share with you today, and the first is focus When we started working together with Google one year ago we created a mission statement, a get to buy The Google Assistant action with that action we wanted to get the digital set of users Those users must see RaboBank bank as an innovative bank, and we want to provide it, we want to accomplish that by proactively serving them signals about the finances that feel seamless and magical Without this kind of focus you will be lost, especially in a corporate but really within a bank So get your focus, get your focus straight Another lesson is in team structure We decided to build speedboats, kind of guerrilla teams Teams who can test the water, move fast, break things, be flexible No RaboBank legacy, no double back loss, just team focused only on one thing Each case in my digital interaction program is such a speedboat And when successful we pull them back into the big oil tanker called RaboBank When not successful or not validated, the speedboat sinks and people are going back to the– yeah, normal work life, or other speedboats We will show before the demo one of the [INAUDIBLE] execution When we started this speedboat, we had some traditional roles Think about the product owner, a developer, some other stuff But also we keep adding roles, and we ended up with this bunch of people for one Google Assistant action But when you look closer at those goals, you still see the more traditional ones like a product owner, like a developer But you also see data science, marketing, and some special ones like conversation design, compliance, legal risk, and a chief pushing officer For example, we edit a conversation design because writing for voice really is different when you compare it to writing a website text, for example Or even when you write a conversation for your text chat bot Voice is a whole new way of having your conversation, so it needed some special attention And of course, compliance, legal and risk Again, we are a bank And to be honest, it was the most painful part It really was a pain to get approved internally Discussions– do we want to be on a third party channel? What is Google doing with our data? All kinds of discussions in the whole bank from– to the top, basically and really to the top, it is terrible But once again, Google helped us Google said, no, we are not doing anything with your data

You are under an Enterprise agreement, it’s all safe And that is the path that let us all came in, the chief pushing officer This is a kind of sponsor, and it’s the latest advice I want to give you When you such an innovation, get a sponsor internally So in the top of your organization who says, OK, I believe in this project, I believe in what you are validating, so I will support you So when we are stuck with compliance legal and risk, the chief pushing officer says OK, now it’s clear, Google said how it’s done, so move on We push it, we are doing this I think without a chief pushing officer, we still not live at the moment So it was a really crucial role, a critical role inside this project So a minute left for a demo, so I will showcase it live without demo effects, I hope Let’s switch the screen There it is, cool Talk to RaboBank AUTOMATED VOICE: All right DAAN GONNING: What’s my balance? AUTOMATED VOICE: Your current balance is 2,177 Euro and 97 cent How can I help you? DAAN GONNING: I want to set a budget AUTOMATED VOICE: The Rabo Assistant can help you setting up a budget I can then tell you how much you spent and send you notifications when reaching certain limits Do you want to set up a budget? DAAN GONNING: Yes AUTOMATED VOICE: First determine the amount and then the period What should your budget be? DAAN GONNING: 200 Euros AUTOMATED VOICE: OK, until which date is this budget? DAAN GONNING: Next Friday AUTOMATED VOICE: Do you want to receive notifications to inform you about your budget status? DAAN GONNING: Yes AUTOMATED VOICE: OK, thank you I will notify you when you reach 100 Euro, 160 Euro, and 200 Euro You can always ask for a budget update See you next time How can I help you? DAAN GONNING: Cool, thank you Lee, please join me for [INAUDIBLE] demo [APPLAUSE] LEE BOONSTRA: Thank you, Daan So hi everybody, My name is Lee Boonstra and I work as a customer engineer for Google I’m mostly working with the banks and insurance companies in the Netherlands, and chat bots, it’s a topic that I get very excited about, but it’s also a topic that a lot of companies are asking me questions about Nowadays I think we shifted Like in a past where companies were building lots of mobile apps, nowadays the companies are more spending their time on building chat bots And one of those reasons why they are doing that is because you can trim a lot of business costs by working on chat bots Like what you see here and his research from Juniper, you can trim over $8 billion That’s a crazy amount of money, right? And now you probably are asking like wow, how can we do this? Well, let’s first take a step back and look at chat bots What is a chat bot? I typically would say there are three types of chat bots You have seen the example of RaboBank with their voice assistant, and Google Assistant, or a Google Home, on a Google hub, those are typically like the voice activated speakers You have that also on Alexa, or on the Apple That’s the voice activated speaker The other example that’s more like that the regular chat bots that you know Like for example, on the website often people see that as an iframe, but you also have them in social media And then the last example, that’s more like the call bots These are integrated in your IVR system or on your phone line Not often people think about this, but this is a very effective way in how you can trim business costs, and I will explain a little bit more about that today I will go through a couple of architectures to show you like, well, this is how you could implement something like that So if we first look into the Google Assistant, typically your users, they talk to the Google Assistant or the Google

Home, and the Google Assistant is connected to Dialogflow Now, Dialogflow, that is the tool that does intent matching So basically you create in Dialogflow, it’s a conversational tool You create your conversations, you have your content people, or your UX designers writing conversations in this tool, and the intent matching is based on natural language understanding So machine learning, AI We understand what has been said, so it doesn’t matter if your users are saying it a different way, or if they spell their questions wrong, we can match their question to a scripted answer in the system Or it can be an answer through an API, but we do this intent matching So that’s Dialogflow And Dialogflow that has automatic integration with the Google systems, so Daan, you start your whole conversation on a voice activated speaker like that Now for a website, that’s typically like your users They write on the website in the chat bot, and again, you can use Dialogflow to do the intent matching Now it’s not voice, but now it’s just text Works the same You can enhance it a little It might not see this a lot that companies are doing, then as soon as the chat escalates we can detect that, for example, based on sent demands, or we can detect that on based on fallback scenarios of the chat bot Then we switch over to a human, and that person takes over the chat He sees the whole transcription and continues the conversation that way But how about a contact center? I mean, like most people probably have a traditional contact center like this where you as a user or your customers, they call through your phone number and the next thing that happens is you’re put on hold You’re in a waiting line, and it takes a while You hear a tape, a recording, and then it starts playing It tells you like, oh, press one for X, press two for B And if you’re very unlucky you have to wait all the way to number nine, and then you press nine Again you have to wait, then somebody picks up the phone, you phrase your questions That person doesn’t know how to answer you, so she forwards you again You have to again rephrase your whole question, and then hopefully you get the right answer What we see a lot at the insurance companies at the end of the year when they’re like, everybody’s changing their insurance policies Then it’s hot within the contact center So what these companies are doing is they’re hiring lots of students to pick up the phones And they’re obviously not as much trained as people that are normally in a contact center, so that way the quality of the call is also not always great So this experience is very painful, and it doesn’t need to be that way because you can create a modern contact center as well And that is exactly with the same tools as I’m explaining first as for the website or for the Google systems, it’s through Dialogflow And in Google we have Contact Center AI which can do as within your IVR system with more components, or you can build it yourself with Dialogflow Because the idea is you call, and at that moment you’re not you’re not put in a waiting line No, the chat bot picks up the phone It answers you, how can I help you? You ask your question Either the bot knows the answer, so you hear a computer voice back like a wave net model– so text to speech– that tells you to the question Like for example, like an insurance company, there are always a lot of questions that are always very common, so let the bot just answer that question And if it’s a more difficult question that the bot doesn’t know, then we can route it to the call center agent– so the human But we can help that call center agent by showing the suggestions on the screen, by showing the full transcription that the person had first with the chat bot And while the person is talking to the human, you see live suggestions on the screen So no longer you’re put on hold, and you always can guarantee that you give the same quality because that person just reads off the screen So these are very great ways Wouldn’t it be nice if you just built one AI solution– one solution that can answer all these questions anywhere? And it is available from everywhere Because what you don’t want is build all these separate solutions one by one I mean, you should stop thinking of Google Assistant as a destination or a website chat as a destination, you start seeing it as a channel So that way it means like the Google Assistant that is a channel, Alexa is a channel, social media, Facebook Messenger, that’s the channel, your website

chat, that’s a channel, and also your contact center, your IVR system, that’s a channel And it’s powered by Dialogflow where we do the intent matching, understanding what’s being said, and then it’s based on this AI platform that you created to route it to the right channel that gives you the answer So either the answer will be on the Google Assistant, or it will be over phone by the live agent that’s reached out to the suggestions or as text in a website And then the very nice advantage of this all is then once you’re building this AI platform solution anyway and you can also power it by analytics So you can really see what are the questions that people are asking, what is my customer thinking about my brand, and how can I improve it over time? So you continue the cycle But even if you just add one new AI channel, I think you can already improve your customer experience and trim a lot of business costs So with that, I would like to thank you [APPLAUSE] So next I’m going to hand over to Jonathan and Druva from Pluto7, and they can talk about the cloud AI platform Thank you DRUVA REDDY TIRUVURU: Thank you so much, Lee So I’m Druva Reddy Tiruvuru, a cloud data engineer at Pluto7 We are a Google Cloud premier partner, and we are one of the key partners for artificial intelligence and machine learning implementation across the verticals such as our high tech manufacturing, and retail, and health care We serve the global market We are based out of Barea, Los Angeles, and India Talking about the use case, today some of the financial institutions are facing problems to view The scalable predictive view of the loan delinquency and then likelihood of the customers being default. And then being part of the Google Cloud professional services organization, we built an accelerated solution which demonstrated how we leverage the capabilities of machine learning model to predict the loan delinquency status of a customer Briefly talking about the data set, we used a low level credit information performance of a customer that Freddie Mac has made it publicly available This data set had around 26.6 million records originated between the years 1999 to 2017 So that [INAUDIBLE] was pretty big Some of the specific credit performance information in their data set includes voluntary repayments and then loans that were foreclosure alternatives, and a few other information such as the expenses, net sales, proceeds, and mortgage insurance repayments And then this is the high level architecture of the solution that we have built The low level data set of the credit performance information about their customers which is made publicly available by the Freddie Mac is stored into the Google Cloud Storage Google Cloud Storage is an online RESTful file storage system which is used for storing and accessing your data on Google Cloud platform’s infrastructure Followed by that, we did our initial data exploration and analysis using BigQuery With simple SQL queries, we were able to scan through millions of records within seconds And then followed by that, we use data prep and data flow Data prep is an intelligent data service tool which helps you to visually cleanse, transform, and explore your data And it also helps us to wrangle the data and prepare the data for machine learning Followed by that, we use Datalab Datalab was built on a Jupyter Notebook environment that helps to bring its equivalent to Jupyter Notebook And the Cloud Source repositories, which is used to push our code and do the code versioning And followed by that, coming to the most important part of the high level architecture which is Google Cloud Machine Learning Engine, that is where we hosted our different approaches and different machine learning models, the way we tried out, which got

deployed and then made predictions all of them ML Engine And Cloud Functions, I’ll briefly explain what it does Cloud Functions, it helps us to connect the different services It gets triggered whenever an event occurs that it’s listening to My colleague will explain or talk about that more in the later part of the session Data preparation So data preparation is very critical in the journey of machine learning So over here, after initial data exploration and analysis we figured out that some of the features had very skewed data Like for example, delinquency types, some of the values occurred more often whereas others were very rare So this will degrade the ability of our model to train if we feed this raw data asset So in order to get out of this, we followed a strategy called binning, which helps us to convert the continuous numeric values into discrete categories And over here, each bin represents the degree of intensity of each class Over here as you can see, the class 0 represented safe, and the class 1 it’s moderately safe, second is moderately risky, and third one is risky We use this as a label for prediction Followed by that, we also did a feature engineering The raw data had almost 40 features, but we did feature engineering by creating an aggregated column such as min, max, standard deviation, average, and so on of each and every column, and then followed by– we feature engineered a column which finds out the frequency of each bin that we have created and likewise recency as well, how recent each and every class occurred And then the final data set we used on different machine learning approaches that we tried like vendor estimators, TensorFlow estimators, XGBoost, and we were fortunate enough to try AutoML when it was in pre-alpha stages since we are part of Google Cloud Professional Services organization All right, this is not letting me view my own video, so I’ll quickly explain So whatever feature engineering that we have done, it’s on top of data prep Data prep helps us to visually cleanse and wrangle the data, and also perform feature engineering And all the series of steps are stored in form of a recipe Once we run the recipe, under the hood what it does is it fires up a Dataflow job Dataflow job is an ETL tool It builds an Apache pipeline for you You don’t have to code it, and it’s fully managed By fully managed I mean that you have to focus on your data and analysis, and you don’t have to worry about the underlying infrastructure Google will handle it for you On your right as you can see, you can see the number of current workers at this present, and their number target workers, and how Google Cloud Platform will scale according to the load And this graphical user interface will help you to see the status of the job, whether it has completed or it’s still under progress Once the flow is completed, the target results will be stored in the Google Cloud Storage bucket And my colleague Jonathan, he will talk about the rest of our machine learning journey [APPLAUSE] JONATHAN JENG: All right, so as Druva mentioned, we use Google Cloud Functions We set up a function called CML epred you see over there It’s triggered by an object finalized event,

and the bucket’s specified there So we created one called loan delinquency input for N19, next 19 So this Cloud Function, what it’ll do is it will take the data, put it in the bucket from data prep, and then it will send the processed data to ML Engine– Cloud Machine Learning Engine– to form predictions And it will take those results and send them to a CSV file– two different CSV files One for containing about this current– sorry, the current predictions, and one to compare it to the previous models And it will put those files in Google Cloud Storage And so it will take those two things We set it up to send it by email because you could actually send it to your own company’s communications system We did not do that, we just sent it by email So the important thing to note here is it is almost in real time So we were getting it via online predictions because we’re interested in speed But if you are not concerned about the rapidity of that, you can request a batch prediction So you can submit a batch prediction job So one email shows you how the model performs Over here we use a metric called mean per class accuracy, and the lower email shows an interpretation of the predictions So you see there is a loan sequence number label, and then it shows you the prediction interpretation for that loan sequence number And also as I mentioned previously, the data file is stored on GCI’s Google Cloud Storage And using that, you can set it up to form a dashboard via Google Data Studio, or Looker, or Tableau So over here we used Google’s Data Studio On the left, we can see certain information about the data that may be of interest to you, such as how the class is distributed And then in the middle, we have visualizations of the current machine learning model So on the top we have a confusion matrix regarding their predictions that were just made On the bottom we have it compared to other models that we use So over here you see a couple of TensorFlow models, the DNN linear combined classifier, and then we also used several vendor estimators as well as XGBoost And on the right, you may notice that it’s blank You can choose to implement different actions based on the predictions that you made in the previous two columns And over on the right, you can have it show you just the results of those actions or recommendations for what actions you may choose to produce And on the top you see a live refresh button So as I said, the data gets dumped into GCS– cloud storage And then you can hit that Refresh button to give you a dashboard with the most up to date data So here, we’ll be talking a bit about interpretability So there are two prominent approaches to interpreting machine learning models So on the one hand you have intrinsic simplicity So you may choose to form a more intrinsically simple model, such as the smartest linear model or decision trees On the other hand, it’s compatible with more complex

models such as neural networks, so what are typically considered black box models And over there, you may want to– it’s called post hocs, and you may want to form global explanations of the model itself And over there we have the most famous approaches would be feature importances and black box beta So black box estimation through proximity Oh, sorry And then you might want to examine locally interpretations So you might want to look at each individual prediction Over there, you can use something called SHAP, just to examine the individual predictions, what led to that prediction– which features and which values were important which helped you determine that class, that prediction So over here we’re going to show the SHAP values So SHAP actually refers to a solution from the cooperative game theory domain So what it does is you can construe a game as a scenario, the game players as the features, and the payouts as the contributions So we used XGBoost ensemble estimator, and for the criterion we chose Gini So we wanted to see how much each feature helped lower the Gini contribution And you’ll notice on the x-axis it says the mean chat value for that feature So what it does is it finds the mean– you see the XGBoost model is an ensemble of many trees, and it wants to take the mean SHAP value observed across the entire ensemble So that gives you a high level overview of the model And you may be interested in– before the evaluation, before the prediction, you may be interested in the entire– if I’m going to get this class for loan delinquency prediction, which features are more important? What should I prioritize? And this will give you an overview of that However, once an evaluation has been made you may wish to examine in more detail your individual case So over here we use a module called ELI 5, explain like I’m five So what it does is it will take the different amounts that it lowers the Gini coefficient In your case, you may choose to use a different criteria on, such as entropy, but we chose Gini And over here, what you will see is the contribution of each feature for each case So over here we’re showing two different individuals of class ones, which as Druva showed, we determined to be moderately risky So class one, two, three They’re a low delinquency range They’re not perfectly safe, but they’re not too risky either But you see two different classes, two different individuals They receive different feature importances like different features were determined to contribute to each prediction And we chose to include only four features because humans tend to prefer more brief, concise explanations whereas– so we only chose to include four And you will see X more positive or X more negative, and those will tell you the difference You can increase it or decrease it as you prefer And I don’t want to bore you about this contribution

or this interpretation, so we’ll just move on And so this is the end of what I want to say I will open up the floor to any questions that you may have for my co-speakers and I [APPLAUSE] AAKASH BORDIA: Thank you, Jonathan Thank you, Pluto 17 So we can go on for a couple questions since we are out of time The first one is an easy one Will this slide deck be available online? So we are checking for you I did post on the Dory The video should be available on YouTube The second question is for Pluto7 So if you guys want to come up on the stage, the question is, how did performance of the AutoML estimators compare to the TensorFlow and XGBoost estimators? So again, how did the performance of AutoML estimators compare to TensorFlow and XGBoost? JONATHAN JENG: So as I showed previously in the dashboard, there are some model evaluation metrics for the different models that we use here So we have mean for class accuracy, precision, recall, and F1 score And we found the best one was actually XGBoost for our case It outperformed TensorFlow We used a deep and wide bottle, and they outperformed vendor solutions and AutoML AAKASH BORDIA: All right, thank you, Jonathan Thank you everyone for attending Have a great rest of the day And if you have any questions for the speakers, we’ll be huddling outside So thank you [MUSIC PLAYING]