Applying AI in Games with DeNA (GDC '19)

[APPLAUSE] SAMIR: Hi, everyone Thank you for joining us today I’m really excited to be here today to talk to you about an exciting topic, which is AI My name is Samir I’m a gaming technical specialist at Google Cloud And I’m based in Tokyo You might wonder also, like, what’s a gaming technical specialist? I have a lot of my friends asking me what I do in my daily job The answer is very simple I just play games But I’m not joking My co-workers are always asking me, what are you doing? They see on my phone I’m just playing But no, more seriously, I’m helping gaming studios in Asia making innovative backends on GCP So today, I’ll be talking to you about what we offer in Google Cloud around AI But then I will give the space to DeNA, who are on the stage with me today, that will talk about their experience on how they simulated human players in one of their more popular games and building that on a scalable and robust backend on Google Machine Learning Engine So they will come back on stage I’ll do a bit of introduction And I think I don’t even have– oh, yeah, this So they will come back in a few minutes Thank you So Google is working with the game industry to help developers succeed And Cloud is at the center of those efforts We want to help create great games and connect with players globally and scale your business So we’re working with other Google teams to bring this vision to life So before diving into the concrete implementation of DeNA in the AI space, let me talk to you about different applications of AI in games And I won’t talk to you about all of them, but just a few that I really like For instance, the first one– being able to play against a bot that simulates the pro players So wouldn’t that be cool to be able to compare your skill to the best, the very best in the game? Usually, you don’t have the opportunity to play against a pro player if you’re probably not one So isn’t that cool to do that? Think also, a similar way, instead of having a bot that simulates a pro player, it could simulate yourself And think about you’re playing a multiplayer game against someone online, and that person that you play against disconnects, and the game ends prematurely Isn’t that frustrating? Me, I play a lot online, and when it happens, I feel really frustrated because I wanted to end the game But think about if you had a bot that could simulate yourself, like the way you play So you’re playing online, your connection drops, but you’ve been replaced at that moment by a bot that plays like you Your opponent might not even notice that he’s playing against a bot, and the game could end Wouldn’t that be a great user experience? There’s also use cases that we’ll talk a bit later that don’t even require expertise in ML But I want you to realize that AI has its place in games And it really helps increase the user experience, but also, in very critical things, like monetization or peer retention So not only the AI has its place in the game, but games are also an ideal environment for AI So games plays a big part on how we develop technology in Google and Alphabet DeepMind is a very good example of that DeepMind is an artificial intelligence research company From Atari to the ancient game of Go, they use increasingly challenging game environment as a testing ground to develop more capable and generalizable systems that can navigate through extreme complexity So the natural next step in their research program was the real-time strategy game StarCraft So StarCraft 2, because it posed yet more hard problems to solve in areas of computer science, like planning, dealing with uncertainty, and special reasonings I don’t know if you have seen what has been done recently with the StarCraft bid They could make a bot that could beat the top pro player of StarCraft 2 So that’s really impressing how you go, how they have done And you can see the vision of how AI is going in game So unfortunately, not everyone is DeepMind, and not everyone has their ML experts

The data is overwhelming The world is lacking of ML experts From the 21 million developers, you have less than a million data scientists and even less ML experts So our goal on the Google Cloud is to help anyone use Machine Learning to build new applications or to add intelligence to existing ones We’ve taken the best of Google Machine Learning and offer it as a fully-trained model via accessible [? VR ?] API’s So all of our pre-trained models are built on years of Google expertise and refined training data From vision recognition to natural language processing, you can use our API to teach your application to see, hear, or interact with users in a complete different way So what are the different AI journeys that are offered by Google Cloud? There is a few If you see on the top, the prepackaged solution– so we have prepackaged yet powerful AI solutions for games to draw new on their AI pass, with very limited resource or very limited ML expertise On the bottom part, for advanced studios with AI journey– like, advanced on their AI journey– who has data scientists or ML experts that can build and train ML models themselves using TensorFlow, Cloud TPU, or Cloud Machine Learning Engine And then, in the middle, you have a new pathway that’s supposed to help you onboard on machine learning, and do more advanced stuff without even that ML expertise So you will find Cloud AutoML So Cloud AutoML helps game studios with limited ML expertise to start building their own high-quality models, but using a GUI And on the right side, you’ll find BigQuery ML So BigQuery, for those that are not very familiar, it’s a data warehouse that Google Cloud is offering on their cloud And it offers an additional functionality where you can create machine learning models just using standard SQL So solutions like Cloud AutoML or BigQuery ML, they’re here to make even AI expert be more productive, but even help– and it will also help the less-skilled engineer that are not very familiar with ML to build powerful AI systems that they could only dream of previously So this is the only sales speech that I will put today in my presentation– just telling you why customers are choosing Google for AI I will go very quickly over it So there’s four points One is scale So we offer a cloud called Google Cloud Platform You start using our cloud, you get access to thousands of instances So if you have heavy computational workload, you get access to the compute that you need to do that Two, the speed So Google Cloud is the only one to offer what we call TPUs So I don’t know if you have ever heard of what a “TPU” is It sounds like a CPU, GPU So TPU is a Tensor Processing Unit So it’s a process [? sword ?] that we custom made for the machine learning use case But to understand how faster is it– usually when you train a model, most of the time you will use a GPU too have a higher computational power And if you compare what Cloud TPU can do, using a ResNet 50 model you can benchmark it, and using a single Cloud TPU– which is eight cores and 64 gig of RAM– it goes 8.4 times faster than an Nvidia P100 GPU, and it still goes five times faster than an Nvidia V100 GPU So it’s extremely fast If you try with other Inception models, like Inception V3, you get similar results So Google Cloud is the only one to offer the Cloud TPU, and can really accelerate your machine learning workload The third one is quality I think you have already tried Google Photo, Google Translator, Google Assistant And you can see how accurate Google Photo can be when you start searching for anything that you have in your Google Photo albums I often search “hamburger,” because I like to show to my friend the last hamburger I’ve eaten And I put “hamburger,” and I get the list of all the burgers that I ate in the past It’s very accurate So even when you search a card– at some time, I took photo of my ID or my credit card, and I put “card” or “ID,” and I will find those photos that otherwise I wouldn’t be able to find

Accessibility, this one is the tool product Like, outside the pre-trained model that we are offering, we have the intermediate solution, like Cloud AutoML and BigQuery ML, that will help you make ML more accessible to everyone So to conclude my part, I have just only three words, “don’t miss out.” AI is at your fingertips, and you can start right now adding intelligence to your application– like existing ones, or to a new one– using those pre-trained model or even custom model using Cloud AutoML If you are already an ML expert, then you can jump directly to TensorFlow Cloud ML Engine to run your own custom model So this is enough for my side Without further ado, I want to introduce you to folks from DeNA that will talk about their experience creating an AI and a deck recommendation system in one of their most popular mobile games in Japan on GCP So I’ll let Jun continue this presentation Thank you [APPLAUSE] JUN: So thank you, thank you, Samir, for the introduction And we’re from DeNA, which is one of the largest game publishers in Japan And today, we would like to share our AI implementation on gaming So far, we have released two applications that we’re going to talk about– the motivation implementation, and the outcomes of our cases So my name is Jun Ernesto, and I’m leading this project Also, let me introduce Ikki Tanaka He’s a data scientist, and built our AI models So in this part, both of us will present So to begin with, this work has been accomplished by four machine learning engineers, and lots of collaborators from gaming team Each of us have our own area of expertise, and I really love this team So let’s start So I think most of you are not so familiar with DeNA Our company is one of the largest internet companies in Japan Although we have various business domains, such as game, automotive, and sports, most of the revenue comes from games Actually, we’re providing dozens of in-house game titles, and the other publishers’ titles Maybe some of you may know that we have an alliance with Nintendo For example, our company is in charge of the backend system from “Super Mario Run.” So this is who we are In this session, we’ll focus on the in-house game title “Gyakuten Othellonia.” So, first, I’d like to talk about the program we had, and how we challenged to solve these issues So this game, “Gyakuten Othellonia,” is a strategy app game The player’s action is based on the board game called “Othello” or “Reversi,” and the players put black on white pieces alternatively What is unique about this game is that each piece has its own skills, which adds strategic components to the battles Actually, when you start playing the game, there are a variety play starts, and endless exploration is required to master the game This title was released three years ago, and fortunately, it’s still growing We have more than 23 million downloads so far, and the service region is expanding gradually And of course, I hope in the future we can release in the US, as well So in “Othellonia,” there is the concept of decks and the characters Before starting a battle, player [? constructs ?] a deck which is composed of 16 characters among more than 3,000 options And each character has its own skills, and the players have to choose consistent characters according to their play styles As you can imagine, the number of combinations is extreme, and it’s not easy for beginners to learn how to build an optimal deck Another issue here is that building an optimal deck requires to learn effective strategies through the process of trial and error So let me explain why this is a problem Generally speaking, if the game has complex features which can distract players from continuing the game, the churn rate increases, and impacts the future revenue Right? For example, there is an estimation that just a few points decrease in the 30-day churn rate can increase the revenue by 20% to 30% in the future Back to “Othellonia,” we actually

have a high churn rate, so we are in trouble This is partly because the game is complex, and lacks sufficient [INAUDIBLE] support Therefore, it’s hard for beginners to construct an optimal deck Also, there is the problem that they don’t have a good place for practice in order to improve their battle skills For instance, we only have two battle options for beginners One is NPC battle, in which they battle against very, very weak opponents, and the other is PvP battle, in which they are frequently matched with skilled players So we needed an opponent whose play level is similar to the beginner’s So that’s our opponent setting So let’s see our approach To support beginners, we have developed two functions The first one is the deck recommendation system, which selects appropriate characters from players’ piece box, and we are using the association analysis to provide a better recommendation The second one is Battle AI, which we call Othellonia Dojo In this function, we developed AI that can play at any levels A deep neural network is used for this AI Later, Ikki will explain how these techniques are used in our cases So, by the way, from the machine learning perspective, we need to prepare the data to build such AIs We are collecting various kind of in-game data since the release of the game, such as deck logs and battle logs So we have a good amount of data before starting this project So in our cases, we used the data for the last nine months, which contains billions of log entries And then we filtered the data by player skills, and augmented the original data in the battle AI cases The image on the right side is an example of a battle log, which is formatted in JSON, and describes enter states and action information of the battle The bottom diagram shows a rough data pipeline, and BigQuery did a very good job at this pre-processing phase For instance, it can handle even a terabyte order of data very fast, and we could reduce the processing time drastically BQ could handle our data in a few minutes, which took an hour on on-prem Hadoop clusters And we also love that BQ is easy to collaborate with other GCP services So here I want to stress that our AI has been successfully built by sufficient amount of data and a reliable pipeline So let’s see how each of the cases works This is a very rough overview of the deck recommendation system So we first exploit a pre-process deck log to extract information about how each characters are associated, which is called associational rules With this analysis, we can score the strength of the relationship across all the pairs of characters Then, if there is a request from the player, the recommendation server receives a set of characters in the player’s piece box, and it fills all of the 16 characters according to their score of the associational rules This function is very flexible, and the player can specify deck archetype, deck cost, and their favorite characters to be included in the deck Regarding KPIs, we are monitoring several metrics For instance, we are watching how many players have used the recommendation system, whether the recommended deck has been accepted, and whether the deck contributed to the players in terms of their win rate in battles So as for the Dojo, players can battle against AI with various archetypes and strengths Here we train a deep neural network from preprocessed battle logs The clients send current game status, and the battle API server returns the inference of neural network at every turn of the game As for the metrics, we are monitoring how frequently the Dojo is used, and whether the strength of the AI was sufficient in terms of win rate So these two AIs have already been released, and we are observing very positive KPIs

So before showing the results, I’ll pass the talk to Ikki He’ll talk about the technical details, and some tips for building AI systems [APPLAUSE] IKKI: Thank you, Jun So hello I’m Ikki, and a data scientist at DeNA So I am one of the developers that created our gaming AI system by using data analysis, data science, and machine learning So today, I’ll talk about the technical details of our use cases with GCP So now, let’s look at the story of the technical part So first, I’d like to standardize our deck recommendation This is the outline of the deck recommend algorithm And firstly, the deck recommendation algorithm receives some input from a player, such as deck archetype, deck cost, their possessed characters, and the characters the player wants to use Next, the algorithm selects a leader by using the result of association analysis and statistical data So in this algorithm, the leader character is chosen first, and the next characters are selected based on the leader The well-associated characters are simply selected based on the result of association analysis If the deck is not full of the 16 characters, another character is selected as the best character After repeating this procedure several times, the recommended deck is proposed to the player If the game is different, the way of fine-tuning is also different So in our game, “Gyakuten Othellonia,” for example, we made additional rules to decide a leader, and we checked its activation with the skills in deck building I’d like to give a brief explanation about how to extract the relationships between characters In our recommendation, we are using association analysis The association analysis is a rule-based machine learning method for discovering interesting relations between contents in large datasets So this method can perform high-speed calculation to extract the main relationships, and extracted relations are easy to understand and very useful in various settings The relations are evaluated with some metrics, like the strength of the connections and frequency of the rules or patterns Our recommendation builds a deck by selecting a character sequentially with strong relationships, one after another So this method can handle various kinds of data easily So I think this technique is applicable to many games Next, I will talk about the system So we built our application such as deck recommendation and the deep learning AI bot with GCP So why did we choose GCP for our ML systems? There are two main reasons The first one is powerful and flexible tools for machine learning and data science In the data analysis, we often try to investigate the data, visualize it, and create the prototype models GCP services for big data analysis make it possible for us, such as BigQuery, Cloud Datalab, and Google Compute Engine The GCP services for cloud storage provides scalable and faster services for large data And it helps us to manage the big data and a lot of machine learning models About Cloud ML engine, we can deploy the AI models to Cloud ML engine easily and quickly And it also provides user-friendly prediction APIs Additionally, GCP provides many services to perform machine learning efficiently For example, we are planning to use the GKE content of service and Cloud ML Engine training with custom functions for scalable and distributed ML training The second one is scalable and robust infrastructure services We can run the application easily by using Google App Engine with several programming languages, like the Python Go language

And we can check many metrics of system performances through stack Stackdriver, Logging, and so on I’d like to emphasize the auto-scaling function of GAE is very great service, because new instances are increased or stopped in response to traffic And the [INAUDIBLE] are distributed to the instances automatically So this GCP service is easy to integrate into the same systems So for these reasons, we are choosing GCP for our ML system I’ll show the system architecture of deck recommendation with GCP First, the deck logs are saved in BigQuery, which are used to extract the rules We send the [INAUDIBLE] schedule to performance the batch processing of the association analysis in Google Compute Engine And in order to catch up the trained decks with latest characters, we are performing the batch processing every day Then Cloud Functions preprocesses the result of association analysis, and updates the data for production environment Cloud Functions checks if the data is updated correctly– for example, if the deck size is too small or large And we are using Google App Engine Python 3.7 for the recommendation API server The server receives a request from a client, then builds the deck, and finally, it returns the recommended deck to the player One of the noteworthy point is GAE Python 3.7 was generally available at the end of last year, so we could use it just at the right time As a result of that [? wait ?] to Python 3.7, we don’t need to change our code that we had already developed in Python 3 So far, I’ve explained the algorithm and the system of deck recommendation Let’s move on to the next topic– Deep-Learning AI-Bot It’s Othellonia Dojo “Dojo” means a training room to improve their playing skills This is an overview of the training in Deep Learning AI Bot The branding is suitable for our case to learn the many complicated features, such as state of board, decks, hands, skills, and combinations of the features The Deep Learning AI Bot receives a state of battles from a client The information of battle is converted to the numerical features, which are used as the inputs of the Deep Learning The network learns the actions of top-tier players as binary classification And finally, the Deep Learning model outputs the values of each action that will be used to decide the next action Here we used more than 10 million battle records in the training Next, I will show the system of AI Bot First of all, I’ll talk about the training phase A large number of battle records in PvP is saved in Google Cloud Storage through BigQuery And the training process consists of preprocessing of the features and the model training, which are running in Google Compute Engine At the preprocessing, the features are generated on demand from the battle records by emulating the game In our case, we created the custom function of Data Generator that creates a feature during the training After the training, the trained AI model is saved in Cloud Storage to deploy onto the Cloud ML Engine Next, I move on to the inference phase So please take a look at the communication between the client inference API server on the left of the slide The battle records are posted to the inference API server from the client, which plays a battle event against AI bot The inference API server reproduces a battle from the record, extracts the features, and it sends those features to the online prediction API with Cloud ML Engine Online prediction of Cloud ML Engine is very useful, because it is optimized to run the AI models with low [INAUDIBLE] in real time

Cloud ML Engine uses the trained AI model starting in DCS and returns the values of each action The values are sent back to the client, and the client moves towards the battle step by selecting a best action according to the values That’s the overall communication between the client and the servers Google App Engine and Cloud ML Engine can auto-scale in response to the load on the server That’s a very good function So by taking the benefit of the auto-scaling, we can manage a large amount of requests Lastly, I’ll touch on the technical topic– the testing of the systems So why should we test the systems? I mentioned in the last slide, Cloud ML Engine can auto-scale quickly– that’s good So yes, that’s true But we didn’t have enough knowledge about how much load the system withstand In the mobile game, generally, the load tends to increase at the moment a new event opens The figure on the slide shows an example of estimated requests per second in a yardstick event This is a trial calculation The requests per second will increase to several hundreds in 10 minutes We have to check in advance if the system can withstand this load As a point of notice, we cannot predict the action of the AI as a batch processing at regular intervals Our system has to respond to the request from the player in real time That is where the AI should put the character in the next turn To achieve this, we chose the Online Prediction API of the Cloud ML Engine We’ve checked if the Cloud ML Engine can satisfy our requirements by performing the load testings in advance of release In our case, the testing was performed in the setting and the requests increase linearly and reach a maximum of 1,000 requests per second We found the Cloud ML Engine can auto-scale quickly in response to the requests, and the average latency of the prediction is less than one second However, there are a few requests that took a longer time than usual Since this might not be possible to request to the player, so we needed to resolve this condition Our DeNA team and Cloud ML Engine team cooperated to investigate the bottlenecks And then we found the timeout has happened when the new instances of app in the process of the auto-scaling due to our heavy load It is challenging to handle a large number of requests which consists of many features at the same time with online [? prediction. ?] But if we can resolve this problem by building an efficient but consistent, we don’t have to warm up the system beforehand, and the system will be more robust So our team took our time to deal with this problem Let’s take a look at we tackled to remove the errors When the error returns within a time we can simply throw it back again to the servers But in the case of a timeout, the system cannot send it back again, because a timeout error is not returned Therefore, when the inference API server does not receive the response within five seconds, the request is sent back to the API again, because something bad has happened in the inference Also, if the destination of the request is the same end point, it will further increase the load on the server As a result, we set on the backup end point to receive the request that fails to be operated in the main endpoint That is, the second request is sent to the backup endpoint, which is prepared independently from the main endpoint This is the result of load testing with backup endpoint

The figure shows the error rate of several settings– no backup, backup with 10 minimum nodes, and backup with 20 minimum nodes In the case without backup endpoint, on the left, about 1.3% of errors remain In the middle setting of the backup with 10 minimum nodes, although small errors still remained, the backup can deal with most of the main endpoint errors We found the reason of these small errors is because the number of backup instances was small a bit So the timeout has still happened at the warm-up of the auto-scaling in the backup environment So right most cases of fine tune setting, where the number of backup instances was set to 20 for covering the errors of main endpoint substantially In this case, we could handle all the requests without errors So in this way, DeNA and the Cloud ML Engine team got over the challenge of high and spiky traffic and latency limit without errors After deployment, this system has been stable in handling many errors So far, I’ve explained the [INAUDIBLE] system of our AI, and the system with GCP can be constructed flexibly to cope with various programs for gaming So now I’d like to hand over to Jun for the KPI result and the future Thanks [APPLAUSE] JUN: Yeah Thank you, Ikki And I think you mentioned, we needed a scalable and robust system to actually implement this AI in the real cases So in this final part, I would like to share what happened after the release So first, for the deck recommendation, here is a chart which shows the number of beginners who use deck recommendation There were constant usage after the release, but at this time, this function was not well-known But the station has been changed after we started an in-game event to promote players to build new decks The deck recommendation system has been recognized more, and even after the event has been finished, many players are still using this function frequently So second, we are very glad that the recommended deck pushed the win rate of the beginners by about 5% compared to decks created without the recommendation system Since our focus was to support beginners, we are very satisfied with these results Also, I’d like to mention that this function has good adoption even for top-tier players This was sort of byproduct, but we’re observing that heavy players usually update their deck frequently And this recommendation system mitigates the cost of building deck from scratch They use recommendation as a first edition, and then customize the deck as they want So let’s look at the system metrics, as well After the release, we have a stable latency of 20 milliseconds, which is very fast enough for our cases But we are very surprised by the result, because the recommendation algorithm is very complex, and we haven’t expected such a low latency The auto-scaling does a good job, as well, and we haven’t had any system trouble so far So briefly, we will explain about the evaluation of Deep Learning AI Bot After the release, we are observing that our AI Bot has good enough win rate as a practice partner for beginners Our AI Bot can beat even top-tier players, and therefore, the players can choose any levels of AI as a sort of mentor Although this was released last week, and we don’t have enough data to evaluate the business impact, Dojo is used by beginner very well, and we hope this can lead good KPI in the future So as to the latency, the inference of Machine Learning Engine is about 200 to 300 milliseconds, and they’re very fast and stable Even for the entire system, the response time is about a second, which is enough fast for our case

Backup endpoint is working well, and we haven’t had any system trouble so far So to the end of my talk, I’ll show items which worked well in our cases So to train the AI that played the game like a human, we needed a battle log that completely reproduced battle status and actions So if you have some random effects in the game, make sure that the random seeds are stored in the logs Secondly, for the evolution of our AI, we needed to run thousands of battles to get performance statistics Headless simulator helped us a lot in this context If you want to try to use reinforcement learning techniques, you also need very fast simulator, because the number of battles that our version goes through really matters And third, we also want to stress that scalable and robust system is required for implementing AI in real services We think that GCP– in particular, Machine Learning Engine and GAE– were a good choice for us The last item is not technical, but critical Generally speaking, we don’t know how well AI will perform and how big the business impact will be before starting a project So in such an uncertain situation, the communication and controlling the expectation is very important Although we are machine learning engineers, we try to understand the game and the business issue correctly And I think that this was a key factor of the success of these projects So this is from me, and I’d like to pass the summary to Samir again Thanks [APPLAUSE] SAMIR: Yes Thank you, Jun-san and Ikki-san, for this amazing presentation I think it’s inspired a lot of people here today, and it’s very impressive, the work they have done to make their game a bit more inclusive and a bit more friendly to beginners And I hope that what they have implemented will be very successful, and that today’s presentation can help you, maybe, have some ideas And so, before closing this session, I want just to show you some simple use case that you could start with using machine learning And I will talk quickly about two that don’t much require any expertise, and one that’s very similar to what they have done So starting by this one, the chat translation So one of the big problem in games is the language barrier And so I strongly see that in the regions where I come from– Japan, where players are just speaking Japanese And you will never see a game that has multiple community of people from Japan playing with Korean and Taiwanese people, because the language And so if you want to break that barrier, you can start by using this kind of pre-trained model, like Translation API, to translate chat messages in real time, and to break those kind of barrier It would be so good to have more integrated community within the game that could go beyond frontiers So this is one use case that you could really start very quickly And if you have ever tried the translation– Google Translation– you see that it’s very accurate And even though it might not be perfect in the language of a game, it will still be better than having communities that cannot communicate with each other The second one is one that I see every gaming company trying to fight, is the player toxicity Sometimes in your game, you might have some people that might bully some other player, and that can break a bit the game And so you don’t want to lose important players because of the bad players’ behavior You could start on Google Cloud Platform– on GCP– using Natural Language Processing API That has a functionality called Sentiment Analysis So what Sentiment Analysis is doing is, it would take a text as an input, and will be able to tell you if it’s a positive input or a negative input that has been done Based on a lot of data that you would be able to take, you could maybe detect very negative insight from players, and to warn them, maybe to ban them, if it’s really at that point But you don’t want, maybe, to go manually to check every single chat or action of players to decide if that person is toxic

You could get this kind of API to bring you insight on potential problems in your game, and potential players that are dealing, maybe, problems in your game, and that can damage it in the end So these are two use cases that can be start very quickly, using pre-trained models The last one, which will require a bit more expertise, and I think that’s very similar to what DeNA has done today, is simplifying games And I think today, what DeNA have done is a perfect example of what is simplification of game using AI And they made their deck recommendation to simplify a bit the construction of the games when you have too many characters, possibly And also, having a way to train against a bot is also a way to train and help people ramp up So you could also think from other use cases And one that I always hear from studios is the Q and A. So the QA testing– not Q and A. QA testing So QA testing costs a lot of money, because you need to test every time you do some changes You need to test if this doesn’t break, for instance, the balance of your game In the mobile space, and even now in console, because you start to see that the latest game, like Fortnite, gets weekly updates to bring you new items in the game If you will be able to automate this kind of testing, you’ll be able to be sure that when you bring a new item or a new card in your game, it doesn’t break the balance of the game And usually, I can see from being in Japan, and playing those Japanese game on mobile that brings new item, new character every two weeks– sometimes a character will break the balance I even see on Fortnite, sometimes they bring a new weapon, and it’s too strong, and they immediately take action But you would like to act before doing that, before going in production with your update If you’ll be able to create the automation, like they did, that would be one of the use cases that could be interesting And I think to do this kind of thing, you’ll be able to use TensorFlow, Machine Learning Engine, and do something very similar to what DeNA did And I hope that what the presentation from DeNA inspired you and gave you some insight of how you bring and approach a problem that’s as complex as, maybe, a human-simulated bot But I hope this will be helpful for you And I think we have a little time– I think we have 10 minutes more So if you have any questions, I think we’re– Ikki-san, Jun-san– please come up on the stage And if you have any questions, I think they have a mic in the back And feel free to ask question If you want to talk a bit more deeper about something, or a challenge that you’ve been having, we can discuss after the session So thank you very much [APPLAUSE] Any question? Ah AUDIENCE: First, thank you for your talk When training your AI based on different skill levels– so to play against beginners versus playing against experts– did you ask the Cloud system, the ML system for multiple inferences? Or did you train it differently? Did you train it on games for people who lose frequently, as opposed to just experts? JUN: So your question is that if we only train with the top-tier tier players [INAUDIBLE]?? Ah Yeah Yeah That’s right AUDIENCE: You want to train it so that it sometimes loses to beginners So how did you accomplish that goal? JUN: You mean that– to train the levels? Yeah Yeah OK So we have invented some schemes for that adjustment So we inference for every turn the score of the move Right? So if, for example, there is ten moves, we sort every latest score, and we pick the most highest score, up to– I mean, to be strong Right? But we are optimizing that score by selecting the second- and third-best move in a certain probability, and that’s a hyper parameter, and we are tuning manually that parameter So that’s why we are– we could balance the levels according to the player AUDIENCE: In building the game bot, did you use reinforcement learning, or improvised learning, or maybe you had different results with different models? JUN: So in this case, we are not using, but we are trying Well, actually, the [INAUDIBLE],, here, is that the [? main ?] function running is very unstable

to learn– especially in this kind of strategic game So while we are trying, but the win rate is not so high, like a supervised learning AI bot SAMIR: Here AUDIENCE: Hello Thanks for your talk again So you had nine months of data, and billions of logs to start If you were to put this system into a brand-new game, nine months is a very long time, and a lot of players may already come and churn So what would be an ideal– what would be sort of the theoretical minimum? Could you put this in at the start and learn as you go? Would you need to wait some certain amount of time to [? accomplish this? ?] JUN: Yeah Actually, like, two week or one month’s data is enough for our case So nine months is a kind of buffer So I think it’s too much to use nine month’s data But you are measuring the [INAUDIBLE] AUDIENCE: If you could– moving this to your next game, you would have this at the startup? JUN: Ah Yeah So in that case, maybe we need like one month data, or a few weeks’ data, before the training SAMIR: Any other questions? Ah Over there Yes By the way, thank you for the very good questions It’s a very good point AUDIENCE: The nature of the games, these days, is that the rules of the games change frequently– especially with game updates And that changes the nature of how the bots should be playing How do you plan to deal with that? JUN: Yeah That’s a good point And actually, we’re releasing a few characters every week, and the game bots change time by time But our deep learning frameworks works even for that cases, because– well, just after using one week or a few days’ data, the AI become to add up to– how do you say– the latest environments So there is kind of a one week delay, but in this case, I think we are acceptable So yeah That’s answer for the question AUDIENCE: What other problems are you trying to tackle? JUN: So your question is what’s the problem, currently, we have? Or– AUDIENCE: Is there any other type of problem that you’re considering tackling, other than what you presented today? SAMIR: If there is any other problem that you want to solve using ML? JUN: Ah Yeah, of course So I mentioned the importance of machine learning But actually, we want to keep the game balance very stable, and before releasing the characters Right? So we want to implement reinforcement learning engine to learn the brand-new characters before release, and then check if the balance is not too much– I mean, strength, or strong or weak, or something like that So that might be the next challenge AUDIENCE: Hi Thanks In regards to your inputs, what data are you discarding? Are you using everything you’ve captured since the beginning, or everything from the last patch, for the last two patches– or updates, rather? JUN: Yeah Actually, we are using almost everything– information, as I say this So we’re using a five-kilo feature– exerted for the board Like what kind of skills is activated, or in the last turn, what happened, or something like that Yeah AUDIENCE: Are you at all worried that the legacy data kind of skews the AI, or the choices, at all? JUN: Um– so sorry I think I missed your point AUDIENCE: Yes As you introduce more characters, more options, this, that, are you worried that the very old data that didn’t know about all this new stuff kind of skews the recommendations, or what the AIs– SAMIR: I guess if you have a change in your game functionality over time, how do you handle the new type of data that you didn’t have before JUN: Yeah, that’s the biggest problem we have now And frankly, we are– so every time we release a new skill, we are tuning the feature extractor, and making to create new features But that operation cost is very high, so we want to somehow optimize that process

So that may be the next challenge to solve SAMIR: Any other questions? OK I think if you want to talk a bit more in detail, we’re staying on the left of the stage So feel free to come by and ask any question Other than that, I want to thank you, again, for today It was really great And thank everyone for joining us today [APPLAUSE]