Tensorflow, there is no spoon (DevFest 2019)

[MUSIC PLAYING] TIFFANY SOUTERRE: Thank you very much for coming to my talk I’ll start with a short video that I really like And I think you know the movie [VIDEO PLAYBACK] – Do not try and bend the spoon That’s impossible Instead only try to realize the truth – What truth? – There is no spoon – There’s no spoon – Then you’ll see that it is not the spoon that bends It is only yourself – The Oracle will see you now [END PLAYBACK] TIFFANY SOUTERRE: OK So first, before starting, who in this room has not seen the movie “Matrix.” Oh There’s always one hand raising OK So for those of you who haven’t seen the movie– and I’m not going to feel guilty by spoiling it because it’s about 20 years old So the “Matrix” is this huge let’s say software that recreates the world around us And we think we’ll live in the real world but we’re actually in this simulation And for artificial intelligence, it’s almost like the same thing as if when Neo sees the spoon, he thinks he is seeing a spoon, because he’s actually only getting the information of seeing a spoon And his brain processes the feeling, the touch, the view, but there’s actually no spoon there It works pretty much the same for machine learning We have machine learning almost now everywhere in all of our apps We trust them for self-driving cars We trust them for face recognition apps and things like that But are they really reliable? Today I’ll show you how we can hack a neural network into seeing things that aren’t actually there And I’m going to show you how to protect against attacks So my name is Tiffany Souterre Feel free to follow me or send me messages on Twitter I am a GDG organizer in Paris And I’m working as a data scientist at JEMS Data Factory And I’m also a Women Techmaker lead So here what do you see? It’s a picture of a panda And you’d be right to say it’s a panda And Googling it, it’s a model from Google– classifies this image as being a panda with 57.7 confidence And now what do you see? It looks the same, right? It looks like the second image is the same as the first one But the second image is classified by the exact same model, the Google net, as being a gibbon with the confidence of 99.3% What is going on here? What’s the problem? What’s the difference between the second image and the first one? The only difference is some added noise that had been added to the first image This noise has been very carefully crafted to make the classification of the panda be the gibbon by the exact same model And it’s been multiplied by a very small scale so the human eye cannot see it but it’s enough to perturb a model So this was published in 2016 by a team working at Google Brain led by Ian Goodfellow Scientists went further, discovering new ways to confuse neural networks Here you have six different examples You have a bus, a bird, a temple, speakers, mantis, and a dog All of the images in the column on the left are the original ones And all the images in the columns on the right are the hacked ones we’ll say

And the only difference is that they added the noise that you see in the middle Again, the noise has been added to a very small scalar so– yes Come in– by a very small scalar so you don’t see the difference But now all of those images that have been modified are classified as being ostriches by AlexNet which was, at the time, the state of the art neural network Oh Do I have Wi-Fi? OK So now you see that we can add noise to images But does it actually translate to the real world? I downloaded the video I’ll show you here Yes Come in All right So here I’ll show you an example of scientists that– hi OK So here is an example It was from an article published in 2016 So scientists try to print original images and images that have been added with noise to it And so here you see the first image was the original one on the left And it was classified as being a library And the second one was being classified as being a prison Here we have a second example with a washing machine So here you see the situation is pretty obvious Here is the original image correctly classified as being a washer and then it’s classified as being a doormat Last example a little bit more so so you can see how it doesn’t need to be that much obvious for it to perturb the model So here you have the original image We can barely see it So the first image is classified as being a washer Then it starts getting confused if it’s a switch or a washer It’s not sure anymore And now it classifies it as being a switch So this actually do translate to real world applications So the step further was can we can we confuse neural networks even more So here it’s a very interesting example Scientists try to craft the noise in the shape of frames as if people was wearing glasses Here in the image on the top left, you see the actress Reese Witherspoon and then added noise around her eyes in the shape of glasses And she is now recognized by the model as being Russell Crowe So it means that you can really design the noise in a certain way and get a very, very different label at the end So what they did is can we do this? Can we try to hack face recognition apps? So they tried two different types of attacks The first one was impersonation And the second one was dodging attack So they 3D printed glasses as the ones you can see on the bottom And they designed it a certain way So in the first column here, you see two people doing a dodging attack, which means that the face recognition app was not able to recognize them anymore And then for the three other ones, I’ll show here, the people on the top row did impersonation on the people on the lower row So this means that this man wearing those glasses was recognized as being Milla Jovovich So if this guy can be Milla Jovovich with those glasses, you pretty much could be anything you want if you know how to craft your glasses The last example that I’ll show you that I really like

So yeah I’ll explain a little bit before what they did So they 3D printed two turtles, a turtle that was supposed to be a normal turtle and a second one that was supposed to be recognized as a rifle, like a lethal weapon So they tested it on Inception V3, which was the state of the art neural network for Google back then, like a few years ago So here is the control turtle It’s recognized as being a turtle by Inception V3 And no matter the angle it’s a good classification And now you have the other model, the perturbed one with noise on it You can see on the shell some of the noise And now no matter the angle, it’s recognized as being a rifle It’s uncanny for me because it still has the shape of the turtle but somehow it’s not recognized by the model anymore All right So I could potentially just tell you [? adversarial ?] attacks do exist and your model is susceptible to be fooled by those attacks But what is interesting is to understand how it works and how you can actually defend against it So I’ll try to explain to you I need to explain to you first how neural network works And for this, I choose a very simple example from the MNIST database It’s a database of handwritten digits And all of the images have been nicely scaled for a 28 by 28 pixel image And it’s black and white All of the pixels in the image that are black have a value of 0 And all of the pixels with white have a value of 1 Everything in between is in a grayscale and it’s in between 0 and 1 So how a neural network works is the following You have first an input layer The input layer makes up for all of the pixels in the image So if you have an image that is 28 by 28 pixels, the first input layer will be 28 by 28, so 784 neurons The value in each of those neurons is the value of the pixel So it’s a value between 0 and 1 The output layer will give you all the different possibilities of labels So if you have a data set of handwritten digits, you only have 10 different possibilities of the outcome– 0, 1, 2, 3 up until 9 And the value in those neurons will be the probability that the model classifies this number as being a 9 or– like let’s say if your model has been really well-trained to recognize this image as being a 9, in the output layer, all of the values in the neurons will be close to zero except for nine, which it will be closer to one And all of the values need to add up to one And what you have in the middle, we call it hidden layers This is where all the math happens And I explain to you in a very second how to calculate the values for each hidden layer So I’ll show you the calculation for the first neuron in the first hidden layer and then you can iterate for all of the other neurons So to calculate the first value of the first hidden layer, what you need to do is first you do a huge sum You add all of the values of the neuron from the input layer, so all of the values of the pixels in the image And for every single one of those values, you multiply it with a weight parameter At the end, you add a bias term which basically sets a threshold beneath which the neuron should not be firing And because this sum can be very, very high or very, very low, we need to squish this number between 0 and 1 So we add this function that we call sigmoid function here But it’s called an activation function And you have different types of function that can squish your number between 0 and 1 So you repeat this step for all of the neurons in the hidden layers And basically, what a neural network will do is a huge matrix multiplication You add your vector with your bias term

and you squish everything between 0 and 1 So what I’m showing you here is that you don’t necessarily need to remember all the steps What you need to understand is that a neural network is just a huge function It’s a function that will take and input 784 values and spits out a vector as 10 entries long and that’s it So it’s not black box voodoo magic It’s just straight linear algebra Why am I telling you this? What you need to know– what you need to understand is that what a neural network is trying to do when it’s learning, it’s trying to classify things It’s trying to draw a decision boundary between different labels So here for a very, very simple way to visualize it, I’m showing you what it would be for a network that is only two dimensions So for an image, it would mean that it’s only two pixels So it’s not like this because I don’t know how to draw graphs in 784 dimensions but you follow me So here for a very simple example, let’s say for the sake of understanding and simplicity that your image is only two pixels And it’s trying to classify between two things Let’s say blue is dogs and red is cats You have the data set The curve here represents all of your data points, so all of your images of cats or dogs, and then all of your images of cats And what the neural network is trying to do is to draw a decision boundary that will separate those two curves And with a neural network that is only one input layer and one output layer, the best separation you can do is a straight line And here you see it’s not perfect This is the best straight line you can draw between those two data sets But you see some of the blue dots are landing here in the red area, and some of the red dots here are landing in the blue area So what can you do? You add one hidden layer Adding one hidden layer to your equation allows the decision boundary to take another dimension And by bending the space here in the decision boundary, you’re able to make better performance with your model to separate the image of cats and the image of dogs These two images are really interesting, because here it’s showing you the graph from the perspective of the input layer But here you have the exact same separation Only here the decision boundary’s a straight line and all the space around it has been transformed This is what the model will do while learning It’s bending the space so it’s trying to find the straight line that would separate two data sets Here you can see a very mesmerizing example With only four hidden layers, what the model is doing while it’s learning is stretching and bending the space in order to be able to draw a straight line between two data sets So let’s say here you have a data set of two things that are intertwined, let’s say like tigers and cats because it’s closer in the space So the model has to apply a lot of transformation to the space to be able to draw the straight line I’ll let you see it a last time because it’s really, really cool Every time it’s stretching, it’s always compressing the space between 0 and 1 It’s really interesting to see it in action And it’s only four hidden layers So why does [? adversarial ?] image work? It’s because, if you understand this, you understand that the only thing you need to do in order to misclassify your input is to push your image towards the decision boundary Let’s say you have an image of a cat and you want it to be misclassified as being a dog You just need to push a little bit every single value of your pixels towards the decision boundary to land in the red area and then it will be classified as being a cat So you just need to apply linear algebra to the values of your pixels So now that we know, theoretically, how to make your adversarial example, we can try to make it So for this example, I chose inception

So the first thing I did was a classification of the different models that was out there And I put in the middle human performance So here is a graph of the top five error rate, which means basically it’s the– yeah When you send it a bunch of images to classify, it takes all of the images that it was wrong in the top five classification So a human– Andrej Karpathy, I encourage you to read his blog because it was really interesting He was curious to know how well he could perform against machines And so he went through the same data sets that machines would go through in order to assess the performance And it was really brave of him because he went through thousands of images And hand labeling them is a pain Like I can tell you if you read this article, it was a very long process for him So this is why I take it as granted for all of human performance on earth, because I think nobody on earth would want to go through the same data set So let’s say humans perform at 5% The lower the better, so I took Inception v3, which currently does better than humans at classifying images And Inception v3– here is the architecture of Inception v3 It’s huge It’s a very, very deep neural network Every single square, like rectangles that you see here, is a hidden layer I showed you earlier only four And this is the complexity you need to be able to classify much more complex images And it’s able to classify between 1,000 different classes so you don’t only have animals but you also have planes, glasses, houses, and things like that And the only thing you need to know here is that all of the inputs must be 299 by 299 images And the depth of the image is the channels of colors, so R, G, and B. So that’s the three So I needed a guinea pig for my experiments I took a picture of my cat And I wanted to classify my cat as being a spoon with Inception v3 Apparently, she didn’t like the joke So first I wanted to know how Inception v3 was classifying my cat The first thing you do is you load your image And you want to size it to a nice 299 by 299 And then you need to apply transformation for the values of each pixels so it’s landing between 0 and 1 and not between 0 and 255 And then I ask Inception v3 what my cat is And Inception v3 classified my cat as being a tabby, which is a striped cat, so it’s exactly what my cat is, with a confidence of 96.96% 86 All right So now I want my cat to be a spoon First thing you’d need to do is to load your model and set your input layer as being the picture and the output layer right now gives me a 0.87% confidence that [INAUDIBLE] is a cat And I checked for the classification of the spoon because this is what I want to reach And I put 0.00 but it was actually much– it’s not exactly zero When I checked, it was about 0 times 10 to the power minus 6 something So it was a really low– like Inception v3 was pretty certain that my cat was not a spoon That was for sure And I thought it was really interesting because I was thinking, OK, I’m going from very far, like the classification of the spoon is really far, so I’m wondering like can I actually reach it So the first thing you need to do is to grab the confidence So basically, what the code is saying here is the confidence of my cat being a spoon is about zero something until the confidence of my cat being a spoon is at least 98%, keep changing the image And how do we change the image? We change it with the gradient descent, like the gradient So for those of you who have already tried to do machine learning, basically what the gradient will give you is a direction in space towards which you need to go to get to your label So imagine you have the classification

of the spoon being around here Again, it’s not in three dimensions It’s in 299 by 299 by 3 So it’s huge dimensions But visualize the area of the spoon being around here and my cat is here in the classification of the cat And I want my cat to land in the spoon zone So I need to know which direction in space I need to push every single pixels in my image And I need to know how far So the gradient will give you the direction And the learning rate will give you the steps So you choose how far of a step you want to take towards the direction of the classification of the spoon If you take too much, like too big of a step, you might overshoot and not land in the area of the spoon And if you take too little steps, your while loop here is going to take a long time So you want to find the sweet spot between taking steps that are not too small but not too big so you’re not overshooting to the classification of the spoon And of course, because I didn’t want my image to be obviously hacked, you need to clip the value of your pixels so it’s not changing too much So I clipped it between minus 0.1 and 0.1 So if the value of my pixel was something like 0.5, it could not go lower than 0.4 and not higher than 0.6 so it still looks like a cat And then you save your image So here are the results I got The first image is the original one classified as being a tabby The second one is the image of my cat classified as a spoon And it’s with a confidence of 98% confidence– 98.65 And here I tried with another classification of pineapple, but you can try with pretty much everything It’s really cool I didn’t do a lot more because it took about three hours for my computer to generate those images, so a long time, but it was running overnight And so when you see those pictures, it’s not obvious that there is any difference Like for me, I got really excited and I was really happy, but I tried to put my face really close to my screen and try to see differences but I couldn’t And it was really frustrating I was really wondering if actually there was any differences So I made another experiment I took a blank white image and I tried to apply the exact same thing to my image So I tried with a classification of a pineapple And here is the image I got Oh, we actually can kind of see the difference, but it’s not much Like it doesn’t need much of a difference to confuse a neural network network into thinking that this white image is now a pineapple And to convince myself that I was actually applying differences to the pixels, I put my image in Gimp and add a saturation of 100% When you do this on a complete white image, you’ll get a gray image Here I got this It’s not gray It’s all colored What it means is that almost all of the values of the pixels in my white image have been changed somehow a little bit And like I don’t know if you see a pineapple here Obviously, for me I don’t, but if you see a pattern, let me know because it’s weird For me, what it means is that you ask your model to draw a pineapple and it’s unable to do it It’s unable to draw a pineapple because it doesn’t have the concept of it And even though it’s better than humans at classifying pineapples, dogs, cats in images, it’s actually unable to know what’s the concept of a cat and what’s the concept of a pineapple So all of what I’m saying is really nice, but you might be thinking, I am using machine learning in my apps And I kind of want to know how to protect against adversarial examples because I don’t want them to fool my neural network So there is no magic trick Unfortunately, there is no magic way to solve this problem One way that scientists have been working on for the last year are called generative adversary networks

And it’s kind of a huge work but you can make it so– I’ll try to explain how you can do it What you do to protect your model against attacks is you copy your model So you have two exact copies of it And you’re going to call one generator and the second one the discriminator And we’re going to play a little game They’re going to fight So at first, you start with your real database of real images And you feed it to the discriminator And the discriminator will classify your image as being a cat And it’s going to say it’s a real image and it’s a cat And you’re going to reward the discriminator for being able to tell that it’s a real image And now the second turn would be to the generator to play So the generator will generate an image, let’s say a fake image of a pineapple And because this is the exact same model at the beginning, the discriminator will recognize this image as being a pineapple Say, oh, I know this image This is a pineapple But you know it’s not a pineapple You know it has been generated by the model So you can punish the discriminator for believing that this image was a real one And you are going to reward the generator for being able to fool the discriminator into thinking that this image was a real one And you play this game iteratively like thousands and thousands of times What is going to happen over time is that the discriminator will be better at recognizing fake images but your generator will get better at generating fake images What happens after a while is this kind of thing At the beginning you have those images that is completely gibberish and doesn’t mean anything But at the end, those images have been generated by a generator And it was trained on the MNIST database So over time, it’s able to recreate numbers Like here I see a 9 I see a 4, a zero Some of them don’t look like anything, but it’s getting closer and closer to being able to fake images Even for me, I would say that this is a nine that has been drawn by someone At some point, what’s going to happen is that if you train your model on adversarial examples, here you see it’s getting better and better Like here is the validation set error, which means it’s doing less and less error The green curve is the model that has been trained on adversarial images And when you show it adversarial examples, it’s doing less and less errors over time while the model that has been trained on standard images is systematically wrong with adversarial examples Here it’s really interesting With clean examples, you see that the green one that has been trained with adversarial examples is doing less mistakes than on clean examples than the model that has been trained on standard training What this means is that not only it’s able to recognize adversarial examples but it’s also getting better at recognizing normal images So no matter what you’re doing with your model, always train it on fake images Always do it, because not only it will make your performance better, but it will protect your model against attacks That’s the take home message All right So now you have a progression of what GAN are now able to do This tweet was tweeted by Ian Goodfellow at the beginning of this year In 2014, we were able to– those images are generated by machine learning Those images don’t exist At the beginning, we were black and white images a little bit blurred By 2018, we were able to create images that look, for me, uncanny, like human-like Even the expression of the face, the smile, it’s really surprising So there is a really cool website I hope I haven’t earned it because I would like to show it to you It’s called thispersondoesnotexist.com And it was created by Philip Wang who works at Uber

And he used a model that was created by [? NDIA. ?] And with this website, what they’re able to do is to generate fake images on the fly Like this person– I just loaded the website This person has never been generated before And every time I refresh a page, it’s going to calculate a new face every time And none of this person exists Some of them can be weird, but it’s really interesting to see how in the last five years, we’ve been able to generate those kind of images, Oh yeah So for the little story, when Phillip Wang created the thispersondoesnotexist.com website, it was a huge blast and everybody was talking about it on the news And so on Reddit, there is a bunch of kids that asked him if he could do the same thing with cats So he created this website called thiscatdoesnotexist.com But the training set was not– so the data set of images was not as clean as the human one so sometimes you get really cute cats like this one And sometimes you get cats that are really creepy like this one And I only played with thiscatdoesnotexist.com for a few minutes and I found really hilarious images of cats that have no sense They have six legs But they do crazy faces like that and it’s really cool Also because a data set of cats is a lot about memes, sometimes it creates a meme, which is like so weird like artificial intelligence get humor I don’t know So yeah I do encourage you to play with those websites They’re really cool There’s also this one, whichfaceisreal.com So what it shows you basically is two faces, one that has been generated by adversarial examples like GANs and another one is real So we’re going to play a little game Who here thinks that the image on the right is the real one? OK And the image on the left? OK So let’s see Yeah It’s the real one Let’s try another one Who thinks the image on the right is the real one? And now left OK Left twins And you’re correct again Well, nice, because usually I do this on conferences and I get 50% but you’re good OK So this is to show you how crazy it is now that we’re able to generate So it doesn’t only apply to images It’s because I show you this, it’s fun, right? But it really has very, very important applications in the real world today We call it– what was the name again? I’m sorry I forgot But what we can do now is not only we can ask neural networks to give us labels, like before, we used neural networks to give us very binary questions like, is this a cat, is this a dog, but now we can ask more complex questions to neural networks We can ask, draw me a cat We can ask draw me a person, a person’s face What it means is that if you have models that can learn from, let’s say blueprints of cars, and you have blueprints of cars that you know that cars are fast and you know the cars are not using that much fuel, you can ask questions to your model like draw me a car that would be ridiculously fast without using as much fuel, or if you have applications– like say you have a data set of chemical compounds And you know that some chemicals are during a certain disease and some of those are toxic, some others are not, then you can ask really important questions like, can you draw me a molecule that would cure this disease without being toxic And like here, only your imagination is the limit And it’s really, really crucial that we still do work with GANs today because they’re going to bring new answers to questions that we haven’t explored in the space of possibilities yet But artificial intelligence can help us and [INAUDIBLE] really on questions like this

And finally– oh yeah This was really recent It’s been published probably three weeks ago And it completely blew my mind And I didn’t really have the time to study it So I’m going to try to explain with the few things I have the time to read so far Yeah You don’t really need the sound So basically what they did with GANs is that– so you have a video sequence where you get the landmarks of a video And then you have a stream of images What they did is that they applied landmarks to a stream of image And even now you can apply landmarks to one image What it means is that you can apply this moving face, like this guy is talking in this video and he is moving, if you have only one picture, you can apply the movement of his moving face to a picture, which is really interesting And I don’t think I have time to show you the entire thing, but what is really interesting is that they applied this to things like– they make pictures speak You can apply landmarks from a video that you’ve registered to pictures that you don’t even need more pictures now if you have your model that has been trained And I think the coolest example that they showed is “The Mona Lisa.” It’s mesmerizing It’s fascinating And it’s scary, I think But yeah So I have the link of the article on those slides I’ll share it later on Twitter if you want if you’re interested But yeah, GANs are really– it’s an incredible technology It’s going really, really far now And if you’re interested in artificial intelligence, you should definitely check it a lot, because it’s going really, really, really fast And I’m almost done Yeah Last example of an actual adversarial example that works on humans and not on machines, it’s optical illusions So we’re not flawless We also have our own adversarial images So thank you very much for being here [APPLAUSE] [MUSIC PLAYING]