Artificial Social Intelligence

[MUSIC] Good morning everybody, welcome to the Microsoft Research Summer Workshop on Artificial Social Intelligence This is the first talk of the workshop and I’m going to introduce you to what the theme of the workshop is, which is artificial social intelligence So in this talk we will primarily look at three things One is, what is ASI, which will be the bulk of the talk Then we will talk about how to do ASI I don’t have frankly much clue about it too, and we will figure it out in the course of the workshop But maybe a little bit of idea about how to do ASI And finally, I will wrap up with some basic things that you should all know about this workshop What are the to dos, and what are the things you should not do, etc Now, let me be very clear that when I say that I’m going to talk about what is ASI I’m not going to give you a very formal definition of ASI That’s because I also don’t know, and I don’t think anybody has defined it yet So what we will do is we will start with some working definitions of ASI, and I will put out a bunch of questions that you can ponder on And probably through that we will come up with a idea of what ASI is And I’m sure by the end of this workshop, all of us will have much better idea about what ASI is Okay, so we all hear a lot about the rise of AI these days, right? Now AI has achieved human parity in speech processing In vision, AI can even beat humans, so we see AI infiltrating almost every sphere of human life And especially, I would like to highlight deep learning, because deep learning is the underlying technology which has made a lot of these things possible And people think that deep learning is going to revolutionize AI, and the world in which we live in And a lot of people are even worried that we will not have jobs, and all the jobs will be taken by robots, and what would people do? Bill Gates has been saying that, okay, then robots should be pay taxes, and all these kinds of things But well, how much of this is true and how much of this is just noise? Now, there was this very interesting article recently, earlier this year, in Business Insider The title of the article says it all It says robots are coming, but you, meaning humans, still have an edge over them Now what is this edge over them? Let’s see, so this article actually takes out a lot of human capabilities So they are primarily divided into, let’s say, sensory capabilities, cognitive capabilities Natural language capabilities, social and emotional capabilities, and physical capabilities And further broken down into other tasks and capabilities that humans can do, and there is rating on how well AI can do it today So green means AI can do it as good as top quartile of humans, red means AI is below the median And yellow would mean that AI is right now not in the top quartile, but at least has achieved median performance as humans And let’s see, if you look at things like recognizing patterns and known categories And lots of physical capabilities like gross motor skills, navigation, etc It’s all green, so AI does pretty well, but where are the red blocks? Now if you see a bulk of the red blocks are here under social and emotional capabilities, now what does that mean? So let’s see, social and emotional sensing, so talking to a person and trying to gather the social as well as the emotional cues that the person is expressing Social and emotional reasoning, now to reason that, okay if you are unhappy then I should not make you more unhappy And I should tell you something which makes you happy kind of reasoning And then social and emotional output, and then of course, the AI also has to be socially and as well as emotionally expressive So these are all hard problems, they are all red, and then there is language, which is also red So language understanding, in its most general form is red Because even to date, there is no robot which can read, let’s say, a Wikipedia article, and understand everything that’s said here and make inferences We can do a lot, but not as well as we would have wanted to, or as a human can do And similarly, natural language generation, we are still better But it’s only the median performance, it cannot do that well

Now, if you think of it, the other problems which are hard are creativity, logical reasoning, and problem solving But if you are generating novel patterns and categories, now if you think of these things, creativity, generating novel patterns and categories, social and emotional sensing, etc What’s common to them? The thing that is common across all of them is the task of the AI system, the goal of the AI system is not well-defined If you remember the slides that I had, the previous slide where I had shown you what AI can do well Speech, vision, in all of this, there is a well-defined end goal In speech, we measure the word utter rate And if the word utter rate is low, the system is good In vision, the classification accuracy of the images, and if that’s high, we say the system is good Now thre’s a very well-defined metric, and according to that metric, we can mathematically model it as an objective function And whether it’s your deep learning system or SVMs or whatever classifier you want to use that optimizes for that metric The problem is when there are no metrics So what is creative? Even for humans, it’s a subjective decision What is a social and emotional sensing? If you are talking to me today, or maybe five people at the same time and all of them are looking at you, different people might get slightly different emotional and social cues of what you have said Whether you are happy, unhappy, whether you are being sarcastic, people may not agree on those decisions all the time So there’s a lot of subjective element on this, and this is why these are hard problems And interestingly, the right side of the chart also shows when do the researchers in the field think that AI is going to achieve the top quartile performance Now, this is the year from 2010 to 2080, and if you’ll see, for social and emotional reasoning, it is actually one of the hardest problems So the prediction is, sometime between 2040 and 2080, we will have AI systems which achieve the top quartile of human performance And if you see, across all these bars, these three are the hardest, which is social and emotional sensing So now comes the question then, what is social intelligence? Now, when we are calling artificial social intelligence There are many definitions, I have picked one which comes from the psychology literature It says, social intelligence is the capability to effectively navigate and negotiate complex social relationships and environments Now, there is an interesting social intelligence hypothesis, which takes it one step further And it says that complex socialization, such as politics, romance, family relationships, quarrels, collaboration, etc We’re the driving force behind this uniquely large human brain, and this large brain today helps us to navigate through the complex social circumstances In other words, one of the bulk of the intelligence that human possess is essentially coming from these complex social circumstances and the social intelligence that we possess So that gives a very central stage to social intelligence in the context of all types of intelligence And language, it goes without mention, plays a very vital role in this entire landscape Because it is language through which we communicate, we communicate our intent, we communicate our relationships and everything Of course, nonverbal cues are also extremely important, but language is a central piece of this puzzle And a lot of people say that if we can solve problems related to language and problems, let’s say, language understanding, then we can solve almost all problems in AI Or in other words, language NLP is an AI hard or AI complete problem So similarly, somebody could say ASI is an AI hard or AI complete problem I’m gonna play a video for you, it’s a futuristic video which shows how AI which is socio-culturally intelligent might look like >> [FOREIGN] [MUSIC] [FOREIGN] [MUSIC] [FOREIGN] [MUSIC] >> [FOREIGN] >> [FOREIGN] >> [FOREIGN] >> [FOREIGN] [FOREIGN] [MUSIC] [FOREIGN]

[MUSIC] >> [FOREIGN] >> [FOREIGN] [MUSIC] >> Now, the question is would you want such an AI companion? Does it really appeal to you? And also, the questions to think for is this video was in the context of Japan Now suppose, this was to be made in another context, let’s say India, or for another target audience instead of young adult, let’s say for children or for elderly people or for somebody who is not a professional Here it was a young male professional, but somebody else What aspect of the behavior of the bot should change according to the target audience and according to the social cultural norm of the particular geography we are talking about? Let’s say Japan versus India? Would it remain all the same or does it have to be different? If you think it has to be different, a lot of things need to be changed Then those are the things which we mean by artificial social intelligence or socio-culturally aware AI Or in other words, as Intelligent systems become more and more ubiquitous, they don’t only need to solve specific tasks But they should behave in a socio-culturally appropriate manner And it is the socio-culturally appropriate manner that we are trying to characterize here Now in this context it becomes important to mention one more very important as well as old concept of AI, a very fundamental concept of AI, which is the strong AI hypothesis and the weak AI hypothesis And I took the definitions by John Searle where he says, the strong AI hypothesis actually intends to build artificial intelligence systems that can think and have a mind So, as opposed to the weak AI hypothesis which says an artificially intelligence system can only act like it thinks, but it actually did not think or it acts like as if it has mind, but it actually doesn’t have a mind So there is a very strong difference between the two So in the weak hypothesis, the AI system can do almost everything that might say that it is actually thinking it has a mind, but we are not making any claims that it is having a mind or it is really thinking On the other hand, in the strong AI hypothesis, we are really making this assumption So, most of the AI today that we talk about is actually the weak AI hypothesis Nobody makes a claim that humans actually reason the way the machines are reasoning However, there is also this concept of artificial general intelligence, which takes the same idea, and says that an AI system or a machine that could successfully perform any task, any intellectual task that a human being can perform, is artificially generally intelligent Now, this is not necessarily linked to strong AI or weak AI An AGI system could still be a weak AI, because it’s not saying that it has to have a mind or think like a human But at least it can solve almost all tasks and it’s actually, until today, we don’t have any system that can do this

And most of the systems that we have are solving only specific tasks, like driving, self-driving cars can only drive Translation systems can only translate, a speech recognition system can only recognize speech There are bots which can talk to you, but they can only talk to you There is not a system which can do all kinds of intellectual tasks that a human being is normally capable of doing So we haven’t solved the AGI problem Nobody even, these days, talk much about the strong AI hypothesis and try to solve it Most of the time, we are in the weak Al realm and solve specific task, and ASI is, I would put it also under the weak AI hypothesis So we are not saying that bots are really socially intelligent or think the way a human would think in a social scenario, or they don’t have emotions We are not saying the bot would have emotions But the bot’s behavior would reflect as if it understands your emotion and it has emotion But really, it doesn’t say anything about whether the bot has emotion or not This is an important distinction I want all of you to keep in mind when we talk about ASI And ASI is a step forward to AGI Now finally, before getting into the details of how we can do AI or ASI, I would like to motivate it further using some more concrete ideas of what ASI could be And I will do it using what I call these Moonshot Projects So I’m gonna talk about three Moonshot Projects The first one, it’s called the Meeting Participant, and I have taken this idea from Professor Alan Black from CMU So he says, imagine that there is a participant in a meeting, which is a bot which is sitting along with all the other human participants So it could be the same scenario, where all of you are sitting and listening to a talk, and the bot is also sitting in this corner and listening to the talk Now, what happens when you want to ask a question? So typically, you’d raise your hand, and then I would look towards you or make a sign or just say, okay, do you have a question? And then you can answer, you stand up and talk, or you ask the question and comment And then I can engage in a conversation If somebody wants to pitch in now, they might just start talking just before you stop your sentence So humans know exactly when the other person is going to stop and the next person can start talking So all these dynamics happen very naturally and there’s a code of conduct For instance, in this case, raising your hand before asking a question, or not speaking when the other person is only midway through So all these are norms, social norms, and we learn them mostly by immersion Now, the bot also has to adhere to all this behavior If I crack a joke, most probably all of you will laugh, or I expect you to laugh even if it’s not funny, and many of you do that just because I expect it So the bot should do this as well So a participant in a meeting which adheres to the meeting norms can interact with other human beings in a very similar manner, as if it was just another participant So if we can achieve this, then that will be a great, so I’m not here talking about the content So when you ask a question, there’ll be a content of the question And that content might be relevant to the discussion or irrelevant to the discussion Even humans might say irrelevant things a lot of the time So that’s a different, important but different question But just these norms that we adhere to, giving the right cues, taking the right cues, laughing at jokes, raising a hand before asking questions, a bot should be able to do all these things So this is the first Moonshot Project The second one, which I call the Creative Partner, is courtesy my colleague Jacki O’Neill, who is an ethnographer And she has this idea that imagine there is a bot which can help you design So suppose you are a reporter or you are a designer And you have some facts in your mind, and you are going to write a report or design a poster, let’s say Now, the bot is like your collaborator Your reporter, your journalist friend, or your designer friend, or just another person who you are talking to And Would help you write your report or would help you design the poster Now, how does that happen in a human interaction context?

So you might say, okay, here are the facts and I want to write a report Should I start with putting the fact A before B or B before A? And your friend might ask, well, that depends on what is the punch line, do you want to emphasize on this point or that point? And you might say this point, and then your human collaborator might respond, well, then I think you should put B first and not A first Because if you put B first, then this will happen And the conversation evolves like this, and as the conversation evolves, the structure of the report also evolves And finally, you have a report, and you might be happy with it, or you might still keep working on it and it goes on I mean, there is no fixed end, but at least conceptually, we know when we are happy and satisfied and when we are not Now imagine, instead of a human, there’s a bot which does the entire conversation with you, so that would be a Creative Partner And the third Moonshot Project I’m gonna talk about is the Recommender Buddy Now, we all know about recommender systems, there are movie recommenders, product recommenders, they’re all over the place Every time you log into Amazon and try to shop something, they will recommend you books or clothes or whatever you are shopping Now, those recommendations only come as items And the only explanation is, people who bought A also bought what you are searching for, also bought the following things That’s the only explanation But is that how your friends recommend you? Most of the time, no, right? So suppose your friend is recommending a movie? So your fiend would say, this is a really nice movie I watched and I think you will like it, you should watch it And probably you would be asking, why did you like it, or why do you think I would like it, or what the movie is about And your friend might not say the entire plot of the movie, but tell the relevant things, which he or she thinks would make you like the movie Or it’s an action movie and you like action movies, so I think you might like it And this also has this particular actor, who you are a fan of, or it could be more complex than just the genre, or it might be a particular thing in the plot And the person might say okay, this particular sequence of instance that happens in the movie, I think you will be able to relate to it So that’s actually going one step further, it’s much harder But even giving basic explanations and trying to negotiate, so you might say, I’m not in a mood to watch an action movie, can you suggest something else for right now? I want to watch some light comedy And your friend might say, maybe then you then you should watch that movie which So this conversation, and coming up with a recommendation through the conversation So this is a Recommender Buddy Now, all of them have to have some sort of social and cultural intelligence so that they can negotiate with you, continue the conversation Now, what do you think is common to all of these cases? So the most important thing, I think, here is the goal How do you define whether the conversation or the entire interaction has been successful or not, is not well-defined The goal and the success matrix are not predefined It’s not a particular answer you are looking for So the Meeting Participant just participates in the meeting, and the success of the meeting is a very complex thing to define We may not even achieve a particular decision in a particular meeting, but that may not be the problem of the bot, that might be the problem of the humans But as a collection in which the bot participated, the bot might still be successful in its interaction So it’s very hard to define when we say that Also, conceptually, it’s not hard, but mathematically, it is hard to define what would be the final result of each of these So you might end up deciding that there are no good movies this week, and you don’t want to watch anything but rather go out on hiking That’s a good satisfactory conversation, and even reaching to that point is important But if we had to mathematically model that success is only when the user thinks I am now happy, I want to watch this movie So then such a metric would fail in such cases So it’s very hard, the goal is not defined, the goal as well as the success metric evolves along with the interaction So this is the first important point, but then, mind you, this is not about chatting about nothing So there are bots these days on social media with whom you can chat about nothing You can just keep checking, and it will say something back and you can say something back I have tried a lot of them, and they are pretty stupid, but still, people like them, I don’t know why But these systems are not like that, they are for a purpose, so

it’s not chatting about nothing The thing is, the goal is defined collaboratively between the human as well as the bot So this is, I think, one of the important aspects of this Moonshot Project And of course it kind of combines various aspects of social and cultural intelligence So these are very complex problems, and a lot of different social and emotional reasoning comes in when you are trying to solve this problem in its most general form So I hope by now you have a better understanding of what is SI, and what are the specific problems we are trying to solve From here, I will move on to how you can solve and build such systems We have an approach, I mean, it’s a very generic approach It doesn’t help take you anywhere, actually at the end, you will realize But it helps me tell you things in a little bit more structured way, so we call it the 3-DEE model So obviously the first thing you have to do if something has to behave in a socio-culturally appropriate way is to understand social norms What is important in a given context, in a given society, and to ensure that it’s conforming to that norms So if it’s a meeting, then you can start talking before the other person ends But if it’s a talk, you should not talk before raising your hand So these are social norms, let’s say, so it has to learn and know these social norms and apply it appropriately Secondly is, it has to understand social cues and respond with appropriate cues So for instance, when you are talking to Cortana or any digital assistant, it has to understand somehow your mood that whether you are happy This is very important in, let’s say, call centers, so apparently, when you talk to somebody in a call center, there are so-called good, what do you call them? Call center employees, and bad call center employees So the good ones are good in the sense they can understand whether you are a no-nonsense person, or whether you need those comforting words So the no no-nonsense thing, they will just say, okay, this is the solution, so go and do this And the comforting one might say, I’m so sorry this happened to you, it keeps happening, right, yeah, very bad, it happens to me as well And a lot of people just, even though the problem at the end doesn’t get solved, just because it was so comforting, at the end think, I had a very good experience with that particular call center So understanding cues, so what kind of personality you are, are you in an angry mood, good mood? What kind of mood you are, and taking those cues, analyzing them, and when responding, using, again, social cues is the other big goal So we wanna understand norms and then show conformity, we have to understand cues and respond with the appropriate cues, broadly And the way we can do it is using this 3-DEE model, which is, so first D is discover, so discover the norms Because it’s not obvious what the norms are, norms might change from society to society, and so on Then based on these norms, we have designed a system, and the system has to ensure conformity and respond with cues, etc So how do you design the system, of course, you have to build a system, because unless you build a system, you cannot evaluate it And the evaluate is the E part of it, which is probably the hardest in this So we will go through each of them, so let’s start with discover So discover is about identifying principles of socio-culturally appropriate behavior Now, how do you discover those, from where do you discover those? >> Depend on the quantity >> Yeah, what kind of behavior you want to observe, and there are five kinds of encounters So encounters of the first kind are between human and human, so that’s probably the first starting point, because that sets the social norms, how humans and humans interact Human-human society is also important, so human-human is mostly a dialogue kind of situation that I am talking about And human-human society is when one person is, let’s say, when a lot of people are interacting simultaneously, and how one person behaves with respect to rest of the other people Similarly, you can talk of human, machine, so how human, machines interact Machine and human-society is another possibility Machine, machine, you could also observe machine,

machine behavior if you think it’s useful [LAUGH] to our studies of socio-cultural norms? Will see if it is useful or not You can also study human-society to human-society, I am leaving it out right now because this will be like political science and other things, right? So, two complex things, so let’s not go there I didn’t talk about machine-societies, so you can think of human and machine-society, machine and machine-society interactions So those, also, let’s leave out for now, IoT people might say there is a machine-society, but let’s imagine, for now, we are just studying these five kinds And I will show you examples of studies in each of those, and how it has influenced the freight So let’s start with human, human, so human, human is the most obvious thing you go out and study And people who study these kind of interactions are called psychologists when it comes to general human behavior, and linguists when it comes to linguistic behavior There might be some other people also, but I just wrote fields which I’m more familiar with Now, that’s why there are linguists in this group, so how many of you are linguists here? So, we have quite a few, almost every team has one linguist The linguists are very important, other ways also, but particularly for this reason, that how you study human, human interaction And there are many methods of study, I want to just take two examples The first one is kind of my favorite, it’s a paper from 2011 dub, dub, dub, by, so, Sue Dumais and Michael Gamon are from MSR Redmond Michael lives in Redmond, and Cristian Danescu-Niculescu-Mizil, so it’s a long last name, he’s in Cornell now, I think So they did this interesting study on, I think it was either social media, I’m trying to recall whether it was Reddit, probably it was Reddit And what it says, is Mark My Words, Linguistic Style Accommodation in Social Media So let’s try to understand what is first linguistic style accommodation, so what happens? Whenever we communicate with somebody we try to mimic that person’s linguistic style This is a very, very natural phenomenon That’s why when people go and live in the US, let’s say for long time, let’s say I’m Indian, going to US and living for sometime, will knowingly or unknowingly pick up the accent And that’s because he or she is talking to the people there and they try to match the accent Even I, I’m giving this talk here, and most of you here are Indians, so I’m talking like this If I was giving this talk in Europe or in the US, really naturally and unconsciously, I would probably talk in a little different tone, I mean accent It’s very natural to all human beings, it comes Now, what you match is not only accent, accent is one of them, even you match how do you say no, how do you say yes So, do you, say, do you nod your head like this for yes or like this, do you do this? What kind of gestures you do, what kind of words you use, like do you say no, nope, mm-hm So these things also, we try to mimic the other person, but we don’t do it always Can you guess when do we try to accommodate and when we don’t want to accommodate? Sometimes we just do the opposite of what the other person is doing A more concrete example would be, we know both English and Hindi, and I start talking in English You come back with a response in Hindi, and I respond in Hindi would be accommodating, I responding back in English would be non-accommodating So when do I accommodate and when do I do not accommodate? >> If you are feeling a bit, you’re hostile >> [CROSSTALK] Hostility, yes >> Hostility So, basically you accommodate when you are being friendly, and when you don’t want to be friendly, you want to make a social distance, you want to maintain a social distance, you don’t accommodate, right? So usually, and people pick up those cues unconsciously, so they will say that person was very rude Why rude? And they can’t put their finger on why rude, but something just did not work about that person So this is called linguistic accommodation What these guys studied is they looked at the logs of dialogs between people, and tried to see if people accommodate on certain aspects of Words like SQL choice, so how they say no, how they say, do you use but, do you use however, do you use nevertheless? Which, there are so many ways to connect to sentences with but,

and which one they use It turns out that if two people are conversing for a while, they try to mimic each other on those kind of conjunctions, the ways of saying affirmative or negative, responses, and stuff like that Now, do you want the machine to accommodate or not accommodate? And the answer, I think, is in general, you would like the machine to accommodate So if you are talking in code mixed English, then you want the machine to respond in code mixed English, that would be accommodation, but may not be always So for instance, I remember my son was talking to Siri the other day He keeps asking this question again, and again, and again, are you a boy or a girl? And, I don’t know whether it’s my ear hearing it that way or it’s really happening, but each time, I think Siri’s voice is getting angrier and angrier [LAUGH] So initially it said, I don’t believe in those categories, and then now it said, I don’t believe in those categories, something like that So you might want to do, I mean Siri might want to not accommodate or not be friendly or warm also So in this way, this is one fine example of how you can use data to identify what people typically accommodate with, and incorporate those in your system And this will give a very subtle but, I’m sure, some improvement in the user experience So the user may not be able to point out that this is why I like this system more, but you might see improvement that this is a friendly system Another example, this is not discover per se, this is after discovering, they have applied it So there’s this company called Mattersight, have you heard about it, anybody, okay So I was reading a book, it’s by on how human beings are going to evolve into gods in the future, so it talks a lot about AI and cyborgs and stuff like that And this is one of the examples it gives, so the author tries to argue that, well, it’s not really the case that, so, AI is going to take over is what his hypothesis is And he says, people say, okay, social things, AI can’t do, and he gives Mattersight as an example where social things AI can do So, Mattersight is a company, it’s a Chicago based company, I think it came our around 2013, 14 What it does is call center automation, I like their tag line, it’s the chemistry of conversation So what they typically do is, when somebody calls up a call center, it tries to analyze from the call automatically two things One is what kind of personality you are, of course, the personality types, they are not like extrovert, introvert and those kind of things They might have some other categories, I don’t know, but some classification of your personality type, and some classification of your mood Based on these two, they connect you to the right person who can take the call So there are some people who are experts at taking angry people, whereas some people, other employees who might be good at comforting you more, and stuff like that So there are different kinds of people, and it tries to connect to the right person And apparently so they have quite a few clients, and apparently it has shown marked difference in the user experience of using that particular call center Just by identifying personality type and mood, and connecting to the right person, that’s the only thing AI is doing here Okay, so let’s go to the next one,human-human society, so the fields that are important here are things like, again, psychology So earlier, social psychology, linguistics, social linguistics, and also ethnography, ethnography is a field which studies human culture So we’re not talking of individuals here, but how an entire culture works, and an individual with respect to how a culture works And we do have ethnographers also here, Jackie, but he’s not here right now And let me take two more examples here to explain what I mean by studying how social linguistics and ethnography, these kind of studies can help So the first study, this is our own study that we published last year in [INAUDIBLE] So we were looking at how people use language on Twitter, especially if they are multilingual Now it so turns out, of course, most of the Indians on Twitter tweet using English And that’s simply because, one, it’s easier to type English, I mean, for technological reason Two, English has a wider audience, you don’t know which Indian language to use because whichever you use will have

probably more restricted audience than English, and definitely not international audience But people do use Hindi also, I mean, Indian languages also, we looked at in the English bilinguals mostly and how they switch, how and when they switch language There’s this is very interesting hypothesis in linguistics which says that people switch to native language whenever they are expressing some emotion So we we were trying to see if a Tweet has sentimental content or emotional content or opinion, does it, I mean, is it true that it is more in Hindi versus if it is factual, it is in English? So we tried to do that study So we built Sentiment Analyzer and all these kinds of things, and it worked with reasonably good accuracy But we did not see such a pattern, we did not see that when it is a fact people are saying it in English versus when it is an opinion in Hindi, it’s all mixed up, almost mixed up But we discovered something really really different in the data and really strong signal that when the sentiment is negative, then people switch to native language When people have to swear, people switch to native language And it’s very very skewed, so 95% of the swearing, I mean, if you take the ratio of swear Tweets versus non-swear Tweets for Hindi, versus swear Tweets versus non-swear Tweets in English and compare those ratios, so it’s 5 is to 95 kind of thing So in Hindi, people swear much more often, I mean, and people also switch to Hindi to swear So these are very interesting sociolinguistic finding And when we try to correlate it with linguistic theories, it seems that because emotional words are overcharged when it is a native language And therefore, you’d try, if you want your swearing to be really impactful and hurt the other person, you’d like to use Hindi for swearing There is another theory which we didn’t test, which says that English is the aspirational language in India, and therefore probably English in mind we connect English more to positive things and therefore, so we didn’t check that But therefore, English is more associated with positive sentiment and stuff like that So, now how this study we can use to build a bot that connects to you more? Now imagine that you have a Cortana or a Siri which could code mix between English and Hindi Now this will tell you when Cortana should switch to Hindi versus when it should switch to English So, this is just one aspect, of course there are many such aspects So this study was done on Twitter as they said The second one, it’s again a very same study from the same guy, same group rather, Christian Danis and then Dan Zurawski and his team in Stanford And this one says something interesting So, communities behave differently, online communities behave very differently And in an earlier study, they showed that if you want to be a part of an online community for a long time, you have to adapt your language to the language of that community So they did this interesting study, this earlier study which they did on Reddit And they showed that some Reddit groups are really very exclusive, they don’t like random people coming and joining them But the way the Reddit system works, it’s difficult to prevent people from joining, most of it is open So people will come and comment, but then these people in the group, I think the group he studied was a beer group And good group talking about different types of beers and stuff like that And they said that people use so specific language that they can immediately find out who is an outsider And when somebody is an outsider nobody responds to their comments So it’s a ready in-group out-group kind of a behavior that they display So similarly a follow-up study here, they studied different groups So this is, I think, about babies, baby bumps, this is seahawks, this is very generic pics, and this is cooking So what this showed is all groups have some very specific words which only the group people use a lot So in babies the example is onesies So this is a word which is only used in that particular group And what happens is in each group the words evolve over time And to be a part of the group you have to catch up with the vocabulary The moment you fall short of catching up of the vocabulary, so the previous people have also observed it, they called it old members, no place for old members That was the title of the previous So, if you cannot catch up with the vocabulary, then you have to leave the group Now, the half-life of how often you have to

keep refreshing your lexicon depends on the group So, there are four groups here, cooking, seahawks, baby bums and pigs Which one do you think will have the vocabulary refreshing at the fastest rate? Answer is written here, but if you don’t read it Seahawks, so Seahawks, every season the vocabulary changes Because new players come in, new techniques come in, and they talk about different things And which one is the most stable probably, won’t change for years >> Cooking >> Cooking, so cooking doesn’t, so it’s very easy to be a part of cooking and be there for a very long time But not true for Seahawks kind of thing So this shows how, a society or group level dynamics work, against an individual And yeah, there are many more studies coming up thanks to social media >> [INAUDIBLE] >> See, as humans, if you are following Seahawks, then probably you know who are the players this season and all those things, right? So from a analysis perspective, when you look at the data you see this is evolving fast But from a human perspective, I mean, going and talking about Seahawks If I had watched them 15 years ago and haven’t watched for a while doesn’t make sense So then I have to catch up on my vocabulary But cooking, that’s not true was the point, basically Okay, human machine, so human machine is obvious The people who study human machine we call HCI people, human computer interaction That’s a very developed field, and lots and lots of work has been done here Any HCI people here? So we have one, we also have And, of course, almost all of you have worked on speech and language have to do a little bit of HCI Otherwise you can’t test your systems So the question here, I mean, what can we study, what principles can we discover by human machine interaction? We are talking of discovering principles, so we can see what actually people like Because we are, I mean, from the human, human and human, society study, we know what happens between humans And as a designer, I might project that and build a system which does that But there’s no guarantee that people will like the same behavior from a machine Because people know that machine is a machine and a human is a human So let’s say the code switching dynamics that I talked about people, and swearing in Hindi when they are switching language If Cortana did that, probably nobody would like Cortana, we don’t know unless we implement that and test Implement or do some other ways of testing, there are also other ways of testing systems without implementing them So the first work I wanted to talk about and probably will talk a lot more about it in her talk, I’m not sure So is my colleague here an HCI scientist So it’s a very recent work, so she studied what kinds of chat bots do young Indians prefer? So, how many of you have heard about Xiaoice? Okay, not many, so it’s a chat board that Microsoft launched in China It chats about nothing, really literally it chats about nothing So it’s like how are you? I am fine, how are you doing today? I am also fine, I am feeling a little tired today I mean if you ask it, who are you today? It will say something, tomorrow it might say something else Because it just looks at the data and learns patterns, looks at previous few turns And based on that gives a response to the next turn So if you have lots of chat logs you can build such a bot right? So it doesn’t have really a personality which persists over chat logs or over chat sessions or even in a single chat session However people still love it, I mean apparently there are people who fall in love with Xiaoice in China Now, the question that we were wondering here as kind of older people Because the people who love it are teenagers, so we are old that way We were thinking, what do these teenagers find attractive in these chat bots? What makes them attractive? What do they really want to see in a chat bot So did what is called a Wizard of Oz studies So she did not implement a chat bot Humans were chatting, but the people didn’t know it was humans So people thought it was a chat bot, which was chatting to them And there are three different chatbots, I mean, one is Maya,

one is Adine, what’s the third one? Evy, so all of them had a very different personality, so Maya was this chatbot who was very mother like, not mother like- >> [INAUDIBLE] >> Was it Maya, okay, the names I forgot, but one of them was very productivity oriented So, it was trying to help you answer questions which are like, which is a good language for coding NLP stuff? Or which system I should use for speech recognition, what is the best deep learning system, etc, etc So all kinds of technical questions and other kinds of questions, it was trying to answer, so that was one The other one was, what was the other one, this is Maya, Evy was more about comforting Emotional one, so it tried to give you emotional support, and say, it opened again And the third one was a jumpy bot, flirtatious bot which will change topic in every three or four turns, and will try to tease you, and do all these kind of things And people who participated was mostly, not teenagers, but I think age group of 20 to 30, kind of And then she conducted interviews to find out, she saw the chart logs, analyzed them, about what people, how it went, how long the conversations were, etc, which is the first signal of whether the bot was engaging The second thing she studied was, she did interviews and asked people, did you like it? And it turned out that at least in this context, most people liked the productive oriented chatbot And they said that because I can learn something from the bot It’s not just talking about nothing, but it’s very context specific, mind you I mean, if you change the, I mean move from India to, let’s say, Czech Republic, and from 20 to 30 user group to, let’s say, 15 to 20, the results might be completely different So what people want from such bots are, I mean, machines are not very clear The other one is also interesting, so this actually looks at human-human communication, so very recent paper, but it mediated So there’s this WeChat platform, so it looks at how people interact over the WeChat platform, and this study that there are, I mean, the title says it all So when people usually say hello, people use emojis a lot more, and when people say goodbye, they use text Similarly, there was another study, again in this year, which says, so they made two bots I mean, bots or handwritten conversations, doesn’t matter, absolutely same, asked people to judge which ones were from human-human, and which one was machine-human, the only thing that changed was the font Some of the fonts were Times New Roman, Calibri, and some were more machine-like font, like Lucida Console and stuff like that And it turns out that it’s 50% more chance of saying something is a machine generated conversation, if the font was, let’s say, Lucida Console So it’s very different, I mean, although we are talking of starting from human-human conversation, it’s really very different how things work when it comes to machines Okay, so this I just talked about, what about machine and human society, and in general, it’s a very, very interesting topic It has not been studied quite a lot, but I want to take just one case study, which you might have already heard about, which is the case of Tay So, how many of you have heard about Tay, the bot that, okay, not everybody, so let me tell you a little bit of a context about Tay So Microsoft, after the success of in China, Microsoft thought a similar bot can be launched in English They launched it on social media, Twitter, and what this bot did was talk to people, converse, and put out tweets there In probably 48 hours, the first two days, the bot turned completely racist It was saying the nastiest things you can imagine, which were very racist and all, and why it would do so? So the bot learned from what it saw, the data it saw, and lot of people were talking racist things to it, and therefore it picked it up and it started talking It became racist, so Microsoft had to take it off And then had to redesign and relaunch

Now, the question is, why would people talk racist things to a bot, right? And it was not one person talking racist thing to a bot, right? As a society, lot of people were talking So, it’s a very interesting psychology, social psychology People try to test the limits of it So, there are lots of things here, right? So one is Microsoft So there might be people who are anti-Microsoft, who are just trying to do something against it And they ganged up and beat something or there could be something just as simple as testing the limits of a artificial system So it comes to our society, the extra and artificial system, it’s also a very interesting thing A lot of follow-up studies have been done on why it happened, what happened today, and that has lots of interesting things I encourage you to go ahead and read There are, I mean, conference papers, Microsoft blogs, just written on why people did that, how it happened and why people did that What was the motive behind this? And lastly, machine-machine communication So it is not clear why machine-machine communication would help people, I mean help social ESI But it turns out it does, especially if you are looking at, >> Supposed two of the teams, you’ll make a chatbot You make one chatbot, you make another chatbot, from different data, different principles and all And we try to play them against each other and look at the logs And that gives a lot of interesting things to study And one, not quite but somewhat related example is if you know about generalized adversarial networks, right? So you are basically playing two systems against each other and both the systems get better over time So people have been doing these kinds of studies also, but I think this is, I mean more to develop systems rather than to discover principles, okay? So with that, let’s move on to the last bit of discover which is MetaGen data So one of the important thing, right, all these things work only when you have some solid methods to discover principle, and good set of data to identify those principles Now in ethnography, let’s see, there are these well-defined ways people do studies So people do use our studies with artifacts So if you already have a chatbot, you can ask people to play with the chatbot and you can see those interactions When you don’t have the artifacts you are trying to design, but you want to understand what kind of design is appropriate, So the Aero, Maya, AV, their example, what they really did is a Wizard of Oz study So you actually make a fake machine interface in the front, but actually a person is sitting behind and talking So these are also other methods And then you can do user studies through participate interviews and other things The problem with all of these are these are really intensive So you’ll need to get the users together, and you have to study each log separately You have to take interviews and spend a lot of time, probably just interviewing 50 users So if our users’ study is done on 50 users, it’s supposed to be a very large user study Usually it might be even less like 20 or 30 users, right? So this, on the other hand, gives you a very good idea about what is exactly happening, why people are doing the way they are behaving, right? So it has the depth in it but it can only be limited So such ways of discovering principles cannot probably work with thousands or tens of thousands of people The other problem is we know always that cultural constructs and these norms are very, very specific to cultures Now, if you just try to run it for each and every culture, imagine how hard it is going to be So you need alternative methods and thanks to social media, we have lots of human-human interaction in social media And probably more human-human interaction in social media than real world today, and we can study them But each of them come up with their own problems So for instance, you could look at Twitter, Reddit, WeChat, Facebook, but they are not the same You will see very different kinds of conversation, very different kinds of motive, intention behind conversation in each of these different media And then there is one more dimension to it which is whether you are allowed to script the data or not After you have downloaded the data, whether you are going to publish it or not So there are legalities and always, will talk in another, I mean talk, I think Wednesday or Thursday Yeah, Thursday, yeah Broadly speaking, as I said, so there are these two dimensions in which these datasets,

this comes from social media you can classify So when I’m calling it the privacy continuum, but privacy is an overloaded term So there are many meanings So what I mean is how private is the conversation? So on one hand, it’s like extremely personal conversation So which could be, your chat, Facebook Messenger logs, or chat logs, with only one person who is your friend, or somebody On the other extreme, it’s like all public You are just throwing it out, so things like blogs, probably, or Twitter Tweets are very very public things And depending on where in this continuum you are, you are gonna see very very different patterns, right? Private conversations working one way I mean whatever you’re going to say in a very private setup, are not necessarily the things which you’re gonna make public So this is one The other aspect is that of what we call the formal versus informal continuum, right? So if you look at social media, or indian real computer media communication, certain things people will use very formal language, and certain things people don’t care about formality So most of Facebook and all these things, are very informal SMS could be very informal, though actually it’s not always true that everything in Facebook is informal It depends on who your target audience is And if you have an office email conversation, right? if you look at the email threads from your office and all that, they might be much more formal And depends also again on the office So Microsoft it might be much less formal than let’s say Infosys, [LAUGH] which might be more formal, or government communication might be much more formal So it totally depends And it is very important to situate your data in this plane, that where exactly you are looking at, or what you wanna look at Okay, so now we’ve moved to the other two d’s, and I will go through them together, design and develop I am not going to spend time on develop, because develop is very very context specific, right? What you want to develop So mostly the design part of it So we already talked about we need these four blocks And we have talked about how we understand norms by looking at these different kinds of data sets Now, how do I ensure conformity understand queues, respond with queues are the questions, which you try to answer in the design thing, right? Now, the main fundamental things that we were talk about here, right, are first is socio-cultural So there are two basic ways you can do it One is, you could put socio-cultural norms as hardcoded constraints in your system So your system is, you discover them separately, put them as hard coded constraint, and your system is never going to behave otherwise So for instance in case of Tay, if Microsoft had anticipated that people are going to behave in this way, so they could have put rule that, okay, here is a way I discover what is racist And I am never going to be racist The problem there is though, I mean, okay, you are not racist, what if people talk about something else? Let’s say adult content And it just talks about adult content, and you just learn and talk about those things So you want to probably check that also To what extent you check those? So it’s a little tricky, but it’s probably slightly easier than this second approach, which is you could assume that the sociocultural norms could be imagined properties So you just make a machine learning system, that learns from data And you assume that it will learn the social cultural norms, appropriate behavior, because the data has appropriate behavior Unfortunately that’s not always true, right? The data may not have appropriate behavior There’s an example we already talked about But there are also other kinds of issues, for which data may not have the appropriate behavior For instance, there’s lot of noise about these systems, search engines like Google, or even Facebook The ads that are shown are highly discriminative So depending on your search query, you might be shown, the famous example I remember was that if you search for a black sounding name, you will see ads which are, is this person in prison? When this person got arrested kind of things that come up randomly Whereas, if you search for a white sounding name, so it happened with Google and somebody sued Google for that And unfortunately, I mean Google said we didn’t do anything I mean, it was not by design It was just how the data was, right? So there is this problem The other thing is certain signals in data might be very very skewed Because these norms might be very subtle, and you may not have lots of signals about the norm, to learn from the data So a whole new field which is coming up Very hot these days, it’s called fairness, accountability, and

transparency iin machine learning Now what they try to do is suppose you have some sensitive variable in the data Let’s say, I’m automatically trying to process resumes and trying to say, okay, these are the people whom you should interview So because, let’s say I’m a firm who take resumes and connect it to different companies and say okay, these are hiring firm, kind of thing These are the people you should look at Now, how do I learn such a thing? So typically I would see all the previous resumes with this company has evaluated, which one they have interviewed Those are my positive examples, negative is evaluated but never interviewed kind of thing, and learn something from there Now, it might sort on out, and it often turns out that a very simple variable like gender often is one of the high, good predictor in computer science So gender female means don’t interview Very, very unfortunate, but it’s true So that’s a very sensitive variable, we don’t want discrimination for gender So how do you make machine learning techniques fair? As in it learns that distribution, but without taking into account that sensitivity, so you can mask it and you can do other things Also, transparency and accountability, right? When you predict something, why did you predict it? If the machine learning has an explanation for that, that’s a very good thing to have With deep learning unfortunately, this is getting easier and easier I mean, how to come up with accountability and transparencies itself one of the big question in deep learning Okay, so I think Allan Stock is gonna cover a little bit about this which is well, how do you plan dialogues well? So, remember the shower wise example, this is all from data Whereas if you are really trying to complete a task, like in Cortana, which wants to book your ticket yet be social You might have to have a dialogue plan in place And how do you do error recovery, and all those things? How do you greet a person, everything has to be nicely formulated So those are the first method where you want to put the social culture norms as some sort of a constraint or design principle in the system itself, rather than let it learn from the data But one of the very, very, important question in all these, right, who is your end user? And when it comes to social culture principles and when ASI, without an end user you can’t even define what are the norms And the bigger problem here is end user cannot be, so there are several dimensions in which things interact So you can think of designing something for everybody, like speech recognizers that can recognize all accents Which is probably very hard or, I don’t know On the other hand, you can actually design speech recognizers which is for one individual It will work fantastically but that’s also logistically, probably a nightmare because you need data from each person, and separately train, and so on So similarly for any cultural norm, right? Most of them are not universal, every individual will have their own idiosyncrasy what they like So somebody said, right, I want to choose what kind of that both video that I showed People said, okay this is the one that I want So those kind of things also needs to be considered Now, what is the right point here where you want to design is not clear So whether you want to go with user groups, demographies or for a entire country And this is a very hard problem, for instance, when we design search engines, today, Bing goes with what is called geographical regions So they have just decided, okay, India is one geographical region And let’s say Europe is one geographical reason, US is one, like that But there’s no reason for doing that, right? You could have divided India into sub-parts also and better So this is a very hard problem, and of course it interacts with other dimensions, like I said, this is from the point of culture But imagine that even within a culture you have people with different personalities And I just said how in the meta site example how personality matters, right? So whether you divide instead of this, divide people by personality and serve different personalities differently, or other kinds of demographic variables Do you define different chatbots to serve to men and women? Maybe there is some commonality across gender or age group Younger versus older versus, so there are demographic variables that cut through this plane, and you have to consider Okay, and the last bit is evaluation, and it so turns out that this is the hardest problem Because you don’t have an end task

Here, right? Since there is no end task it’s very, very difficult to say what you really want And the only way to do it is all about user experience You do experiments and see whether the user is liking it Or you just deploy the system and see if the user is using it and things like that But how do these entangle the SI from other respects of the UX? So, why people like it, there will be so many aspects of the system, and you have to do very, very, systematic experiments I’m not going to talk anything more about this because I think this is pretty open about how do you evaluate Okay, so I’m gonna end u with showing two videos which is, I mean some interesting, implementations of a ASI kind of system So this one is an example So this is not the video So this actually takes us to cues, from visual and other kinds of gesture related cues and use this in the communication So, I will show an example of a similar thing that was developed in Microsoft research Look at her eyes how she’s watching That’s her visual field Its visual field >> Hi there Are you here looking for Zack? >> Yes >> Are you here for the 2 o’clock meeting with Zack? Sorry did you say you were here for the 2 o’clock meeting for Zack? >> Yes >> Is one of you John? >> Yes >> Sorry, I can’t tell who is speaking when you stand so close together Which one of you said they’re John? >> I did >> Right, hi, John, Zack is expecting you, will you be joining the meeting? So, sorry, will you be joining the meeting? >> Yes >> Alright, I’ve let Zack know you’ll be joining his meeting with John I’m sorry, I think Zack is running a little bit late I’m pretty sure he’s on his way Just to be safe, I’ll send him a note to let him know that you’re here Feel free to have a seat while you wait for him Guess I’ll see you later then Bye-bye >> Okay, so the system was built by Dan Moore who said Eric, my colleagues from [INAUDIBLE] Redmond and I guess Dan was your student Alan? Yeah So the thing you see here, right? Her facial expressions and then how she used the hesitations like so to attract the attention of the person, right? So most of the time these robots today would just say, okay blah blah blah blah, but it’s trying to track whether the people are attending to her and looking at her and then only after making sure, she would say something And the other example is that of, okay, you can see >> Morning Can I help you find something? >> Yes >> What room number are you looking for? >> Conference room 3,800 >> To get to conference room 3,800, go to end of this hallway, turn right and go straight down the hallway for a bit Conference room 3,800 will be the first room on your right >> Thank you >> You’re welcome >> Also where can I find John Doe? >> John Doe is in office number 40120 To get there, take the elevator up to the 4th floor, turn right when you exit the elevator and continue to the end of this hallway John’s office will be in that wing of the building >> Thanks >> No problem, bye, bye >> [LAUGH] Okay, so the interesting thing and interest and they fill you in and we go to Redmond and it just stands outside the elevator And we can interact, though it never responded to me, but I don’t know [LAUGH] So I tried various accents and everything, but it nevertheless responded But I think what’s interesting here is, did you see the gestures? Like when it says up, right like exactly like how human would do And that actually increases the understandability, intelligibility of what is being said and also the overall experience, right?

Okay, so now very quickly in five minutes I will tell you a little bit about the technical program of the school, I mean workshop So, this is what you should have done before the school so team assignment was done by us I hope you have done the other things, data set preparation and everything, so you are ready to go And the first week will be more lectures as you have seen in the schedule and lot of tutorials to three, three actually tutorials, and we will have fewer and fewer lectures as we go So second week, we’ll have approximately one lecture per day Third week, we’ll have two or three lectures overall and fourth week, nothing And we will have evaluation at the end of the thing, right? Yeah, so this is just, I mean we had said this earlier Also, everybody is expected to work primarily on this You can think of it as a hackathon, just that a longer one, a month long hackathon, because, of course, there’s a prize to be won at the end And students, if you think the faculty’s gonna take away all the prize, talk to me what you can get out of it [LAUGH] Okay, these are the lectures and so tomorrow, we will have two lectures, one on the Dialogue Systems and Chat Bots by Alan, and I mean I don’t need to emphasize why this is important I’ll talk about NLP & Social Media on 7th And this is mostly going to be about what kind of language processing you need to do on Social Media Data because a lot of you will be primarily using Social Media Data Then on 8th June, Indrani is gonna talk about User Studies and this is what you will need probably in you second or third week, because when you, if you have some system and if you are ready to evaluate how do designs such User Studies and all Kalika is going to talk about Data Ethics and Privacy So, which kind of data you can collect, what data can be shared publicly, what not, what are the different things of licences 9th June the whole day is Speech recognition day So Preethi and Sriram are going to do two lectures in the morning about how to do speech recognition and there’s a tutorial in the afternoon Then next week, 12th June there will be a talk by Seza, this will be over Skype on Computational Sociolinguistics, as I just said why this is so important And Kalika the next day on Multilingualism This is mostly the kind of things we do and even more broadly, what happens in multilingual societies, the languages in that Then Ritesh and I will talk about Computational Pragmatics Pragmatics is how do you infer things that are unsaid just by logical deduction? And then on 15th, we have a talk by Thamar on her bookwork, how you can take use from the text of the book in predicting whether the book will be successful of not Saptarshi is gonna talk about Social Media Analytics And, again, it will be useful, I mean this is not from the NLP side of things, this is more from the network side of things How do you analyze the network and model the network, etc This is also probably very useful for a lot of groups And, Sunayana is going to talk about a very, very, important thing which is Error Analysis So, when you have the systems ready and you do the User Studies, you see it impact us, what to do with those, right How do you quantify and characterize the others, and how do you use them to make your systems better There will be two mission learning talks by Prateek and Praneeth We didn’t keep any basic machine learning talk because I know that all of you are not aware of machine learning, probably, because you come from various backgrounds, but that’s not needed There are too many aspects of every project that you can work on The machine learning talks are primarily gonna focus on advanced topics like fairness, accountability, and transparency And then on 23rd, there’ll be talk by Jackie on Ethnography for AI Now this was all very special topic because I don’t know if anybody else or who else in the world works on such a topic So these are gonna be very, very special talks, initial talks, the first week are mostly things which you will be needing, but it’s not necessary that everything here is something that will come handy for your project, but that’s intentional We wanna give you a holistic view of the field, what is ASI and what are the different aspects and it’s very interdisciplinary as you can see So a little bit of everything So every morning probably there will be one talk And then in some afternoons, mostly this week, we will have some tutorials Today there will be an Azure tutorial which you all need to attend because you all need to use Azure So how you can use Azure, and then tomorrow there will be a tutorial [INAUDIBLE] how to build Chatbot Because that solves a common to some of the teams, you need probably to build something

And in ninth, we have Bipreethi and Sriram again on Kaldi, which is a tool for building speech recognition software Okay, some special things, so these are more important actually They may not be there in your schedule So, on 7th, from 11 to 12, we have something called Lab Sabha So Lab Sabha is when the entire lab comes and meets, and we introduce new people Since all of you are new to the lab, so all of you will have to introduce yourself So the way we are going to do it is, and I request the PIs for that, each team prepare a five minute presentation And give the slides in advance, so that we don’t need to change the slides or change laptops, etc So it’ll prepare, it’ll keep it there So the team can come one by one, introduce themselves, and the PI also introduces themselves As I said, you’ll have to say one funny thing about you, by which people can remember you, so think about it also And then 5 minutes talk, so maybe in 10 minutes, we’ll done with one team, maybe less than that, and similarly for four teams, so that’s 40 minutes, roughly So usually, there is a talk in the labs about technical talk, but this particular Lab Sabha, you are the guys who are giving the talk, so we’ll have four short talks On 19th in the morning, we will have a status update, so what you have done so far in the project, what are the pin points and all And so for that, I request each team to prepare a 20 minute presentation of that type You don’t have to be too formal, PowerPoint slides are okay If you don’t want to use them, that’s also fine So it’s going to be informal We just want to understand where you are And 29 June is the final presentation and the banquet So there, each team gets 30 minutes to present and plus 15 minutes for Q and A And we have a panel of experts, four or five people coming from other organizations And they will be our judges who will decide which project wins the prize So this is very, very super important and it’s important that you get something working by that, so that you can show and impress the judges And then we will have banquet, 30th is the last day So we haven’t kept anything on 30th, just wrap up, say, by two people, and we have some fun event wind down in the afternoon So this is broadly the thing, and you know most people here Yeah, that’s it, okay, thank you >> [APPLAUSE]