Keynote: Smart Enough to Work With Us? Foundations and Challenges for Teamwork-Enabled AI Systems

– Please join me in welcoming Barbara Grosz – Thanks very much for that wonderful introduction – Thanks very much for that wonderful introduction It’s really a great honor to be here giving a keynote at Faculty Summit It’s really a great honor to be here giving a keynote at Faculty Summit where the theme is focused on AI at the Edge It’s a time where it’s really important for those of us doing research in AI to think about what capabilities systems have, what our methods can do, what they can’t do, where the challenges are and what we may be doing in the future And for everyone in computer science to think about how we can design systems that actually do good in the world, and don’t do harm This is perhaps, this is Turing’s best-known conjecture about building machines with intelligence And by the way he equated intelligence with thinking He thought they were, he wasn’t talking about intelligence as being able to do research that gets a Nobel Prize So his most well-known conjectures are about what he called being able to participate well in the imitation game, now talked about as passing the Turing test That conjecture, that we would have machines able to do that by the year 2000 was not realized But the conjecture that you see on this slide was People now regularly talk about machines as though they were thinking Of course, they frequently also ask what were they thinking, when they do things that don’t make sense So about five years ago, when we were celebrating the 100th anniversary of Turing’s birth, I was asked to give a talk at a Turing symposium, and I was trying to figure out what I could say that was novel, because people have talked about Turing and written about Turing And I decided it would be interesting to think about if Turing were with us now, I mean there are people who live to 100 years If he were with us now, what would he be Just to the people watching the clock This clock is wrong I am not four minutes over So you might want to reset it OK They didn’t hear me I’ll try to keep to my time without the clock OK so, and of course no one knows what Turing would say They got it But I thought the world had changed a lot and I wanted to put a challenge to the AI community I hoped that other people would also do that, so I still welcome other challenges And this is the challenge that I came up with And it’s really important to notice that this is about working as a team member over the long term, not one-shot deals; not just one question, one answer, we can do that just fine And I also want to say, it says here that you won’t notice that it’s not human It’s not to say that the systems would be indistinguishable from humans, it’s fine to know you’re working with a computer system But it has to behave sensibly So I’ll give you an example from healthcare A smart partner that searches documents much faster than any human being could do, if it delivers sensible results, passes the test If it delivers nonsense it doesn’t OK why now? Because AI has really made it out into the world The focus in this meeting, the descriptions of the focus of this summit all make it very clear that we now have AI capabilities entering daily life So we really ought to think about how the systems we’re building are going to work with people Another thing that’s happened, as AI technologies have made it into the world, we see widespread investments in AI, we see lots of claims about them We see lots of fantasies, not only that you can let your car drive while you play games or watch movies Don’t do it We’re not there yet But also in healthcare, in elder care, in education And it is crucial as we think about this to think not simply about the systems, but about the people that they interact with And of course, we also have a lot of nightmares I will say the nightmares are not new People have been thinking about building machines, could a human being build a machine, create something that could think? That was like a person? For centuries It goes back at least to the Golem of Prague Certainly to Frankenstein And more recently we have things like the movie Ex Machina

So on the one hand, this android dream set a context for Turing’s conjecture On the other hand, it makes clear that there has, at least in western culture, it’s very different in Japan, by the way, they think of robots as cuddly And I’ll leave it as a question for you to think about the cultural part So my argument is, if we want to get away from the philosophical focus, and Turing was asking a philosophical question And away from the science fiction of machines gone mad to get real intelligent behavior, we need to think about building machines that augment our capabilities Not that replace us or replicate human intelligence So here’s my quip We know how to replicate human intelligence The process is known to be fun The product is babies Even when they grow up, they have only human intelligence We don’t need to build machines that have more of that, we need to build machines that complement that And I’m going to very quickly talk about five reasons why we want to aim for coordination and teamwork now The first two are about how times have changed since Turing Then about our responsibilities And then because we’re doing research, we want to do exciting research So that’s how people use computers In Turing’s day, one person, one machine Now we have many people, in many places, in many systems, working at many times There are several arguments from the cognitive sciences, I’ll just say a couple briefly Brain development We know that in the first two years of life if you don’t have social interaction, your brain is smaller and it generates less energy And you’re basically cooked for life if you don’t have that social interaction in time We also know that language learning depends on interpersonal interaction There’s some wonderful work by Patricia Kuhl at the University of Washington I’ll give you one example She looked at babies, American babies, no exposure to Mandarin Looking at the issue, so we’re all born, for those who don’t know, we’re all born able to recognize the phonemes of all languages in the world By the time we’re 12 months old, we know only the phonemes of the languages that we’ve been exposed to in those first 12 months Somewhere between eight and 10 months of age, English-speaking babies understand the difference between L and R, and Japanese-speaking babies have lost that distinction Many other examples She compared exposing babies between 8 and 10 months to American babies, to a speaker of Mandarin face-to-face and watching that same person on video The Mandarin phonemes were retained by the babies who had the face-to-face interaction, not by the ones watching the video OK what about engineering? Well, if we want to build good systems, we have to think about the people that we’re building systems for and how their different cognitive processes work I’ll come back to language later in the talk, but here’s an example to illustrate getting things wrong I asked my phone, I won’t say which one, where’s the nearest gas station? I’ve got a list of 16 I asked which ones were open, and it offered to search the web for the phrase which ones are open, which would have not have gotten me the answer I wanted OK I think it’s really important for all of us building systems to focus on the errors the systems make, rather than bragging about the great successes that they have, because the errors are telling us where there’s a mismatch with our expectations The same thing is true of Watson jeopardy, and I hear that is now proving true of deep learning systems with captioning of images So look for the errors to understand where there’s a mismatch Finally there’s an ethical argument for considering working with people Autonomous vehicles give a lot of promise, the potential is huge I’m very excited about them But they have to be able to work with us, otherwise they’re going to run into trouble and we’ve already seen many of that As wonderful as the cars are, because we don’t do what they expect But hey, we’re here first, we’re building them for us People are a pain in the neck How many people here have tried to do experiments with people? Really, we don’t behave very well But too bad You got to deal with them And finally, there are all of these really wonderful scientific challenges

So, let’s go for it I’ve listed some here There are many, many more The last one is actually about the challenge of designing experiments that let us know how people work OK So I want to shift now from the argument for thinking about this, to thinking about the research There are many domains in which this challenge arises, many kinds of teamwork I’m going to mention those and then I’m going to talk about some of my own work in developing foundations Theoretical foundations for modeling teamwork, and then putting those to use So education, health care, writing papers, driving We can think about groups of people and groups of systems, groups of people that systems help The last two on this slide, autonomous system, human collaborations and crowd work present really interesting challenges where we as people can teach machines to do better But we have to design them so that they’ll learn from us, and so that we’re teaching them things that are interesting, not making up, not doing boring things because they are stupid OK So shift to talking about some work There are several different theories of collaboration that have been developed I’m going to talk about mine because it has some features I think are really important But all of them have mostly the features I’m going to talk about So the first thing to note about building systems that can be team members is you have to design the team work from the start, and you’ll see that in the formalism that I show you And it’s partly because teamwork is not simply the sum of individual plans You can’t take my plan to do something and Eric’s plan to do something individually and just smoosh them together And you’ll see that now in the theory that I’m going to talk about So this is all in English It was actually developed as a formal theory in logic I’m going to talk about this as I give you these principles I want you to understand this is not just soft, blah, blah, blah But in fact, if any one of these principles doesn’t hold for a system, you can show it will fail at teamwork at some point in time So first we need each team member to commit to the team doing its job That commitment is framed as an intention, which means that you have to believe that the team can actually do the work You have to reach consensus on how you’re going to go about doing it, and on who you’re going to allocate subtasks to Now that doesn’t mean that everybody, in fact it importantly doesn’t mean that everybody has to know everything about what everyone else is doing But you need to know about the points of join, so not all the details Just with this much you can see that there is a requirement that the group have some decision making processes They know when they’ve come to an agreement, and that they have mutual belief of the decisions that they have So we all have to know what we’re all doing In addition, two things that I’m going to illustrate with some examples of research are that we have to commit to the tasks that we’re given And again, that commitment is an intention and you can’t commit to something you can’t do That you know you can’t do So our systems cannot commit to doing things they can’t do You also have to commit to each other’s success, which means that you have to try to, the systems have to be able to, when they act, think about whether they are helping us get our jobs done or interfering OK So that’s the delegation challenges We’re going to talk about three different roles for these theoretical frameworks First, system specification, then design guide, and then as an analytic framework And I hope that you’ll take away from this that not only are these roles, but for each of these roles there are different ways you could implement the system So you could use machine learning or Bayesian reasoning or gridding theory or logical rules It’s the properties that the theory says the system has to have that you need to get into the system within whatever implementation you’re going to use I want to read you a quote from Yao Shou in a recent CACM article, when he was talking about the usefulness of logically solid theories

He said, “This matters because it helps you build software that’s beautiful on the inside, and if the internal structure isn’t right, if it doesn’t cohere with how humans think of the world, you’ll never have a truly beautiful user experience.” So I hope you’ll keep that in mind OK, so first theory as system specification And I’m going to talk about the problem of interruption management So the shared plan principle here is that you have to commit to each other’s success So if you’re going to interrupt somebody, you ought to have information that’s valuable to them, to interrupt only when it’s useful How many of you have been interrupted by dialog boxes warning you of things like your battery dying in three hours that you don’t really care about now? OK So just go forward without me? There we go So there are two insights that we had about the interruption management problem, and these are insights from A.J Comar’s PhD dissertation work A.J is here now The first of these is that when we are collaborating, most of the time, most of what we’re doing is individual Remember, we break the tasks down into subtasks, we delegate to other people And if we take advantage of that in your composability we can get computational savings The second insight is that we have to have efficient ways of reasoning about each other’s activities, even when we don’t know in detail what other people are doing So let me look at the first This is work on seeing that the problem can decompose into individual actions and joint actions, and then A.J designed a formalism which he called nearly decomposable MDPs The idea is you’re maximizing a joint function, but you have independent, individual and joint action sets, and you decouple the transitions depending on whether there’s a dependency, when you need to communicate, or things are independent The result of this is that you get exponential efficiencies in computation time You can actually compute things you couldn’t compute, if you try to do it fully, as a fully coordinated system OK Second, the second problem… oops! This is running ahead of me OK I have this fear it’s on automatic advance OK So that’s an example where there’s a specification that you commit to each other’s success and therefore that you don’t interrupt when you don’t have something useful The second example I want to use is work on what we’ve called collaborative interfaces, and this is in an educational setting Here the domain challenge is we have lots of interesting, exploratory software for kids in schools to learn math and science and evidence shows that by exploring on their own, they learn more than just being lectured at The constraint that we get from shared plans in designing these interfaces, is that you can’t assign something to the computer system that it isn’t capable of doing So the task allocation, so we looked at, could we build interfaces that would help teachers where we leave with the teachers the things that they do best; so having broad knowledge of students and how to interact with them But assign to the computer recognizing patterns and what the students are doing And I’ll give you two examples of this On the left you see an exploratory system for teaching middle school students probability and statistics So what happens when you flip a coin? And on the right you see a system for teaching freshmen chemistry What we did in this work was to expand on plan recognition algorithms, the plan recognition algorithms that had been developed in AI up to this point were all algorithms that assumed that we were well-behaved Eighth grade children, given exploratory software, not only make mistakes sometimes, that’s part of it, so you have to look for things that are not right and don’t fit But they also like playing around and exploring in ways that maybe the designers of the system or the designers of the problems hadn’t thought So the plan recognition algorithm had to deal with those kinds of errant behaviors

And the result was that we developed new plan recognition algorithms So this is a place where taking on the challenge of building something that worked well with people, we actually came up with hard scientific problems These systems were both validated off line by through surveys of teachers and finding out how well they worked, and also proving correctness of algorithms One of the most interesting responses for me happened after the summit a few years ago There was a workshop on AI and education, and the chemistry faculty member at CMU who had developed virtual labs said the interesting thing for him was that he learned from the system how his students were solving problems and it wasn’t the way he thought they were So this is part of what a system can do to really help in education But notice, we’re not replacing the teacher; we’re complementing the teacher More recently, Coby Gaule, who’s at Ben-Gurion University who worked on S-Cast and then did virtual labs, has been working So this was plan recognition in a very different domain, teaching seventh grade geometry where the students are working in small groups, and they’re chatting with each other Again, the teacher can’t listen in on all the conversations, and the technical question was, could you detect critical moments in those student conversations and alert the teacher to groups that needed to be encouraged to do something else? You can’t do full natural language processing, it’s not just dialogue, it’s multiple students talking at once at once But they could use machine learning to detect those critical moments And they’ve just done some early classroom validations that show that when you are using the software, the teachers are actually intervening at better moments and the duration of interventions is more appropriate for what they are, because they’re not trying to do too many things at once OK So the third thing that I want to look at, my current research on healthcare coordination And this is where we use shared plans not for system design, at least not yet And not in a system specification way, but to look at what was going wrong in health care coordination So the setting is clinics for children with complex conditions This could be from genetic problems, it could be from birth problems, it could be from illness at an early age The children see somewhere between 10 and 15 care providers No human being can keep track of 14 other people Some of us try, but it’s really not possible So the domain challenge here is, could we build a computer system that would assist the care providers in keeping track of what each other is doing? I’ll just make a note that electronic heath records does nothing to help with this problem They may exacerbate it and I’ll give you an example in a minute So this little sketch just shows you how much communication goes on in different modalities among people on a typical team You can feel very sorry for the PCP in the middle, who is like the quarterback and worse with no technology to help him OK, what we learned from looking at the work that people were doing was that this was a very different kind of teamwork from the tightly coupled teamwork we had studied before and that our theories had originally covered, and that our plan recognition algorithms were working in So the first thing is the team is relatively flat The PCP might be in the middle of things, but he’s not in charge and he can’t dictate to other people what to do They’re also loosely coupled, which means that they’re like the nearly de-coupled plan Sorry I’m not pushing the button It’s just working It’s autonomous I don’t know how it became like that It is not collaborating well with me Like, I’m not there yet, and I’m not running out of time yet OK So the bottom line here is, we have a new kind of teamwork What we did before will not work So the first thing we did, having observed this by looking at what people,

the care providers are doing, by talking to Oh wait, I forgot this really great quote Sorry I have to go back to this, because it shows you one of the things that for me or the graduate student working on this learned from, asked the physicians that she interviewed was, was there ever a time that they couldn’t find the information they needed in the electronic health record Here’s my favorite example Answer: there isn’t an example when I wasn’t missing information And the parents say, we need to relay information back and forth So the parents, who aren’t medical experts, are trying to figure out what’s important to tell people We wanted to see if the system could help with that So here’s what shared plans would tell you about this setting First, if they’re going to reach consensus on the recipe, then we could provide support for people not just making up a care plan on a piece of paper that they put in a drawer or the electronic equivalent, but something that’s dynamic and active That in fact takes into account that as the child’s condition changes, the plans evolve We could support communication and coordination at the appropriate levels So figuring out who needs to know what, when In order to do this, we have to solve a really hard information sharing challenge, which is to figure out who needs to know what when you don’t have the complete plan information So not only don’t you, it’s not only that you don’t know exactly what somebody else is doing, so the PCP doesn’t know exactly what the GI doctor is doing, nor does the physical therapist, but you may not even know the full set of plans You may not even have the probabilistic recipe tree with a probability over them OK So what Ofra did was to look at this loosely coupled nature and recognize that it was complemented by the extended duration of the work that people were doing So it’s not one-shot deals, and you can take advantage of that to learn collaboration patterns over time And from those collaboration patterns, you can learn about task dependencies and task allocation, and you can reason about information sharing based on those learned collaboration patterns Well, this need is not only in healthcare, but also in various kinds of office work For example, when we write papers with each other, how many of you when you’re writing papers with three or four other people are sending e-mail and making phone calls and getting together? Right? As wonderful as track changes is in Google Docs So what she did was to implement the mutual influence potential representation in the MIP DOI algorithm for collaborative writing; compared giving all changes, which is what systems today do, with randomly filtering out some so you have a smaller load, with personalized filtering based on the MIP DOI algorithm The results, very briefly, were that not surprisingly, if you limit the number of edits you show people they’re more productive, because they’re looking at fewer things If you base the change, so this is what’s important about the algorithm If you base the change on what you’ve detected about interactions between the work that people are doing, then it’s judged more helpful by the writers and independently the results are judged higher quality by people looking at what comes out So this personalized change awareness makes a difference I’ll just note here that this work was really a wonderful blend of AI and HCI, and I think we’re at a time in the development of both fields where it’s crucial for them to work together They split apart a long time ago, and it’s really important to come together So now I want to take a step back and actually look at some of the lessons from the research that I’ve done over many decades And as a way of just summarizing where we are with respect to this collaborative work and also leading into some discussion of dialogues, because there’s so much going on in the dialogue and language world these days So the first thing, so I actually got into collaboration by doing research on dialogue The first thing was that we noticed that dialogues aren’t linear sequences,

they aren’t random collections, they’re actually structured, and you’ll see an example in a minute And the language that we speak indicates the boundaries and indicates discourse information At the root of this structuring of language is the intentions, the purposes of the people having the conversation And this is maybe the most important lesson for those of you who are working on dialogue systems It’s not just the streams of words It is the underlying purposes you need to understand to really participate in a dialogue OK We then moved in trying to, Candy Sidner and I, in trying to move from understanding this to understanding how to model those intentions And that’s when I became interested in collaboration, because you can’t represent intentional structure as plans of individual agents You have to, and that’s where shared plans came from And then there was a lot of, we did it for dialogue and then a mathematician friend of mine said to me, come on, two, do N And it turns out as any basketball coach will tell you, N is a lot harder than two I think there’s a big challenge here at the edge already with this work, which is how to integrate the principles from shared plans for dialogue and collaboration in a way that melds with the data driven ML approaches How are we going to bring these two together? Finally, from the collaboration to loosely coupled plans, I told you about that research There is here also a challenge, which is how do we do a broader challenge? How do we deal with all the different kinds of teamwork and coordination which have widely different information-sharing circumstances? And now, before I move on to the next thing I want to talk about, which is the ethics aspect of all of this, I want to read a quote from Zak Kohane who is very, very active in health informatics and a big advocate of using AI to understand and support diagnosis He has a mother who has a cardiac issue, and he was interviewed about this He said that on the one hand it’s really great to collect data and use AI for that On the other hand, there are family and friends And here’s the quote “AI may be good at chess and go, and developing expertise once reserved for doctors in arcane areas such as reading X-rays.” Radiologists might disagree about it being arcane, but anyway “But AI does not do well at understanding the wide world, at picking up mood or subtle signs of distress, at convincing a resistant human to listen to the doctor We don’t need AI for that We need a caring village.” So again, an argument for the complementarity of AI, building systems that complement us rather than thinking about systems that replace us or simply replicate us So now I want to turn to look at the question of ethics And I’m going to do that via an example of a child’s toy and dialogue Because as I said in the beginning, AI is out there in the world We’re using it all the time on phones and for search and in many other ways, and it’s even now entering toys for children And I think that really highlights the importance of anyone building a system with any AI in it Frankly I think anyone building a computer system that people are going to use for anything other than programming need to take into account So this is for those of you who didn’t see this article in the fall of 2015 Mattel announced a new Barbie doll that could talk, and Mattel advertised it as your daughter’s new best friend Here’s an example dialogue taken from the New York Times I teach a course on intelligent systems design and ethical challenges And the students in the course bought a Barbie doll, and found the 300 pages of scripts that the Barbie doll follows So there are many more examples that are even worse than this, but we’ll deal with this So Barbie knows this kid has a sister, and asks, what nice things does she do for you?

The child says nothing Barbie says, what’s the last nice thing? OK, Barbie hasn’t understood the word nothing Not in the script It’s this little word, negatives are hard, anyone who studies natural language knows that OK The child tries to answer, but it’s got another kind of negative word Destroyed it Barbie says, oh yeah Tell me more The kid gets a little annoyed I left that part out More than a little annoyed And Barbie says, have you told your sister lately how cool she is? At this point, I believe the child said, that’s it Barbie I forgot that part OK So here’s the thing This is not just awkward but it raises, I mean this is kind of funny for us, but the whole idea of having these kinds of interchanges with a kid raises some ethical concerns And I’m just going to talk about two of them, there were many more One is expectations that are violated with respect to dialogue behavior, and the fact that this doll as my philosophy student PhD student teaching fellow said, unremittingly pleasant, like no human being I’ve ever met Kind of this emotional equivalent of being too skinny OK So language learning The doll is designed for three to eight-year-old children They’re supposed to be learning language and how we have carry-on dialogues What they’re learning is you just ignore what the other person says, and go along with what you’re saying This is not exactly what we would like to teach three to eight-year-old children, boys or girls How about child development and friendship? So unremittingly pleasant How many of you are unremittingly pleasant? I didn’t think so Thank you for being honest You know, we don’t want a three-year-old to think she has to be unremittingly pleasant On the other hand, we do want a child to know that there’s a penalty to pay if you’re mean to somebody, and Barbie doesn’t care what the child says It just goes along with its script I’ll just make a note here that I have some worries about anybody who thinks we should create the Westworld environment, given that most of us would stop somebody beating a dog on the street Why do we want to teach people that it’s OK to do what goes on in Westworld? I don’t know Anyway All right, so there are other questions Questions about trust and creativity, but I won’t be able to go into those OK You can ask me about them later OK so it’s not just Barbie It’s any approach to dialogue that misses some of the key facts we know about human dialogue And one of those is that dialogues are structured by purpose They’re not just linear sequences So here’s one of my favorite examples of why that matters Most of us were taught that a pronoun referred to the last thing that it matched in number and gender The last thing that them in blue matches is number and gender is kids How many of you think that what this person means is that they put the kids away? How many of you, I just have to see if you can raise your hands How many of you think it’s something else? OK So here’s what actually went on in this dialogue There was some intonation John came by and left the groceries. Stop that you kids. And I put them away after he left There’s no question you didn’t have a question about what’s going on Now this is a narrative about groceries I have to say I was in the airport on Saturday, my plane looked like it would be canceled I called the MSR travel people I could have had exactly the same conversation but with it being about travel interrupted, by saying something to the noisy kids who were unhappy their plane might be canceled OK So finally, dialogue is not just adjacency pairs It’s Twitter conversations aside, but you know, they have their limitations It’s not just question-response pairs So just from looking at this, you can see some of the challenges for chat bots Open domains are going to be harder than closed domains because you have to deal with many more kinds of purposes Short dialogues that look just like question-answer pairs are much easier than when you get into a real dialogue OK

What are the roots of the ethical challenges here? Well we already saw missing dialogue capabilities It’s also the case that if you give people something, if they think a system can do one thing, they will generalize to its being able to do another So the last time I asked my phone where the nearest E.R was, I did this, I gave a bunch of talks for phi beta kappa traveling around the country It always could told me where the nearest E.R was When I asked where I can get a flu shot, it understood what I said, but this is answer gave me Actually two different answers I don’t know why I can’t figure this out More importantly, when I asked where can I go to get a sprained ankle treated, this is the answer I got Now it’s not so bad, really, to tell me how to treat the sprained ankle But it would have been really bad if this had been about a heart attack OK So we need to get it right We need to be clear about what the competence is and what it’s not And we need to take seriously this idea that we can’t build systems that commit to doing something they’re not capable of doing But there are many other ethical challenges that come up, and they affect us not just as individuals, but they can affect whole communities, they can affect the world at large And I have just some of them here that I hope people will think about Privacy and security is always brought up, but there’s also a place where there’s a clash of the incentives for business and social good And I’m really glad to see things like the ETHER effort here at Microsoft and the Partnership for AI, because it’s really crucial that companies think about this So we saw this clash in the concern for getting more clicks and selling more ads in the spread of fake news There are also questions of justice, and Eric brought up some of this in his talk about bias If you’re predicting the future based on the past, at the same time that you’re trying to change communities and lower crime, for example, then there’s a clash between this progressive cultural change you want and the predictions that you’re making Also, and this came up in a State Department workshop I chaired last fall There’s great concern about who owns the data And here we have data, it is not just Google and Facebook, by the way, it’s the cars that people are driving Who owns that data? How is it going to be shared? Who’s going to determine what regulations there are? And then there are questions of equality, and making sure that the benefits of the AI systems we develop are spread around the world and of building trust Well, there’s a much longer list The second thing I want to say is that when you think about AI and ethics, we should think not only about not doing harm, which is really important, but about doing good And there was recently a AAAI spring symposium in which people talked about many, many different areas including health care, including sustainability I was glad to learn that MSR has an effort on AI for Earth And so I think we really could all be thinking about, there’s just many, many aspects of life where we could be building systems that would be doing good, and not just worry about what we shouldn’t be building OK, so then one of the issues that people think about a lot, and I think we as AI people should think about as computer scientists, by the way I don’t think this is just an AI problem It’s a problem of all the technology that we’re building It is about jobs, and about whether we’re replacing people, augmenting people or complementing people So let me just give an example from the legal and policy arena, where we can think about things that already computers are really great at, like document retrieval and search Things that they can do adequately and well enough in some areas like drafting straightforward documents And things that at least, in the near-term future, they’re unlikely to do well, like producing a new persuasive foreign policy initiative So that’s the law environment, the policy environment Health care is similarly nuanced, as is education So the idea that we’re going to replace doctors Rather than thinking about how we can replace them to save money,

we could think about how to build systems that will enable them to do their jobs more effectively, just as we were doing with the teachers in the examples I gave you OK So these are all high level things I think there’s some things that as AI people, as technology people, as computer scientists, we should be thinking about right now because these are things we can directly control We can’t control foreign policy And that’s the question of the jobs now, not just in the future And here I’ve just given you two examples, one is crowd-sourced piecework OK, so people working, under what conditions? Who regulates those conditions and how? The second thing is as we think about designing systems that have some AI capabilities, we could think about, and we would think about them as complementing people rather than mostly replacing people And when they can’t do it they just fail, and then some poor person has to take over and make up for their failures, which happens to me all the time when I get an automated customer care agent We might think about figuring out a way to build a system that would help, let’s say those customer care agents do their jobs better So I would like people to think about that, because I think we can do better than we have done so far So far we have thought about, wow, could we replace the person, OK if it fails it goes to the person That makes the person’s job one of handling, trying to figure out what the other person, really the person who called in really wanted, and how to make them happy now that they’re extremely unhappy Rather than enabling a person to establish a relationship with somebody, and then get them an answer more quickly So let me turn to some general ethical principles People have asked about who should decide the values, and I think that’s a really important thing to leave with people, not machines I also want to urge people to stop thinking about and help those of us who are already working with the press and others, to get them to stop thinking about robots taking over, because it’s not happening anytime soon and it’s a distraction from issues that are important now So I personally think we have more to fear from dumb systems that somebody has told us are smart, than from intelligent systems that know their limits And I know that Eric talked earlier about unknowns and known unknowns I think this is really a system, a really important thing to aim for is a system that knows its limits So there are different ways for handling the ethical challenges that come up The one we hear about most is regulation and policy There’s also design We can design the systems differently, and that’s really the main message I want people to take away from here, is that we can design systems differently One of the things that many people are thinking about is, could we build in ethical reasoning capabilities? And that would be great, but it’s hard enough to build in reasoning capabilities, so I think this is a long term aspiration Certainly an interesting research area, but I’d like to see us focus on design right now And finally, when we put systems out in the world, we could accurately describe their capabilities Mattel should not have said the Barbie doll was going to be your daughter’s next best friend OK So I’m actually unclear whether this is my time or the total time OK – That’s the total time But we’ll make it work – OK All right So let me, this is my last major point I said earlier that teamwork capabilities had to be designed in from the start Ethics also has to be designed in from the start And to get it right in the future, we need to do two things One is those of us who teach need to integrate ethics into our curriculum We’re doing this at Harvard now We experimented this past spring with having a small ethics module in four courses We’re going to do six in the fall A quick example from my class They study the Facebook emotion contagion experiment on a Tuesday Lots of discussion of ethics, by the way These students, 150 students applied for a class of 30, so they really wanted to get in and they cared about ethics

Forty-eight hours later on Thursday, I gave an assignment and I did an in-class activity on designing a recommender system, including needing to list the characteristics of the users they would collect After they reported out, I asked how many of them had thought about the ethical consequences of the information they were collecting None of them, none Well actually, this year one Last year, none If we’re focused on efficient, effective design with all this stuff, you know efficient code, et cetera, et cetera and ethics isn’t on the list, it ain’t going to happen So that’s the second part Companies building systems need to make ethics part of the product design OK I will end by saying it’s not smart if it doesn’t work with us, and we want it pulling together, not pulling against us I want systems that make us feel smarter, not dumber And conjectures are really important Anybody who is early stage and wants to know how Chomsky and McCarthy discouraged me, can ask me later They told me what I was doing was impossible or useless And when we first did shared plans, people told us to think harder and we could do it as individual plans So if you’re young and you have a good idea, go with it We need people to take risks now Computers aren’t islands We’ve got to do this These are all the people that helped me, including the funders And I’m happy for questions – Thank you, Barbara We have time for a few questions There are people running around with a paddle, so if you have a question, please raise your hand And please say who you are and where you’re from – Hi Brent, I’m from UC Berkeley – Wait, wait Where are you? I didn’t even call on anyone yet – Over here – OK – Paddle 3 – Hi, I’m from UC Berkeley Thanks for the wonderful talk I had a question about foundations for teamwork You talked about how designing it from the start is really critical for success How do these other, higher order, social activities build into that foundation? What is sort of the status today? Things like competition, justice You mentioned ethics a little bit How does that factor in? – Well, that’s for people here and their students and post docs to start working on Number one over there – Hi My name is Richard Banks I run the design team in Microsoft Cambridge Lab, and I spent a long time with design students particularly in the interaction design discipline And I’m wondering, what way do you think we should be educating these students who will be forming the design part of a multi-disciplinary teams, in terms of both understanding AI as a technology, and starting to deal with some of the ethical issues that you were describing? – That’s terrific And I can tell you best by telling you a bit about my course, which is that it’s aimed at students who have had one programming course I co-teach it with somebody who is either a philosophy graduate student or next, this coming year, actually a philosophy faculty member I think it’s really, really important to have somebody who has deep expertise in ethics there with you Almost, and let me tell you about what we did at Harvard in the spring We had a recent PhD in ethics meet with faculty to talk about the systems, the courses, and what they were teaching and then find a particular example where ethics mattered So for example, in the HCI class, they looked at accessibility Many apps are not accessible to people who are blind or deaf, so that was an issue that they talked about Needless to say, in social networking, they talked about fake news So you can find, that’s what I would recommend There have got to be a lot of good people who are philosophers and who have written, who have studied ethics In fact, the Trolley problem comes out of Britain, by the way And really, so make it an integrated part of the design that they’re doing and the projects that they’re working on, but bring it So I’m a disbeliever in professional ethics I think that’s if you follow these rules, you won’t get sued You really, really have to go deeply to become a thinker about ethical issues – OK, thanks Number two – Hi, Gershon Dublon from MIT Media Lab So one of the last things you said was we need systems that make us feel smart

And I wondered if you could comment on this question of kind of, how long it should take to learn to use the system? How easy should they be? And I ask that because I mean, teamwork as an example, is not something that is necessarily something that we’re well-suited for at the outset We learn to work together over long periods of time And as we build these AI systems, what if it takes two years to learn how to work with them? How should we kind of think about that question? – OK So first on systems in general, I’ll just quote Don Norman, who said for those of you not in HCI, a really important person in the HCI community He said, I don’t need a manual to use my toaster Of course that might not be true anymore So, it’s true that as people we need to learn with respect to each team how to work with those team members I think it should take no longer to learn how to work with a team computer agents than it takes to learn with each other, and that depends on the people and whether they’ve, how much they’ve been parts of teams before or not But the systems, I think we should design to be the best of team players – OK, thanks The last two questions have been from Cambridge, and continuing that trend, I think question number three was from Cambridge as well No more Cambridge questions after this – Hi Other Cambridge, David Karger, MIT – Hi David – Hi And I guess alumnus of your 1988 AI class So you talked about the sort of foundational principles of collaboration, your theory of collaboration and such It struck me as interesting because you talked about sort of the need for consensus around goals and shared plans and things One of your colleagues, Madu Sudan, did some really interesting work a few years ago on communication with aliens, and sort of developed a theory about how you could actually carry out communication and say the communication was happening, even when the two parties had no shared goals, no shared understanding of the universe And I’m wondering whether that sort of, you know, how that relates to your arguments that you have to have these consensus goals? And bringing it closer to home, if you think about user studies, where subjects are often not aware of what they’re doing, or are even deceived about what their goals are How does that play into what you were talking about? – So, remember what I said is if the principles are violated, you don’t have real teamwork So the answer with Madu is to do it justice, is much longer, but he and I actually disagree on when you have communication and dialogue So, and I would have to look at the user studies to give you a particular answer, but it’s not saying, you can have things that look like teamwork happen accidentally But if you don’t have these conditions, I can take that group of people and whatever they’re trying to do and give you an example where it would fall apart and they would fail So that’s the point of the principles It’s not that you can’t. Like you can have a system that looks like, remember my point about errors? You can have a system that looks like it’s smart until it looks like it’s really dumb So you can have something that looks like it’s working like teamwork, and then it falls apart I can give you an example from the 1990s at Harvard I love this example Women Basketball The Ivy League always sends one team to the NCAAs Harvard went Their first match was with Stanford, because they were at the bottom and Stanford was at the top Stanford’s best player, I don’t know, tore her ACL or something Harvard won the game and it was played at Stanford Why? Because they had teamwork Ask their coach They could, Harvard could lose their best player Stanford couldn’t – OK, let’s take one last question here Number two – Hi, Melanie Mitchell, Portland State University – Hey – Hey You made the really great suggestion that products that have AI in them, like the Barbie example, should advertise their limitations – Yeah – But I’m wondering if you can comment a little bit more on how feasible that actually is, given that many of the systems that seem to be the most successful are the hardest to understand? – So first, let me say that a system knowing its own limits is one of the hardest problems I know of With respect to the Barbie doll, I’d like to get to the first step

of not being deceptive about the capabilities that a system has And the designers know So I didn’t say this with Barbie, but it’s one of my quips about it Most of us have known three-year-olds I don’t know a single three-year-old who follows a script I mean, does anybody? If you do, raise your hand, because I want to talk to you I know a lot of parents who would like to know how this happened They always, OK We don’t follow scripts That’s why some of these chat bots have trouble So I would just like them to say, look, this is a cute, it’s a, you know, for a normal kid in a healthy environment with great parents, it probably doesn’t matter The kid will throw it away after two days I would like to start with not exaggerating the capabilities – OK Let’s thank Barbara again for her stimulating talk, and she’ll be around for the next few days