Interactional expertise and the imitation game

4 Interactional Expertise and the Imitation Game Robert Evans and Harry Collins Introduction Interactional expertise provides one solution to the problem of coordination created by the existence of different cultures. Though it is not the only resolution, it has particular relevance for social scientists as it justiies their own status as experts. Put another way, if there was no such thing as interactional expertise, interpretive sociology would be impossible unless social scientists more or less completely shared the physical experiences of those they research. But, to give one counterexample, a criminologist can succeed without irst committing crimes. Indeed, if shared practice were always a prerequisite for understanding, then not only would social science in general become impossible, but each of us would live in small private boxes of practice conceptually opaque to our collaborators, family members, and lovers. We call the expertise that bridges distinct practice through a deep sharing of discourse “interactional expertise.” Interactional expertise can be acquired by the usual techniques of social science ieldwork—participant observation and immersion in the discourse of a community. The strong claim is that immersion in the discourse is just as good as immersion in the practices, so long as the aim is competence in tasks in which practice is not required (Collins 2004, 2007; Giles 2006). In a trading zone, a person with interactional expertise could move smoothly between different social groups, “translating” the concerns of one into the language of the other and vice versa (Ribeiro 2007a, 2007b; Shrager 2007; Collins, Evans, and Gorman, this volume). Such abilities are what make the division of labor possible. The idea of interactional expertise is, therefore, important even though, in practice, separating the linguistic and practical elements of expertise is not easy. In what follows, we set out the idea in more detail before describing an experimental protocol L for investigating it empirically. Denning—The Innovator’s Way 8351_004.indd 53 5/24/2010 4:55:03 PM 54 Robert Evans and Harry Collins Periodic Table of Expertises The idea of interactional expertise is part of a more broad-ranging approach to the study of expertise known as studies of expertise and experience (SEE). First proposed in the paper “Third Wave of Science Studies” (Collins and Evans 2002) and subsequently developed as a book (Collins and Evans 2007) and illustrated in a special issue of Studies in History and Philosophy of Science (Collins 2007), the SEE approach sets out a normative approach to expertise that starts from the sociological axiom that knowledge is grounded in the life of a community. In short, expertise is the outcome of successful socialization. This emphasis on experience and socialization serves as the link between SEE and mainstream work in contemporary science and technology studies (STS), which we dubbed Wave Two. Like Wave Two STS, the sociological approach of SEE stresses the importance of socialization in the creation, transfer, and application of knowledge. This is important because tacit knowledge is an essential part of any social practice, including science, but the only known way of transferring tacit knowledge is by social interaction. In other words, without social interaction, tacit knowledge cannot be acquired and practice will invariably fail in new contexts. High-level expertises like interactional expertise therefore depend on mastering the tacit knowledge needed to speak a language luently and to respond to novel problems. This same emphasis on experience and socialization also marks the SEE perspective as different from Wave Two STS. Whereas Wave Two is primarily concerned with documenting how scientiic controversies and practice unfold over time, SEE is more concerned with intervening in real time to make a difference in the way scientiic controversies are understood as they happen. Put slightly differently, SEE is concerned with the difference between what we have dubbed the problem of legitimacy and the problem of extension (Collins and Evans 2002). Wave Two STS directs attention to the problem of legitimacy by showing how the boundary work of the scientiic community and their supporters serves to exclude those with other kinds of expertise and experience, and shows how the legitimacy of such decisions could be increased if more heterogeneous forms of participation were developed (Irwin 1995; Funtowicz and Ravetz 1993). In contrast, SEE is concerned with the emergent problem of extension, which is created by the inability of Wave Two to draw a boundary around those who might be counted as legitimate contributors to technical debate. By classifying and distinguishing between different types of L Denning—The Innovator’s Way 8351_004.indd 54 5/24/2010 4:55:03 PM Interactional Expertise and the Imitation Game 55 expertise, SEE aims to rescue the idea of expertise from the corrosive implications of STS; Wave Three is intended to show what it means to “know what you are talking about.” Three Types of Expertise Lest this all sound too abstract, imagine that a social scientist wants to investigate the nature of scientiic work in a particular ield. Assuming he or she starts as a novice, the researcher’s irst task will be to understand the ideas, institutions, and actors that make up the ield. He or she will read books and journal articles and may attend lectures, will probably go to conferences, and will deinitely want to speak to the scientists themselves. As the project develops, one marker of the researcher’s growing expertise will be the quality of these conversations with the scientists. Discussion of the science being researched will change from simple if rather stilted interviews to increasingly natural and engaging conversations. But increasing linguistic luency need not, and often will not, involve the development of practical skills. For example, it is unlikely that the social researcher will ever take an active role in the substantive research work of the science he or she studies by designing or running crucial experiments. Similarly, while the social scientist would be expected to publish in the journals of his or her own discipline, publishing papers in the ield being researched would be unusual and would certainly not be a requirement. These differences between novice, social science expert, and practitioner expert were formalized in the threefold categorization of specialist expertise developed in the paper that launched SEE (Collins and Evans 2002, 254): 1. No Expertise: That is the degree of expertise with which the ieldworker sets out; it is insuficient to conduct a sociological analysis or do quasi-participatory ieldwork. 2. Interactional Expertise: This means enough expertise to interact interestingly with participants and carry out a sociological analysis. 3. Contributory Expertise: This means enough expertise to contribute to the science of the ield being analyzed. Beginning to categorize expertise in this way leads to two important observations. First, because expertise is related to experience, and experiences vary, each expertise must have a distribution. Not everyone can be an expert in everything even if everyone is an expert in something. Second, because not everyone has the same experiences, there must be some domains in which at least some individuals have no expertise. Taken together, these two observations underpin both the problem of legitimacy (i.e., L Denning—The Innovator’s Way 8351_004.indd 55 5/24/2010 4:55:04 PM 56 Robert Evans and Harry Collins Table 4.1 Periodic Table of Expertises Ubiquitous Expertises Interactive Ability Relective Ability Dispositions Specialist Expertises Ubiquitous TACIT KNOWLEDGE Beer Mat Popular Knowledge Understanding Specialist TACIT KNOWLEDGE Primary Source Knowledge Interactional Expertise Contributory Expertise Polimorphic Mimeomorphic Metaexpertises External (Transmuted expertises) Metacriteria Ubiquitous Discrimination Credentials Internal (Non-transmuted expertises) Local Technical Discrimination Connoisseurship Experience Downward Referred Discrimination Expertise Track Record the need to recognize expertise where it exists) and the problem of extension (i.e., the need to recognize the absence of expertise where there is none). The Periodic Table of Expertises The three-stage model of no expertise, interactional expertise, and contributory expertise has been expanded into a “Periodic Table of Expertises” (see table 4.1; see also Collins and Evans 2007, esp. chs. 1 and 2). The top row of the Periodic Table indicates the ubiquitous expertises that every member of a society must possess in order to live in it. This is important because the sociological model of expertise presumes a community that shares a natural language within which individuals are socialized and where expertise can be generated, held, and shared. The second row identiies some dispositions or personal qualities that are necessary for gaining expertise but do not, in themselves, constitute an expertise. For example, in order to be socialized, one must interact with other members of that community, but being “easy to get along with” does not make you an expert in anything in particular. The next three rows are the most interesting. The row labeled “Specialist Expertises” L lists the different kinds of expertise that can be developed in any substantive domain. Starting at the left-hand side, the table identiies the irst and lowest level of specialist Denning—The Innovator’s Way 8351_004.indd 56 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 57 expertise as beer mat knowledge. This corresponds to knowing simple facts that feature in games like Trivial Pursuit or Who Wants to Be a Millionaire. The name itself derives from the fact that this kind of information can sometimes be found on the coasters provided in bars. The next level of expertise is popular understanding, which refers to the kind of expertise that can be developed by reading popular accounts of a particular domain. Unlike beer mat knowledge, in which facts are known but not interconnected, popular understanding begins to link different facts together. Examples of this genre include popular science magazines and books that aim to explain science to a general audience. The next step up is primary source knowledge. The difference here is that knowledge is now based on specialist publications like peer-reviewed journals that are written by practitioners for practitioners. Although there is a clear progression from beer mat knowledge to popular understanding to primary source knowledge, real specialist expertise requires more. Because all three competences mentioned so far can be acquired without speaking to a practitioner, they exclude the specialist tacit knowledge held by the expert community of practitioners. It is for this reason that interactional expertise and contributory expertise, both of which depend on socialization within the expert community, are qualitatively different. The difference between primary source knowledge, the highest form under the left heading (ubiquitous tacit knowledge), and interactional expertise, the irst form under the right heading (specialist tacit knowledge), is vital. The counterintuitive conclusion that SEE draws from this is that the difference between contributory and interactional expertise is not great in those many practical settings where experts interact through words rather than deeds, whereas, in the same circumstances, the difference between interactional expertise and primary source knowledge is crucial.1 The next row of the table identiies the types of meta-expertise that are a corollary of the socialization model outlined above. If it is impossible for everyone to be an expert in everything, then everyday life becomes a puzzle: how do we make choices about issues in which we have no particular expertise and where experts appear to disagree? The idea of meta-expertises provides the solution by recognizing that there are expertises about expertise that can be used in the absence of any specialist expertise. This does not mean such judgments are to be preferred to those of specialist experts; in many cases they would not be. Nevertheless, meta-expertises are necessary because without them we would be unable to make many of the choices we have to make to live in contemporary society. The different kinds of meta-expertise capture the different ways in which judgments L about experts can be made. As with the types of specialist expertise, there is a distinction between types of meta-expertise that use ubiquitous tacit knowledge and those Denning—The Innovator’s Way 8351_004.indd 57 5/24/2010 4:55:04 PM 58 Robert Evans and Harry Collins that use tacit knowledge acquired through some specialist training. These differences can be explained as follows: • External meta-expertises: These are judgments that are based on the widely shared ubiquitous tacit knowledge of everyday life. Discrimination refers to the ways in which experts and the organizations they represent are routinely judged by their look, demeanor, and reputation. The idea of local discrimination reminds us that some communities will have had additional dealings with particular experts or organizations before (e.g., because they live near a particular industrial site), and these experiences will shape their views even in the absence of any signiicant substantive expertise (e.g., previous promises may have been broken). In these cases, even in the absence of any specialist knowledge, local communities might reach different conclusions from those reached by more distant communities. • Internal meta-expertise: This category contains those judgments that require some appreciation of the criteria used by the experts they judge. Connoisseurship refers to expertise in consumption rather than production. For example, connoisseurs of wine would typically be familiar with the conventions and techniques of winemaking and, therefore, what counts as a good wine, even though they might not be winemakers themselves. Downward discrimination applies most readily to relatively settled areas of knowledge and refers to the ability to identify a mistake and, on this basis, to discount claims as being made by someone with recognizably less expertise. Finally, referred expertise highlights the ability to use experience acquired in one domain to make judgments in another. The most detailed example (Collins and Sanders 2007) refers to the way managers of large, multidisciplinary science projects use their experiences as bench scientists in other domains to do such things as set realistic targets, judge between competing claims, and recognize when something is “good enough” to be an acceptable solution even though other scientists might reject it because it is not the best possible solution. The inal row of the table identiies some criteria that might be used for identifying or choosing between experts. The least powerful criteria are qualiications because they exclude expertise based on experience. Track record is slightly better but also undervalues experience and is unavailable in any genuinely novel setting. Instead, given the emphasis on understanding expertise as social luency, the best indicator of expertise is experience, and the more extensive and recent it is, the better. Interactional Expertise Each of the categories of expertise set out in the Periodic Table of Expertises could L give rise to a research program in its own right, but, to date, interactional expertise Denning—The Innovator’s Way 8351_004.indd 58 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 59 has been subject to the most systematic research. Interactional expertise is interesting because it touches on both philosophical issues, such as the relationship between the body and language, and practical concerns, such as how to do interdisciplinary research. The philosophical signiicance of interactional expertise is the challenge it poses to the conventional distinction between explicit and embodied knowledge. In the Periodic Table, explicit knowledge is represented by everything up to primary source knowledge, while embodied knowledge is found within the category of contributory expertise. Interactional expertise has no place when explicit knowledge and embodied knowledge are seen as the only possibilities. For the phenomenologist, for example, if knowledge is not embodied, it must be explicit. SEE, on the other hand, implies that there are two different kinds of embodiment. Traditionally the language and conceptual structure of the world are taken to be a function of the physical characteristics of the human body (for a discussion of this point, see Selinger, Dreyfus, and Collins 2007). SEE does not challenge this claim, but sees it as a matter of the bodily form of the human species rather than the individual. According to SEE, an individual can gain the interactional expertise pertaining to a species, including all the relevant tacit knowledge, taken-for-granted assumptions, and so on, so long as he or she has only a minimal body—i.e., one that can do no more than hear and speak. In principle, a body can acquire the interactional expertise pertaining to seeing, feeling, and doing in the way that humans see, feel, and do, without itself being able to see and feel and do (Collins and Evans 2007, esp. ch. 3; see also Selinger, Dreyfus, and Collins 2007). If this were not true, then, as has been argued above, each of us would know only our private world of sensation. Thus, disabled people would not be able to understand the world of the able-bodied, and vice versa, and in each case their embodiment would be immediately apparent from their speech. Still worse, there would be no division of labor, no interdisciplinarity, and no hope of translation between the languages. The difference between explicit, interactional, and contributory expertise can be summed up by reworking the distinction between “talking the talk” and “walking the walk.” If “talking the talk” corresponds to primary source knowledge (knowing what has been said), and “walking the walk” corresponds to contributory expertise (actually being able to perform the task), then interactional expertise corresponds to “walking the talk”—that is, being able to use the language in novel settings in much the same way as a contributory expert might. Phrased this way, the deinition of interactional expertise also suggests its own empirical test: Can those who L have it really “walk the talk” or not? It is this ability that is tested by the Imitation Game. Denning—The Innovator’s Way 8351_004.indd 59 5/24/2010 4:55:04 PM 60 Robert Evans and Harry Collins The Imitation Game Method The Imitation Game originated as a parlor game in Victorian Britain. In this form, the game was played between a man, a woman, and a judge who could be either male or female. The man and the woman go to separate rooms and write answers to questions submitted by the judge. The challenge is for either the man or woman to pretend to be a member of the opposite gender while the other answers naturally. For example, the man might have to answer as if he were a woman, while the woman answered as herself. The task for the judge is to work out, just from the answers to the questions, who is the real woman and who is the man pretending to be a woman. The idea of the Imitation Game retains a contemporary relevance because of its use by Alan Turing as a deinition of machine intelligence (Turing 1950). Under the Turing Test protocol, one of the participants in the Imitation Game is replaced by a computer, and the challenge for the human judge is to work out which answers are produced by the real human and which by the computer. Turing claimed that if the computer succeeded in fooling the human judge for ive minutes or more, it should be deemed intelligent. There is a clear connection between the Imitation Game and the idea of interactional expertise. Given that the Imitation Game is based solely on sequences of questions and answers—i.e., it is purely linguistic—a person with interactional expertise should be indistinguishable from a person with contributory expertise. In contrast, a person without interactional expertise would not be able to reproduce the discourse of the contributory expert and so would be readily identiied. In other words, success in the Imitation Game can serve as an operational deinition of interactional expertise. We have developed a Web-based application that allows Imitation Games to be conducted over the Internet (details are available from http://www.cardiff.ac.uk/socsi/ expertise). Like most experiments, the Imitation Game is much more complicated in practice than it irst appears. The method is now briely summarized in a form that is meant to help those who want to run such games for themselves. Step One: Decide on the Topic The irst task is to decide on the topic and hypothesis. That is, do you expect that the person charged with pretending really has interactional expertise or not? If you think the person pretending has interactional expertise, then the hypothesis is that they should succeed in the Imitation Game and that the judge will be unable to tell who L is the “real” expert because both are experts (one contributory, one interactional). Denning—The Innovator’s Way 8351_004.indd 60 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 61 Over a series of experiments of this kind, you would expect judges to be right as often as they are wrong, because they are essentially guessing. This is called a chance condition run. If, on the other hand, you think the person doing the pretending is being asked to display an expertise they do not possess, then your hypothesis will be that the judge should be able to identify who is who because lack of expertise will be revealed in the answers of the person who is pretending. This is called an identify condition run. The two experimental conditions are summarized in igure 4.1. Chance conditions occur when the person claiming to have interactional expertise is (or is expected to be) well integrated with the domain they are being asked to mimic. In this case, interactional experts have lots of interaction with contributory experts, which provides the opportunity for them to learn the language of that community. In contrast, identify conditions are generated when the person who is pretending has little or no social contact with the other social group. In these cases, the would-be interactional experts have little or no interaction with the contributory experts and remain within their own communities. If possible, it is best to run experiments in which reversing the role of the participants produces a switch from the identify condition to the chance condition so that the outcome of the two conditions can be compared. = Contributory Experts and Judge Chance Condition = Potential Interactional Experts Identify Condition Figure 4.1 Chance/identify condition diagram. L Denning—The Innovator’s Way 8351_004.indd 61 5/24/2010 4:55:04 PM 62 Robert Evans and Harry Collins Step Two: Recruit Participants The next task is to recruit participants and perform the experiments. Again, the basic rules are simple, but the devil is in the details. In the ideal experiment both the judge and the person answering naturally will be contributory experts. This is to ensure that the test is as hard as possible for the person who is pretending. Following this rule, in the original gender-based Imitation Game, the man pretending to be a woman should have to fool a female judge rather than a male judge. It is best if none of the participants knows the identity of the others. That is, even if the participants are known to each other in real life, they should not know who is taking part in a speciic experiment. The reason for this is to make sure that judges cannot use information about the particular individuals involved to inform their guesses; thus, they will be forced to consider the general skills exhibited, such as familiarity with male or female culture. Using the Internet as the medium of exchange makes it possible for participants to be located at an arbitrary distance from each other, and this helps to avoid the problem of judges knowing who is who. On the other hand, it is vital that the research team can authenticate the identity of the participants, and some existing reports of false identities being successfully assumed in Internet chats or games are vitiated by the problem of knowing who is really who. It can be dificult to recruit participants from small minority groups. In such cases, identify condition runs (in which both the judge and one other participant has to be a member of the minority group) are dificult to organize. Step Three: Data Collection Once participants have been recruited, the experiment involves a series of questions and answers. The judge sends the same questions to each participant. When both participants have typed their answers, these answers appear, simultaneously, on the judge’s computer screen. The judge then makes a provisional judgment about who has given which answer and records his or her level of conidence in this judgment. The judge then sets the next question and the process continues until the judge is sure they have worked out who is who or feels they cannot make further progress. Developing the questions is the key to the Imitation Game and raises several interesting issues. For example, while the quasi-experimental nature of the experiment gives rise to concerns about demand characteristics and the need for standardization, the logic of the Imitation Game method leads in the opposite direction to conventional wisdom in the social sciences. In the case of demand characteristics, for example, the analogy with a “true” experiment is misleading as there is no direct manipulation L of an independent variable that demand characteristics might mask.2 Clearly the Denning—The Innovator’s Way 8351_004.indd 62 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 63 instructions given to the participants matter, but the researcher in the Imitation Game can be completely open about what is required as there is no intervention to hide. Researchers using the Imitation Game should, therefore, provide judges, contributory experts, and interactional experts with clear guidance about the role they are to play. Judges, for example, must know that one person is pretending and that one is answering naturally. Contributory experts must know they should answer naturally and (would-be) interactional experts need to know that they must pretend to be something they are not. For judges, the key requirement is that they know they are trying to identify a member of a social group rather than a speciic individual. In most cases this can be accomplished by ensuring that participants remain anonymous, although it may be useful to remind judges that they should not try to work out individual identities. Interactional experts, in contrast, can be advised that it might be useful to base their answers on a real person who has the relevant expertise or experience, if they know one. In this sense, the literature on demand characteristics (e.g., Orne 1962) supports the experimenter: to the extent that participants want to “do the right thing,” there is nothing in the Imitation Game method that prevents the researcher from explaining what “the right thing” is. In other words, the more perfectly the participants share the researcher’s understanding of what the experiment consists of, the better the results should be. Other kinds of advice can also be useful for both the judge and the interactional expert. For example, questions that require beer mat knowledge can prove problematic. These questions can often help judges discriminate between contributory experts and those without any expertise, but they do not, as a general rule, reveal much about the social experience of being a member of a particular culture. In these cases, the most practical response is to encourage participants to make better use of Internet search engines and other stocks of explicit knowledge to force judges to ask questions that touch upon depth of cultural integration rather than mere surface knowledge. Thus, judges should be reminded that, because the experiment is being run over the Internet, the person who is pretending can simply look up answers online; thus, a good strategy is to avoid questions that can be answered in this way. Similarly, the participants charged with pretending can be reminded that this strategy exists. In this way, participants can be directed away from questions that rely on explicit knowledge, however obscure, and toward those that require experience. The use of the Internet also points to another advantage of letting judges choose their own questions. While the temptation might be to try to standardize questions, the role of tacit knowledge in expertise implies that standardization comes with a L signiicant cost. The reason is that as questions and their answers become standardized Denning—The Innovator’s Way 8351_004.indd 63 5/24/2010 4:55:04 PM 64 Robert Evans and Harry Collins and explicit, they move down the ladder of specialist expertise and so become less discriminating. Letting judges choose their own questions gets round this problem by allowing judges to set their questions in the context of their own, up-to-date, lived experiences. In this sense, the Imitation Game becomes a formalized “ethno-method” in which participants, be they judges, contributory experts, or interactional experts, automatically adjust their contributions to relect the current state of knowledge within their respective communities. For all these reasons, the Imitation Game generates a range of data that can be used to explore similarity and difference across and within cultural groups. At present, the software developed for the Cardiff experiments automatically records questions, answers, levels of conidence, and the judge’s reasons for guessing one way rather than another. More data could also be collected, however. In most cases the experiment lasts about thirty minutes, but this includes several fairly lengthy waits. For this reason it is often useful to have researchers with the participants who can use these pauses to collect more detailed data about the strategies being used by the judge or the experiences participants are drawing on in order to compose their answers. Alternatively, the opportunity could be taken to record basic face-sheet data and other details that might help with interpretation and the contextualization of individual results. Step Four: Data Analysis When the series of experiments is complete, both quantitative and qualitative data are available. The quantitative data consists of the guesses and levels of conidence, and can be used to see if the hypothesized differences between the chance and identify conditions have emerged in practice. In the ideal scenario, the identify condition would show 100 percent correct guesses, all with very high conidences. In contrast, results from an ideal chance condition would show equal numbers of right and wrong guesses, with the proportion of “Don’t know” responses relecting participants’ willingness to make deinite judgments. Because of the variation in “Don’t know” responses, the fundamental measure of successful identiications is not the absolute number of correct guesses, which will be inluenced by the propensity of participants to make conident guesses, but the excess of right guesses over wrong guesses (i.e., “conident and correct” minus “conident and incorrect”) as a proportion of the total number of Imitation Games in that condition. This number, which we call the “identiication ratio” (IR), can then be used to compare results across topics and over time. In practice, things are never quite as clear-cut, but it is often quite easy to see that the two conditions are different, and so long as the sample size is large enough, these L differences are generally statistically signiicant. Denning—The Innovator’s Way 8351_004.indd 64 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 65 The qualitative data consists of the sequences of questions, answers, and judgments contained in the text captured by the software plus the ield notes and other records made as participants were interviewed and observed. This qualitative data can be used to answer a number of different questions. For example, looking at the topics used in the questions provides an insight into what judges think differentiate the two social groups, i.e., what it is that makes their group unique and what kind of insights will be beyond the capacity of the person who is pretending. In contrast, looking at the answers shows what kinds of knowledge are shared and how people with interactional expertise are able to reproduce the discourse associated with experiences they have never had. When complemented by interview and ield note data, these sequences of questions and answers provide powerful insights into the diversity of different cultures as well as how they overlap and intermingle. Even though its primary purpose is to investigate interactional expertise, the Imitation Game also allows exploration of some other categories in the Periodic Table of Expertises. For example, looking at the chance condition runs and the reasons judges give for choosing between apparently similar responses provides some insight into the strengths and limits of discrimination. This is because some chance condition runs do produce correct identiications and some produce very conident guesses. Examining how these guesses are made provides insights into folk theories of lying (e.g., Which are an indication of honesty: short answers or long answers?) and the ways in which minor discursive cues are built up into strong convictions. Boosting the Sample Size: Phase Two Experiments Running Imitation Games in real time can be logistically complex. For the ideal experiment you need three participants and three researchers to be available at the same time. Fortunately there is a way to gather additional data from the initial live runs: the transcripts from real-time runs can be collected and sent to new judges, so that new judgments can be made which will conirm or disconirm the live data. Using this approach, much larger data sets can be created and a larger number of judges recruited than if only real-time experiments are used, boosting statistical signiicance as well as subjective conidence in the validity of the outcomes. In addition, as the sample size increases, the sophistication of statistical analysis can increase as it becomes possible to distinguish between different subgroups. In a case where very few live runs have been conducted and these include systematic errors, there is a danger that a spurious result can be given spurious statistical signiicance because “secondary” judges, reading only the lawed transcripts, will be L in a position to make only the same judgments as the original judges. Secondary judges Denning—The Innovator’s Way 8351_004.indd 65 5/24/2010 4:55:04 PM 66 Robert Evans and Harry Collins could thus reinforce a “chance” judgment by being unable to make good guesses from the transcript of a poorly conducted initial interview. Likewise, they could reinforce an “identify” judgment by making the same correct guess which was the obvious outcome of a poor answer given by a respondent. In such cases, the problem of secondary judgments is akin to what psychologists call a “stacking effect.” Empirical Examples So far the Imitation Game has been used by the Cardiff group to investigate a number of minority groups with a view to demonstrating the idea of interactional expertise, testing for the possession of interactional expertise, and testing for membership of social groups. The best known of these is probably the Imitation Game involving Harry Collins and gravitational wave physicists, which has been featured in Nature and several Internet news sites (see Giles 2006; slate.com). In the experiment, Collins, who has been researching the gravitational wave physics community for almost three decades, and a gravitational wave physicist provided answers to questions posed by another gravitational wave physicist. The two sets of answers were then shown to a series of other gravitational wave physicists, who were asked if they could tell which answers came from Collins and which from the physicist. The result was that the physicists were unable to reliably identify Collins, demonstrating both that interactional expertise exists and that intensive social science ieldwork is one way in which it can be acquired. The other experiments that have been written up and published, this time in the peer-reviewed literature, are on color blindness and perfect pitch. The results have been described in detail elsewhere (Collins et al. 2006; Collins and Evans 2007), so what follows here is only a brief summary. Color blindness was chosen because the ability to distinguish between colors is taken for granted by the majority of the population, and everyday discourse relects this. There are, however, a small number of people (about 5 percent of the population) who are unable to distinguish between some or all colors and who are, therefore, classed as color-blind. The hypothesis is that, because the color-blind have been brought up and socialized within the society of color perceivers, their language will be indistinguishable from those with normal color vision. In the Imitation Game, the color-blind should succeed in fooling a judge because they will have interactional expertise in color language. If the roles were to be reversed, and a person with normal color vision were asked to pretend to be colorblind, we would expect them to fail. In this case, the hypothesis is that, because most color perceivers have little or no experience of the problems faced by the color-blind, L they will lack interactional expertise. Denning—The Innovator’s Way 8351_004.indd 66 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 67 In the case of perfect pitch, the roles are reversed. Perfect pitch is the ability to identify the musical note that corresponds to a sound, but, unlike normal color vision, perfect pitch is relatively rare, and so the reduced form of perception (“pitchblindness”) is the norm. In the Imitation Game we would thus expect people with perfect pitch to be able to mimic the absence of this ability. In contrast, if the roles were to be reversed and people without perfect pitch—the “pitch-blind”—were asked to mimic the discourse of perfect pitch, they would fail for the same reasons that normal color perceivers cannot pass as color-blind. Using these two groups created four different experimental conditions—two chance conditions and two identify conditions—based on whether or not the experience of the judge was expected to be shared by both of the other participants (see table 4.2). The chance conditions represent the “proof of concept” runs, for the hypothesis is that the pretender has the interactional expertise needed to successfully imitate having the target experience. The identify conditions act as a kind of experimental control by demonstrating that, where the socialization is absent, so too is the expertise. As detailed elsewhere, the results of these experiments were consistent with the hypothesis, and statistically signiicant differences were found between the chance and the identify conditions. Our conclusion is, therefore, that the results support the claim that interactional expertise exists: the color-blind can pass as color perceivers relatively easily because they have been immersed all their lives in the language of color. In contrast, the pitch-blind cannot pass as pitch perceivers because they have not been so immersed. In more recent work, which we hope to publish in due course, we have run Imitation Games on a number of other topics. These have included the extent to which the blind can pass as sighted, and vice versa; whether members of ethnic minority communities can pass as white, and vice versa; and whether or not gay and lesbian participants can pass as heterosexual, and vice versa. In addition, Theresa Schilhab, at Table 4.2 Expected Outcomes of Imitation Game Experiments Pretender is A: B: C: D: Color-blind Color perceiver Pitch perceiver Pitch-blind Imitates Imitates Imitates Imitates Target Expertise Expected Outcome Color-perceiving Color-blind Pitch-blind Pitch-perceiving Chance Identify Chance Identify Adapted from Collins and Evans 2007, 96. L Denning—The Innovator’s Way 8351_004.indd 67 5/24/2010 4:55:04 PM 68 Robert Evans and Harry Collins the Danish School of Education, has run Imitation Game experiments on whether midwives who have not given birth can pass either as lay mothers or midwives who have given birth. Taken together, this research suggests a bright future for the idea of interactional expertise and the Imitation Game method. Conclusions: Interactional Expertise in Action The Imitation Game experiments bear in two ways on the concept of interactional expertise and its relationship to the wider typology of trading zones set out in chapter 2. First, they demonstrate that “walking the talk” is indeed possible and that interactional expertise is an observable empirical phenomenon. The Imitation Games provide positive support for the idea of trading zones in general and for the development of fractionated trading zones in particular. The experiments also showed that nothing can be taken for granted. For every chance condition we created, there was a corresponding identify condition characterized by the absence of a shared language to carry disparate experiences across social boundaries. While there is nothing in principle that prevents the interactional expertise needed for one group to understand the other from developing, acquiring this expertise is dificult. In the case of the chance conditions we have studied, the participants with interactional expertise had all been immersed in the majority/contributory culture for periods measured in years, not months or days. Transferred into the context of interdisciplinary research, multiagency teams, or a multicultural society more generally, the clear implication is that those charged with creating mutual understanding must be alert to the scale of the problem and not underestimate the time and effort needed to understand the perspective of another. In the case of research funding, in particular, where interdisciplinarity is often encouraged, research projects will need to include speciic time periods for gaining the interactional expertise needed to understand each discipline’s concerns and problems. By demonstrating that interactional expertise is real, the Imitation Game experiments help us understand how many of our existing social institutions work. Peer review groups, advisory committees, and multidisciplinary teams of all sorts already function because their members are able to communicate with each other. What the idea of interactional expertise provides is an explanation for why these teams and groups can function that does not founder on either the absence of tacit knowledge that undermines any explanation based on formal knowledge alone or the need for actual experience that the phenomenological approach claims to be a necessity. By L being both laden with tacit knowledge and only minimally embodied, the idea of Denning—The Innovator’s Way 8351_004.indd 68 5/24/2010 4:55:04 PM Interactional Expertise and the Imitation Game 69 interactional expertise provides both the glue that binds social groups together and the lubricant that allows them to mix and mingle. Notes 1. The distinction between polimorphic and mimeomorphic actions, which appear beneath the row of types of specialist expertise, refers to the extent to which actions—in this case, expertise— can be mimicked by machines. Actions that can be reproduced by machines are said to be mimeomorphic. Actions that rely on an understanding of tacit social rules and cannot, therefore, be reproduced by machines are said to be polimorphic. The distinction is not particularly important in this context and is explained in more detail in Collins and Kusch 1998. 2. To the extent that there is such a variable, it is the socialization of participants prior to taking part in the experiments. References Collins, H. M. 2004. Interactional Expertise as a Third Kind of Knowledge. Phenomenology and the Cognitive Sciences 3 (2):125–143. Collins, Harry, ed. 2007. Case Studies of Expertise and Experience. Special issue of Studies in History and Philosophy of Science 38 (4). Collins, H. M., and Robert Evans. 2002. The Third Wave of Science Studies: Studies of Expertise and Experience. Social Studies of Science 32 (2):235–296. Collins, H. M., and Robert Evans. 2007. Rethinking Expertise. Chicago: University of Chicago Press. Collins, H. M., Robert Evans, Rodrigo Ribeiro, and Martin Hall. 2006. Experiments with Interactional Expertise. Studies in History and Philosophy of Science 37 (4):656–674. Collins, H. M., and M. Kusch. 1998. The Shape of Actions: What Humans and Machines Can Do. Cambridge, MA: MIT Press. Collins, H. M., and Gary Sanders. 2007. They Give You the Keys and Say “Drive It!” Managers, Referred Expertise, and Other Expertises. Studies in History and Philosophy of Science 38 (4):621–641. Funtowicz, Silvio O., and Jerome R. Ravetz. 1993. Science in the Post-Normal Age. Futures 25 (7):739–755. Giles, Jim. 2006. Sociologist Fools Physics Judges. Nature 442 (8). Irwin, Alan. 1995. Citizen Science: A Study of People, Expertise and Sustainable Development. London: Routledge. Orne, Martin T. 1962. On the Social Psychology of the Psychological Experiment: With Particular Reference to Demand Characteristics and Their Implications. American Psychologist 17:776–783. L Denning—The Innovator’s Way 8351_004.indd 69 5/24/2010 4:55:04 PM 70 Robert Evans and Harry Collins Ribeiro, Rodrigo. 2007a. Knowledge Transfer. Ph.D. dissertation, Cardiff University. Ribeiro, Rodrigo. 2007b. The Language Barrier as an Aid to Communication. Social Studies of Science 37 (4):561–584. Selinger, Evan, Hubert Dreyfus, and H. M. Collins. 2007. Interactional Expertise and Embodiment. Studies in History and Philosophy of Science 38 (4):722–740. Shrager, Jeff. 2007. The Evolution of BioBike: Community Adaptation of a Biocomputing Platform. Studies in History and Philosophy of Science 38 (4):642–656. Turing, A. M. 1950. Computing Machinery and Intelligence. Mind 49:433–460. L Denning—The Innovator’s Way 8351_004.indd 70 5/24/2010 4:55:04 PM