Sensing defects: Collaborative seeing in engineering work

991919 SSS0010.1177/0306312721991919Social Studies of ScienceSargent et al. research-article2021 Article Sensing defects: Collaborative seeing in engineering work Social Studies of Science 1–19 © The Author(s) 2021 Article reuse guidelines: sagepub.com/journals-permissions https://doi.org/10.1177/0306312721991919 DOI: 10.1177/0306312721991919 journals.sagepub.com/home/sss Adam Sargent1 , Alexandra H Vinson2 and Reed Stevens3 Abstract This paper explores how professional engineers recognize and make sense of product defects in their everyday work. Such activities form a crucial, if often overlooked, part of professional engineering practice. By detecting, recognizing and repairing defects, engineers contribute to the creation of value and the optimization of production processes. Focusing on early-career engineers in an advanced steel mill in the United States, we demonstrate how learning specific ways of seeing and attending to defects take shape around the increasing automation of certain aspects of engineering work. Practices of sensing defects are embodied, necessitating disciplined eyes, ears, and hands, but they are also distributed across human and non-human actors. We argue that such an approach to technical work provides texture to the stark opposition between human and machine work that has emerged in debates around automation. Our approach to sensing defects suggests that such an opposition, with its focus on job loss or retention, misses the more nuanced ways in which humans and machines are conjoined in perceptual tasks. The effects of automation should be understood through such shifting configurations and the ways that they variously incorporate the perceptual practices of humans and machines. Keywords engineering work, perception, human-machine interaction, automation 1 The University of Chicago, Chicago, IL, USA University of Michigan, Ann Arbor, MI, USA 3 Northwestern University, Evanston, IL, USA 2 Corresponding author: Adam Sargent, Social Sciences Collegiate Division, The University of Chicago, 1116 E 59th St, Chicago, IL 60637, USA. Email: sargenta@uchicago.edu 2 Social Studies of Science 00(0) The buzz in the annealing furnace of Large Southern Steel Mill (LSSM) is deafening. Leah, an electrical engineer, and Harold, a technician, have to yell to be heard over the noise and the earplugs they wear (all names are pseudonyms unless otherwise noted). Leah pulls a notebook, a pen and a handheld device called a digital sniffer from her tool bag. She yells to Harold that she’s ‘going down’ before descending a narrow metal staircase. On the lower level she slips around a large tangle of pipes to get close to a long yellow one that is interrupted at regular intervals with short pieces of grey piping shaped like an accordion. She moves close to two flanges that join a baffled section to the rest of the pipe and, pulling out a roll of duct tape, wraps a piece of tape around one of the flanges. Poking a hole in the duct tape with her pen, she holds the digital sniffer up to the hole. After a few moments the device begins beeping and flashing a red light. Leah notes the reading on the device’s LCD screen and moves to write the information down in her notebook. As she’s doing this, Harold comes by holding a device called an ultrasonic wand that allows him to hear movement inside the pipes. Leah walks toward Harold, who shouts, ‘what’d you get?’ Leah says, ‘it had like 63%’ and Harold moves over to listen to the flange. After some time, Harold yells out to Leah that he’s picking up something. Harold points out a section of pipe near where Leah had been testing and says that he can hear a leak. Leah uses the digital sniffer but can’t get a reading. ‘Where’s – what’s coming out of it then?’ Harold asks, adding, ‘That’s why I wanted you over here’ cause I could smell it. It’s almost like air.’ Leah inspects the area again and turns back to Harold with a look of frustration on her face. Moving toward Harold, she says, ‘It has to be gas’, to which Harold replies, ‘It’s gotta be!’ and Leah repeats, ‘It’s gotta be.’ They both move on to checking other parts of the furnace, but Leah remains vigilant about the sensitivity of the sniffer. She double checks flanges to make sure that the sniffer is giving accurate readings. By the time she is finished, she has located three flanges that she is confident are leaking, which, she tells Harold on her way out of the furnace room, is better than the twelve leaks she found a week ago. Contained within this relatively short mundane work interaction is a complex set of perceptual practices that professional engineers engage in on a regular basis: the detection, identification and interpretation of defects in production processes. Because the engineers rely on their perceptual capacities, they are sensing defects. For our purposes, ‘defect’ is an ethnographic term that brings out similarities in tasks that engineers were given in which they had to identify, interpret and even repair aspects of a product or production process that detracted from the value or function of the finished product. As we will see, this notion of the defect shaped the contexts in which engineers engaged in their perceptual work. However, sensing defects does not refer only to the internal processes of the engineer, but rather highlights the complex human-machine configurations through which defects are perceived, identified and interpreted. For the engineers we studied, sensing defects always occurred across agglomerations of human and machine capacities. We argue that attending to the work of sensing defects can ultimately help us rethink processes of automation, focusing on how human labor may be displaced rather than merely replaced in such configurations. In taking perceptual practice as a key analytical framework for investigating engineering work, we draw on traditions in both science and technology studies (Alac̆, 2008; Sargent et al. 3 Baim, 2018; Daston and Galison, 2007; Latour, 1986; Lynch, 1985) and interaction analysis (Goodwin and Goodwin, 1996; Lindwall and Lymer, 2017; Stevens and Hall, 1998) that have argued for an approach to perception as a situated social practice rather than an individual biological fact. Defects are not natural, pre-given aspects of an environment or product; rather, defects emerge in relation to modes of what Stevens and Hall (1998) call ‘disciplined perception’, a term that draws attention both to the particular forms of perception that support claims to knowledge and expertise (Carr, 2010; Goodwin, 1994) as well as to the fact that these forms of perception are learned practices (Baim, 2018; Butler, 2018; Goodwin, 1997; Jasanoff, 1998; Vertesi, 2012). It was because engineers had learned particular ways of looking at and assessing steel products and production methods that certain perceptible phenomena became recognizable as defects. Disciplined perception saw the difference between a steel slab that was merely brittle and one that was defective. In engineers’ everyday work, detecting defects was more than just listening or noting a number on an LCD screen – defects emerged with and through learned modes of perception. In attending to perception, we contribute to research that has challenged popular images of engineering as primarily mental and rooted in calculation (Jocuns et al., 2008; Stevens et al., 2014; Suchman, 2000; Trevelyan, 2014). Approaching sensing defects as a form of ‘disciplined perception’ (Stevens and Hall, 1998) also draws attention to networks of tools, infrastructures and language within which properties of the environment take on certain types of significance (Giere and Moffatt, 2003; Hutchins, 1995). Here, mundane objects like figures, graphs and charts, as much as digital sniffers and ultrasonic wands, facilitate and shape cognitive aspects of practice (Goodwin, 1994; Lynch, 1985). In our analysis, we attend to the ways that the perceptual abilities of human and non-human actors are brought together in different constellations depending on the task at hand (Hutchins, 1995; Star and Strauss, 1999). This is not to downplay the significant differences in the affordances that particular technical objects bring to such constellations; rather, we mean to track such differences while also eschewing assumptions that automated sensing technologies come to replace the bodies and sensory capacities of engineers (Elish and Hwang, 2015; Helmreich, 2009; Lehman, 2018; Stacey and Suchman, 2012). Scholars of science and technology have long noted that scientists and engineers can form intimate and embodied connections through various remote sensing technologies. Thus, for example, hydrophones are used to render underwater sounds audible to the human ear (Helmreich, 2007), or machine sensors make the force of nuclear explosions perceivable to, but not annihilating of, the human body in underground tests (Masco, 2010). Here the sensorium of the scientist, or in our case the engineer, intermingles with technological objects that enhance human perceptual capacities so that they can hear, see and taste what is normally imperceptible. Yet there are crucial differences here. The hydrophone recreates underwater soundscapes, with locatable sounds, for the human ear, while the sensors in underground testing facilities transform detonation into graphs of nuclear explosions. In both cases material representations created by technological instruments become the focus of analysis (Alac̆, 2008; Latour and Woolgar, 1979), yet in the case of the production of graphs there is also a transposition of physical phenomena into a visual and mathematized mode (Lynch, 1988). When Leah uses the digital sniffer to ‘smell’ for gas, she is engaged in a visual practice of reading a mathematical 4 Social Studies of Science 00(0) representation of the air quality. Our cases focus on such transpositions of sensory data that allow engineers to sense defects at scales and in ways quite different from direct sensory experience. Our focus on how human perceptual practices are redistributed and transposed in new technological assemblages informs our approach to the increasing automation of engineering work. Much of the academic and popular debates around automation have focused on the critical question of how, and which, human jobs will be replaced by non-human machines and computer systems (Autor, 2015; Autor and Salomons, 2018; Brynjolfsson and McAfee, 2014). Our approach contributes to these studies by shifting focus from the politics of whose jobs are lost (Benjamin, 2019; Brussevich et al., 2019) to also consider the politics of work with increasingly automated systems. Other ethnographic analyses have urged us to focus on projects of automation as they unfold in particular industrial and organizational contexts (Birtchnell, 2018; Bissell, 2018; Shestakofsky, 2017; Zuboff, 1988). This work finds that corporate efforts at automation often overstate the human labor replaced by technical apparatus and find instead that human labor is displaced, transformed or reorganized rather than eliminated in the name of automation (Ekbia and Nardi, 2017; Hamid et al., 2017; Irani, 2015; Johnson, 1988). Our analysis of sensing defects contributes to an understanding of how automation shapes human labor by focusing on a kind of work that does not match the repetitive forms of work most easily replaced by technological systems. As in the case of Leah, certain aspects of sensing defects can be and are carried out by machines (e.g. the digital sniffer detects and measures the presence of gas) while others cannot (e.g. coordinating with Harold’s findings). At the same time, sensing defects produces value for corporations like Large Southern Steel Mill by eliminating defects from products and the production process. This particular type of work, which cannot be completely automated, allows us to see the ways that projects of automation transform work, distributing some tasks to machines while simultaneously refocusing the sensory practices of workers around value-producing activities. Our attention to the disciplined forms of perception involved in this work provides a nuanced account of how automation reconfigures work practices across and within jobs. It also contributes to studies of perception in technoscience by drawing attention to the ways that these practices are reconfigured as different elements of work are automated. In what follows we present an expanded case of sensing defects in the context of LSSM’s efforts to further automate steel production. We focus on a range of engineers, machines and other workers involved in pouring steel slabs from molten metal and in checking these slabs for various types of defects. The cases that we analyse are drawn from ethnographic fieldwork conducted by Alexandra Vinson and Pryce Davis as part of a larger research project on the ways that early-career engineers learn on the job. Over the course of five weeks in 2016, Vinson and Davis conducted fieldwork at LSSM. Using a combination of video-recording, semi-structured interviews and traditional field notes, we investigated workplace learning by accompanying a total of seven early career engineers as they did their daily work. Six of the seven engineers had been participants in a rotational program designed to allow new engineers to work in several departments before selecting an assignment in one department. In addition to these Sargent et al. 5 focal engineers, we conducted interviews and observations with more senior engineers and technicians who worked with and taught the focal engineers as they learned to do their jobs. We begin by setting the scene of Large Southern Steel Mill and the history of automation in this industry in order to contextualize the work interactions that we analyse. The following sections focus on different engineers as they learn to sense defects in steel slabs. We begin with the work of creating a daily quality report from various data sources. This work is at risk of being automated by the introduction of a camera system equipped with computer vision. While it may seem that this is a case of a job being lost, we show how the automation of defect detection actually provides data for, and becomes part of, other ways of sensing defects that intervene in production at larger scales. We argue that the effect of automation was precisely this shift in human work from the identification of defects to their prevention via the optimization of production, rather than the replacement of human workers. Locating LSSM and the work of sensing defects Our analysis of sensing defects centers on the quality department of LSSM’s melt shop, which was responsible for ensuring that the chemical and physical properties of each batch of steel matched the necessary standards for production and sale. The members of the quality team worked in a converted trailer, separate from but next to the melt shop itself. As the name suggests, the melt shop was where LSSM melted shipments of scrap metal in a large electric arc furnace, mixed the molten metal with other required additives and cast the molten steel into slabs. From the melt shop, steel slabs could either be sold directly to clients or sent on for further processing into coils. These coils, in turn, could be sold to clients or further processed to varying degrees, including custom cut pieces of stainless steel. All of these processes took place inside large industrial buildings that were differentiated by color schemes on the exterior walls that changed based on the functions they housed. These massive structures stood out prominently against the flat topography of the grounds. It was not only LSSM’s physical presence that was imposing, its influence on the local economy was equally large. The mill had recently come under new ownership which received coverage in the local news, as the new ownership brought with it the promise of new rounds of hiring. Despite this, during the time of our fieldwork LSSM operated at a loss, a fact that engineers mentioned on multiple occasions. Indeed, LSSM’s financial distress led it to eliminate the rotational program, causing four of our seven focal engineers to be laid off. While engineers were generally more insulated from job loss due to automation than technicians and operators, they were not immune to the increased precarity in the industry at large. The concern over LSSM’s profitability shaped the work of the quality team because effective quality control was crucial for allowing LSSM to enter new markets. The automotive industry, for example, was a lucrative new market for LSSM but had especially stringent quality specifications for steel. In this context, the detection, identification and making sense of defects had significance not only for the efficient production of steel but also for the continued commercial viability of LSSM. 6 Social Studies of Science 00(0) LSSM’s financial precarity is tied up with global transformations in the steel industry. Between 1950 and 1984, the US went from producing almost half of the world’s steel to producing less than 12% (Tarr, 1988: 175). This precipitous decline was due in part to a period of economic stagnation in the US during the 1970s, which ultimately led to the globalization of production and finance (Harvey, 1990). Falling global demand for steel was especially destructive to the large integrated steel mills of Pennsylvania and the Midwest and the unionized labor on which they relied. While large US steel producers were failing, production in Japan, South Korea, and later China increased, and US manufacturers began importing cheaper steel from these countries. By the early 2000s, many of the large steel producers had declared bankruptcy, shuttering mills and laying off large numbers of workers. At the beginning of the crisis in 1974, the US steel industry provided 512,000 jobs. This dropped to 399,000 in 1980 and further to 142,000 in 2015 (AISI, 2015; Tarr, 1988). This period of distress led to large-scale restructuring in the industry. Steel companies consolidated into large multinational corporations that bought up derelict facilities and invested in creating new, technologically advanced mills. The downturn also led to rounds of automation to enable increased labor productivity. This was an intensification of a longer history of the implementation of computer control systems in steel production that began in the late 1950s (Noble, 1986: 60). Taking advantage of emerging methods of melting metal (e.g. electric arc furnaces), computer-automated casting and communication technologies, new steel plants employed far fewer workers. The number of man-hours per finished ton of steel went from 10.1 in 1980 to 1.9 in 2014 (AISI, 2015: 4). The use of cutting-edge technologies to boost productivity and ensure precision steel products has become a key economic strategy globally, but especially in US and European steel mills, where it enables mills to stay competitive despite the higher cost of labor. In this context, much of the human labor involved in steel production has already been replaced, and further efforts at automation focus on optimizing the production process itself. Across its many departments LSSM employed just over 900 people. The mill was organized by production stage, including the melt shop, cold rolling works and finishing, as well as mill-wide functions such as maintenance, where Leah and Harold worked. Within these areas were different departments dedicated to specific machinery or functions. For example, in the melt shop different teams of operators and technicians operated the raw materials yard, where scrap metal and other materials were delivered and organized, the electric arc furnace, where the scrap was melted, the ladle station, where additions were made to the molten steel, and the caster, where molten steel was cast and cooled into steel slabs. Entry level operators, who worked on the mill floor, generally did not need any education above a high school diploma, although a BA and previous experience was preferred. Technicians often had or were in the process of getting college or vocational degrees. Indeed, a number of the engineers we spoke to in our research had begun working at LSSM as technicians directly after or during their education. Operators and technicians were overseen on the melt shop floor by process engineers who were responsible for coordinating operations with the directives of managers and engineers in relevant departments such as planning and quality. Sargent et al. 7 The melt shop’s quality department (hereafter simply ‘quality department’) was responsible for ensuring that the slabs cast in the melt shop met both internal standards set by LSSM and external standards set by ASTM International (formerly known as American Society for Testing Materials) for different grades of steel. These standards specified visible characteristics such as size, coloration and surface imperfections, as well as invisible qualities such as chemical composition and processing technique (especially the temperatures at which casting occurred). Monitoring these aspects of the steel meant that quality team engineers had to collect data from and coordinate with other departments further down the production stream. In part this was because many types of defects in the steel would not be discovered until the slab was further processed, at which point surface imperfections could be rolled into the steel, debris enclosed in the slab could be discovered when it was rolled into a thin coil, or chemical imbalances and missteps in the cooling process might cause the steel to become brittle, crack and tear. Of course, the further along the production process a slab went, the more costly it was to correct a defect, which often meant re-melting the metal and beginning again. Maintaining quality standards meant that the work of the quality department included both daily reports and troubleshooting as well as longer-term tasks aimed at improving quality control. Both of these practices required the sensing of defects. Daily tasks included the creation of a quality report that collected and analysed production data from entries made by operators and readings made by automated sensors on the machinery itself. The purpose of the report, which was discussed at the quality meeting each afternoon, was to collect data for monthly analyses of production and to ensure that all defective slabs had been properly dealt with. In addition to these tasks, engineers in the quality department also worked on what they called ‘projects’, which were generally longerterm tasks aimed at improving quality control through analysing and reducing variation in the production process. Many of these projects involved analysing historical production data to assess connections between different factors of production, such as whether molten steel flowed freely out of the ladle or not, and the rate of defects for that batch of steel. The findings of such projects were used to help LSSM redesign its production methods and achieve more stringent quality specifications. ‘Seeing’ with data During much of our fieldwork the task of compiling the daily quality report was done by Gwen, who was working at LSSM as a co-op student while she finished an engineering degree in materials science – a co-op is a relationship where an engineering student agrees to intern multiple times with the same company over the course of their undergraduate program with the high likelihood that they will be offered a job once they graduate. In part, Gwen had been given this task because of her relative lack of status in the quality department. The tedium of creating the daily quality report stemmed from the fact that the work largely consisted of scanning through spreadsheets, graphs, shift reports and other documents, and compiling the data into a master spreadsheet where she also recorded which slabs were likely to have defects and whether action should be taken to repair, scrap or crop the slabs. 8 Social Studies of Science 00(0) During the research period, Gwen was learning how to create the quality reports. Her relative lack of experience meant that the constituent elements of this practice were more visible as Gwen learned to order, interpret and collect the different forms of data involved in creating the quality report. Especially in our early visits, this work required consulting her notes and lists of allowable chemical tolerances, as well as developing new techniques of looking and attending to the lines on graphs, the numbers on spreadsheets and the language of reports. Moreover, sensing a defect through the data required a particular distribution of perceptual work. Key elements involved practiced modes of reading data, such as scanning columns and rows on a spreadsheet, and also the more difficult assessment of complex graphs. Technological mediation did not mean that the work of the engineer was distant and analytical. Rather, quality department engineers engaged the representations of steel slabs themselves – chemical compositions, graphical representations of pouring and photographs of slabs – as objects of perceptual engagement (Lynch, 1985). Rendering this data into information about defects in the batches of steel required Gwen to engage in a highly specific form of visual practice that she learned through repeated engagement with the data and guidance from more senior engineers (Goodwin, 1994; Stevens and Hall, 1998). This became especially clear when Gwen interpreted a complex graph that charted the change in multiple aspects of the casting process, including the position of the stopper rod that regulated how fast molten steel could enter the caster. Changes in the speed at which molten steel was cast could potentially cause air bubbles in a slab due to uneven flow of molten metal into the mold. Changes in the stopper position, along with other indicators, could also suggest that the sand plug used to keep the molten steel in the tundish had disintegrated and become mixed with the molten steel, creating inclusion defects in the slab that would only be exposed when it was rolled into a coil. Changes in the stopper rod position were plotted on a graph that was generated automatically by sensors in the casting machine and recorded by a proprietary software system that stored and displayed the data. The graph consisted of multiple brightly colored lines on an x–y axis. Above the graph was a legend indicating what the different colors referred to. By moving the cursor across the graph, Gwen was able to see the numerical value of the lines at any point. In the following exchange Gwen describes how she interprets the graphs to Vinson. 1. G: So, this shows me all this fun – all this important information [points with pinky finger to legend at top left corner of computer screen]. 2. And what I mostly look at is the stopper position [left index finger to column above graph], 3. the level change [left index finger taps a different column], 4. and the weight in the tundish [left index finger to a different column]. 5. I don’t even really look at that much [runs finger around other indicators in legend] 6. because that’s all within happy range. But the stopper position is very important [right hand grabs mouse]. 7. And so what I’m doing now is I’m holding my cursor [right finger moves up and down tracing the vertical height of the graph where the cursor is] Sargent et al. 9 8. at the – at a good spot. [(soft voice) there we g] [moves cursor left and right slightly]. 9. So, the blue is the slabs [touches the flat spots on a dark blue line that steps up across the graph with right finger]. 10. So because this is the first [moves cursor across an area of the graph], we’re always gonna grind the first [slab]. 11. So I can ignore it. [touches finger to the relevant area of the screen] 12. And I just go through and I watch the level change here. 13. Ahhh [closes window that has popped up on screen] 14. And I watch the level change–this light blue line [runs finger across a light blue line at the top of the graph]. 15. And I watch to make sure it doesn’t deviate [turns to camera and moves hands up and down as if drumming] 16. within a slab over 3 and over the entire thing more than 6. 17. Except for that first one. [points to lower left corner of screen] […….] 26. G: It isn’t moving a lot, so that’s good. And it ends at 3.3, that’s fine. 27. A: ‘Cause it looks pretty horizontal, that line 28. G: It looks pretty horizontal. And so this is actually a really good one Rather than interacting directly with the slabs, Gwen’s perceptual practice is focused on a visual representation of the slabs: the graph. As a visualization, the graph mathematizes the casting process, both eliminating excess information and transforming it (Lynch, 1988). On one hand, the graph leaves out information that a picture or visual inspection might yield, such as the smoothness of the surfaces of the slabs. On the other hand, the graph transforms the complex process of casting molten steel into a series of measurable indicators represented by different colored lines plotted on a grid. As Lynch (1988) argued for doctors examining brain scans, the visualizations, not the objects they represent, become the focus of attention. That is, when Gwen says that ‘this is actually a really good one’ (line 28), she is not necessarily referring to the slabs but to the lines of the graph (Mol and Law, 1994). As she learned to evaluate the graph, Gwen got a sense for which patterns of lines were normal and which were ‘a mess’. While a messy graph suggested something was wrong with the slabs, the explicit focus of attention was on the image itself rather than the object represented. At the same time, the distance between slab and representation is collapsed in these comments. That Gwen is dealing with a visualization of the steel slabs and not directly with the slabs does not make her practice of sensing defects any less embodied. In lines 10–17 Gwen describes a disciplined form of looking at the graph. The graph, like Leah’s digital sniffer, transposes the dynamic qualities of moving molten steel into a perceptible and quantified form, given the correct technique of looking. This practice incorporates LSSM standards, as when Gwen ignores the level change information that corresponds to the first slab because it is always sent to the grinder (lines 10 and 17). In this way Gwen comes to see the graph like a member of the quality team (Vertesi, 2012). While visualizations were the direct objects of Gwen’s perceptual activity, to ‘see’ the slabs through her data she had to connect these visualizations to different aspects of the 10 Social Studies of Science 00(0) slab. Some connections between indicators and defects had been sedimented into company policy that Gwen could learn without necessarily knowing what defects the data indicated. As she gained experience, she became more familiar with the phenomena that different data indicated. Yet the relationship between data and phenomena was itself constantly being intervened upon by other engineers in the quality department who were quite aware of the potential gaps between the slabs and their representation in the graphical data. Because of this, Gwen often had to calibrate her data-based perceptions to direct visual observation of the slabs. Gwen’s direct supervisor and the head of the quality department, a chemical engineer we call Will, carried out regular visual inspections of the slabs and coils. After one such inspection, Will returned to the office and had the following conversation with Gwen. 1 2 3 4 5 6 7 8 9 Will: I’m sending the first heat of {grade of steel} to all grind [holds out his iPhone with a picture of slabs] because they look like… G: [leaning close to phone] Yeah, that looks ugly W: Yeah, dog poop G: Which one is that? W: Uh .. Seventy One [shows Vinson phone] A: Ooh. W: So, uh … and look a little closer up … so all those little undulations in the surface could end up turning into laminations or other surface-type defects Here Will uses the photograph as a record of his direct visual inspection of the slabs, giving Gwen access to what the slabs ‘look like’ (line 2). She correctly evaluates the surfaces of the slabs as ‘ugly’ (line 3), and Will confirms and amplifies the judgement by comparing the slabs to ‘dog poop’ (line 4). In this way, Gwen and Will mutually align to the slabs as defective. A few turns later, Gwen asks Will if the undulations on the slabs look like what would be found on the initial slab of a heat before grinding. Will says this is not the case, but in posing the question Gwen works to associate physical phenomena with patterns that she can see on the graphs. This helps her to see slabs through their representations in data. At the same time, Gwen uses the visual information, and Will’s account of it, to see the data differently when she pulls up the graph of the heat that Will mentioned. 1 2 3 4 5 6 7 8 9 10 G: A: G: That’s the one he was just telling us looked like poo. [looking at graph] Y::eah, you can see. You can see. The pink? Well, [gestures to screen] look at that, [moves finger across graph on screen] the- that’s a huge spike afterwards. And then that’s still [moves cursor over another area of the graph], it’s not super high .. it’s not where we would grind it just because of that, but it’s still kind of high and then the [moves cursor back to far left of graph] mold lev – and then the stopper position … is – goes up and comes back down [moving cursor left to right] Sargent et al. 11 12 11 again, it’s not where we would necessarily do it just because of that, but when he got the pictures, it really did look like shit. Gwen confirms that she can ‘see’ (line 2) what Will had told her about in the data presented in the graph. In explaining the graph to Vinson, Gwen directs attention to different features of the lines on the graph, noting ‘spikes’ (line 5), ‘high’ values (lines 7–) and other movements (line 10). Her analysis of these forms as problematic hinges on having already been oriented to the pictures of the slabs. She notes that the numerical values themselves are not out of range (lines 7 and 11), but by coordinating with the evidence of direct observation Gwen is able to see the lines of the graph as indicating the presence of defects. Not only does Gwen’s work turn on a disciplined form of perception, it must also be coordinated with other forms of perception if the quality department is to accurately sense defects. Many types of defects are not visible on the surface of the slabs and thus are only perceptible via the kind of data-assisted perception that Gwen engages in. In addition, Gwen’s perceptual practices take shape within an organizational hierarchy in which she must align her work and interpretations with those of her bosses (Will) and mentors (senior engineers). Gwen and Will’s perceptual practices mutually inform each other as Will’s findings guide Gwen’s interpretations of aberrant data and as she fits her perceptual practices into the accepted standards of the quality department. From sensing to sensors In some ways, Gwen’s tedious labor in making the daily quality reports was something of an anachronism. Many of the other departments at LSSM used computer programs to automatically collect and tabulate the data for their quality reports. Indeed, Will had begun to look for an automation solution for the quality department as well. One element of this solution involved hiring a third-party contractor to install specialized cameras that would use thermal imaging and algorithms to detect surface defects on the slabs, in addition to recording length and width dimensions for each slab. Ultimately, this system would be able to inspect each slab and send automated messages to the operators so that slabs with surface defects could be routed to the grinder for resurfacing. Such automated sensory systems would seem to render the work of sensing defects redundant. We might therefore inquire into the jobs that could be eliminated with the installation of this camera and attendant software to automate the creation of the daily quality reports. However, while these are certainly important concerns, we suggest that a singular focus on the creation or elimination of jobs as a result of automation may blind us to a more subtle politics of automated technologies, one highlighted by an attention to the distributed networks of sensing defects. Such an analysis highlights how the particular activities making up a task are variously distributed across human and non-human actors. In this view, automation marks dynamic shifts in the distribution of such activities. At stake then, is not so much the existence, or not, of a particular job but rather the way that the addition of automated systems requires human labor even as it also transforms the quality of human jobs. The addition of a computer vision enabled camera would eliminate the need for certain elements of human work, such as visual inspection 12 Social Studies of Science 00(0) of slabs and interpretation of graphs to identify defects. Yet the automated system itself requires human labor to set up and would become infrastructure in the long-term projects of quality department engineers. The work of sensing defects is not removed from humans through automation; rather, its distributional network is transformed. Much of the work of enabling the automated system was done by Walter, a newly hired mechanical engineer. Walter had recently begun working in the melt shop and was eager to work on a project for the department. He described the goal of the camera project, while speaking with Vinson. 1 2 3 4 5 6 7 W: A: W: Eventually they want to make it all automatic where it kinda takes care of itself, so the computer says ‘here’s a problem’ and it automatically sends the- it tells the grinder there’s a problem. Um.. but in the first stages they’re going to have to kinda I think kinda work out the bugs and do it more manually = uh ..= =Gotcha= But yeah where I come to play in this is basically just mounting it, you know figure out something to hold this camera uh … where you place it. Here Walter describes a future in which a camera will be able to communicate directly with the computer system that Gwen used to create quality reports. This communication would completely automate the process of sensing a surface defect, from detection to documentation to reparative action. While the camera would not be able to detect defects in the chemical composition of the slabs, this detection presumably could be automated in other ways. Even this automated system required human input. First, as Walter notes (lines 3–4), the system must be properly calibrated, which would require a great deal of work on the part of the camera company. Second, the existing LSSM machinery must be adapted and new infrastructure added to accommodate the specialized equipment (lines 6–7). When complete, this work would create the infrastructure for automated surface defect detection and should be understood as part of the distributed network of sensing defects. Our usage of ‘infrastructure’ draws on Star’s (1999) formulation. She notes that infrastructure is fundamentally relational, so that what may be infrastructural for one actor is a topic for another. For Walter, the table that holds the camera is the topic of his work. In the context of sensing defects at LSSM, however, Walter’s work became infrastructural to the extent that it supported, and was folded into, an invisible background for the task. Whereas in the previous examples non-human actors provided the infrastructural backbone for sensing defects (e.g. presenting data, recording perceptions), in this case it was human labor that provided the initial infrastructural backbone for the sensory perception of machines. What is different about the case of automated defect sensing is the place of human and machine sensing, not their existence. For the computer vision system to work, cameras needed to be very precisely positioned at a particular distance from the slabs and also from a light source. Walter’s task was to design and procure fixtures to which the cameras could be mounted. During our research he was working on the design for a table to mount a camera underneath the rollers that received hot slabs as they came out of the caster. The process of creating a design for this table was a complex and emergent one that would itself be incredibly difficult to Sargent et al. 13 automate. Walter began by making a rough sketch of the area under the rollers and taking measurements to come up with an initial idea of how big the table needed to be. He then went out to LSSM’s scrap metal heap to see if there were any pieces of metal that could be used to make the table. With a clearer idea in mind Walter drew a more formal sketch of the area under the rollers and began to make an even more formal representation of the table design in AutoCAD, complete with specifications for the types of metal to be used. Despite their popular representation, automated systems are often far from autonomous (Ekbia and Nardi, 2017; Irani, 2015; Shestakofsky, 2017), but rather depend on vast networks of human labor and resources. If we are to take seriously the distributed nature of sensing defects, we must include Walter’s design work, as much as the camera’s computer vision algorithms. Of course, this does not mean that all positions in the network are equally valued; Walter’s design work was eventually enfolded into the operation of the automated camera as part of its infrastructure. Sensing with sensors The automation of sensing defects has led to transformations in networks of human and machine labor. As we have shown, Walter’s design work became the unseen infrastructure for automated defect detection. This automation could eventually render Will’s visual inspections obsolete and reduce the time it would take Gwen to make daily quality reports. In the long-term, the addition of these technologies might mean that Gwen would not be hired on, but it is more likely that her job would shift its emphasis. This is because even though automating defect detection may replace certain human practices, namely surface defect detection, in so doing it may also provide data that would enable engineers like Gwen to sense defects in different ways. We can begin to see this in Hannah’s description of the camera. Hannah was another engineer in the quality department and was involved in a number of projects for improving quality control in LSSM’s production processes more generally. She described the camera project to Vinson as follows: 1 2 3 4 5 H: so the machine coming in it’s gonna u:m be able to detect {(rising intonation) defects for us} it’s a camera system and then we’re also getting length measurement off of it which would be hu:ge for us because right now we know approximately what we’re casting to but then as far as like a (lase) width we h- just assume what hot strip mill gives us is correct on it Hannah’s description draws our attention to two important points. First, the camera will only automate one part of the perceptual work of sensing defects, namely their detection. The camera will not be able to classify these defects or make sense of them. Second, the camera functions not only as a replacement for visual inspection by a human but also as another source of data about the process of casting. In doing so, the camera will enable Hannah to gain increasing control of the production process as a whole. To the extent that technologies like the camera would be able to automate work like the daily quality report, it would allow engineers like Hannah and Gwen to focus on their own projects. These projects usually consisted of collecting, organizing and interpreting 14 Social Studies of Science 00(0) historical quality data in order to improve the production process. This was sensing defects but at a different scale, one oriented toward the improvement of production techniques themselves rather than with respect to particular slabs. One such example on which both Hannah and Gwen worked at different points during our research was a project called the flipped slab trial. The trial was designed to make sense of where and why surface defects were occurring. Such defects often became visible in the step immediately downstream from the melt shop, in which the steel slabs were rolled into long thin strips and stored as coils. As slabs are rolled into strips any air pockets or foreign matter cast into the slab will come to the surface. At the same time, it is also possible for the process of rolling itself to create defects, such as when the rollers press debris into the metal. This step in the production process was overseen by a different company that received the slabs and returned coils, which were then further processed by LSSM. For this reason, sensing defects was doubly crucial here: The steel was changing form and making defects visible, but it was also changing organizations, making responsibility for defects a contentious question. The flipped slab trial involved sending the metal slabs to the hot strip mill in different orientations. Engineers in the quality department would then track which slabs were found to have defects when they came back from the hot strip mill, and compare this to historical data. If the defects were from the casting process, flipping the slabs would affect whether defects were found on the top or the bottom. If the defects were coming from the hot strip mill, slabs would be found to have defects on the top regardless of their orientation, because of the way the rollers pushed particles into or scraped particles along the metal. As a practice of sensing defects, the flipped slab trial brought together multiple human and non-human actors. It relied on the disciplined perception of Hannah and Gwen to find patterns in the collected data. It also relied on the operators and their machine sensors both in the melt shop and the hot strip mill to record relevant qualities of the steel. Finally, it relied on proprietary software such as the program that Gwen used, as well as Excel spreadsheets. Only across this expanded network did the defect generating practices of the melt shop and the hot strip mill become sensible at scale. The quality department carried out similar trials to determine the likely effects of different production techniques on steel quality. At this scale, the daily quality reports and other defect data became the foundation for engineers’ acts of disciplined perception, which would ultimately allow them to see and intervene in production practices. At stake was not a particular batch, but rather whole methods of production. While engineers in the quality department referred to tasks like those involved in making the daily quality report as ‘busy work’ or boring, projects such as the flipped slab trial were considered more valuable because they had the potential to improve production techniques, thereby eliminating not just defects but defect-causing processes. In this context, the automation of visual inspection that the camera represented, and even the automation of data collection involved in the daily quality reports, would itself operate as crucial infrastructure for the projects of the quality department. Indeed, Hannah and Will often mentioned that they wished that more of the data collection was automated, as it would give them more time to engage in the sort of pattern interpretation that constituted sensing defects at this scale. That is, the automation of detecting defects would displace human labor onto the work of interpreting defects and onto the processes of production from which defects emerged. Sargent et al. 15 Taken ethnographically as ongoing projects and new technologies, automation is as much a process of displacing and transforming labor as it is of replacing it. As our cases suggest, the introduction of a camera system capable of automatically detecting and even recording defects may not eliminate engineering jobs, although it would eliminate certain tasks. The framework that we have been developing calls attention to the ways that projects of automation shift the configurations of human and machine practices that go into sensing defects. With the addition of the camera system, Walter’s human labor is enlisted to provide the physical infrastructure for a machinic sensor which can then assume the task of detecting and recording defects. To the extent that the data collection process is automated, engineers like Gwen and Hannah will be pushed into tasks oriented toward the larger-scale control of defects. Importantly, these tasks produce value for LSSM not only by eliminating particular defects, although this is certainly part of what they do, but by opening up new markets. As we mentioned above, the automotive industry has stringent quality specifications; being able to sense defects at the scale of production techniques is crucial for accessing this lucrative market. In this regard the case of LSSM demonstrates how the perceptual work of engineers can be reconfigured under automation, becoming ever more closely tied to the creation of economic value from the commodities they help produce. Conclusion Scholarly and popular debates around automation have been dominated by the questions of job creation and loss. While much work has been focused on the problem of job loss, other research has focused on the potential of job creation and human-machine cooperation brought about by automation (Brynjolfsson and McAfee, 2014). These debates tend to operate at the level of nations, markets or sectors. At this level, losses in certain kinds of jobs due to automation may actually increase demand for labor in general, due to economic expansion, rising consumer demand and other factors (Autor and Salomons, 2018). In measuring jobs lost and searching for evidence of new jobs created by automation, these debates miss the more complex effects of automation (Shestakofsky, 2017). More ethnographic studies demonstrate that automation does not simply substitute human for machine labor but rather shifts the distribution of tasks that human and nonhuman actors undertake at work (Hutchins, 1995; Johnson, 1988; Shestakofsky, 2017; Stacey and Suchman, 2012). As Shestakofsky (2017) argues, automation must be approached as a project whose effects depend on the diverse contexts in which it is undertaken (see also Birtchnell, 2018; Bissell, 2018). His study of automation in a computing startup firm stresses the temporal dimension as the company’s efforts at automation create successive forms of human and machine collaboration. Our work follows such ethnographic approaches to automation. Our cases demonstrate how an as-yet unfinished project of automation would fit into and transform the existing landscape of engineering work practices at LSSM. Thinking beyond job loss, we focus on the perceptual practices involved in a particular form of work that many engineers in manufacturing environments engage in, namely sensing defects. Such an orientation brings into relief the human-machine configurations within which the autonomy of machines or humans is enacted (Stacey and Suchman, 2012). As Gwen 16 Social Studies of Science 00(0) learns how to ‘see’ defects with graphs and spreadsheets, she comes to rely on the software, operators and machine sensors that populate the LSSM melt shop. Yet for all that it may be removed from the hot steel slabs coming out of the caster, Gwen’s work still ties her intimately to the steel slabs. By learning to look at the computer screen in particular ways, Gwen came to be able to see defects in the slabs. Rather than amplifying sounds or illuminating depths, Gwen’s disciplined perception allows her to see phenomena that are not necessarily themselves visual. She can see why slabs may look terrible, but she can also see the too-cool temperature of a slab, or the too-slow movement of a ladle to the caster. Interfaces between human and machine are often sites where the mode of perception itself is transposed as well as amplified. This does not make the work of sensing defects less embodied, but it does redistribute the actors and perceptual capacities involved. Our approach here stresses the fact that automation often displaces rather than replaces human labor and suggests a framework for analysing the shape of such displacements. By considering the different scales at which engineers at LSSM sense defects, we demonstrate how the addition of automated technologies will likely shift the emphasis of engineering work from defect detection to the sensing of defectgenerating processes in the production process. This work is no less reliant on disciplined forms of perception, but it is oriented toward a different horizon. While the everyday work of creating the daily report may one day be nearly fully automated, these automated layers of defect detection and data collection become the infrastructure for longer-term defect sensing work such as the flipped slab trial. Human labor is displaced from the work of identifying and eliminating particular defects in slabs and channeled toward projects that aim to optimize the production process itself. As we have noted, the work of sensing defects is deeply shaped by the imperatives of creating value for the company. In the case of smaller-scale practices such as those engaged in by Leah and Gwen, this occurs through identifying potential sources of negative value either in leaky furnaces or defective slabs. In the case of larger-scale practices, such as the flipped slab trials, this involves the increasing control over production such that defects do not occur in the first place. Projects of automation are crucial elements in shifting the focus of human engineering labor from the first sort of practice to the latter. The increasing automation of new tasks is undeniably transforming the contours of work and the jobs that people do. Yet the urgency of these transformations does not mean that automation is only a question of job loss and creation. Attending to how professional engineers sense defects organizes an approach to automation that highlights other ways in which automation reorganizes human work. Taking a widespread and common task in engineering work, we were able to demonstrate different cases of how this perceptual work is distributed across humans and machines. Our cases illuminate the complex relations between heterogenous elements that are involved even in seemingly straightforward instances of visual perception. Yet they also demonstrate that situations that appear to be straightforwardly automated rely on embodied human action as well. At a larger level, our approach to sensing defects as an embodied and distributed work practice brings in to focus the diverse human-machine configurations on which technical production depends. 17 Sargent et al. Acknowledgements The authors would like to thank our research participants for taking the time to discuss their work experiences with our research team. Our thanks to Ella Butler, Anna Jabloner, Erin Moore, Malavika Reddy and Jay Sosa for providing input on early drafts of this paper. We would also like to thank the three anonymous reviewers and the editors for their close readings and useful suggestions. Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/ or publication of this article: This research was supported by a grant from the National Science Foundation (#1252372). Transcription key () [] […] {} = == . … ? : unclear audio, text is best approximation nonlinguistic or paralinguistic activity material omitted from transcript information redacted for purposes of anonymity latching where there is no pause between the turns of different speakers a section of overlapping speech falling intonation pauses with number of periods denoting the relative length of the pause rising intonation extended syllable. The number of colons indicates relative duration ORCID iD Adam Sargent https://orcid.org/0000-0003-0302-5492 References AISI (2015) Profile 2015: American iron and steel institute. Available at: https://www.steel.org/ (accessed 2 Junuary 2021). Alac̆ M (2008) Working with brain scans: Digital images and gestural interaction in fMRI laboratory. Social Studies of Science 38(4): 483–508. Autor D (2015) Why are there still so many jobs? The history and future of workplace automation. Journal of Economic Perspectives 29(3): 3–30. Autor D and Salomons A (2018) Is automation labor-displacing? Productivity growth, employment, and the labor share. Report for National Bureau of Economic Research. Report no. 24871, 18 July. Baim AD (2018) Getting the picture: Visual interpretation in ophthalmology residency training. Medical Education 52(8): 816–825. Benjamin R (2019) Race After Technology: Abolitionist Tools for the New Jim Code. Cambridge: Polity Press. Birtchnell T (2018) Listening without ears: Artificial intelligence in audio mastering. Big Data and Society 5(2): 1–16. Bissell D (2018) Automation interrupted: How autonomous vehicle accidents transform the material politics of automation. Political Geography 65: 57–66. Brussevich M, Dabla-Norris E and Khalid S (2019) Is technology widening the gender gap? Automation and the future of female employment. Report for the Fiscal Affairs Department. Washington, DC: IMF (International Monetary Fund). 18 Social Studies of Science 00(0) Brynjolfsson E and McAfee A (2014) The Second Machine Age: Work, Progress and Prosperity in a Time of Brilliant Technologies. New York, NY: W.W. Norton and Company. Butler E (2018) Tasting off-flavors: Food science, sensory knowledge and the consumer sensorium. Senses and Society 13(1): 75–88. Carr ES (2010) Enactments of expertise. Annual Review of Anthropology 39(1): 17–32. Daston L and Galison P (2007) Objectivity. Brooklyn, NY: Zone Books. Ekbia H and Nardi B (2017) Heteromation, and Other Stories of Computing and Capitalism. Boston, MA: MIT Press. Elish MC and Hwang T (2015) Praise the Machine! Punish the Human! The Contradictory History of Accountability in Automated Aviation. New York, NY: Data and Society Research Institute. Giere RN and Moffatt B (2003) Distributed cognition: Where the cognitive and the social merge. Social Studies of Science 33(2): 301–310. Goodwin C (1994) Professional Vision. American Anthrpologist 96(3): 606–633. Goodwin C (1997) The blackness of black: Color categories as situated practice. In: Resnick LB, Säljö R, Pontecorvo C, et al. (eds) Discourse, Tools and Reasoning: Essays on Situated Cognition. New York, NY: Springer, 111–140. Goodwin C and Goodwin MH (1996) Seeing as situated activity: Formulating planes. In: Engestrom Y and Middleton D (eds) Cognition and Communication at Work. Cambridge: Cambridge University Press. Hamid OH, Smith NL and Barzanji A (2017) Automation, per se, is not job elimination: How artificial intelligence forwards cooperative human-machine coexistence. In: 5th International conference on industrial informatics (INDIN). Emden: IEEE (Institute of Electrical and Electronics Engineers), 899–904. Harvey D (1990) The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change. Cambridge, MA: Blackwell Publishers. Helmreich S (2007) Ananthropologist underwater: Immersive soundscapes, submarine cyborgs, and transductive ethnography. American Ethnologist 34(4): 621–641. Helmreich S (2009) Intimate sensing. In: Turkle S (ed) Simulation and Its Discontents. Boston, MA: MIT Press, 129–151. Hutchins E (1995) Coginition in the Wild. Cambirdge: The MIT Press. Irani L (2015) The cultural work of microwork. New Media and Society 17(5): 720–739. Jasanoff S (1998) The eye of everyman: Witnessing DNA in the Simpson trial. Social Studies of Science 28(5–6): 713–740. Jocuns A, Stevens R, Garrison L, et al. (2008) Students’ changing images of engineering and engineers. In: American society for engineering education annual conference and exposition. Berkeley, CA: ASEE (American Society for Engineering Education), 13.1113.1–13.1113.28. Johnson J (1988) Mixing humans and nonhumans together: The sociology of a door-closer. Social Problems 35(3): 298–310. Latour B (1986) Visualisation and cognition: Thinking with eyes and hands. In: Long E and Kuklick H (eds) Knowledge and Society: Studies in the Sociology of Culture Past and Present. Greenwhich, CT: Jai Press, 1–40. Latour B and Woolgar S (1979) Laboratory Life: The Construction of Scientific Facts. Princeton, NJ: Princeton University Press. Lehman J (2018) From ships to robots : The social relations of sensing the world ocean. Social Studies of Science 48(1): 57–79. Lindwall O and Lymer G (2017) The dark matter of lab work: Illuminating the negotiation of disciplined perception in mechanics. Journal of the Learning Sciences 17(2): 180–224. Sargent et al. 19 Lynch M (1985) Discipline and the material form of images: An analysis of scientific visibility. Social Studies of Science 15(1): 37–66. Lynch M (1988) The externalized retina: Selection and mathematization in the visual documentation of objects in the life sciences. Human Studies 11: 201–234. Masco J (2010) Nuclear technoaesthetics: Sensory politics from trinity to the virtual bomb in Los Alamos. American Ethnologist 31(3): 1–25. Mol A and Law J (1994) Regions, networks and fluids: Anaemia and social topology. Social Studies of Science 24(4): 641–671. Noble DF (1986) Forces of Production: A Social History of Industrial Automation. Oxford: Oxford University Press. Shestakofsky B (2017) Working algorithms: Software automation and the future of work. Work and Occupations 44(4): 376–423. Stacey J and Suchman L (2012) Animation and automation - the liveliness and labours of bodies and machines. Body and Society 18(1): 1–46. Star SL (1999) The ethnography of infrastructure. American Behavioral Scientist 43(3): 377–391. Star SL and Strauss A (1999) Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer Supported Cooperative Work 8(1–2): 9–30. Stevens R and Hall R (1998) Disciplined perception: Learning to see in technoscience. In: Lampert M and Blunk M (eds) Talking Mathematics in School: Studies f Teaching and Learning. London: Cambridge University Press, 197–229. Stevens R, Johri A and O’connor K (2014) Professional engineering work. In: Johri A and Olds BM (eds) Cambridge Handbook of Engineering Education Research. Cambridge: Cambridge University Press, 119–138. Suchman L (2000) Embodied practices of engineering work. Mind, Culture, and Activity 7(1–2): 4–18. Tarr DG (1988) The steel crisis in the United States and the European community: Causes and adjustments. In: Baldwin RE, Hamilton CB and Sapir A (eds) Issues in US-EC Trade Relations. Chicago: University of Chicago Press. Trevelyan J (2014) Observations of South Asian engineering practice. In: Williams B, Figueiredo J and Trevelyan J (eds) Engineering Practice in a Global Context: Understanding the Technical and the Social. Boca Raton, MA: CRC (Chemical Rubber Company) Press, 223–243. Vertesi J (2012) Seeing like a rover: Visualization, embodiment, and interaction on the Mars exploration rover mission. Social Studies of Science 42(3): 393–414. Zuboff S (1988) In the Age of the Smart Machine: The Future of Work and Power. New York, NY: Basic Books.