See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/266653790

IPhone in vivo: Video analysis of mobile device use

Article · April 2013

DOI: 10.1145/2470654.2466132

CITATIONS READS
50 305

3 authors, including:

Moira Mcgregor
Stockholm University
18 PUBLICATIONS   263 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Moira Mcgregor on 23 September 2017.

The user has requested enhancement of the downloaded file.

Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

iPhone in vivo: Video Analysis of Mobile Device Use

Barry Brown Moria McGregor Eric Laurier
Mobile Life @ Stockholm University Centre HCI Design School of Geosciences
SE-164, Kista, Sweden City University, London University of Edinburgh
barry@mobilelifecentre.org moira.mcgregor.1@city.ac.uk eric.laurier@ed.ac.uk

ABSTRACT by their very nature lightweight and used in a range of

Despite the widespread use of mobile devices, details of uncontrollable, distributed settings.
mobile technology use ‘in the wild’ have proven difficult to
collect. This paper uses video data to gain new insight into To address this, the paper here presents a study of mobile
the use of mobile computing devices. Our new method device use drawing on video recorded from multiple
combines screen-capture of iPhone use with video portable wearable cameras. We combine this video with
recordings from wearable cameras. We use this data to recordings from their smartphone made with screen capture
analyse how mobile device use is threaded into other co- software. This allows us to relatively unobtrusively record
present activities, focusing on the use of maps and internet device use and surrounding user activity, in naturalistic
searches. Close analysis reveals novel aspects of gestures settings without the presence of potentially distracting
on touch screens, how they serve ‘double duty” - both as equipment. We studied 10 city-trips, both visitors and locals
interface gestures but as as resources for ongoing joint on an ordinary non-work day. Participants made use of their
action. We go on to describe how users ‘walk the blue dot’ iPhone as they would normally, except recorded on video.
to orientate themselves, and how searches are occasioned Through this we collected just under 24 hours of video of
by the local environment. In conclusion, we argue that mobile activity, and around four hours of video of mobile
mobile devices - rather than pushing us away from the device use, with recordings of the surrounding
world around us - are instead just another thread in the environment, conversations and interactions with others.
complex tapestry of everyday interaction. Existing research on mobile device use emphasises the
Author Keywords interaction with the device, the importance of particular
Video methods; smartphone use; mobility; ethnography applications and gross patterns in application use [1, 3, 8,
18]. In contrast, the study here focuses on the relationships
ACM Classification Keywords between the social and physical environment, and how
H.5.m. Information interfaces and presentation (e.g., HCI): device use responds and reshapes action in that
Miscellaneous. environment. The use of the mobile devices captured was
General Terms shaped by the contingencies of particular situations - such
Human Factors as checking a map while ordering a drink, having a
conversation around a web search, or visibly and
INTRODUCTION accountably dis-attending to others while reading content
Any observer of technology cannot have missed the on a mobile device. This data would have been difficult to
ongoing growth in mobile device use. While conventional collect using existing mobile research methods.
keyboard and mouse based computing has hardly
disappeared, the mobile phone’s increase in functionality, We focus on two popular uses of mobile devices:
sophistication and raw sales, has changed fundamentally the information search and map use. We describe the
assumed platform for technology use. By some estimates occasioned nature of search on mobile devices, where
mobile internet-use will supersede fixed internet-use by information searches happen just before or at the point
2015 [17]. Computing use appears to be bifurcating into the when that information is needed to be used, drawing on the
existing mouse and keyboard based interaction (usually on environment and conversation. With our examination of
a laptop) contrasting with touch interaction on smaller map use we describe how directions come to be oriented to
hand-held devices such as tablets and mobile phones. the local environment and used to navigate and find suitable
destinations. The challenges of users in overcoming the
This shift has contributed to the growing interest in mobile inaccuracies of sensors and maps are also outlined.
computing within HCI. Yet we would argue that we still
have only schematic understandings about how mobile This new method, while relying on earlier analytic work
devices are actually used. In part this is because of the using video and interaction analysis, provides a new
methodological challenges of working with devices that are viewpoint on mobility and mobile device use. This allows
us to document device use as not simply something done
Permission to make digital or hard copies of all or part of this work for while co-present with others but action intimately
personal or classroom use is granted without fee provided that copies are connected with and dependent upon others in the
not made or distributed for profit or commercial advantage and that copies environment. We conclude the paper by arguing for seeing
bear this notice and the full citation on the first page. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior mobility not in terms of physical movement or device
specific permission and/or a fee.
CHI 2013, April 27–May 2, 2013, Paris, France.
Copyright © 2013 ACM 978-1-4503-1899-0/13/04...$15.00.

1031
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

design, but in terms of the embedding of device use in the diary-method examined issues surrounding cost and plan
world of co-present others. usage, relatively neglected for research since then [26].
PREVIOUS WORK Focusing specifically on smartphone use, three studies are
The massive growth in mobile phone use, and the more revealing. The first is Barkhuus and Polichar’s study of
recent explosion in mobile device functionality, has been smartphone use [1], where they document the importance of
one of the most notable changes to our modern world. As smartphones as combining multiple functionality in a
Merchant puts it “Mobile phones have rapidly been relatively ‘messy’ way. While this work documents
absorbed into the fabric of our day-to-day lives”, [22]. A revealing aspects of use, its reliance on interviews and
range of statistics can be drawn upon to argue for the diaries means that its description of use is indirect. There is
importance of mobile phones as a computing device. a lack of description of actual cases of smartphone use.
Smartphones outsell PCs, and recent surveys have reported
that mobile phones are used for longer than all other In contrast, Böhmer et al. [3] used logging of application
computers combined [17]. Tablet use is also one of the launches on the Android platform to present a detailed view
quickest growing, new, multi-purpose technology forms, on which applications are used on smartphones, alongside
followed by the growth in ebook readers and other devices an analysis of time and length of use. An interesting
with a similar form factor. Indeed with laptop computers concept introduced in this work is ‘application chains’ -
outselling desktops, one might even say that the majority of where users switch from one application to another in a
computing is already ‘mobile’. ‘chain’ of activity. While they report that the majority of
sessions consist of only one application launch, 32% of
While smartphones can be relatively simple to demarcate, sessions have two or more applications. Although this paper
the question of what devices are ‘mobile’, and what gives us an important view on what applications are being
mobility means for research can be harder to define. One launched, there is relatively little information on what is
approach is to see mobility in terms of the mobility of users then being done with the applications themselves.
themselves. For Brown and O’Hara [6], ‘mobility’ is a
concept that has analytic value as a worker’s concern. This Techniques of device logging such as this have much in
research documents how ‘mobility’ is something that has to common with experience sampling, such as MyExperience
be dealt with in terms of managing place - when and where and Momento [7, 9]. While these systems pioneered using
work will be done, who will be there, and what artefacts are instrumentation and user alerts on mobile devices to collect
need to be deployed to make places ‘work-place-able’. In a data about use and broader user interaction, they have not
similar way, Luff and Heath [20] discuss mobility yet focus been used as part of studying modern mobile device use.
instead on the manipulation of artefacts in collaborative One exception is the work of Church and her collaborators
settings. Here, their interest is on what they term ‘micro [8], which has focused on mobile search. This research
mobility’: the ways in which information - particularly makes extensive use of experience sampling (prompted
paper documents - come to be deployed in face to face through SMS messages), and web-based surveys of mobile
interactional settings. The dynamic is between the co- search. This work is particularly strong in understanding the
present revelation and obscuring of information. context and motivation behind mobile search. For example,
reporting on the importance of conversations as a prompt
Alternatively, the predominant user interface paradigm built for web search. Yet it also suffers from analysing reported,
into different devices can be seen as providing a definition rather than actual, use.
of mobility. Perhaps the most notable recent change has
been the growth in finger-based interaction with smaller Previous research thus leaves much about modern mobile
mobile devices. In terms of user interaction, devices can be device use unanswered. In particular, there is little in the
divided between ‘touch’ systems (predominately iOS and way of understanding the details of usage of particular
Android) and keyboard and pointer based systems applications and what shapes usage beyond simply what
(predominately Windows and OSX). With a market in flux applications are used when. For example, what prompts the
this distinction is open to revision - although it does use of particular applications at particular times and
highlight a fairly distinct class of touchscreen devices with locations? What purposes are different applications put to?
distinct forms of use. As we will argue, this demarcation of How might application use be influenced by the setting,
mobile devices as touch systems also reveals that we have what role does task have in use, and how might smartphone
relatively little understanding of the differences in how use influence action more broadly? To be answered, these
these touch systems are used. sorts of questions demand a different method.
USING VIDEO TO STUDY MOBILE DEVICE USE
A number of classic studies have documented the use of
As we can see, the predominant methods in studying mobile
mobile phones, however this work mainly discusses the use
device use have been either post-hoc interviews, diary,
of conventional mobile phone features, such as text
logging or experience sampling based. While these methods
messages or phone calls. Weilenmann discusses shared
have considerable strengths they face two immediate
phone use [32], Murtagh describes mobile phone use in
challenges. The first is a lack of of coverage. Interviews
public [24], and Taylor [2] documents some of the broader
take place after use, diary reports are made just after use,
interactions around mobile phone use amongst teenagers
and experience sampling depends upon sampling situations
being some of the most notable. Palen’s voicemail-based
of use. These reports are not likely to capture every

1032
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

situation of use. Respondents prioritise certain incidents METHOD

over others, or simply forget brief uses [29]. This issue is To address some of these issues we developed a relatively
compounded by the lack of detail. Again, post-hoc lightweight video method that made use of wearable
interviews offer only outlines of usage, filtered again cameras, combined with screen recording software. This
through what an interviewee may consider relevant or method allows us to gain something of a new perspective
important. While diaries can be used to ask detailed on mobile device use and, importantly, lets us study in
questions, there are limits to what can be practically asked detail how the environment and device are connected in
of an informant [25]. It is not that these methods are not use. Our participants wore multiple portable wearable
useful - of course, any method has its shortcomings and cameras that recorded their actions relatively unobtrusively
limitations - but that these absences limit our view on (figure 1). These videos were combined with phone screen
device use since much of the data is recorded after the point recording software to allow us to unobtrusively capture the
of use. In contrast, while log or instrumentation studies use of smartphones. We combined this with experimenting
collect data at the point of use, these logs offer only a with studying multiple participants together, each wearing
schematic outline of device usage. This is compounded by portable cameras, and obtaining multiple viewpoints of
analysis which seems to describe general use, rather than device use, with video recordings of the user of a device
attend to understanding specific cases of use. (from their companion) alongside video recorded by the
device user, and a recording of screen use. For individual
Our goal for this paper was therefore to experiment with users, one of the authors accompanied them for the first
new methods that might capture more of the situation of hour or so of their visit. This allowed us to collect a range
usage, and allow us to take a different analytic frame. Video of videos of both collaborative and solo device use.
recordings have been one longstanding method used in CHI
and CSCW for a range of different analytic purposes, but
particularly to document interaction with and around
technical artefacts [13, 16, 31]. This research has
extensively studied interaction around screens, capturing
on-screen interactions as part of ongoing complexes of
work practice (e.g. [14]). Video has proven valuable in
illuminating aspects of activities neglected by previous
methods, particularly drawing on conversation analysis to
understand technology in use. In some senses video forces
attention on the moment-by-moment production of
technologically mediated action. A recent example is our
recent research on driving with GPS [5] where cameras
mounted inside cars were used to record drivers navigation.
This work elaborated on the detailed ‘problems’ overcome Figure 1: Participant is wearing camera in map bag
by drivers using GPS, problems on the whole missed by around her neck. The left picture is taken from one
more experimental studies. wearable camera, with the image from the iPhone
Using video methods to study mobile use, however, recording and the second camera on the right.
presents considerable challenges. Since the users In our previous work we have long struggled with obtaining
themselves are largely mobile, fixed cameras are of limited good video images of smartphone screens. Due to glare,
use. Small digital wearable cameras, while potentially video recording smartphone screens outdoors is very
portable, also have a number of limitations in battery life difficult. We dealt with this problem by developing
and flexibility. Camera angles can be vital in capturing the software that unobtrusively records the mobile display in
situation of use and mobile cameras are more difficult to real time without interfering with its use. The software
aim, constrain or control. While fixed cameras can be takes a static image of the screen four times per second, and
pointed at desktop computer screens this is not as easy with then uploads the images to a server. The frames are then
mobile devices which may be held at any angle, in sunlight, combined to produced videos of device use. While this
and may have particular software limitations. limits the exact timing of some recorded actions, it was
More broadly, ‘sessions’ of desktop computer use can be more than sufficient for our analytic work. Combining this
organised as part of an experimental task. Yet mobile device screen capture with the wearable cameras we aimed to
use is something that takes place sporadically throughout produce real time recordings that would give us access to
the day and is frequently dependent on the local setting. the user actions and the environment on and off-screen.
Constraining device use to a specific place or time would We recorded a diverse set of everyday practices of
likely distort somewhat the nature of use. Applications smartphone use, centred around the use of smartphones as
which rely upon the environment or relevant setting are part of a city ‘day trip’. We recruited a mix of locals and
much harder to study as part of a pre-arranged experiment. visitors to Stockholm, Sweden who regularly used an
iPhone branded smartphone. In total fifteen mobile device
users participated in the study, five recruited as individuals
and five recruited as couples, with ten trips studied in total.

1033
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

The participating smartphone users were recruited from services via the websites instead. Prominent in the
advertisements in local cafes, and adverts on visitor smartphone usage captured was the use of the internet to
websites. We recruited a mix of five groups of locals and browse for information, and the iPhone’s maps application
five groups of visitors, volunteering to let us record them to find and navigate on foot and public transport.
during a ‘typical’ day out. Four of our individuals were
1. Visit to ferry and music concert
female, and one male, and two of our couples were male/
2. Walk through the city to collect bike from repair shop,
female, and three all female. Our participants were recruited
followed by cycling to music concert.
with the request they take a ‘typical day out in Stockholm’.
3. Lunch followed by shopping trip
All the participants were fluent in English, and we asked
them to interact in English - although some participants 4. Visit to botanical gardens
switched to other languages for parts of their trip. 5. Visit to church and walk across the Stockholm
6. Visit to local town outside city
Our screen capture app was written for the iPhone. Due to a 7. Visit to Reimersholm and then to the city hall
range of issues, seven of the ten participants made use of an 8. Visit to Old Town to watch changing of the guard
iPhone supplied by ourselves. When running the trial, two 9. Visit to food hall
participants carried Android phones which could not have 10. Shopping trip and visit to gardens
their screens recorded. Another three participants requested Figure 2: list of activities
that we did not install any software on their phone and two
participants did not have data access enabled on their Multiple involvements: Divergent and convergent
phone. For these reasons, only three of our participants had One of the most prevalent features of the video recordings
our recording software installed on their own phone. For the is the extent to which device use takes place alongside
remaining seven participants we instead supplied them with involvements with co-present activities and other people.
our own iPhone with the recording software pre-installed. These interaction can be characterised as ‘multi-modal’, in
For the participants that went out carrying our device, they that interaction takes place with the device but also through
nearly exclusively used our device which was recorded. In spoken conversation, or ‘talk-in-interaction’, [27]. What is
one case a group of two participants used both devices - one also clear is that the device use itself shapes the interactions
using their own phone, and one our iPhone. In two that unfold. If mobile device use takes place with others
additional cases text messages and phone calls were who are co-present, device use is frequently brought into
conducted on the users’ own device, but the supplied the interaction - what Mondada [23] characterised as
recording device was used for their other smartphone use. ‘convergent’ interaction. In contrast we can identify
To record their interactions we equipped participants with ‘divergent’ interactions [23] where the use of the device is
small GoPro portable cameras worn over their neck in map- kept relatively separate from interactions with others.
bags. Audio was recorded with a small external microphone In figure 3, one tourist is talking to a researcher while on
attached to the outside of the bag. We combined the screen the subway. She pauses in conversation before moving on
recordings, with the multiple camera angles to produce to interacting with her phone to read some details about the
composite videos of each participant’s trip. These ranged in section of Stockholm she is visiting. She returns to the
length, but were limited by the battery life of the GoPro conversation by reading out a name from the website she
cameras (around four hours). has found, and then returns to reading. In this clip the
The analysis of the data grows out of ethnomethodology reading of the webpage takes place mainly without
and conversations analysis. Our goal was not to provide a interaction with the researcher who is sitting in the seat
statistical breakdown of the uses of the mobile device. In opposite. Moreover, the visitor ‘reading’ is something that
many ways this is already well catered for in the existing is visible in their orientation to the device. In the short
log-based literature. Rather our goal was to better exchange with their companion, the user lifts their head and
understand the details of particular situations of use. In reads aloud, keeping her gaze on the phone. In this clip the
individual and group data analysis sessions we surveyed the involvement of the participant with the conversation and
journeys, editing the 24 hours of video down to 205 one to reading are kept separate - her attention, gaze, and body
two minute clips for focused analysis, of which 12 were compartment used to keep apart her engagement with the
transcribed and analysed in depth. Our goal in this analysis device from her participation with her companion.
was to understand both in a broad sense how mobile device In studies of talk and interaction one recurrent concept is
use was organised, but also how specific incidents of use that of ‘multi-activity’ settings where participants are
unfolded, and what contingencies users brought to bear. engaged in multiple streams of activity simultaneously [11,
RESULTS 15]. One well researched example of this is in driving and
While obviously the device usage we recorded was conversation - where the activity of driving plays a role in
fashioned to an extent by our recruitment, our participants regulating the flow of conversation. Drivers and passengers
conducted a range of different visits (figure 2). In doing so regulate their talk around the commitments of the driver to
they also used a wide range of applications including: Yelp, safe and competent driving. In her analysis of car travel
Facebook, Instagram, Google Maps, Phone, Web browsing Mondada refers to cases of ‘convergent’ or ‘divergent’
and SMS. Users provided with our own phone did not use interaction.
the iPhone’s Facebook or mail clients, but accessed these

1034
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

B: But artists like joni mitchell

A: uuuh
B: are sammy
A: i:s she sa:mi
B: according to wikipedia
A: no ill use the phone [brings
out device]
B: who’s the other one
B: the other actress the one who
[Participant A is reading the “Time Out” guide. used to go out with jack white
After about one minute she lifts [A types Joni m [*], then selects
her head, but maintains her gaze Joni mitchell]-------------------
on the phone] B: the blonde girl
[A starts new search editing the
A: Oh [*] yeah------------------- previous search term, adds ‘sa’
the gonda [.4] to the end]
gondalean bar i think A: how do you write sami [*]----
is it two a:s
[A Turns head down to look at [types extra a]
phone again] 
B: S a m i [*]------------------
B: hhh
A: [*] hhhhh--------------------- [saami suggested by google]

[50 seconds reading] A: sa:mi i think

[types mi]
[A looks across to space above B: its just how you say it sami
B’s head] A: Funny record for your research
A: okkhhhhha [6.0]
A: Thats interestinghhh[*]------- [reads search results]
[Raises head and points to poster A: Lapland people lappy [*]------
hung above B’s seat] B: ahha
B: dont ponder it too much you:ll B: is this are you on
have nightmares wi:wiikipedia sammi people
A: hhh A: i always thought joni mitchel has big teeth ok
B: just a bit weird well:hhhhhh
A: very random

Figure 3: Talk and device use - divergent Figure 4: Talk and device use - convergent
That is, settings where the participants converge in mutual The participant reads out a spelling, at the same time as
orientation, or alternatively, diverge from that common typing it in. This spelling is confirmed by the companion,
interaction. Co-participants: “can mutually align [...] and but at this point Google makes a suggested autocompletion
display shared attention towards common objects; they can of the incorrect spelling (from saa to saami). The tourist
dis-align as well, in not attending to the other’s talk.” [23] then reads out the suggested spelling emphasising an
As she explains, even in cases of ‘divergent’ attention, elongated syllable. On conducting the search a conversation
parties are still engaged with each other and can time ensues about the search before a dismissal of one of the
utterances and actions with respect to each other. Figure 3 is result (“big teeth”). Clearly the search and typing are
one example of ‘divergent’ talk and device use, in that the arranged in concert with the conversation that is taking
participant keeps each involvement relatively separate. The place. The search responds to and is a source for the
two or so minutes spent reading the webpage are spent conversation. While in this case the device screen does not
mostly without talking to their companion. seem to be directly available to the second conversationalist
the participant reads out aspects of the search (“sami”,
In contrast, in figure 4 the search and the conversation are “lapland people”, “big teeth”). In this way although her
more closely related. Here device use and interaction with focus of attention is on the device, her involvement in the
others is convergent, in that the results of the search at conversation continues. The use of the device is part of the
different points are brought into the conversation. same activity as the conversation.
Interaction here takes place between the researcher and the
participant about the Sami people and a singer. This At a smaller scale of interaction - with specific user
conversation prompts a web search. As the search is interface events, we can see in a similar way how the device
conducted ("Is Joni Mitchell sami?") the conversation does use can converge with interaction. Figure 5 is an example of
not stop, but rather the search provides topical resources for where a user-interface gesture is also an interactional
the conversation. Particularly interesting is the way in gesture. Here the user is talking about an area of the map
which the spelling of sami is approached. and as she does this, the participant puts two fingers at each

1035
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

A: i just check [2.0 ] which road do we take Occasioned search and search chains
B: yes The three data extracts above display interactional features
A: eh yes we will see i think we actually where of mobile device use. They are also examples of
is it where are we  information search. In figure 3 the participant reads
A: hh [*] e:r [*] e [*]
information about areas of Stockholm that they might go
and visit. In figure 4, a web search is initiated around Joni
Mitchell. Lastly, in figure 5, the participant checks where
they are and what route they need to take to go through a
forest that is nearby. As we mentioned earlier, much of the
mobile device use we captured can broadly be characterised
[Pinches to zoom in on the blue dot] as ‘information search’ of some sort. Since our participants
were making visits to the city this provided a context to
B: ah ha much of the searching for information. For example,
participants searched for information about what they could
[A moves map over to
do next, the area they were exploring, public transport
the right, revealing
the forest and hiding information about how to get to particular destinations, or
the suggested route] information about local cafes, restaurants or shops.
As a time and attention-consuming activity, search is
A: because I want to
something which requires - at least some of the time - focal
go through the
forest[*]---------- attention. Some search we would refer to as pre-meditated
search, that is search which takes place before an activity to
Figure 5: double duty gestures be part of planning or organising that activity. In earlier
work (pre-smartphone) on tourists this was characterised as
side of the area referred to, and pinches outwards to zoom
"pre-visiting" [4], getting information about a place before
in. The pinch enlarges the area of the map where the
the actual visit from printed and online sources.
referred to point is, as well as highlighting the point
between the fingers. While the finger movement is away What was striking about the information searching in our
from the desired destination, it does prevent the fingers study was how it was conducted on the way to or even as
from obscuring the gestured target. part of the activity itself; information would be gathered on
the way to a particular part of the city, or even on arrival at
We would describe this as a form of 'double duty' [30]. In
the attraction or area of interest. Figure 3 is an example of
this case the highlighting of an item worked in combination
this, in that the participant is on public transport on the way
with a user interface gesture. What it also underlines is that
to the city, and they start to read through a list of ‘city
interface actions frequently come to have a role to play in
highlights’ from the Time Out website. The use of resources
the conversation too. At times in our videos, interaction
in the environment is one reason why search may be
with the mobile device was done as a solitary task - either
delayed until the last minute. As you move through an
by a visitor on their own, or on a device turned away from
environment you gain information about that place, about
those co-present. Yet devices were also very frequently
the setting, your own mood and what a place is like. Any in-
shared and seen by others. For example, at the end of figure
situ search can then draw directly upon that information,
5 the participant scrolls the map to the right, revealing the
whereas a search conducted earlier could be premature or
forest as she does so. Her reference to the forest is then
waste time. A change in the weather, mood and the like,
easily understandable as ‘this forest’ (she also later points).
might affect the planned activity and any subsequent
As has been extensively documented in studies of searching that might be undertaken. We are reminded of
interaction around fixed displays, ‘awareness’ of what is March’s description of decision making in organisations,
going on amongst co-present interactionists extends to co- where decisions are postponed until as late as possible to
present objects and devices in this case, the screen and map allow for the collection of maximum amounts of
which is being manipulated [14]. Using the device, when it information [21].
is visible to someone else, produces actions that are
This last-minute searching that we observed we titled
‘oriented-to phenomena’ [12]. That is, those who see the
‘occasioned searches’, where searching is triggered by the
screen can be expected by others to have seen the action
environment or local events. For example, one visitor
and to understand what that action means. Indeed, they
(travelling on their own) walks past a postal museum. He
often comment on, correct or repair the on screen actions of
searches for information on his iPhone about the museum,
the device user. Broadly speaking, much of our mobile
reads the short summary and then walks into the museum.
device use shares this ‘double duty’ quality - with usage
After picking up an information sheet (which he quickly
oriented to the device itself (to get it to do what the user
discards) and a quick browse of the bookshop he returns to
desires) but also as something that is seen by others. This is
walking along the street. In this case the physical
an interactional role of gesture that goes alongside the user
surroundings prompt an online search, which informed the
interface role.
visit. The search was occasioned by the circumstances. In a
related way, in figure 4, this search was occasioned by the

1036
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

conversation taking place, where the search term came from

a query over a singer’s heritage. By “occasioned” here we
mean that the details of the particular situation produce the
reasons for the activity, and that search is tied into what will
be done next. So, for example, coming across an interesting
looking tourist attraction might occasion a search.
Alternatively, one might come out of a subway and decide
to find some lunch - with the search taking place there and
then to find a restaurant or cafe. Perhaps surprisingly, many
of these searches resulted in failure of some sort, such as
not finding the desired establishment, or failing to find the
information required, and so participants would fall back on
simply waiting to see if they chanced upon what they were
looking for (such as a good restaurant), or giving up on the
particular activity (such as finding a particular attraction).
Another interesting aspect of information searches was the
ways in which the web and the iPhone’s inbuilt map
application would be switched between in an 'application
chain’ [3]. This would take the form of searches for
information in the web browser and then switching to the
inbuilt map to find a place (with the user often typing in the
name of a place). This movement between the map and the
web took place in part because searches were on names of
particular sites and attractions - with the location of those
sites being important, but also what those sites were too.
So, for example, in figure 6 a participant sees a church of
interest on their map, and then searches for that church on
Google. After reading about the church, they then search for
the particular area where that church is. This leads them to a
shop which is listed at the top of Google’s listing, next they
click on the map and view the shop on the map. Then they
return to Google and visit the shop’s website. Finally, they
return to the map and study the distance between their Figure 6: Eight part search chain. User switches
current location and the shop (and the searched-for area). between map and browser while sat on a park bench
The movement of the map between their current location incomplete and require the wayfinding skills of those
and the shop emulates the route they will later take, and present to be successfully followed. For iPhone users public
without specifically planning a route they look at the transport information was one potential problem, as the
general set of paths they then take to walk down to the maps app only displayed one subway exit per station, and
searched for area. described trains by route numbers rather than their subway
MAPS line colours - in one case this even led a participant to try to
One of the most used apps in our study was the map get a bus rather than enter the subway.
application. The success of smartphone map applications, A broader point concerns how the map comes to be used,
alongside the use of GPS based turn-by-turn driving and in particular, the orientation work that is necessary to
systems, has affected the sales of paper maps in recent years move between the map’s representation of the local
[28]. While paper maps have advantages of tangibility and environment and the visible features of the environment. In
size, this is countered by the much more dynamic abilities map studies the classic notion of map orientation concerns
of mobile device map applications. In our trial, as we the orientation of a paper map to the environment and the
mentioned above, searches would often start in the browser map reader [10]. A well established finding is that map
(with Google) then move over to the Map application as the users find it easier, and make less mistakes, when they are
location of the searched for item became relevant. orientated behind the map, facing the direction of 'up' on the
Using electronic maps to navigate is dependent to a large map. This allows for the direct visual alignment of items of
extent on overcoming ‘normal natural troubles’ [5] - such as the map and the environment - a road to the left then also is
inaccurate positions, incorrect maps, unclear maps, on the left on the map [19]. We noticed similar orientation
miscalculated routes, and problems in specifying work by our users, with the phone turned to bring the map
destinations, locations and so on. However, these problems into orientation with the environment. In figure 7, for
are not so much issues that demand redesign, since many of example, the phone is held so that the street straight ahead
them are practically unavoidable. Maps are always is on the map aligned up.

1037
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

Figure 7: Street aligned with the phone, the blue

line and the blue dot
The iPhone that we used in the study feature a 'rotating
map' view that should turn the map to align with the
environment. Yet the compass functionality is error-prone,
and much of the time it would misalign the map with the
environment (and users accepted and expected the errors).
In some situations (like figure 5) a conversation is taking
place around the map and turning the phone to manually A searches for “cheap eats old town” and finds a
comment about a restaurant called vapiano. They
align it had advantages, in that particular streets could be
type vapiano as a destination into the map and
accurately (although manually) aligned, and that this could start to walk across the street [pictured, map
be done relatively quickly so long as one knew the current frame 1]. As they do so, the blue dot (indicating
orientation. Physical rotation also made alignment of the current location) moves along the blue line
map and environment available to others. With the map towards the destination the red pin [2]. After 18
oriented to the environment participants could then seconds, it changes direction [3] then starts
following a recommended route, with the blue line of the moving away from the destination [4]. After
route would stretch up and continue visually with the road arriving at a street junction the visitor realises
being walked along, giving a satisfying visual confirmation they have been walking in the opposite direction
of being on the correct route (as in figure 7). from the restaurant. They give up their search for
Vapiano and find the closest cafe.
Finding the orientation could however cause problems.
Some users chose to discover their orientation themselves. Figure 8: The visitor walks up small side street seemingly
This in part could have been due to problems with the following the suggested route.
compass, or they were simply unaware that pushing the
when driving. Our pedestrians seemed much more likely to
‘locate’ button twice enabled the map rotation. Whatever
deviate from the planned route, than in our earlier studies of
the rationale, we noticed that some users instead 'walked the
driving. Moreover, since pedestrians can walk on a wider
blue dot' to try to find their orientation. This entailed
variety of terrain than cars (which hopefully remain mostly
walking in one direction then checking to see the new
on roads) plotting a route across a city can be difficult.
position relevant to the old position. The street walked
Shortcuts which are obvious to a pedestrian (such as an
along could be aligned to the street revealed by the change
open gate, or an unmarked footpath) are not available to the
in position. If the aim was to follow the route a user could
device. Pedestrians might also pick a more desirable route
also then see if their location was 'following' the blue line.
(such as across a park, rather than under a highway)
Unfortunately, one feature of the iPhone's navigation encouraging them to deviate them from the path. City
system (at least in iOS 5) disturbed this. To improve the environments are also three dimensional, something
accuracy of the positioning system the iPhone sometimes difficult to plot or model on a two dimensional map.
'snaps' a users location onto the current route. In many ways Suggested pedestrian routes then might often be
this is sensible. Much of the time one will be following this suboptimal, or downright wrong. If public transport was
route, and a users location, is more likely to be on that route taken then this also could lead to inaccuracies in the route
than elsewhere. Yet when on foot and distances are small suggested, since public transport data - although valuable
and GPS readings inaccurate, this assumption fails. Here, and useful for many of our participants - was also
one might accidentally walk tangentially away from the approximate in many ways.
route. In an attempt to 'correct' GPS signals the iPhone
The route snapping technique is particularly problematic for
would then display the pedestrian walking on the route,
pedestrians when they deviate from the route, in that it
giving incorrect confirming of the pedestrians choice of
provides wrong information, often at a point where
direction. Indeed, when not-driving there are a number of
accuracy is crucial. Indeed, it can seem to encourage
issues that arise in maps use that are less likely to arise

1038
Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

deviations from the route, since these will not initially be ordinary use because of how that device fits with
rendered. Figure 8 shows one example of a pedestrian interaction itself. Luff and Heath describe ‘micro-mobility’
walking in the wrong direction, which is seemingly - the ways in which a device would fit with physical bodies
confirmed by the map. Yet after 18 seconds it is revealed and interactional commitments.
that the map has incorrectly plotted their moving position.
Developing this notion further we can see that the form of
When a route was deviated from, at least it was often not a mobility here goes beyond simple portability. This
large challenge to navigate in the general direction of the mobility-in-interaction encompasses the broader range of
suggested route, or to navigate back to the route. The blue commitments and activities that we undertake in our daily
line on the map was still useful even if it was not literally lives. That mobility is the way in which a drink can be
followed (see, for example, figure 5). It is clear that much ordered and that transaction interleaved with looking
of the design of the Maps is optimised for driving. This something up on a map. Mobility is what allows
perhaps is wise since the majority of journeys in the US are information to be looked up as one walks past a museum,
taken by car. Yet for pedestrians this means that the map immediately seeing electronic sources as one walks into the
tool, while useful, does have shortcomings. actual physical museum.
DISCUSSION It may be the case that the value of mobile devices come
We found that the recording methods described here less from their explicit portability, and instead from how we
provide a unique and powerful view on interaction around can weave themselves into our ordinary actions and
mobile devices. The data we have included highlights a interactions. That is, mobility not simply in terms of
range of different aspects of device use. The news here physical size but mobility in vivo. Mobile devices fit with
includes the ‘double duty’ of interactional gestures, the our lives and actions in ways that conventional technology
ways in which information search is occasioned by local never could. We would argue that mobility in vivo is a topic
activity, and features of using the map such as ‘walking the that still rewards renewed research interest.
blue dot’ This would have been impossible to find without a
CONCLUSIONS
method that supported this sort of detailed analysis.
The data here reveals a world of device use that using
There are of course some limitations. As with any complex previous methods is hard to access. It reveals how
device the variety of applications and use are determined interaction with smartphones such as the iPhone in vivo is
largely by the user and their situation. Our data then is threaded through other activities. We documented the
obviously somewhat constrained in that it focuses on usage differences between convergent and divergent mobility,
on a particular type of day (non-work), in one city. Our how gestures on the device are seen by others, the ways in
reliance on providing a device for the majority of our data which searching for information is arranged, and lastly we
collection also constrains considerably the type of talked at length about how the map is used. Our focus
application and device usage we collected (alongside our throughout this analysis has been on understanding the
reliance on one model of mobile device). We have no relationships between device actions, interactions, physical
examples of game playing, for example, even though this is actions, movement and conversation.
one of the most popular categories of applications. With the
In closing we would argue that the method we have
reliance on mobile cameras, the perspectives and view that
provided here gives us considerable potential to move to a
is recorded can at times inadequate for analysis. Yet this is
better understanding of device use, and of course how these
balanced by the inclusion of a clear uninterrupted recording
devices might be better designed. We have resisted detailed
of the onscreen activity on the mobile phone. This makes
design suggestions, instead attempting to build a more
available the ‘workplace objects’ for analysis in a way not
substantive understanding of what it is to use a mobile
previously possible; something that would be harder to
device. Indeed, it may be that the touch and surface
access with traditional ethnographic methods. In turn the
computing literature is too focused on developing the latest
use of portable and wearable cameras supports collecting
interaction technique to engage properly with situations of
data on interactions that would have been impossible to
mobile use. Device use is dependent upon and threaded into
record using fixed cameras.We have also developed a
what goes on around us. It may be that rather than pushing
version of the software that records audio alongside screen
us away from the world around us, our mobile devices are
interactions, and we hope to capture the device camera as
instead just another thread in the complex tapestry of
well. We intend to make this software available to other
everyday interaction.
researchers, to assist in future studies.
ACKNOWLEDGEMENTS
Mobility in vivo
We thank Marek Bell for writing the recording software, the
A second concept that our data casts light on is mobility
helpful comments on the paper by the anonymous reviewers
itself. While mobility is a foundation for much technology
and our colleagues at mobile-life. The research was made
design, it has had surprisingly little analytic attention within
possible by a grant from the Swedish Governmental Agency
HCI. Our research builds on the work of Luff and Heath - in
for Innovation Systems to the Mobile Life VinnExcellence
particular the move away from mobility as mere portability,
Center, in partnership with Ericsson, Microsoft, Nokia,
to ‘mobility in interaction’ and the myriad ways in which a
IKEA and the City of Stockholm.
particular device may or may not fit with situations of

1039
 Session: Reflecting on Phones CHI 2013: Changing Perspectives, Paris, France

REFERENCES 17. Ipsos-MediaCT/Google. Mobile Internet & Smartphone

1. Barkhuus, L. and Polichar, V. E. Empowerment through Adoption: New Insights into Consumer Usage of Mobile
seamfulness: smart phones in everyday life. Personal Devices, the Shift to Smartphones & the Emergence of
and Ubiquitous Computing, 15, 6 (2011), 629-639. Tablets, http://googlemobileads.blogspot.se/2012/01/
2. Berg, S., Taylor, A., et al. Mobile devices for the next new-research-global-surge-in-smartphone.html. 2012.
generation: device designs for teenagers. In 18. Kamvar, M. and Baluja, S. A large scale study of
Proceedings of CHI 2003, Fort Lauderdale, Florida, wireless search behavior: Google mobile search. In
ACM Press (2003). Proceedings of the SIGCHI conference on Human
3. Böhmer, M., Hecht, B., et al. Falling asleep with angry Factors in computing systems (2006),701-709 ACM.
birds, facebook and kindle: a large scale study on 19. Laurier, E. and Brown, B. Rotating maps and readers:
mobile application usage. In Proceedings of MobileHCI praxiological aspects of alignment and orientation.
2011 (2011),47-56 ACM. Transactions of the Institute of British Geographers, 33,
4. Brown, B. and Chalmers, M. Tourism and mobile 2 (2008), 201-216.
technology. In Proceedings of ECSCW 2003, Klewer 20. Luff, P. and Heath, C. Mobility in Collaboration. In
Academic Press (2003), 335-355. Proceedings of CSCW 1998, ACM Press (1998),
5. Brown, B. and Laurier, E. The normal natural troubles 305-314.
of driving with GPS. In Proceedings of CHI 2012 21. March, J. and Sevon, G. Gossip, information and
(2012),1621-1630 ACM. decision making. In Decisions and Organzations, Basil
6. Brown, B. and O'hara, K. Place as a practical concern of Blackwell (1988).
mobile workers. Environment and Planning A, 35, 9 22. Merchant, G. Mobile practices in everyday life: Popular
(2003), 1565-1587. digital technologies and schooling revisited. British
7. Carter, S., Mankoff, J., et al. Momento: support for Journal of Educational Technology, 43, 5 (2012),
situated ubicomp experimentation. In Proceedings of 770-782.
the SIGCHI conference on Human factors in computing 23. Mondada, L. Talking and driving: multi-activity in the
systems (2007),125-134 ACM. car. Semiotica, 2012, 191 (2012), 223-256.
8. Church, K. and Oliver, N. Understanding mobile web 24. Murtagh, G. Seeing the 'rules': preliminary observations
and mobile search use in today's dynamic mobile on action, interaction and mobile phone use. In Wireless
landscape. In Mobile HCI 2011 (2011),67-76 ACM. World, Springer (2001).
9. Froehlich, J., Chen, M. Y., et al. MyExperience: a 25. Palen, L. and Salzman, M. Voice-mail diary studies for
system for in situ tracing and capturing of user feedback naturalistic data capture under mobile conditions.
on mobile phones. In MobiSys '07 (2007),57-70 ACM. ACM, City, 2002.
10. Golledge, R. G. Human Wayfinding and cognitive 26. Palen, L., Salzman, M., et al. Going wireless: behavior
maps. In Wayfinding behavior, cognitive mapping and & practice of new mobile phone users. In Proceedings
other spatial processes, The John Hopkins University of CSCW '00, ACM (2000), 201-210.
Press (1999). 27. Psathas, G. Conversation analysis: The study of talk-in-
11. Goodwin, C. Notes on story structure and the interaction. Sage Publications, Incorporated, 1994.
organization of participation. In Structures of social 28. Rodriguez, B. Amid GPS boom, paper maps hang on,
action (1984), 225-246. 3rd July 2012. USA Today(2012).
12. Greiffenhagen, C. and Watson, R. Visual repairables: 29. Scollon, C. N., Kim-Prieto, C., et al. Experience
analysing the work of repair in human–computer sampling: Promises and pitfalls, strengths and
interaction. Visual Communication, 8, 1 (2009), 65-90. weaknesses. Journal of Happiness Studies, 4, 1 (2003),
13. Heath, C. and Luff, P. Technology in action. Cambridge 5-34.
university press, Cambridge, 2000. 30. Tse, E., Shen, C., et al. How pairs interact over a
14. Heath, C., Svensson, M. S., et al. Configuring multimodal digital table. In Proceedings of the SIGCHI
awareness. Computer Supported Cooperative Work conference on Human factors in computing systems
(CSCW), 11, 3 (2002), 317-347. (2007),215-218 ACM.
15. Heath, C. C. and Luff, P. Collaboration and control: 31. vom Lehn, D., Heath, C., et al. Exhibiting Interaction:
Crisis Management and Multimedia Technology. Conduct and Collaboration in Museums and Galleries.
CSCW, 1 (1-2)(1992), 69-94. Symbolic Interaction, 24, 2 (2001), 189-216.
16. Hindmarsh, J., Fraser, M., et al. Object-focused 32. Weilenmann, A. and Larsson, C. Local Use and Sharing
interaction in collaborative virtual environments. ACM of Mobile Phones. In Wireless World: social, cultural
Transactions on Computer-Human Interaction and interactional aspects of wireless technology,
(TOCHI), 7, 4 (2000), 477-509. Springer Verlag (2001).
2

1040

View publication stats