Des Form 2013

Lin-Lin Chen, Tom Djajadiningrat, Loe Feijs, Simon Fraser, Jun Hu, Steven Kyffin, Dagmar Steffen
Design and semantics

of form and movement
DeSForM 2013
DeSForM 2013 is supported by:
A catalogue record is available from the Eindhoven University of Technology Library.

ISBN 978-90-386-3462-3
COVER: 2013, Experio project (see Experio: a laser-triggered dance music generator, page 170).
PHOTO: TOM VAN ROOIJ
Lin-Lin Chen, Tom Djajadiningrat, Loe Feijs, Simon Fraser, Jun Hu, Steven Kyffin, Dagmar Steffen
Design and semantics

of form and movement
DeSForM 2013
Contents
005
DeSForM 2013 Program Committee
006
Foreword
010
Programme DeSForM 2013
012
014
015
016
017
Keynotes
Chris Dorsett At a moments notice, according to the pleasure of the holder. the semiotics of paper
David Frohlich Framing the design of new media experiences
Stephan Wensveen Design Research through Practice. From the Lab, Field, and Showroom
Xiaoyou He Interaction & Fusion
Xiangyang Xin Intention, not just in the minds of Designer
018
019
020
021
022
023
Seminar
Steven Kyffin Design research in the academic context: practice vs scholarship
Loe Feijs Product semantics: Quo Vadis
Dagmar Steffen The interplay of design practice and design research
Edgar Rodriguez How practicing designers can engage in academic research
Tom Djajadiningrat Meddling or helpful? The Semantics of Decision Support UIs
Christoph Bartneck Academic Publications For Designers
024
034
041
053
064
071
085
100
108
119
128
137
143
152
159
Papers
Alan Young The Value of a Semiotic Sensibility in Graphic and Communication Design
Ozge Merzali Celikoglu Three dimensions of lace: Reading the meaning of a traditional product from
different perspectives
Ehsan Baha, Dirk Snelders, Yuan Lu and Aarnout Brombacher Retracing an Evolution of Meanings
for Design-Driven Innovation
Evelien Van de Garde-Perik, Federico Trevia, Adam Henriksson, Luc Geurts and Helle Ullerup
Getting a GRIP at the Design of a Nature Inspired Relaxation Space for Work-Related Stress
Christoph Bartneck Robots in the Theatre and The Media
Sara Colombo and Lucia Rampino Beyond Screens. Exploring product dynamic features as
communication means
Heather Rene Barker Hive-Design for Big Data
Colin Kennedy Designing to Scale: Social Relations Design
Sophie Brenny and Jun Hu Social Connectedness and Inclusion by Digital Augmentation in Public Spaces
Loe Feijs Multi-tasking and Arduino. Why and How?
Gustavo Ostos Rios, Mathias Funk, Bart Hengeveld and Joep Frens EMjam: Jam with your Emotions
Christopher Henley, Yaakov Lyubetsky, Ajay Ravindran, Canute Haroldson and Mark Baskinger
Project Loci: Haptic Interactions Influence Situational Awareness
Edgar Rodrguez Ramrez and Kah Chan Smart interactions for home healthcare: A semantic shift
Jing Gu, Yu Zhang and Jun Hu Design for Elderly with Dementia: Light, Sound and Movement
Dominika Potuzakova and Loe Feijs Oris: Bonding through maternal scent
Design and semantics of form and movement
170
175
182
185
189
192
196
199
204
209
211
053
Interactive Demos
Manon Junggeburth, Luca Giacolini, Tom van Rooij, Bastiaan van Hout, Bart Hengeveld, Mathias Funk
and Joep Frens Experio: A Laser-Triggered Dance Music Generator
Tetske Avontuur, Eveline Brink, Iuliia Malyk, Derec Wu Sentiment Ninja: Sentiment Mining on Twitter
based on Topic and Location
Cheng Zheng Visual-Tangram: Tangible Puzzle for Childrens Intelligence Exploration
Zhiyuan Zheng, Linkai Tao, Loe Feijs and Jun Hu Two design cases of social network for parents in the
context of premature birth
Yumei Dong, Xiaohe Luo, Xiaoyu Tan, Miaosen Gong, Wei Wu Mobile Fetal Surveillance:
A product service system design for pregnant women
Martijn ten Bhmer, Eunjeong Jeon and Kristi Kuusk Vibe-ing: Designing a smart textile care tool for
the treatment of osteoporosis
Shengxiong Zhang, Tiantian Yang and Feng Wang Social Blobs, an Interactive Art Installation in an
Urban Public Space
Daniel Cermak-Sassenrath Makin Cake and the Meaning in Games
Kang Kai, Tiantian Yang and Feng Wang Interactive Art Installation for Creating Sense of Belonging in
an Working Environment
Hanqi Zhang, Dan Gao, Miaosen Gong, Yawei Yin and Yiyi Zhang EVGO: A tour service system with
electrical vehicle
Joep Frens, Mathias Funk, Jun Hu, Shengxiong Zhang, Kai Kang and Feng Wang Exploring the Concept
of Interactive Patina of Culture
DeSForM 2013 Program Committee
PC Chairs
Jun Hu (Chair)
Guest Professor, School
of Digital Media, Jiangnan
University, China
Associate Professor,
Department of Industrial
Design, Eindhoven University
of Technology, The Netherlands
Feng Wang (Co-chair)

Professor, School of Digital
Media, Jiangnan University,
China
Loe Feijs (Co-chair)

Professor, Department of
Industrial Design, Eindhoven
University of Technology,
The Netherlands
Seminar Chair
Jun Hu
Guest Professor, School
of Digital Media, Jiangnan
University, China
Associate Professor,
Interactive Demos Chair

Feng Wang
China
Design Competition Chair

Feng Wang
China
Conference Secretaries
Tiantian Yang
School of Digital Media,
Jiangnan University, China
Xingwei Wang
Visual Design
Xingwei Wang
DeSForM Committee
Lin-Lin Chen
National Taiwan University of
Science and Technology, Taiwan
Tom Djajadiningrat
Philips Design Eindhoven,
The Netherlands
Loe Feijs
Eindhoven University of
Technology, The Netherlands
Simon Fraser
Victoria University of
Wellington, New Zealand
Jun Hu
Steven Kyffin
University of Northumbria,
Newcastle Upon Tyne, United
Kingdom
Dagmar Steffen
Lucerne University of Applied
Sciences and Arts, Switzerland
DeSForM 2013 International

Program Committee
Lin-Lin Chen
National Taiwan University of
Science and Technology, Taiwan
Tom Djajadiningrat
Philips Design Eindhoven,
The Netherlands
Loe Feijs
Simon Fraser
Jun Hu
Steven Kyffin
University of Northumbria,
Newcastle Upon Tyne, United
Kingdom
Dagmar Steffen
Lucerne University of Applied
Sciences and Arts, Switzerland
Christoph Bartneck
University of Canterbury,
New Zealand
Mathias Funk
Technology, the Netherlands
Joep Frens
Technology, the Netherlands
Michael Lyons
Ritsumeikan University, Japan
Michael Hohl, Bauhaus
University, Germany
Edgar Rodriguez Ramirez
Miaosen Gong
School of Design, Jiangnan
University, China
Stephan Wensveen
University of Southern
Denmark
Foreword
DeSForM 2013
Welcome to Wuxi! Welcome to Jiangnan University!

After successful conferences in Europe and Asia, and
an exciting and inspiring long trip to New Zealand in
2012, DeSForM comes to China, where the making has
been happening and innovation is landing. The School
of Digital Media of Jiangnan University is honored to
host the 8th international conference on Design and
Semantics of Form and Movement.
Imaging Laboratory and The Digital Interactive

Technology Laboratory. As one of the first and leading
Chinese universities in the discipline of design, Jiangnan
University has an ambition in not only continuing leading
in traditional design, but also prevailing in designing
digital and interactive products and systems. We hope
that the DeSForM conference will contribute to this
ambition and the new development in this direction.
Wuxi is a beautiful city located in southeast China,

being famous for its culture, architecture, waterway
transportation and various art forms. It is also one of
the birth places of modern Chinese manufacturing and
textile industries, having a history of business people
involved in modern Shanghai commerce since the early
20th century and a splendid history of over three
thousand years. Wuxi claims to be the Pearl of Tai Lake.
The DeSForM conference is intrigued by the notion that

the nature of things, the essence of what a fabricated
object is, was about to be and really should be
completely and fundamentally questioned, re-defined, and
explored. We are proposing some challenging questions,
which in some way define the scope and the key themes
for the conference; these include
The Aesthetic of Interaction & Digital Representation:
What are the key DeSForM insights concerning: Design
and meaning, design and aesthetics, new forms, new
materials, new media.
Co-Creation and Globalism: What are the current
challenges concerning Global design, multicultural
design, translational design, cooperative design,
especially perhaps in the China context.
Social Computing and the Nature of Dynamic STUFF:
How might Designers respond to; and what are the
key changes and challenges we see in the role of
design, the nature of systems design, service design
and social design; especially when considering the
Emerging from the Department of Animation in the

School of Design and the former Department of
Digital Media Technology in the School of Information
Technology, the School of Digital Media of Jiangnan
University was established in 2009 to meet the strong
demands of the increasingly rapid development of
digital media industry for high-level design and technical
professionals. The school consists of the Digital Arts
Department, the Digital Media Technology Department
and the Experiment Center. The Experiment Center
includes the Digital Virtual Laboratory, the Digital
6
potentially pivotal effects of computational networks

on Social Cohesion in the Global contexts (i.e. Social
Computing)
DeSForM is about Design and Semantics of Form and
Movement. The making of meaning through matter
is of increasing relevance to the practice of design in
the twenty-first century. Rapidly evolving techniques
and processes are creating pathways for new modes
of interactivity, expression, and experience. These
modalities offer expanded opportunities and contexts
for design to communicate meaning through form,
movement and experience.
Since the first DeSForM conference, the field of design
has been changing a lot, but not as much as it should
have, to respond to significant societal changes. One
important external change is the development and
widespread application of social computing. Social
media fundamentally change the relationships between
people and it changes their behavior. This is significant
for design and at DeSForM 2013 we ask: how should
design respond. Any innovative system, service, or
related product is a combination of material concepts
and digital concept. Material form-giving and digital formgiving. Which are the new tools that the design research
community will offer to cope with this new complexity?
we assign meaning to the movement of artifacts and our

interactions with them without relying on the meaning
of their three-dimensional form? These questions are
becoming urgent now meanings are modulated and
amplified by the Internet, smart phones and social
media. We should also be prepared for new actuators:
smart polymers, memory metals, scent and light as
media. What will our design process look like when the
constraints of the traditional form are disappearing?
This years DeSForM conference tries to address these
questions from different perspectives by authors from
the Netherlands, Italy, Czech, Denmark, Turkey, United
Kingdom, United States, New Zealand and China,
covering wide range of topics in 15 long papers and
by 11 interactive demos, which are selected from 42
submissions.
We are also honored to have five renowned keynote
speakers (3 from Europe, and 2 from China) joining us
to address these questions for the conference:
Prof. Chris Dorsett from Department of Arts, School
of Arts, Design and Social Sciences, Northumbria
University to share with us his interests in the semiotics
of paper, in which he explores the diverse forms of
graphic innovation required to realize symbolic value
transactions on paper.
Another change is continued globalization. The

countries and the peoples of this planet are becoming
more and more connected, through travel, migration,
trade, the Internet, and new media. Communication and
exchange between cultures is an essential ingredient of
design. Already since the beginnings of Industrial Design,
the designed objects have been considered culturally
important. Nowadays we understand that the output of
our design work has the power to change the behavior
of people. DeSForM always has been trying to pave the
way for objects displaying active behavior. We have to
ask: what are the best state-of-the-art examples where
design is mediating between cultures, countries, and
people. And the new tools that help us when designing
in a multicultural setting?
Prof. David Frohlich, director of Digital World

Research Centre at the University of Surrey and
Professor of Interaction Design. In his talk, he will
examine the relationship between products and media,
especially in the early stages of product design when
their properties are fluid and functionalities are still
being defined.
And still some of the core-questions of DeSForM

have been answered only partially so far. These are
the questions about our fundamental understanding of
meaning and movement. How does meaning arise? Can
Prof. Xiaoyu He, vice president of Nanjing University

of Arts. He will share with us how we could introduce
Chinese traditional culture to modern design and focus
on research on the fusion state of people, product and
Dr. Stephan Wensveen, associate professor

Interaction Design at the Mads Clausen Institute,
University of Southern Denmark. He will talk about
Constructive Design Research. Multiple examples will
illustrate this type of design research that highly values
imagining and building new things while describing and
explaining these constructions.
environment. It will emphasize not on the interactive

relations between people and product, but on the
fusion state of people and creations.
Prof. Xiangyang Xin, dean of the School of Design,
Jiannan University. He will tell us that it is important
to recognize the role of intention in the realization and
utilization of the things. His lecture will look for sources
intention, and searches for preliminary strategies for
identifying attributes and functions of intention, not just in
the creation but also the utilizations of man-made objects.
This years DeSForM is also preceded by a seminar
Design for Research, in which the experts will share
their observations and experiences how design can
contribute to research, and discuss how designers in the
academia can meet their design background with the
academic challenges.
Prof. Loe Feijs will raise questions around product
semantics and share his knowledge about the structure
of product semantics and how they can effectively
communicate and help users to effectively communicate.
Dagmar Steffen will argue that the interplay of
design practice and design theory as it takes place in
practiceled design research creates a mindset that is
beneficial for the discipline in general and especially for
design education since it contributes to bridging the gap
between unconsidered design practice on the one hand
and abstract design theory, bereft of any relevance for
practice on the other.
Dr. Edgar Rodriguez will discuss how the practice
of design produces knowledge, why academia expects
research to produce explicit knowledge and how
practicing designers can engage in academic research
without compromising the quality of their practice.
Prof. Steven Kyffin will introduce some of Northumbria
Universitys Design Research work in the context of the
paradox of Practice and Scholarship and discuss some of
the challenges facing Design Academics as they develop
their design careers in Academia.
Dr. Tom Djajadiningrat will share with us his
experience at Philips Design where research has its
culture, showing us that there are different levels of
8
decision support in a design process, from merely

offering information which helps informed decisions,
via guidance which nudges the user towards particular
behavior or choices, up to pro-active identification and
correction of poor decisions.
Dr. Christoph Bartneck will talk about academic
publications for designers. In his opinion, the academic
publication process has some fundamental flaws that
inhibit the publication of innovative ideas. He will reflect
on some of the underlying problems of putting design
work into the scientific context and will present insights
into how we can overcome some of the issues of the
academic publication process.
Alongside this years DeSForM conference, for the
first time, we will also organize a design competition.
Five student teams from top Chinese design schools
are invited to the competition in the area of designing
for public spaces using digital augmentation to physical
spaces or objects. All the teams are provided with
earlier technology support over distance and five
scale models of buildings or statues will be revealed
on the first day of the conference and the challenge
is to augment these models with digital media and
with meaningful interactions. The experts from the
DeSForM conference will provide guidance throughout
the competition we hope that the knowledge from
the DeSForM conference will have a direct influence
on the young designers, and contribute to their growth
and development. The winning team will also get an
award flight tickets to the next DeSForM conference,
to promote the DeSForM conference among the young
designers.
The conference can only take place in this form thanks
to the excellent local administrative and financial support
from the School of Digital Media, Jiangnan University.
We would also like to thank the sponsorship of Philips
Design, and the TU/e DESIS Lab and the Out of
Control theme of the Department of Industrial Design,
Eindhoven University of Technology. Thanks to the Wuxi
Museum of Chinese Calligraphy and Paintings, the
conference can be held at a place with a rich culture
and history, in a beautiful nature. The IFIP WG14.7
workgroup in Art and Entertainment, the DESIS
network and the Cumulus Association endorsed the
conference by spreading the CFP to their members.
Special thanks to our local organizers and the

supporting team, not mentioning them all, especially our
local co-chair prof.dr. Feng Wang, and our conference
secretary ms. Tiantian Yang and ms. Xingwei Wang, for
taking care of the details with a great amount of effort
and patience.
We hereby offer the 8th DeSForM Proceedings. We
would like to thank all the authors who have submitted
their work to DeSForM 2013 as well as the reviewers
for their constructive and critical comments. We hope
the presentations and demonstrations will raise many
questions and inspire lively discussion and debate about
the increasing sophistication and complexity of design
research, theory and practice in a constantly changing
global context.
Wuxi, August 18, 2013

Lin-Lin Chen,
National Taiwan University of Science and Technology,
Taiwan
Tom Djajadiningrat,
Philips Design Eindhoven, The Netherlands
Loe Feijs,
Eindhoven University of Technology, The Netherlands
Simon Fraser,
Victoria University of Wellington, New Zealand
Jun Hu,
Steven Kyffin,
University of Northumbria, Newcastle Upon Tyne,
United Kingdom
Dagmar Steffen,
Lucerne University of Applied Sciences and Arts,
Switzerland
Programme DeSForM 2013

10.00 19.00
13.00 18.00
19.00
September 22
Conference Registration
Seminar Design For Research
Reception dinner

09.30 10.00
10.00 11.00

11.00 11.30
12.00
14.00 15.00

15.00 15.30
15.30 17.00

19.00
September 23
Opening speeches
Keynote 1
David Frohlich
Framing the design of new media experiences
Opening (design competition)
Lunch
Keynote 2
Chris Dorsett
At a moments notice, according to the pleasure of the holder. the semiotics of paper
Tea
Paper session 1: Meaning and semantics
Alan Young
The Value of a Semiotic Sensibility in Graphic and Communication Design
Ozge Merzali Celikoglu
Three dimensions of lace: Reading the meaning of a traditional product from different perspectives
Ehsan Baha, Dirk Snelders, Yuan Lu and Aarnout Brombacher
Retracing an Evolution of Meanings for Design-Driven Innovation
Dinner

09.00 10.00

10.00 11.00
11.00 12.30

12.30 14.00
14.00 15.00

15.00 16.00
September 24
Keynote 3
Stephan Wensveen
Design Research through Practice. From the Lab, Field, and Showroom
Tea / Demos
Paper session 2: Design, exploration and frameworks
Christoph Bartneck
Robots in the Theatre and The Media
Sara Colombo and Lucia Rampino
Beyond Screens. Exploring product dynamic features as communication means
Lunch
Keynote 4
Xiaoyou He
Interaction & Fusion
Tea / Demos
10
16.00 17.30

19.00
Paper session 3: Big data and social design

Heather Rene Barker
Hive-Design for Big Data
Colin Kennedy
Designing to Scale: Social Relations Design
Sophie Brenny and Jun Hu
Social Connectedness and Inclusion by Digital Augmentation in Public Spaces
Dinner

09.00 10.00

10.00 11.00
11.00 12.30

12.30 14.00
14.00 15.30

15.30 16.00
16.00 17.00
17.00 17.30
19.00
September 25
Keynote 5
Xiangyang Xin
Intention, not just in the minds of Designer
Tea / Demos
Paper session 4: Interaction and experience
Loe Feijs
Multi-tasking and Arduino. Why and How?
Gustavo Ostos Rios, Mathias Funk, Bart Hengeveld and Joep Frens
EMjam: Jam with your Emotions
Christopher Henley, Yaakov Lyubetsky, Ajay Ravindran, Canute Haroldson, Mark Baskinger
Project Loci: Haptic Interactions Influence Situational Awareness
Lunch
Paper session 5: Healthcare
Edgar Rodrguez Ramrez and Kah Chan
Smart interactions for home healthcare: A semantic shift
Jing Gu, Yu Zhang and Jun Hu
Design for Elderly with Dementia: Light, Sound and Movement
Dominika Potuzakova and Loe Feijs
Oris: Bonding through maternal scent
Tea
Presentations (Design Competition)
Closing speeches
Closing Dinner
September 26
Cultural events
11
Keynotes
prof. Chris Dorsett
Professor of Fine Art, Department of Arts, School of Arts, Design and Social Sciences,
Northumbria University (UK)
At a moments notice, according to the pleasure

of the holder: the semiotics of paper
For contemporary graphic artists the detachment of the ocularcentric virtual from the sensory pleasures of the hand is accepted
as a necessary consequence of moving ones attention from a
computer screen to a printed page. In the Art and Design sector
we continue to celebrate well-made products but enthusiastically
embrace the creative restlessness of ongoing deferral made
possible by electronic information, an enthusiasm that encourages
us to write many more emails than we would letters and build
a series of websites when, in the past, we would have curated
a single exhibition. As a creative practitioner whose research is
realised primarily through drawing and photography I wrestle with
the potent finiteness of blank sheets of paper on a day-to-day
basis and it is the redefinition of this experience in relation to the
indefiniteness of digital technology that will serve as the topic of
my presentation.
And so: do Ming Dynasty circulating treasure notes raise important

questions about graphic images on a computer screen? My answer
will be yes, and these questions help us set out parameters for
a contemporary semiotics of paper in which, following debates
about touch-based experiences in David Howes Empire of Senses
(2005) and my own Things and Theories: the unstable presence of
exhibited objects (2011), the pleasure of the holder must play
a significant role.
Barry, K. (2007) The Aesthetics of Paper Money: national differences during

the period of enlightenment and romanticism, in Duff, D. & Jones C. (eds)
Scotland, Ireland and the Romantic Aesthetic, Lewisburg, PA: Bucknell,
pp.55-76.
Paper Money and English Romanticism, Times Literary Supplement, No.
4899, 21 February 1997: 14-16.
My title quotes the British political economist David Ricardo

(1772 1823) who argued that the introduction of paper money in
late 18th century Britain would lead to uncontrollable fluctuations
in the value of currency unless the scope to convert each note
into metal was, for the bearer of these new fangled sheets of
printed paper, executed on demand that is, at a moments notice,
according to the pleasure of the holder (Barry, 2007: 68). It is
difficult not to see parallels to our current situation. For Ricardo,
paper money severed the representational force of a signifier
from the intrinsic value of that which was signified and this same
semiotic rupture, I suggest, shapes the virtual and artefactual
tensions of today (see, for example, Rawsthorn 2013)
However, the literary historian Kevin Barry (1997, 2007) has linked
the deferred promise to pay of the first paper pounds to the
open-ended character of the 18th century creative imagination
and my presentation will explore this same symbiosis of monetary
and aesthetic exchange in our own period of economic fluctuation.
I will also offer scope for creative reflection on curatorial
treatments of printed currency at Princeton University Library
(Stahl, 2010) and the coin and banknote displays at Oxford
Universitys Pitt Rivers Museum. This latter collection represents
numismatic problem-solving and innovation in the widest
geographic and historical contexts offering, it is my contention,
endless opportunities to model the restlessness of the creative
imagination in general.
12
Dorsett, C. (2011) Things and Theories: the unstable presence of exhibited

objects. In Dudley, S., Barnes, A. J., Binnie, J., Petrov, J., & Walklate, J. (eds)
The Thing about Museums: Objects and Experience, Representation and
Contestation, London and New York: Routledge.
Howes, D (2005) Empire of the Senses. Oxford: Berg
Rawsthorn, A. (2013) Hello World: where design meets life, London: Hamish
Hamilton.
Stahl, A. (curator) (2010) Money on Paper: Bank Notes and Related Graphic
Arts from the Collections of Vsevolod Onyshkevych and Princeton
University, Leonard Milberg Gallery for the Graphic Arts, Firestone
Library, Princeton University.

Chris Dorsett is an artist and exhibition
curator whose career has been built
on cross-disciplinary collaborations
with collection-holding institutions such
as the Pitt Rivers Museum in Oxford
and the Royal Swedish Armoury,
Stockholm. His curatorial activities also
include fieldwork residencies in the
Amazon (organised with the Centre for
Economic Botany, Kew) and the walled
village of Kat Hing Wai (commissioned
by the Arts Development Council of
Hong Kong). Dorsett seeks to resituate
the changing aesthetic and political
ambitions of the visual arts within the
widest range of historical and scientific
contexts and, as a Professor of Fine
Art at Northumbria University, he
continues to interrogate museological
claims about the construction of
knowledge through exhibition display.
He publishes with Routledge for the
museums studies sector and was
recently commissioned to lead Cast
Contemporaries, an exhibition about
the fate of anatomical and sculpture
cast collections in art schools (2012
Edinburgh Arts Festival).
13
Keynotes
prof. David Frohlich
Director Digital World Research Centre
University of Surrey
Framing the design of new media

experiences
Many modern products and services involve the presentation of
digital media content of various kinds. Sometimes this content
defines the form and function of the product, as with televisions,
music players or e-books. In other cases the relationship is less
clear, as with smart phones or computers, which are really conduits
for multiple types of media and experiences.
In this talk, I examine the relationship between products and
media, especially in the early stages of product design when their
properties are fluid and functionalities are still being defined. I argue
for attention to five key factors which frame the design of new
media experiences, and shape their character and meaning. These
include Form, Reason, Audience, Material and Extent (FRAME). To
illustrate this framework, I show how the factors interact in the
design of three digital photography systems originating from both
corporate and university research labs. These include an interactive
desk for playing the sound of printed photographs, a live digital
storytelling system, and a kind of visual twitter display. In each case,
it is possible to see how small changes in framing shift the resulting
design and experience, and how the trick of good design is to find
an alignment of factors that works for users. The complexity
of interactions between factors often leads to surprises in the
behavior of such systems, and suggests an open-minded exploratory
approach to their design.
David Frohlich is Director of Digital

World Research Centre at the
University of Surrey and Professor
of Interaction Design. He joined the
Centre in January 2005 to establish a
new research agenda on user-centred
innovation in digital media technology.
Current work includes a mixture of
PhD and Research Council projects
exploring a variety of new media
futures relating to digital storytelling,
personal media collections, and
community news and arts (http://
www.dwrc.surrey.ac.uk/). His recent
2012 book with Risto Sarvas contains
the first history of personal digital
photography:From snapshots to
social media: The changing picture of
domestic photography. Prior to joining
Digital World, David worked for 14
years as a senior research scientist at
HP Labs, conducting user studies to
identify requirements and test new
concepts for mobile, domestic and
photographic products. This allowed
him to pursue ongoing research
interests in tangible interfaces to
computing, new media design, and the
global digital divide. Some of this work
was documented in two books entitled
Audiophotography: Bringing photos
to life with sounds, andContextual
Innovation: Creative approaches to
innovation in emerging markets.
David has a PhD in psychology from
the University of Sheffield and postdoctoral training in Conversation
Analysis from the University of York.
He has also worked as a Human Factors
Consultant and Research Psychologist,
and held visiting positions at the
Helen Hamlyn Research Centre, Royal
College of Art, and the Department
of Psychology, University of York.
He is currently Visiting Professor at
Manchester Business School and is
founding editor of the international
journal Personal and Ubiquitous
Computing.
14
Keynotes
dr. Stephan Wensveen
Associate Professor
University of Southern Denmark
Design Research through Practice.

From the Lab, Field, and Showrooms
This presentation talks about Constructive Design Research
as brought forward in the co-authored book Design Research
Through Practice. From the Lab, Field, and Showroom (Koskinen,
Zimmerman, Binder, Redstrm and Wensveen, 2011). Multiple
examples will illustrate this type of design research that highly
values imagining and building new things while describing and
explaining these constructions. The basis for the book and the
presentation are three observations:
Methods in design research proliferate, but there are three
main methodological approaches: Lab, Field and Showroom.
These approaches share a theoretical and philosophical basis
that sets them apart from earlier attempts to turn design into
a science.
Design is not only the object of research, but the very means
of doing it.
This last observation will be deepened by focusing on the roles that
prototypes and the act of prototyping can play in design research.
In this presentation we will celebrate the diversity of how designing
can generate knowledge in design research.
Stephan Wensveen (1970) is Associate

Professor Interaction Design (since
2011) at the Mads Clausen Institute,
University of Southern Denmark. He
studied, educated and researched
Industrial Design Engineering at
Delft University of Technology. His
research on the relationship between
emotions, expressivity and product
design started in 1999 and resulted
in a PhD thesis, which is seen by the
design research community as an
canonical example of Research through
Design. The main contributions were a
validated prototype of an emotionally
intelligent alarm clock and a theoretical
framework for interaction called
Interaction Frogger. He has enjoyed
to be part of, and be of influence
on internationally renowned design
research groups, i.e., the Delft IDStudioLab (lead by Kees Overbeeke,
Pieter Jan Stappers, Paul Hekkert &
David Keyson), the Eindhoven Designed
Intelligence and Designing Quality
in Interaction group (lead by Loe
Feijs, Kees Overbeeke and Caroline
Hummels) and the Snderborg
Participatory Innovation Research
centre (lead by Jacob Buur).
His interest is in using the power of
design to integrate research, education
and innovation, which he demonstrated
as project leader for the /d.searchlabs and as initiator and research
director of Wearable Senses and the
Dutch nationally funded project on
Smart Textile Services. He has any of
his papers are part of the standard
curricula in interaction design schools
and he is co-author of the book Design
Research through Practice. In 2011
he expanded his horizon on multidisciplinary design, when he joined the
Snderborg Participatory Innovation
Research group of Prof. Jacob Buur
at MCI to focus on the Aesthetics of
Participation.
15
Keynotes
prof.dr. Xiaoyou He
Vice President of Nanjing University of Arts,
PHD Supervisor
Interaction & Fusion

Since Bill Moggridge of IDEO formally proposed the concept of
interaction design in 1990, it has become a hot topic in design
thinking and methods.
Interaction design seems to be an alternative concept to product
design. Traditionally, product design concerns the function,
structure, human factors, form and styling, color, environmental
effect and other design elements such as technology, methods
and means of achieving other functions. While interaction design
emphasized on the interactive behaviors between product system
and end-users, on the technical support to realize the defined
functions and emotional communication between both interactive
sides. These are believed to have the direct impact on the end users.
We have no doubt on the development of this concept as well as the
changes it has caused. In the meanwhile, we would like to ask is this
the most reasonable design idea?
Interaction design, is still emphasized on the interaction between
this and that. According to this, it is the interaction of two
individuals. If we can change another way to think, if the problem
we are going to solve is not the problem between product and
people, it is the problem of people themselves, what it is now?
If we trace back to Chinese traditional wisdom of creation, there are
a lot of ideas worthy of research on design thinking and methods.
Chinese traditional thought of the theory that man is an integral
part of nature
Chinese traditional philosophy of Yin Yang complementarity
Chinese traditional artifact of two is made of one
Chinese traditional human relations of a man of kindness is
benevolent people
In Chinese traditional culture, you are among us and we are among
you is much more recommended than you and he. This describes
a fusion state which is believed to inherit the spirit of symbiotic
harmony.
By introducing Chinese traditional culture to modern design, we
can focus on research on the fusion state of people, product and
environment. It will emphasize not on the interactive relations
between people and product, but on the fusion state of people
and creations. All in all, it is not to design an object, but to design
people in order to create a fusion state of great coordination.
16
Deputy Director of Committee of

Design Education, Ministry of Higher
Education
Research Fellow of China Institute of
Chinese Art
Research Fellow at Public Art in China
National Art Academy
Director of Committee of Design
Research, China Industrial Design
Association
Deputy Director of Committee of
Art & Design, China Association of
Higher Education
Member of Industrial Design Art,
China Artists Association
Vice President of Association of
Industrial Design in Jiangsu Province
Keynotes
prof.dr. Xiangyang Xin
Dean, School of Design, Jiangnan University, China
Intention, not just in the minds

of Designers
While designers are expected to create future
scenarios of peoples lives through objects, services
and/or systems, it is important to recognize the role of
intention in the realization and utilization of the things.
Often people think that intentions locate in the minds
of designers, and lead to the choices of problems to be
addressed, things to be made and details that ensure
things to be appreciated. It is true that designers
intention play important roles in the creation of our
material environment. What is not understood is that
intentions are not independent things live the minds of
designers, but outcomes of complex social interactions.
This lecture looks for sources intention, and searches
for preliminary strategies for identifying attributes
and functions of intention, not just in the creation but
also the utilizations of man-made objects. Instead of
providing developed understanding of intention, here
the goal is to promote interests and conversations of
issues surrounding intention.
Xin holds a PhD in Design from

Carnegie Mellon University with
research interests in interaction and
service design, cultural studies, and
product development. He looks
at how design, both professionally
and philosophically, contributes to
transformations of lifestyles, businesses
and societies. Xin is currently leading
a Redesign Design Education
movement in China. It aims to reform
design education responding to the
extending scope and evolving principles
of design, from logic of things to logic
of behaviors. Xin has internationally
spoken and conducted workshops
on interaction design and cultural
innovation, served as a guest professor
at different universities, actively served
for several professionally communities
including CIDA, IDSA, IXDC, IXDA. His
professional practices covers a variety
of industries with partners including
United States Postal Services, Efficiency
Unit of Hong Kong Government, P&G,
Changhong, ASTRI (Applied Science
and Technology Research Institute Ltd.,
Hong Kong), Philips and etc.
17
Seminar: Design for Research

prof. Steven Kyffin
Dean of the Faculty of Arts, Design and Social Sciences,
Northumbria University
Design research in the academic

context: practice vs scholarship
Many of us have experienced that the notion of Design Research has often
been the cause of much conflict, creative conflict, in both the Academic and the
Commercial contexts alike. Design is a Research activity in itself. So, what does
Design Research mean to us in Academia and in the 21st Century?
How do we understand these concepts, how do we make sense of them and
how do we contribute to Transforming our futures; the future of our people,
the future of our world and the future of our Discipline while at the same time,
being and acting as a Designer?
Ever since I joined the Royal College of Art in 1981 I have experienced
something of the paradox, which the notion of Research proposes: one the
one hand an activity, which seeks to UNDERSTAND the world more fully
through Objective Scientific Objective Experimentation in Laboratory contexts,
while on the other hand the other strives to CHANGE the world through often
Subjective, Exploration in Community contexts. At a dangerously simplistic
level, one appears to be an act of Creation through Practice and the other an
act of Scholarship through Study, and yet somehow we can bring them together.
Both the Analytical and the Synthetical approaches do of course come together.
Of course as I believe that we are all Creative and as such have a desire to
create a better world for the future. But none the less the two approaches do
set up a tension in the academic contexts, which in time I believe can become
a most creative one from which we can all grow!
This is something that Id like to discuss during this seminar while at the same
time sharing some examples of the work we create as part of our Design
Research Programmes at Northumbria University.
18
Professor Steven Kyffin (Master of

Design, Industrial Design, Royal College
of Art, London & Honorary Doctorate
Northumbria, Newcastle) is Executive
Dean of the Faculty of Arts, Design
and Social Sciences at Northumbria
University, Newcastle in the United
Kingdom.
In this function he is responsible for all
aspects of the management of the Faculty,
one of four in the University. From 2010
until 2012, Steven was Dean of the
School of Design within Northumbria.
Northumbria Design focuses it teaching
programs on Fashion, Communication,
Industrial and innovation Design across
its Undergraduate and postgraduate
programs. Design Innovation is one of
the three core Research territories of
the school. The others include; The
future Roles of Design in Socio-Economic
contexts and Design Futures; a Design
led exploration of future qualities on Life
in the Health and Wellbeing and Digital
Life contexts.
Kyffins own Research has focused on
multi and trans disciplinary Innovation
in general and in particular on the roles,
which Experience Design and prototyping
can play in leading the Business and Social
Innovation processes.
Steven previously held the post of Senior
Global Director of Design Research
and Innovation at Philips Design in the
Netherlands. He was responsible for
Design Research for Philips Electronics
world-wide.
As a member of the Philips Design Global
Management Team, Steven headed up
Philips Designs European Commission
Research Relations Programme and led
the Philips Design University Relations
Programme.
Steven Kyffin is a member of
Northumbria Universitys Executive
Management Team, acts as a European
Commission research program reviewer
and holds Adjunct Professorships in
the schools of Design at Hong Kong
Polytechnic University and at JiangNan
University, Wuxi, China.

prof.dr.ir Loe Feijs
Department of Industrial Design,
Product semantics: Quo Vadis

Product semantics is a central theme in the field of Industrial
Design. Forms, either abstract or concrete always carry meanings.
It is the responsibility of designers to make good use of the
possibilities of these meanings. It is important for designers to
have access to and a full understanding of the structure of product
semantics and the relevant types of knowledge so that they can
effectively communicate and help users to effectively communicate.
In this presentation I will review a number of classical theories
about product semantics mostly Western theories. These include
the notions coming from semiotics, such as the work of Charles
Peirce and Umberto Eco. We shall go back the classic senderreceiver model of Claude Shannon, next to achievements of the
Offenbach school of product semantics and compare these with the
more recent semantic turn by Klaus Krippendorff. In engineering
traditions such as computer science, formalization has always been
a first step towards making more powerful tools and in this context
my own Commutative Product Semantics (CPS) will be explained.
Then I will introduce the terminology of closed and open semantics.
Whereas closed semantics is based on Peirces and Shannons classic
ideas, open semantics is about designs which work as a medium, a
carrier, or a tool. Open semantics allows the user to put his or her
own meanings into the design and connect these to new rituals.
Product semantics becomes a participatory process. When talking
about media, the theories of McLuhan are helpful too. I shall argue
that open and closed semantics both play and essential role, usually
even within a single design. All concepts will be illustrated by classic
and contemporary examples. The last part of the presentation will
be about the future of product semantics. Questions addressed
include: What is the impact of Social Computing upon the field of
product semantics? Why do most of our design tools not have any
semantic knowledge embedded? Can we deploy Semantic Web
technologies to advance the field of product semantics?
Loe Feijs (The Netherlands,

1954) is professor of Industrial
Design at Eindhoven University of
Technology. He holds a Masters
degree in Electrical Engineering
and a Doctor degree in computer
science. He has worked on formal
methods in Philips Research
Laboratories, he was director of
the Eindhoven Embedded Systems
Institute and Vice dean of TU/e
ID. At present he holds the chair
Industrial Design of Embedded
Systems. He teaches topics such as
Integrating Technology and Creative
Programming. His research interests
include creative programming,
product semantics, design for
neonatology, medical simulation and
biofeedback.
19

Dagmar Steffen
Lucerne University of Applied Sciences and Arts, School of Art
and Design (Switzerland)
The interplay of design practice and

design research
The relationship of scientific research in the field of design and design practice
was and still is under debate. For many decades, both activities were strictly
separated, but during the last two decades an increasing number of universities
of art & design around the world introduced so-called practice-led design
research or research through design. In my talk I will focus on three issues.
Firstly, I will argue, that the inclusion of material practice (doing & making)
in scientific research is not specific to design; indeed, in various disciplines
material practice has a long tradition (i.e. medicine, physics and archaeology)
and inevitably contributes to knowledge creation. Nonetheless the role
or function of practice in practice-led design research (including doctoral
research) demands closer examination and justification. Thus, secondly I will
elaborate the commonalities of and the differences between material practice
resp. the role of the artefact in research in the before mentioned disciplines
and in design research (the design object: as an experimental, hypothesestesting or explorative artefact; as a non-discursive symbolism; as a work in its
own right). Finally I will argue, that the interplay of design practice and design
theory as it takes place in practice-led design research creates a mindset that is
beneficial for the discipline in general and especially for design education since
it contributes to bridging the gap between unconsidered design practice on the
one hand and abstract design theory, bereft of any relevance for practice on the
other.
20
Dagmar Steffen is a researcher and

lecturer at the Lucerne University of
Applied Sciences and Arts, School
of Art and Design (Switzerland). She
works primarily in the areas of product
semantics, design and innovation and
design and sustainability. She studied
Product Design at the Academy of Art
and Design Offenbach (Germany) and at
the Edinburgh College of Art (UK). From
2000-2003 she has been a member of the
research staff at the Academy of Art and
Design Offenbach; from 2004-2007 she
was a visiting lecturer at the Bergische
University Wuppertal (Germany), while
joining part-time the SeFun (Semantic
Functions in Design) research project
at the Aalto University School of Art
and Design (Finland). She is author of
numerous publications (books, articles
and conference papers) and holds a PhD.

Dr Edgar Rodriguez
Victoria University of Wellington
How practicing designers can

engage in academic research
With the move in many countries to assess academics according
to their research performance in measurable ways, many design
practitioners moving to academia have faced the challenge of
justifying their practice as research. This has had several outcomes.
Firstly, some highly skilled designers receive low assessments as
their practice is not considered research, which in turn discourages
them to work in academia and it also discourages academic
institutions to hire them. Secondly, some highly skilled designers
reframe their activities and produce other kinds of academic
research outputs (i.e. journal articles) to the detriment of their
practice. Thirdly and most challenging, some highly skilled designers
frame their practice around well-defined research questions that
can only be answered through the practice of design. Their outputs
are sophisticated designs that add new tacit and explicit knowledge
to the discipline. These designers need research and practice skills
that, while unusually taught together, can benefit the academic and
practice areas of the discipline and bring them together to fill a gap
that shouldnt exist.
Through examples of design and research work, this seminar
will discuss how the practice of design produces knowledge, why
academia expects research to produce explicit knowledge and
how practicing designers can engage in academic research without
compromising the quality of their practice.
Dr Edgar Rodriguez is the

Programme Director in Industrial
Design at the School of Design,
Victoria University of Wellington,
New Zealand. His career as
industrial designer includes working
for design studios Sismo Design
(Paris), MDA (Mexico City),
Studio Santachiara (Milan) and
Samsung Electronics (Seoul). He
has produced designs that include
furniture, lighting, digital cameras,
TV sets, wearable computers,
medical devices and experimental
designs. He has exhibited and
published his research in dozens
of international conferences and
exhibitions. His current research
studies how human data including
physiological responses, behaviour
and emotions can be studied to
design smart interactions that help
people improve their lives.
21

Tom Djajadiningrat
Creative director innovation design at Philips Design
Meddling or helpful? The Semantics

of Decision Support UIs
The spiralling costs of healthcare are creating a need for medical products and
systems which can be used by less trained staff or even by patients themselves.
At Philips Design, this has led to an increased interest in clinical decision
support (CDS). The common understanding of CDS systems is that they assist
physicians and other health professionals with decision making tasks, such
as making a diagnosis based on patient data. However, we are interested in
applying the concept of CDS not only to diagnostics for healthcare professionals
but also to consumers struggling to make healthy choices.
An often neglected aspect is how user interfaces can offer CDS to the user.
Through an analysis of many decision support UI examples from fields other
than healthcare, I would like to show that there are different levels of decision
support, from merely offering information which helps informed decisions,
via guidance which nudges the user towards particular behaviour or choices,
upto pro-active identification and correction of poor decisions. The semantics
of the design execution of such CDS Uis largely determine whether CDS is
experienced as meddling and misinformed or quietly helpful and intelligent.
22
Tom Djajadiningrat is creative director

innovation design at Philips Design. In his
role as leading co-creator in a team of
complementary disciplines, his focus is
on helping Philips re-position itself as a
Health & Well-being company. Prior to
joining Philips, he worked in academia at
the ID-StudioLab of Delft University of
Technology, The Mads Clausen Institute
of the University of Southern Denmark
and the Design IntelligenceGroup of
Eindhoven University of Technology.

Dr. Christoph Bartneck
HIT Lab NZ of the University of Canterbury
Academic Publications
For Designers
The number of published papers is one of the main key performance
indicators for academics. Designers working at a university have
to live up to this expectation, but struggle with translating their
research output into disseminations that are being recognized
by colleagues from different disciplines. To make things worse, it
has been shown that the academic publication process has some
fundamental flaws that inhibit the publication of innovative ideas. I
will reflect on some of the underlying problems of putting design
work into the scientific context and will present insights into how
we can overcome some of the issues of the academic publication
process.
Dr. Christoph Bartneck is an

associate professor and director
of postgraduate studies at the
HIT Lab NZ of the University of
Canterbury. He has a background
in Industrial Design and HumanComputer Interaction, and his
projects and studies have been
published in leading journals,
newspapers, and conferences. His
interests lie in the fields of Social
Robotics, Design Science, and
Multimedia Applications. He has
worked for several international
organizations including the
Technology Centre of Hannover
(Germany), LEGO (Denmark),
Eagle River Interactive (USA),
Philips Research (Netherlands), ATR
(Japan), Nara Institute of Science
and Technology (Japan), and The
Eindhoven University of Technology
(Netherlands). Christoph is a
member of the New Zealand
Institute for Language Brain &
Behavior, the IFIP Work Group 14.2
and ACM SIGCHI.
23
Alan Young
AUT University, New Zealand
alan.young@aut.ac.nz
The Value of a Semiotic Sensibility in

Graphic and Communication Design
Abstract
At a time when multiplicities of forms, media,
technologies and pedagogical approaches abound, this
paper argues for semiotics as a unifying theoretical
foundation for the range of communication design
practices. This is not to say that semiotics should
be the only approach to design educationfar from
it. I argue that other approaches, including that of
connoisseurship have much to offer design students,
but that without semiotics, they stand to do as much
harm as good. The only viable approach to design
education is one that is holistic, and responsive to ever
changing global and local conditions and requirements.
As a way of making visible the often latent, yet potently
communicative aspects of all design artifacts, semiotics
offers a firm foundation through which all other
pedagogical approaches can benefit and from which
the meaning of design artifacts and production can be
understood. Semiotics enables one to explore, analyse,
and greatly enhance the creation of works not only
in graphic design, photography, and advertising, but
also sound production, product design, and any other
system where artifacts communicate to an audience or
market. Although this paper concentrates on graphic
and communication design, its reasoning can be applied
equally to these other areas. Although some might
suggest semiotics had its hay day in the 1970s and 1980s
and is now on the decline, I would argue it has never
reached its full potential in design discourse and there
24
is now, more than ever, the need for a reappraisal

of the value of semiotics in design pedagogy.
Keywords
Semiotics, design process, graphic design, communication
design, design history, social innovation, social justice
1 Introduction
Semiotics is the study of signs and their relation to
culture. Originally proposed by Ferdinand de Saussure
in his Course in General Linguistics, he saw it as a
science that would show what constitutes signs, what
laws govern them, but importantly, that studies the
life of signs within society [1]. Out of philosophy rather
than linguistics, Charles Sanders Peirce developed a
more relational model of the sign with the additional
consideration: It addresses somebody, that is, creates
in the mind of that person an equivalent sign, or perhaps
a more developed sign [2]. The connection of signs
and culture was developed by a number of theorists,
of whom perhaps the most significant for designers is
Roland Barthes, who defines the study thus:
Semiology aims to take in any system of signs, whatever their
substance and limits; images, gestures, musical sounds, objects,
and the complex associations of all of these, which form the
content of ritual, convention or public entertainment: these
constitute, if not languages, at least systems of signification. [3]
What Barthes is referring to include key aspects of

graphic design and it might therefore seem surprising
that semiotics, whilst appearing as a component part
of many graphic design courses, has not featured
more significantly within the discourse. Kress and
van Leeuwen [4] make the observation that:
Within the media, visual design is still the province of
specialists who generally see little need for methodical
and analytically explicit approaches, and rely on creative
sensibilities honed through experience (p. 12).
Storkerson [5] notes of semiotics in graphic design
education its apparent lack of broad visibility,
which he attributes to antinomies or contradictions,
due largely to incompatibilities in the key semiotic
approaches of Peirce and de Saussure, as well as a
range of other difficulties, namely: defects in the
theories, difficulty in understanding them or their
obscure terminology, difficulty in applying them, or...
that graphic designers are averse to semiotic theories
or theories in general (p. 7). Whilst Kress and van
Leeuwen and Storkerson make valid points, and there
are certainly differing viewpoints of how semiotics
should be theorized (see, for example, Krippendorff
[6]), I suggest the more powerful antinomies are those
not within the theories of semiotics, but within the
historical emergence of the discourses of design and
graphic design. The first half of this paper describes this
historical emergence to show how it militates against
any significant incorporation of semiotics, whilst the
second half is a response to specific arguments against
semiotic approaches in design discourse.
I use the term discourse in the Foucauldian sense [7],
whereby certain practices are brought together as
a unity. Through this process, previously disparate
components of the discourse are linked and officially
sanctified through a professional language with
concurrent systems of accreditation which specify
what may be practiced and who may practice it. An
important part of the process of gathering together
these disparate practices is the defining of various
sub-categories such as graphic design, industrial
design, interior design and the like. In graphic design,
the traditional practices of aesthetic construction of
type elements on a page and the laying out of images
and type together, as well as many other traditional
and modern practices, are drawn together under the

graphic design banner, whilst other previously related
practices, such as much of the preparation work for
printed media are excluded. Precisely what is chosen
to be included and what remains excluded from the
category graphic design and thus what is and is not to
be considered within the boundaries of design discourse
are seen here in terms of the historical power struggles
of various social and institutional groups.
2 The Emergence Of Graphic Design

Discourse
I have argued previously [8] that graphic design
emerged, as does any discourse, out of a confluence of
numerous events, approaches and conditions. Social,
economic and political changes occurred in the printing
and advertising industries in the Twentieth Century,
which allowed for the emergence of graphic design.
This new discourse subsumed much of the creative
practices hitherto the purvey of printers, but with the
addition of qualities, aspects of historical trajectory,
values and language of fine arts discourse. There had
been for some time an uneasy relationship between
fine and commercial arts, where commercial arts was
seen as somewhat of a poor cousin, and in some senses,
even the antithesis of the disinterestedness of fine
art. The emergence of graphic design can be seen as
an alignment with the wider discourse of design, and
involved a re-signification of many practices previously
seen as art practices to design practices. Alongside
this was a new historical trajectory, which saw the
foundations of graphic design, less in fine art discourse
and more in the design based practices of the Bauhaus,
which emphasized and legitimized utility and industrial
practices. The shift was facilitated by a new aesthetic in
print production. This promoted the clean layouts of type
and photography that came out of, and fitted more easily
to new mechanical processes, than did the previously
more illustrative treatments. This shift was complicated
by the changing nature of design discourse itself.
In design discourse, Modernism originally had two
distinct forms. The first, commercial Modernism was
primarily American. It was dramatic and emphasized
the spectacle, taking its guide from Hollywood, and by
association, the visually resplendent art deco, and was
firmly based in popular culture. Years later, Raymond
25
Loewy, doyen of its legacy the aerodynamically

curvaceous Streamform would state his favorite curve
was the sales curve heading upwards [9]. As Heller [10]
notes: It was the profit motive, not any utopian ethic,
or any esthetic ideal, that paved the way for commercial
modernism in the U.S. In opposition to this, High
Modernism could be seen to have a closer connection
to fine art. It had come from Europe, and like art
discourse, had a powerful hierarchy of key figures like
Le Corbusier and Mies van der Rohe, a new language
of design principles (form follows function, less is more,
and the like), a privileging of academic discourse and
institutions, a legitimacy conferred by important and
expensive projects, a multitude of publications, and a
show devoted to its work in the Museum of Modern
Art [11]. Berman notes how the publication produced
by MOMA would define the modern canon for the next
forty years [12] Proponents of High Modernism would
often see the popular alternative as vulgar [13].
As graphic design to some degree came out of, and
aligned itself with design discourse itself, it embraced
much of the language, historical trajectory and values
of High Modernism. Yet much of the initial work for
graphic designers came from advertising agencies
indeed it can be seen to a large extent as having sprung
from the changing needs of these agencies to make their
productions more visually appealing. The professional
look required a more nuanced appreciation of type and
layout than had previously been needed.
One of the reasons the emerging discourse of graphic
design was predisposed to an alignment with High
Modernism can be seen in one of the dominant
historical trajectories which led to, and became a
component of the discourse: that of book design and
the related notion of typography. Indeed, the term
graphic design, was coined by advertiser, but also book
and typeface designer, W.A. Dwiggins [14] and Heller
[15] notes: it suggested a higher calling than does the
now unfashionable term, commercial artist. Whilst
typography had been primarily the concern of printers,
a refined version of typography had emerged in the late
1800s. This is a historical trajectory that emphasises
the creative as connected directly to art discourse. This
lineage tends to follow the same historical progression
as that of printing history up until about the time of the
Arts & Crafts period, when fine art and printing reach
26
a climactic embrace. In 1891 William Morris founded

the Kelmscott press, one of the key highlights of typo
graphic histories in design literature [14]. Morriss press
was to spawn a generation of artistic printing but more
importantly was to set a language for typography as it
could emerge in the discourse of design. Shortly after
the Kelmscott Press opened, Charles Ricketts founded
the Vale Press (1896-1903) and Kinross [16] makes the
following observations:
He may stand as one of the clearest representatives of a
new figure who appears in printing and publishing at this
time: the book designers... Before the appearance of the
book designer, designer had, in the context of publishing,
meant essentially illustrator. The work of Ricketts, and
other designers for commercial publishers of the late
nineteenth century, represents the incursion of art into
machine production (p. 38)
Kinross notes that artistic printing occurred at the
moment when a change in taste became apparent:
between highbrow and lowbrow, between a minority
and a mass-market (p. 40). Thus, graphic design can
be seen to be the emergence, initially at the fringes
of the printing industry, of a practitioner with a more
recognized artistic sensibility. To what degree the new
aesthetic was a cause or a consequence of this change is
debatable. What is important is how this aesthetic focus
shaped graphic design.
In 2002, Satellite [17] interviewed a noted typographical
designer and the prelude to this interview is revealing:
He walks into the basement of Eshleman, our interview spot,
and homes straight in on a framed poster leaning against
a corner. THIS IS A PRINTING OFFICE, the elegantly
printed and not entirely subtle sign says, CROSSROADS
OF CIVILIZATIONS, REFUGE OF ALL THE ARTS. Andy
Crewdson stands next to the sign, his neck craned down at
it, and says practically without introductions: Thats
a Beatrice Ward from 1932. Thats really cool.
I look at the sign and hes right (the sign is a reprint), but
hes right on both counts. Andy, a senior who has more or
less devoted his free time to typography and reviving long
out-of-use typefaces, goes on for about a minute about
the history of the sign (Ward created several important
typefaces), the methods of printing it (Letterpress), and its
rarity (You dont see many of those). He has all this to say
about a sign hanging in my own office, a sign that Id only
given half a look at about three years ago, and it occurs to
me that this is what graphic design is all about: recognizing
type. (n.p.)
We can see here how the author recognises the
language of typography as the language of graphic
design, but it is equally important to understand the
nature of this language that is, one which has all of
the mystification of the discourse of fine art. Aside from
the misspelling of Wardes name (a common mistake),
there is an unmistakable quality of traditional fine art
connoisseurship within this article. Thats a Beatrice
Ward from 1932 smacks of the elevation of the artists
significance over the work, similar to how one might say
The Picasso is in the hall or Weve just purchased a
Rembrandt. The author is quick to point out the work
is not an original but a reproduction, even though the
work is a piece of printed copy. The language of fine
art connoisseurship is unmistakable. I argue that in
most design departments of universities and colleges,
graphic and communication design is taught largely
through a connoisseurship model. This is especially
so, when it comes to typography. One of the most
popular typography textbooks, Stop Stealing Sheep [18]
describes the difference between the expert and the
amateur:
It is a bit like having been to a concert, thoroughly enjoying
it, then reading in the paper the next morning the conductor
had been incompetent, the orchestra out of tune, and that
the whole piece of music is not worth performing in the first
place... The same thing happens when you have a glass of
wine. While you might be perfectly happy with whatever
youre drinking, someone at the table will make a face and
go on at length why this particular bottle is too warm, how
that year was a lousy one anyway... (p. 17)
Although Spiekermann & Ginger use their metaphor
to emphasise that there are different ways to appreciate
type, their association of the type expert with the
traditional connoisseur points to notions of refined
versus unrefined taste.
There are numerous benefits offered by the
connoisseurship approach. In the first place it offers
individuals a sense of belonging. Well-respected
Australian designer Alistair Morrison had this to say

about typography:
I like the discipline of it. Ive always had the feeling of
satisfaction of belonging to an international and almost
timeless brotherhood which included people like Bodoni and
Aldus Manutius, a feeling that I am just another link in the
chain. [19]
For students, this sense can be a powerful incentive
to learn, and staff have commented that students
have found themselves after discovering the world
of typography. These students are given constant
reminders of their newfound abilities, skills in type
recognition and aesthetic discrimination by the world
around them. Every menu, road sign, and website offers
up delicacies to be savoured, or crimes to be abhorred.
Every unrecognised font becomes a challenge and each
challenge becomes a reminder of the club of which
he or she is now a member. As Garfield [20] notes:
Identifying a particular font can be the most infuriating
task, and designers can spoil their whole day by walking
past a shop and seeing something they cant name
(p. 174). Students become more immersed in their art
than any teacher could otherwise hope for, and soon
know as much, if not more about the subject, than
their professors. Genuine engagement is the best of
tutors. The joy felt by the student is a combination of
deep aesthetic appreciation and the kind of excitement
one has at demonstrating (if even only to oneself) a
finely tuned skill. This can be an immensely rewarding
experience that sees them not only develop an interest,
but often, sets them on a career path, with a fascination
and deep sense of enquiry that lasts a lifetime. The
flow on effects in terms of self-confidence and personal
sense of direction are immeasurable.
There are a number of problems, however, with the
connoisseurship system. The fascination with what
is, that is, with the tangible, can work against the
recognition of what is not; of what is missing from the
picture. The connoisseurship model privileges object at
the expense of context; detail at the expense of wider
philosophical or sociological perspective. Furthermore,
the connoisseurship model acts to maintain and
propagate class distinction, not only in the producer
of type, but also in the consumer. For the model to
work, a correlative degree of mystification around
27
aesthetic values is supporteda form of distancing

of the language of type from the uninitiated. Bourdieu
[21] reminds us that the language of art is a language
of exclusion which privileges certain social groups
over others, noting that the naive spectator cannot
attain a specific grasp of works of art which only have
meaningor valuein relation to the specific history
of an artistic tradition." (p. 4) The danger of this in
graphic design education is that students who have
learned their art through this approach are well-suited
at designing logotypes for banks, law firms, and other
elite clients, but often tend to see what they constitute
as good design, as universally appropriate. This means
that, as graphic designers, they are limited in their
abilities to communicate to anyone outside of the high
design aesthetic. The emphasis of the connoisseurship
model is very much more on the producer, rather than
on the consumer, and although it may be recognised
that what is deemed good design changes over time,
there is still a certain essentialism in the value system.
This approach can thus also set up an antagonism to
more commercial interests and some designers find
themselves antipathetic to areas such as marketing and
advertising. Heller [22] makes the following observation:
The word advertising, like commercial art, makes
graphic designers cringe. It signifies all that sophisticated
contemporary graphic design, or rather visual communi
cations, is not supposed to be. Advertising is the tool of
capitalism, a con that persuades an unwitting public to
consume and consume again. (p. 112)
Although an artistic sensibility was present in the
emergence of graphic design discourse, it must be
noted that this is a simplified picture. Ken Cato notes
that while some advertisers were initially wary of
the emergence of independent design studios, others
saw the potential benefits and were quick to form
relationships with them, and many graphic design
studios came out of designers originally learning their
craft in advertising agencies [8]. Designers also come
from many different backgrounds and have different
perspectives on the value of connoisseurship approaches
to design. However, graphic design education has often
not had the diversity of voices present in industry.
Jones [23] notes how the majority of design teachers
in the new graphic design courses in polytechnic
colleges of England had only fine arts training.
28
Questions around the degree to which art teachers

have a place in design education have surfaced a
number of times over the past forty years, particularly
since small colleges have been absorbed into larger
institutions. Ashwin [24] made the comment in
1978 that changing conditions demand a concurrent
questioning of the place of art history in design studies
and went on to note how the study of design often
requires an economic, technological or sociological
mode of analysis which plays little or no part in
conventional courses in the history of art (p. 99).
Although this has changed significantly for many art
courses, I would argue the connoisseurship approach
of fine art was a comfortable fit with that of the High
Modernist typographic and book arts trajectory of
graphic design and has been thoroughly embedded
in graphic design pedagogy since its establishment in
tertiary education systems.
Ameliorating some of these differences are shifts in
industrial practices, whereby graphic designers have
worked closely with marketers and found, on occasion,
that they are not completely bereft of creative thinking,
or an appreciation for aesthetic considerations. At the
same time, the phenomenal rise of the Internet as a
commercial landscape has reshaped design pedagogical
practices. Website designers cannot limit themselves to
the graphic treatment of a site, but must also consider
its navigation, and with users able to choose different
navigational pathways the design of the experience has
become a crucial consideration. Even logos, which were
at one time simply two-dimensional graphics, can now
have 3D aspects or may be built to deliberately change
over a period of time. In the workplace, companies began
to recognize their branding was far more complex than
graphic elements on a page and could involve even
the way someone answers the phone. This has led to
branding that looks at the entire structure of how a
company communicates with their clients, and previous
graphic design companies have become brand architects.
These more holistic perspectives saw the emergence of
communication design courses, which moved the designer
one step closer to industry and further from visual arts.
3 The argument for a semiotics foundation

in design education
In design discourse, a number of compelling arguments
against semiotic pedagogical approaches are often

raisedthat semiotics is more directed at analysis
rather than creation; that semiotics is too theoretical
whilst design is practical, and that it has leftist under
pinnings that can turn design students off their intended
vocation. However, I would make the case that there
are strong arguments against each of them, and many
powerful reasons for a deeper and more comprehensive
utilization of semiotics in design education.
That semiotics is often seen as passive has been
addressed in the past through attempts to re-read
semiotics as visual rhetoric. Rhetoric comes from the
notion of constructing a persuasive argument, and some
approaches have attempted to use this to promote a
more active notion of semiotics. Whilst the area has
spawned much valuable research, I would argue that
as an approach to designing, it is flawed. It tends to
reproduce linguistic models which result in recipe
book approaches incompatible with the design process.
It seems ludicrous, for example, to suggest a designer
might ponder whether what is needed in their image
is chiasmus (two figures conveyed with attributes of
each other) or antimetabole (two characters portrayed
where one is upside down or otherwise uncomfortably
portrayed). Whilst a repertoire of rhetorical tropes in
visual form may provide inspiration when the well is dry,
it does little else. Instead, I argue for the development
of a semiotic sensibility an inculcation of semiotic
approaches (employing notions of connotation, myth,
and the like) to expansive subject matter, and with
specific attention to how designs work on audiences.
These may be developed in the student psyche by
analyses of design artifacts as diverse as film, advertise
ments, packaging, logos, typography, products and even
architecture. A great strength of semiotics is the ability
to apply the same system of analysis to a variety of
products and to be able to use a single language for the
communicative aspects of wide ranging subjects.
Any criticism of semiotics as passive neglects the
critically important reflective and evaluative components
of the design process. Creative aspects of this process
are often described as flashes of insight, or the magic,
working off intuition or tacit knowledge. What is
referred to here may be more usefully seen as the
subconscious formation of connections between
different, often unrelated elements, leading to
fortuitous results. We develop the propensity for

this ability through creative process techniques,
including brainstorming and the like. It has also long
been recognised that the more experience one has,
the more cognitive potential there is to draw upon,
and many design tutors will suggest to students that
they not spend all their time in books but experience
the world for themselves. A semiotic sensibility
can not only contribute significantly to ways of
processing experience, it is also directly applicable as
documented in the field of marketing [see, for example
25, 26, 27] to the reflective and evaluative aspects of
the design process. Evaluating, whether quantitative
or qualitative, often involves the use of intuition or
tacit knowledge. The rational (or reason-centric)
model of the design process includes production and
evaluation loops at various points in the process.
This might be formalized in the sense of a number of
iterations [28] or be informal, in the sense of thinking,
developing, back-tracking, re-thinking, and so on, in the
plan-driven problem-solving approach [29]. In more
action-centric models, exemplified by Schns [30]
reflection-in-action paradigm, the act of designing is
as much about problem-making as it is about problem
solving, yet primary emphasis is placed on reflection and
evaluation. Design thinking paradigms [31-33] formalize
the prototyping and evaluation stages of designing [34],
where abductive reasoning is an important feature [31].
In abductive reasoning the pool from which the best
guess can be made is enriched by a semiotic sensibility.
Furthermore design thinking foregrounds the user
experience, which is also a strength of current semiotic
approaches [25, 27]. Understanding how designs work
on audiences can lead to asking the right questions. Far
from being relegated to the strictly theoretical realms, a
semiotic sensibility can add significantly to design practice.
One of the most powerful aspects of semiotic analysis
is its ability to view a text from different contextsto
denaturalize the text. In this way a text can be explored
in terms of its possible social and political implications,
where the contexts most commonly used foreground
issues of class, gender and race. As Kress and van
Leeuwen [4] argue:
Pictorial structures do not simply reproduce the structures
of reality. On the contrary, they produce images of
reality which are bound up with the interests of the
29
social institutions within which the pictures are produced,

circulated and read. They are ideological. Pictorial structures
are never merely formal: they have a deeply important
semantic dimension. (p. 45)
critique it. A designer who is ignorant of the effects of

his or her actions, still acts. A semiotic sensibility has
potential to contribute significantly to the formation
of globally aware and ethically responsible graduates.
As it is, fears that students will be put off the course are
This raises the criticism of semiotics in design discourse

that it politicizes the design process. This can be seen
as unnecessary or worse; that it may put students off
becoming designers. We might ask whether designers
even need to take this extra step; whether we need to
know any more than what Hall [35] calls the preferred
meaning of a text. When we are aware of how an
object or message speaks to consumers, we raise
consciousness of how that artifact acts in culture
which beliefs or values it supports and propagates.
For example, a purely aesthetic treatment for a new
brand of deodorant might present few moral difficulties
for the designer; whereas an awareness of how a
product can appeal to a womans insecurities about her
self-image that might result in higher sales, creates a
moral dilemma. The designer who believes they are only
communicating an aesthetic is, of course, communicating
much more, although they may be ignorant of the
inherent messages in their work. I suggest there are
two important reasons to ask what a text does, that
is, what values it promotes, or ideological positions it
propagates, beyond the preferred reading. The first is
that this leads to a deeper awareness and appreciation
of how our designs communicate messages, developing
our semiotic sensibility. Secondly, almost all university
design courses have identified globally aware and
ethically responsible as desired graduate attributes. I
take Freires [36] perspective that everything is political
that, in fact, one acts politically whether one recognises
it or not. As Atzmon notes:
unfounded. Not only has the course I teach in semiotics

had continuously excellent student feedback, but also
I have had numerous students impart to me that the
course has changed their life, and added immeasurably
to their appreciation of design. Even many years later,
ex-students have told of the lasting impression from
their course in semiotics and design, which positively
affect their work still.
If they are not trained, designers may not even be

aware of the meanings underlying their process and the
communicative power of the choices they make. Whatever
the design process, though, the designers beliefs and
attitudes shape choices made about the material form. The
designers aesthetic convictions are necessarily influenced
by the culture(s) in which the designer is immersed, and the
material form of the design object embodies these beliefs
and attitudes. [37]
If there is social imbalance and one acts within the social
sphere, then one acts to maintain that imbalance or to
30
I have worked for a number of years at both RMIT

and Swinburne University in Melbourne and currently
at AUT University in Auckland developing a semiotics
approach to design. This is initially imparted through
a series of lectures and tutorials. Lectures include
an introduction to semiotics, with explanation of
signification, notions of myth and ideology; Marxist
theory; and critical approaches to the politics of gender
and race. A series of assignments move the student
from a general familiarization with the language of
semiotics to gradually more applied semiotic analyses:
1. Visual diary: students are invited to take the concepts
discussed in lectures and apply them to their lives
outside of the university environment. Each week,
they bring their photographs, found images and
other media, along with annotations discussing these
in terms of semiotics. This works well in allowing
students to discover implications of our semiotic
evaluations for their own world.
2. Film analysis: this is an essay where students are
given articles that use semiotic theories to analyse
films. They then choose a particular perspective
from the lecture series (Marxist, gender or race
politics) and using this perspective, analyse a film
taken from a list provided. Students must identify
signifiers, and myths and suggest ideological impact.
For example, a Marxist analysis will involve identifying
signifiers of class in the film, and myths related to
class, and discuss how the film therefore critiques or
propagates certain ideological positions with respect
to social class.
3. Label swap: this project has long been a part of
many design curricula. Two products that appeal to
different target markets are chosen. Keeping the
A report of this project

can be found at [38]
Young, A., and Gray, B.,
2012, "Empowering Art,"
Cumulus Conference
2012: Northern Mandate,
Helsinki, Finland.
For a report of this

project, see [39] Gray,
B., and Young, A.,
2013, "Communication
Design and Community:
Pedagogy and
products in their packaging, and leaving the words

on those packages, all other signifiers (colour,
typefaces, layout, style of images, and the like) are
swapped, such that one products expressive language
is transferred to the other and vise versa. The
importance of the assignment is in the awareness
it brings of precisely which elements in packaging
constitute signifiers which speak to the target
market in a particular voice and which do not.
4. Magazine swap: this works in the same way as the
label swap except involves exchanging the signifying
elements of two magazines with differing target
markets. For example, in a Marxist approach, the
signifiers (apart from the actual words) of Womens
Day might be swapped with those of Style. The
results are then analyzed in terms of how the
re-signified magazines now speak to their target
markets in terms of class.
5. Product analysis: students again take on a particular
perspective from the lecture series, and apply this
to the analysis of packaging and advertising of a
simple supermarket product. Again, the student
attempts to elucidate the voice of the product as
it speaks to its target market, along with any myths
used to appeal to that market, and the possible
ideological impact of the use of these myths.
Empowerment," Proc.
Crafting The Future:
10th European Academy
of Design Conference,
Gothenburg, Sweden.
4

project, see [38] Young,
A., and Gray, B., 2012,
"Empowering Art,"
Cumulus Conference
2012: Northern Mandate,
Helsinki, Finland.

project see [39] Gray,
B., and Young, A., 2013,
"Communication Design
and Community: Pedagogy
and Empowerment,"
Proc. Crafting The Future:
10th European Academy
of Design Conference,
Gothenburg, Sweden.
Students are then given the opportunity to apply their

semiotic sensibility in real world social innovation
projects with important social justice outcomes. This
is vitally important as it offers alternative ways for
design students to envisage their future, rather than the
usual maintenance and support of a consumption-based
economy. Some of the projects students have worked
on include:
1. Such Lives: This project1 involved the creation
of animatics as part of a large-scale theatre
production with residents of the Fitzroy housing
estate in Melbourne. The residents were a deeply
underprivileged community who had lived on the
estate for three or four generations. Students
reported that the experience not only dramatically
impacted on their lives but importantly, on their
sense of the meaning and potential of design.
2. Unreserved: A series of animations to save an
important historical and social landmark from
developers. Working with oral histories from
past workers and residents at the site, students
constructed animations, which successfully brought

together the community and saved the site. Students
and their parents reported that they could not
believe that design students could do such significant
and powerful work as part of a university course.
The work is now part of the permanent collection
of the Australian Centre for the Moving Image,
Melbourne.
3. Same Difference Exhibition: in this project 2 students
worked with the ALSO Foundation (an organization
that represents the rights of GLBTI communities)
and the Department of Justice to produce posters to
reduce discrimination. An exhibition of these posters
was launched at the Justice Museum in the week that
a report that identified potential strategies aimed
at reducing homophobia was handed over to the
Attorney General. As a result of the two pronged
approach, the written report and the highly visual
exhibition, significant legislative reform is now being
considered and funding was provided to the ALSO
foundation to mount a statewide anti-homophobia
campaign, which was launched in August of last year.
4. Equal Service: a digital storytelling project3 that took
place in Australia as an adjunct to a larger project
initiated by the State Government in conjunction
with the Homeless Persons Legal Clinic (HPLC) and
the Council To Homeless People (CHP). The aim was
to work cooperatively to address the discrimination
that people experiencing homelessness receive whilst
trying to participate in public life. Students produced
a DVD of animated stories to be sent out to service
providers in Victoria. The project was launched by the
Attorney General and was so successful, it was decided
the DVD was to be sent to all states in Australia.
5. Altered Lives: Students from Auckland University of
Technology (AUT) collaborated with Positive Women
to produce a series of digital storytelling projects4
aimed at de-stigmatizing what it means to live with
HIV. This was sponsored by Positive Women and
Mac Cosmetics and was launched this year. Focus
group testing has revealed the digital stories have a
profound effect on audiences, with one respondent
recently diagnosed with HIV commenting that she
had considered suicide until this life-affirming work
gave her hope for a future.
These projects are not only dynamic and powerful
tools for instigating change in communities, but also
31
work to demonstrate first hand to students that their

communication skills can have dramatic and very
personal benefits for others. There can be no proof
that a semiotic sensibility helps students to design
better, just as we cannot prove that reading will
make someone a better novelist. Yet students report
that learning semiotics has significantly enhanced
their understanding of how ideas, values and beliefs
are communicated through designed artifacts. The
combination of imparting the language of semiotics
with an ability to decontextualize design artifacts
illuminating how they communicate with their audiences
is overwhelmingly valued in student evaluation reports
of the subject. The development of their skills through
real projects seems to be greatly enhanced by their
semiotic sensibility.
References
[1] Saussure, F. D., 1974 (orig. 1916), Course in General
Linguistics, Fontana, London.
[2] Peirce, C. S., 1958 (orig. 1897), The Collected Papers of
Charles Sanders Peirce, Harvard U.P., Cambridge (Mass.).
[3] Barthes, R., 1995 (orig. 1964), Elements of Semiology, Hill
and Wang, New York.
[4] Kress, G., and Van Leeuwen, T., 1996, Reading Images: The
Grammar of Visual Design, Routledge, London and New York.
[5] Storkerson, P., 2010, "Antinomies of Semiotics in Graphic
Design," Visible Language, 44(1), pp. 5.
[6] Krippendorff, K., 2006, The Semantic Turn: A New
Foundation for Design, Taylor & Francis, London.
[7] Foucault, M., 1972, The Archaeology of Knowledge,
Tavistock, London.
[8] Young, A., 2005, "A Genealogy of Graphic Design in
Victoria," PhD Doctoral Thesis, RMIT, Melbourne.
Semiotics is neither an a priori, nor a universal, fixed

system. It is a culturally specific model that can be
used to structure and categorize ways of seeing. It can
be applied equally to the products of graphic design,
industrial design, architecture and any other system that
produces artifacts that communicate to an audience
or market. Rather than a recipe book approach to
understanding design, I argue for a gradual inculcation
of a semiotic sensibility, which provides students with
a wide range of contexts through which to understand
both the connotative meanings of their work, as well
as the wider social and political implications it carries
for specific audiences.
[9] Schonberger, A., 1990, "Raymond Loewy: Pioneer of

American Industrial Design," Prestel-Verlag, Munich.
[10] Heller, S., 1995, "Commercial Modern: American Design
Style 1925-1933," Print, 49(5), pp. 58-69.
[11] Lupton, E., and Miller, J. A., 1989, Graphic Design in
America: A Visual Language History, Walker Art Center,
New York.
[12] Berman, M., 1988, Design after Modernism, Thames &
Hudson, London & New York.
[13] Zander, A., and Medworth, F., 1941, Design in Everyday
Things, Australian Broadcasting Commission, Sydney.
[14] Meggs, P. B., 1998, A History of Graphic Design, John Wiley
& Sons, Inc., New York.
[15] Heller, S., 1999, "W.A. Dwiggins: Master of the Book," Step-
Although the historical trajectories of design and

graphic design discourses have worked to militate
against any significant inclusion of semiotics in graphic
design pedagogy, semiotics can provide a rich founda
tional system for design education. This will not only
advantage students in their design practice directly,
but will significantly contribute to the other important
pedagogical systems through which they learn to
become better designers and more informed people.
As design approaches are becoming more centered
on the people who use our designs, rather than on
the expert or connoisseur designer, and as global issues
demand more aware and ethically responsible design,
a semiotic sensibility is vital if design pedagogy is to
be relevant and inspirational.
By-Step Graphics, Jan-Feb(pp. 108-113.

[16] Kinross, R., 1992, Modern Typography: An Essay in Critical
History, Hyphen Press, London.
[17] Chen, K., 2002, Man of Letters, 24 October, http://www.
readsatellite.com/culture/2.4/crewdson.chen.2.4.1.htm
[18] Spiekermann, E., and Ginger, E. M., 1993, Stop Stealing
Sheep & Find out How Type Works, Adobe Press, Mountain
View, California.
[19] Caban, G., 1983, A Fine Line: A History of Australian
Commercial Art, Hale & Ironmonger Pty Ltd, Marrickville, NSW.
[20] Garfield, S., 2010, Just My Type, Profile Books, London.
[21] Bourdieu, P., 1984, Distinction: A Social Critique of the
Judgement of Taste, Harvard University Press, Cambridge,
Massachusetts.
[22] Heller, S., 1997, Looking Closer 2, Allworth Press, New York.
[23] Jones, J., 1973, Readings in Art and Design Education: 2
after Coldsteam, Davis-Poynter Limited, London.
32
[24] Ashwin, C., 1978, Design History: Fad or Function?, Design

Council, London.
[25] Floch, J.-M., 2001, Semiotics, Marketing and
Communication, Palgrave, New York.
[26] Valentine, V., 2008, "Semiotics and Market Research (the
Other Half of the Equation)," The Marketing & Management
Collection.
[27] Floch, J.-M., 1988, "The Contribution of Structural Semiotics
to the Design of a Hypermarket," International Journal of
Research in Marketing, 4(pp. 233-252.
[28] Cross, N., 2008, Engineering Design Methods: Strategies for
Product Design, Wiley, New York.
[29] Ralph, P., 2010, 5th International Conference, Desrist
2010, St. Gallen, Switzerland, June 4-5, 2010. Proceedings.,
DESRIST, St. Gallen, Switzerland.
[30] Schn, D. A., 1983, The Reflective Practioner: How
Professionals Think in Action, Basic Books, A Division of
Harper Collins, New York.
[31] Dunne, D., and Martin, R., 2006, "Design Thinking and How
It Will Change Management Education: An Interview and
Discussion [Editorial Material]," Academy of Management
Learning & Education, 5(4), pp. 512-523.
[32] Brown, T., 2008, "Design Thinking," Harvard Business
Review, June(pp. 1-10.
[33] Kimbell, L., 2011, "Rethinking Design Thinking: Part 1,"
Design and Culture, 3(3), pp. 285-306.
[34] Nussbaum, B., 2011, "Design Thinking Is a Failed
Experiment. So Whats Next?."
[35] Hall, S., 1997, Representation: Cultural Representations and
Signifying Practices, Sage, London.
[36] Freire, P., 1985, The Politics of Education, Bergin & Garvey,
New York.
[37] Atzmon, L., n.d., "Forms of Persuasion: The Visual Rhetoric
of Design Artifacts," The Radical Designist, 2(pp. 1-15.
[38] Young, A., and Gray, B., 2012, "Empowering Art," Cumulus
Conference 2012: Northern Mandate, Helsinki, Finland.
[39] Gray, B., and Young, A., 2013, "Communication Design and
Community: Pedagogy and Empowerment," Proc. Crafting
The Future: 10th European Academy of Design Conference,
Gothenburg, Sweden.
33
Ozge Merzali Celikoglu

Technical University of Istanbul, Istanbul, Turkey
ozgecelikoglu@itu.edu.tr
Three dimensions of lace:

Reading the meaning of a traditional
product from different perspectives
Abstract
This paper examines the artifact of lace in three different
contexts in Turkish culture: It is the product of a craft
smanship tradition; it is the object of lace making activity;
it is the signifier of the home concept. The aim of the
study is to approach to this artifact with a culturally rich
background, from different perspectives and provide an
insight to understand the daily social life. Focusing on
the meaning of products and artifacts, it is studied how
a product of material culture can transport different
meanings and be constantly transformed within the inter
pretation of its makers/designers and users/non-users.
in the house. There are several countries and regions,

which have their own characteristic lace depending on
their traditional cultural background; some of them are
French lace, Turkish lace, Italian lace, Hungarian lace and
Finnish lace.
This study, methodologically positioned in the inter
section of product semantics and symbolic interaction,
examines the artifact of lace with its three dimensions:
First, it is taken as a product of tradition; second, it is
examined as the object of a joint action, in terms of lace
making; and third, it is examined as the signifier of the
home concept, in Turkey.
Keywords
Lace, material culture, craftsmanship, home, design
research, symbolic interaction
1 Introduction
Lace is defined as a fine open fabric of cotton or silk,
made by looping, twisting, or knitting thread in patterns
and used especially for trimming garments (Oxford
Dictionary 2012). It can be made by hand or by knitting
machines. Based on how it is made, lace can be classi
fied, such as, needle lace, cutwork, bobbin lace, tape
lace, knotted lace, crocheted lace, knitted lace, machinemade lace, chemical lace, and the examples can be
increased with every new developed technique. Lace
can be used as part of clothes or as decorative element
34
2 About Turkish lace

Among many other types of lace, this study within
its scope focuses on the Turkish lace. Turkish lace is
used as an essential decorative ornament in Turkish
houses. It is made using crochet hooks and needles.
Considering the traditional structure of Turkish lace
the most preferred threads are cotton and silk, and
the most preferred color is white. However, there is a
huge variety in the type and color of the threads, which
are used for lace making. There are basically two main
variations of Turkish lace: One of them can be applied
on several types of cloth surfaces and edges, such as,
towels, curtains, bed linings, headscarves, and etc.
This kind of lace is called needlework; their patterns
and names change according to the region it is being

made. Today, the needlework is also being used in
order to design fabric jewelry and decorative goods
(Yalcinkaya 2012).
interpretative process, which is needed to understand

human behavior and sensitivities because many human
cultural values are embedded in the interpretation of
phenomena.
Adding same or different patterns together and building

a surface by using crochet hooks make the second
variation of lace. With this technique, tablecloths,
curtains, bridal dresses, bedcoverings and clothes can
be made. This kind of lace is usually made in geometrical
shapes and it consists of unique patterns, which come
together and build the whole lace. The size and number
of patterns can vary.
With respect to the claim Design is making sense of

things. (Krippendorff 2006), the meaning of an artifact
sits in the core of its design. However, this meaning
cannot be absolutely determined by the designer but has
to be recreated in every use and every context by the
individual user through a process of interpretation. This
product semantics approach takes the absolute meaning
giving authority from the designer and supports to
see users and non-users as individual interpreters of
the artifacts and find their own meanings, similar to
Barthes (1967) dead author/ity. Considering that
there is no embodied knowledge in the artifact until
it is interpreted (Mkel 2006), the approach of the
design research gets closer to the perspective and
method of symbolic interactionism.
Leaving its physical characteristics aside, lace with all its

connotations related to tradition, cultural practice and
home concept in Turkish culture is more than a domestic artifact in Turkey: It has several meanings for its
maker, its user and even for its non-user. Therefore to
examine lace and lace making activity provides an insight
for understanding the Turkish culture and everyday life.
3 Perspective and method: The inter

section of design research and symbolic
interactionism
Product semantics, as a field of design research, deals
with the meaning and interpretation of artifacts. It
aims to understand users understanding of their
practices of interfacing with designed things and provide
strategies for designing products that can either afford
or supportively intervene in that understanding. ()
Product semantics recognizes that people surround
themselves with things they are familiar with, are able
to handle flawlessly and can arrange so as to feel com
fortable among them (Krippendorff and Butter 1984).
The approach of product semantics brings back the
individual user who is capable of understanding,
practicing and interpreting products instead of the
imaginary average user which needs fool-proof
products. Thus, it focuses on the meaning of the
product which is settled by the individual user and
assumes that Design is making sense of things, as
Krippendorff (2006) has stated.
Considering that design research is not as quantifiable
as in science and engineering it requires an
As Blumer (1969) states, symbolic interactionism has

three main premises:
1. H
uman beings act toward things on the basis of the
meanings that the things have for them.
2. The meaning of such things is derived from, or arises
out of, the social interaction that one has with ones
fellows.
3. These meanings are handled in, and modified through,
an interpretative process used by the person in
dealing with the things he encounters.
The symbolic interactionism sees meanings as social
products as creations that are formed in and through
the defining activities of people as they interact. This
approach corresponds to the attitude where a product
is best evaluated according to its meaning for its user
or non-user. Thus, it is the context and the social
interaction, which helps the individual to make sense
of objects and products.
To summarize, the methodology of the paper combines
both perspectives of product semantics and symbolic
interactionism focusing on the meaning of a traditional
cultural artifact, an element of the Turkish material
culture. As Denzin (1992) states, The production,
distribution, consumption, and exchange of cultural
objects involves issues of ideology and the political
35
economy (semiosis) of signs, including how these signs

are worded or photographed, where they circulate, who
buys them, and so forth. The systems of discourse that
shape the meanings brought to any cultural object must
also be examined.
An ethnographic inquiry among a group of people
revealed various experiences and approaches related to
lace and lace making. Considering that ethnography is
the work of describing a culture the attitude is based on
learning from people rather than studying people. Thus,
instead of collecting data about people, the aim is to
learn from people, to be taught by them (Spradley 1979).
With this perspective, semi-structured interviews are
conducted with people from different age, gender and
educational backgrounds. The interview questions are
formulated in order to get information about lace and
lace making, its usage at home, its cultural perception
and its meaning for Turkish people. Thus, the aim of the
ethnographic inquiry is to have an understanding of the
existence and use of lace in everyday life; to determine
its relation and relevance to Turkish culture, and,
finally, to identify its users and different forms of usage
including past and recent situations.
Besides semi-structured interviews, conversations, as
a part of the methodology, are conducted many times
with these people. Considering that conversation is
a cooperative venture, it has a direction and it allows
new understanding (Feldman 1999), the transmission of
knowledge through an informal conversation provides
the transmission of some kind of tacit knowledge
regarding the issues of daily life. Furthermore,
observations support the data that has been reached
through interviews and they help to recognize the
unstated situations and, photographs have been taken in
participants houses so that a visual analysis can be made
with the documentation of data (Pink 2007).
4 Different approaches to lace in Turkish

culture: A product of tradition, an object
of joint action and a signifier of home
4.1 Lace as the product of tradition
Lace as a traditional handcraft is very popular in Turkish
society and lace making is one of the favorite activities
of most Turkish housewives. It is a transmitter of a
36
tradition and culture with a strong emphasis on the

importance of hand labor. Grandmothers pass this
tradition to mothers, mothers to daughters, and so on.
Examining the lace in its traditional and cultural context,
its connotations can be read through the reading of the
tradition and craftsmanship concepts.
In order to get an insight about the meaning of a
traditional craftsmanship product and this concepts
contribution to the perspective of this study a brief look
at the meaning of tradition and craftsmanship would be
helpful. Starting with the definition of tradition, it is the
transmission of customs or beliefs from generation to
generation, as well as, an artistic or literary method or
style established by an artist, writer, or movement, and
subsequently followed by others (Oxford Dictionary
2012). Tradition contains every kind of belief, images
of individuals and events, techniques and institutions
(Shils 1981). Thus, cultural traditions are a continuation
of the past, as well as a projection into the future,
actually being elements of the continuity of a societys
history. At this point, craft products can be taken
as one of the transmitting elements of culture and
tradition. Craftsmanship and so, craft products, are
material beings in the intersection of culture, tradition
and society since historical production techniques
and rituals generate them. It is important to see craft
products with their cultural depth because they are not
only products of traditional hand-labor but also carriers
of the tradition itself. A craft product offers the user to
get informed about its cultural background visualizing
the traditional usage of the artifact. Due to this kind of
communication, the relation between the product and
the user becomes a cultural interaction happening on a
personal experience for each individual in the society.
Also, this situation allows the user or the subject to
share the traces of the unique objects or craft products
moments of creation (Baudrillard 1968):
The fascination of handicraft derives from an objects
having passed through the hands of someone, the marks
of whose labor are still inscribed thereupon: we are
fascinated by what has been created, and is therefore
unique, because the moment of creation cannot be
reproduced. Now, the traces of creation from the actual
impression of the hand to the signature, is also a search
for a line of descent and for paternal transcendence.
Various interpretations of culture and tradition show
that these two concepts have the power to define and

transform each other. Craftsmanship, in addition, is a
product of them both; a more concrete outcome of
their reflections on the society. Thus, the concept of
craft products includes histories, cultures and traditions
of societies; like myths, which transform a meaning into
a form (Barthes 1957). Craft products, like all other
products, have their own language, however their
languages are configured over a longer period of time
and in a cumulative way. Thus the relationship between
the user and the product is defined by a cultural and
historical involvement with the related tradition, i.e.
an internalization of the tradition by every individual
since the craftsmans cumulative work is actually the
cumulative work of every individual by means of society.
Several patterns, techniques and even names of lace
types, which vary according to the local region they
are being made, are a reflection of the related culture,
tradition and life style. Thus, beyond any kind of
meaning that is attributed to it, the lace is a transmitter
of tradition and culture. It builds a connection from
past generations to the present and future carrying the
traces of a tradition and being regenerated with every
new contribution. During this process, its meaning is recreated by each individual and transformed to the next.
This dynamic situation provides a constant transmission,
transformation and interpretation of the meaning and
artifact of lace in the context of tradition.
4.2 Lace as the object of joint action
Lace making activity is identified with women, especially
with housewives in Turkish culture. It is the activity,
which is being done by housewives at their home
meetings. Home meeting is the social activity of house
wives where a small group of friends gets together with
certain periods. The meeting happens every time in
another womans house, and the owner of the house is
responsible with hosting the others with food, service,
and etc. Housewives enjoy home meetings and they
have conversations together about their lives, recipes,
practical solutions at home works; they share their own
experiences and thus learn from each other during
these conversations. One of the subjects of these
conversations is handicraft, which shows the talent
of the housewife. Completed laces are shown and
sometimes they are sold in the group; new patterns
are shared and taught to those who ask for them.
Thus, lace-making activity accompanies the

conversation, although it is generally not the focus
of the conversation.
According to Mead (1967), objects consist of whatever
people indicate or refer to. Considering lace as the
object of lace making activity, it is the object of a joint
action covering the socialization through exhibiting,
teaching and practicing together. Quoting Blumer (1969),
Objects are social products in that they are formed and
transformed by the defining process that takes place in
social interaction. The meaning of the objects is formed
from the ways in which others refer to such objects or
act toward them. For the one who exhibits her talent,
lace is the object of a show; for the one who teaches to
make lace, it is the object of being the master; and for
the one who is practicing, it is the object of a common
sharing. These differing meanings define the differing
positions of each individual in this group, because they
tend to act on the basis of their own meanings.
Lace, as the object of the joint action lace making
is constituted through social interaction in terms of its
material being, meaning and use. Blumer (1969) defines
joint action as the larger collective form of action
that is constituted by the fitting together of the lines
of behavior of the separate participants. Considering
that anything can be an object to an individual, it is one
of the examples, which has a variety of connotations
and meanings. Within the context of this study, some
of them are material culture, tradition, craftsmanship,
womanhood, social interaction and home concept.
As mentioned above, the meaning varies according to
the interacting individual and the context, and lace is
one of the objects, which can help to understand the
culture and lifestyle of an individual or a group of people
by identifying it and its environment. As Blumer (1969)
also states, Since people are set to act in terms of the
meaning of their objects, the world of objects of a group
represents in a genuine sense its action organization,
and this corresponds to the action organization in the
housewives group, positioning themselves according
to the meaning of lace making activity.
4.3 Lace as the signifier of home
Lace has always been a signifier of the home in Turkish
culture. Generally, it is identified with the womanhood
of the housewife, her talent and dominance over the
37
house and her success in turning the house into a warm

home. According to (Bachelard 1969), our home is an
extension of our existence and it is the representation
of our existence. Thus it reflects us as well as our
position in life, and it is our first point of communication
with life (Fidanoglu 2001). The concept of home is
always remarkably distinguished from the concept of a
house or a flat in Turkish culture. Most of the time, it is
the signifier of a family, warmth and a feeling of safety
that is the capturing of a certain piece of the world.
The very first examples of the Turkish house were
built on soil ground, with various isolation material
and it consisted of a single huge room for different
activities, such as, eating, sleeping, sitting and storing
(Goker 2009). Since rooms in traditional Turkish house
were designed as volumes having many functions,
objects used in these rooms became also portable and
multifunctional, such as cushions, floor-beds, floortables, and many others constructed with cloths. A
characteristic of traditional Turkish house was the sofa,
the center area of the house to which all other rooms
have direct connections. Sofa was the place where all
members of the family can spend time together and
also, where guests are hosted. Within the urbanization
process sofa has lost its power and importance in
gathering people together and, as a result, sofa disappeared in urbanized areas (Eric, Ersoy and Yener 1986).
For the new structure of Turkish house from 1960s to
late 1990s in urbanized areas, it is possible to claim that
the house consisted of two main parts: The inner house,
where the family lives, and the salon, where guests are
hosted. Conceptually, the inner house means sincerity,
nearness, warmth, comfort, directness, density and
familiarity, while salon is the world of relationships
with disturbing strangers (Ayata 1988). Due to this
sharp distinction of spaces in the same house, the
family living in there has two distinct worlds, socially
and psychologically. The salon was like an exhibition
area for guests where all valuable goods of the family
are displayed; these could be goods which have been
bought and are expensive, such as, a new television,
video player, furniture set, or goods which carry the
memories or talent of the housewife, such as, handwoven or hand-knit cloths and laces.
Considering the situation today, the separation between
salon and living room is not as distinguishing as it was
38
before and the fast urbanization bringing the lack

of space is one of the factors which make the living
room serve both as salon and living room. So, the
differentiation between the inner house and outer
house is fading. However, among many artifacts, which
resist the process of modernization in the Turkish
house, lace is the most significant one, which is being
adapted to many different situations in the house. Lace
in Turkish home concept, is mostly used in order to
cover or pack goods. It is used as tablecloth, coffee
table cloth, display cabinet cloth, bed linen, wardrobe
cloth, and etc. It serves to create a warm, dressed
home out of a naked house. This warmth basically
signifies the family and safety. As mentioned before,
the home itself and its objects become an extension of
its user. Putting laces on furniture is like dressing the
furniture in order to emphasize its existence, and laces
under some decorative objects serve to underline these
objects a certain kind of possession (Baudrillard 1981).
Today, lace is still used everywhere in the house for
different purposes: They are placed in the living room,
in the bedroom and even in the kitchen. In the kitchen
it is placed on visible and covered shelves, under plates
and glasses, and etc., in the bedroom it is placed in
chiffoniers and closets. In the living room, the lace
serves to cover the dinner table, the coffee table
(Figure 1), and the shelves of the display cabinet where
it becomes an element of decoration (Figure 2).
Fig. 1.Lace on coffee tables
5 Conclusion
Fig. 2.Lace in the display cabinet
It is placed under vases and other decorative objects

so that direct contact and thus a possible scratch to the
furniture is avoided (Figure 3).
Fig. 3.Lace under decorative objects
Considering that the organization of a human being

consists of his objects, that is, his tendencies to act
on the basis of their meanings (Blumer 1969), it is
the objects, which serve to the creation of the home,
of life and identity. It is because there is something
referring to old habits, different aspirations, desires,
emotions, dreams, individual perceptions of life, and etc.
Everyday objects that have become invisible to the user,
mostly have become an essential part of his/her life,
especially at home. Since people are set to act in terms
of the meanings of their objects, the world of objects
of a group represents in a genuine sense its action
organization. To identify and understand the life of
people it is necessary to identify their world of objects
and this identification has to be in terms of the meanings
objects have for the members of the group (Mead 1967).
Young generations definition of the house is mostly a
place where little time will be spent, these people still
use the word home for the place where they used to
live with their parents and with laces although most of
them criticize theirmothers use of lace, and they prefer
to use flat where they live with their friends.
There are several reasons and many other perspectives

on the existence, use and meaning of lace; however, it
is examined in this study in the intersection of product
semantics and symbolic interactionism. Starting from
its material being and existence in Turkish culture, lace is
examined through its position in craftsmanship tradition,
as an object of lace making activity in terms of creating
a process in its process of creation and as a signifier of
the traditional Turkish home in its final destination.
Examining lace as a craftsmanship product of Turkish
culture provides a brief understanding about the existence
and use of lace in its connection to traditional values and
life styles: It is one of the many bridges, which connects
the past to the present and future and transmits a
shared common past. Furthermore, lace as the reason
and the object of lace-making activity becomes the
organizer of a certain piece of social life where its makers/
designers are positioned according to their relation to it:
This is where teaching, learning and sharing activities are
included and formulated into a type of conversation.
Having investigated the lace in its material existence and
process of creation within its social circle, the following
stage comes as its signification of home. Since one of the
main reasons of making lace is to display the talent and
leave a memory and a gift to next generations it is a part
of creating the warm home. This is where generations
have different approaches: For instance, considering
younger generations, especially living in metropolitan
cities, they do not prefer to use lace in their houses.
It is neither a symbol of hand labor nor an element of
decoration for them; it is some unnecessary ornament
and a house would be more plain and modern without
them while older generations believe that a house would
be empty without lace.
To conclude, lace, like many other objects, has several
meanings in different contexts: Within the scope of this
study, it is the carrier of a tradition and thus the connection
between the older and younger generations; it is the object
of lace making, and thus the starter of a social organization
process in small housewives groups; an finally, it is a
remarkable signifier of the traditional Turkish home. Lace
in all these different contexts, however, stands with a very
strong characteristic, since it is the key object, which pro
vides the repeat of a social act and creates its own ritual.
39
6 References
18. Oxford Dictionary 2012.
1. Ayata, Sencer. 1988. Kentsel Orta Sinif Ailelerde Statu
19. Pink, Sarah. 2007. Doing Visual Ethnography: Images, Media
Yarismasi ve Salon Kullanimi.Toplum ve Bilim(42): 5-25.

2. Bachelard, Gaston. 1968. The Poetics of Space. Boston:
Beacon Press.
3. Baudrillard, Jean. 2005. The System of Objects (J. Benedict,
Trans.). London & New York: Verso Books. (Original work
published 1968).
4. Baudrillard, Jean. 1981. For a Critique of The Political
Economy of Sign. St. Louis, MO: Telos Press.
5. Barthes, Roland. 2000. Mythologies (A. Lavers, Trans.).
London: Vintage Books (Original work published 1957).
6. Barthes, Roland. 1978. Image, Music, Text. USA: Hill and
Wang. (Original work published in 1967).
7. Blumer, Herbert. 1969. Symbolic Interactionism: Perspective
and Method. California: University of California Press.
8. Denzin, Norman K. 1992. Symbolic Interactionism and
Cultural Studies: The Politics of Interpretation. Cambridge:
Blackwell.
9. Eric, Murat, Halil. Y. Ersoy, and Nuran Yener. 1986. Gunumuz
Konutunda Rasyonel Donatim. Istanbul: Teknografik
Matbaasi.
10. Feldman, Allan. 1999. Conversation as methodology
in collaborative action research. School of Education,
University of Massachusetts, Amherst MA. Retrieved
March 20, 2012 (http://people.umass.edu/~afeldman/
ActionResearchPapers/Feldman1999.PDF).
11. Fidanoglu, Esra. 2001. Evin Yeni Adi;
Yolculuktabarinmabarinmadayolculuk. in Gstergebilim
Tartismalari, edited by E. Onatve Yildirim S. Ozgencil
Yildirim, Istanbul: Multilingual Yayinevi.
12. Goker, Muge. 2009. Turklerde Oturma Elemanlarinin
Tarihsel Gelisim Sureci.Zeitschrift Fr Die Welt Der
Trken1(1): 163-169.
13. Harvey, Lee. 1987. Myths of the Chicago School of Sociology.
Aldershot, UK: Gower.
14. Krippendorff, Klaus and Reinhart Butter. 1984. Product
semantics: exploring the symbolic qualities of form.
Annenberg School for Communication, Departmental
Papers (ASC), University of Pennsylvania.
15. Krippendorff, Klaus. 2006. The Semantic Turn: A New
Foundation For Design. USA: Taylor and Francis Group.
16. Mead, George H. 1967. Mind, Self And Society: From The
Standpoint of a Social Behaviourist. Chicago: University of
Chicago Press.
17. Mkel, Maarit. 2007. Knowing through making: the role
of the artifact in practiceled research.Know Techn Pol,
(20):157-163.
40
and Representation in Research. London: Sage.

20. Shils, Edward. 1981. Tradition. Chicago: University of Chicago
Press.
21. Spradley, James. P. 1979. The Ethnographic Interview. USA:
Harcourt Brazce Jovanovich College Publishers.
2 2. Yalnkaya, Nazime T. 2012. Anadolu Kltrnde Dantel.
Retrieved June 8, 2012 (http://turklacemuseum.blogspot.
com/2012/02/anadolukulturunde-dantel-n_26.html).
Ehsan Baha1, Dirk Snelders1,2, Yuan Lu1, Aarnout Brombacher1

1
Department of Industrial Design, Sub-department Business Process Design, Eindhoven University of Technology,
Eindhoven, The Netherlands

2
International Design Business Management, School of Business, Aalto University, Helsinki, Finland
{s.e.baha, h.m.j.j.snelders, y.lu, a.c.brombacher}@tue.nl
Retracing an Evolution of Meanings

for Design-Driven Innovation
Abstract
In this paper we explore how to enrich design-driven
innovation by considering the dynamic nature of such
innovation as a result of history and evolution. Designdriven innovation takes distance from users in their
current context, but instead proposes radical new
meanings to users that address new potential needs.
Here we look at how design-driven innovation can
be based on a thorough understanding of a product/
services current meanings and lost meanings of its
predecessor(s). We investigate this assertion with an
action oriented case study using a research through
design approach. Within the context of recorded music,
and using script analysis theory to define meaning, we
studied the evolution of album covers. As a result,
we were able to come up with two radical meaning
innovations for album covers. We conclude that the
investigation of the evolution of meaning of a series of
products/services from the past up to the present
can help designers to depart from current meanings
more radically, and more purposefully. We thus hope
to inspire design to go beyond studying meanings in
temporal isolation, taking into account meaning as
a result of history and evolution for the purpose of
design-driven innovation.
Keywords
Design-driven innovation, script analysis theory,
evolution of meaning, (design) history.
1 Introduction
1.1 Background
According to Akrich, artifacts contain scripts (messages)
from their designer (and other producers) to the user
describing the products intended use and meanings [1].
The term script here is a metaphor for an instruction
manual which Akrich claims is inscribed in an artifact.
Products/services have meanings inscribed by their
designer (and other producers) within the context of
production. Within the context of use, these scripts can
be: (1) subscribed to, by users, when inscribed meanings
are interpreted and accepted as intended by the
designer (and other producers); and/or (2) de-inscribed
by users, when scripts of intended meanings are
ignored/discarded/rejected/interpreted differently and
used for other meaning intentions as intended by the
designer. Thus, the meanings of products and services
can be found in between a context of production and a
context of use [3, 4, 5, 6, 7].
A common assumption has been that the meanings of
a design are given; one could attempt to understand
them but one cannot innovate on them. Recently,
Verganti has argued that meanings can, and should also
be innovated upon, using a separate, more culturally
informed strategy for innovation [8]. Verganti calls
this strategy design-driven innovation and in his book
he explains that this is an innovation strategy that
leads to products and services that have a radical new
41
meaning: those that convey a completely new reason

for customers to buy them (p. viii). While Verganti
has emphasized that his book is not about design but
management, in this paper we are interested in designdriven innovation both from a management and design
perspective. The question we ask is: How can designers
plan for innovations in the meaning of the products and/
or services they create?.
Our research in this direction started by an investigation
of what happens when radically new meanings are intro
duced by a design and designers want to stimulate the
adoption of those meanings by users [9]. We worked
in the view that meanings of products and services
are organized around sociocultural regimes [10, 11,
12]. With this is meant a regulation of the production
of cultural symbols, in combination of cultural norms
and values in (parts of) society, and artifacts and
technologies with particular symbolic meanings [13,
14]. For example, think of the dos and donts during a
Japanese tea ceremony (regulation), in connection to the
etiquette and experience of drinking tea for Japanese
users (norms and values), and the symbolic meaning
of the used tableware (artifacts and technologies).
In our previous work, we explained that the trans
formation from one sociocultural regime to a new
one could lead to meaning gaps, where radical new
meanings are alienating people. The new sociocultural
regime is too far from the previous regime, with some
existing meanings from a current sociocultural regime
being made redundant, causing feelings of culture and
identity loss [15, 16]. In addition, we explained how
meaning gaps can be bridged through balancing new
and existing meanings. From this we learned that the
meaning carried by products and/or services have
been constructed over a wider, and historical set of
predecessors [7]. Meanings evolve over time, and
during sociocultural transformations meanings can
be: introduced, disposed of, preserved, and/or reintroduced [9].
Yet, in design and innovation research and practice,
meanings are often perceived, measured, and dealt
with exclusively on basis of a product/service at hand.
However, within the current sociocultural regime only
part of a product/services meanings can be perceived,
measured, and dealt with. To uncover, and possibly
42
restore all potential meanings of a new design, it is

vital that we look to the evolution of designs over
multiple sociocultural regimes. We take our inspiration
from Aristotle, according to whom The essentials
of a phenomenon are best understood if one tries to
explore their rise from the very beginnings. [17].
1.2 Objective
Inspired by the dynamic nature of meaning and
meaning as a result of history and evolution, in this
paper we investigate how overall established meanings
of a product/service in the here and now, can more
thoroughly be understood by studying the evolution
of that product/service. I.e. how a product/service
has become what it is today. In addition to this, we
investigate how a thorough understanding of a product/
service as well as related possibly lost established
meanings can serve the design of radical new meanings.
Finally, we investigate the implications that this approach
could have for design-driven innovation.
We have depicted album covers as a product for our
research, which we base on an action oriented case
study regarding the design of a radical home audio
system.
1.3 Structure of this paper
We start by a literature review describing script analysis
theory and design-driven innovation. We continue by
introducing the case study and our research context
and methodology. Then we present the results of our
meaning analysis followed by a radical new meaning
synthesis. Lastly, we derive conclusions and define our
future work.
2 Literature review
2.1 Script analysis theory
A methodological tool that can be used for analyzing
and understanding the meaning of products and/or
services is the script analysis theory. Madeline Akrich
developed this tool within the conceptual framework
of actor-network theory (ANT) [1, 2, 29]. ANT is a
theoretical framework introduced in the latter half of
the 1980s by Bruno Latour, Michael Callon, and John
Law to emphasize that technology and society are
mutually constitutive, not separate spheres influencing
each other [7, 29]. While ANT is concerned with
transformation of meaning through artifacts (nonhumans as well as human actors) as they form and move
through networks and act as mediators, the idea of
product script has been developed to facilitate close
analysis of how meaning is transported and transformed
by products [7].
From a practical and design point of view, we have
clarified some terms from the vocabulary of script
analysis theory used in this paper. Our clarification is
based on the works of Akrich, Latour [1, 2], and Fallan
[6, 7] (Table 1).
Term
Definition
Example
Script
Messages from
designer (and other
producers) of a
product/service to the
user describing the
products intended use
and meanings
Heavy keys of
hostels with intention
do not forget to bring
the keys back to the
front desk [2, p. 259]
Description
Interpretation of
messages of a product/
service to generate
meanings. This is
usually done by
the analyst or user
and is the opposite
movement of
inscription
Interpretation of
heavy keys of hotels
would for example be
do not forget to bring
the keys back to the
front desk [2, p. 259]
Inscription
Translation of
meanings to messages
of a product/service
by the designer (and
other producers)
Translating the
message do not forget
to bring the keys back
to the front desk
by heavy weights
attached to keys to
force clients to be
reminded to bring back
the keys to the front
desk [2, p.259-260]
Table 1. Vocabulary of script analysis theory
2.2 Design-driven innovation

Design-driven innovation (a.k.a meaning-driven
innovation) is an innovation strategy introduced in
2009 by Roberto Verganti as an addition to the two
mainstream and dominant innovation strategies of userdriven innovation (a.k.a. market pull innovation) and
technology-driven innovation (a.k.a. technology push
innovation) [8]. While user-driven innovation usually

departs from user/market studies and technologydriven innovation departs from technological inventions,
design-driven innovation departs from the innovators
culture. This strategy is about taking distance from the
user/market and sometimes also about the integration
of new or existing technology in a new market or
sociocultural regime in order to avoid incremental
change and aim for change which is more radical.
Verganti describes this idea well using his framework on
which the different strategies are mapped together with
a hybrid strategy what he calls Technology Epiphanies
(Figure 1). Technology epiphanies occur when there
is radical change interplay between both meaning and
technology.
In more recent work [18] on design-driven innovation
Verganti has proposed a hermeneutic framework to
look at radical innovation of meanings based on the two
main acts of interpreting and envisioning.
Figure 1. The two innovation dimensions and related innovation

types adopted from Norman & Verganti [19]
In this work Verganti has also provided a clarification on

how he defines a product meaning. Namely: To clarify,
when we mention product meaning, we relate to the
purpose of a product/service as perceived by the user.
It is about the purpose for why a product is used, not
how it is used (the user interface), nor what the product
consists of (its features). The hermeneutic framework
and definition of product meaning brings design-driven
innovation close to design. However, up to now, the
43
research on design-driven innovation has been based

on post-hoc analyses of selected cases in design, rather
than on a design activity that synthesizes new proposals
that will depart from current meanings.
predecessors of album covers were selected and

described by the design researcher using various
documentations on the Internet relating to album cover
history [1]. In total a number of 25 predecessors from
1870s up to 2006 were selected and described.
3 Research context and approach
To complete our research for reporting, we also

performed a post-hoc informant inquiry using social
media in which we asked informants: What do album
covers mean to you?. This question was answered
by a number of 32 informants who did not have any
knowledge about script analysis theory; i.e. they
informally described what album covers mean to them
and motivated these descriptions.
In this paper we focus on two how questions: (1)

Understanding how the meanings carried by a product/
service have been constructed over a wider, and
historical set of predecessors?; and (2) How insights
from such a study, can be beneficial for and used to
design radical new meanings?.
In order to answer these questions a 12-week action
oriented case study in the form of a design research
project was set up between the Industrial Design
Department of Eindhoven University of Technology and
the Department of R&D&I of Philips and executed by
the first author in 2006.
3.1 Design case study and research focus
The design brief for the practical case above was to
design a radical home audio system. By radical we
mean a product/service created from a radical meaning
and possibly technology interplay innovation. In
particular, we were interested in a differentiation from
existing home audio systems (in 2006) by means of what
Verganti later called a technology epiphany (Figure 1) [8].
For the sake of clarity, within this paper we only focus
and report on the album cover relating parts of the
radical home audio system that was designed. This
means: (1) understanding how the meanings carried by
album covers have been constructed over a wider, and
historical set of predecessors; and (2) how insights from
such a study, can be beneficial for and used to design the
next generation album cover with radical new meanings.
3.2 Research approach
This case study was based upon a research through
design approach and reflective practice. In this
approach design action and reflection on action are
considered creators of knowledge, and the design
outcome is considered the physical proof of the
knowledge generated [20, 21, 22, 23].
Our research through design approach was also
supported by desk research. Within this desk research
44
4 Results
We start by a summary of the informal (i.e. not
based on script analysis theory) meaning descriptions
inquired from informants through our social media
inquiry (Section 4.1). Then we present the evolution
of album cover meanings described based on a desk
research using script analysis theory (Section 4.2). This
is followed by a reflection of the designer on both the
informant and meaning evolution descriptions towards
forming a strategic design vision (Section 4.3). Finally
we present a radical new meaning for album covers
within the radical home audio system that was designed
(Section 4.4).
4.1 Album cover meaning descriptions from
informants
Table 2 provides a summary of the analyzed results.
In total a number of 13 meanings (The center column)
were described by the 32 informants (random mixed
group) that participated in the explorative study.
In order to give a good indication of how the study
was done, we have depicted and included a description
example for each described meaning (The left column).
In the right hand column, we have indicated the number
of times that a certain meaning was referred to in the
informant descriptions.
Most of the informants describe the meaning of album
covers from the artist point of view; i.e. as a platform
where the artists can express themselves. Moreover,
album covers can be seen as a channel through which
Description example
Meanings
# of informants
that provided this
meaning in their
description
They say something about the artist and the environment

where the music has been created.
Artist/Band Expression
The best ones I sense as graphic poems.
The Album Message/Story
If the album cover has more bright colors it tells me the

music is more light
(Emotional) Representation of the

Songs/Content
Album covers mean forgotten faces, lost memories, and that

we are growing old.
Cues for Memories
If the album cover is ugly I would not buy it.
Aesthetic Art; Personal Identification
They should have a relation with the songs.
Visual Representation of the Song
Album covers are fun.
Fun; Enjoyable
They gave me a feeling of what type of music I can expect.
Reflection of the Album Style/Genre
I always look at the album cover to see if there has been

enough care put into creating it. That says something about
the quality of the work and whether I would buy it or not.
Trigger for Sales
A good cover adds another layer to the album experience.
Another Layer to the Album Experience
They protect the CD or LP.
Protection of the Recordings
They help me navigate through my iTunes library.
Affording Navigation
I like the lyrics so that I can sing with the music.
Container of Lyrics
Table 2. Summary of the album cover meaning descriptions from informants
artists can tell a story about themselves and/or their

work. From the audience point of view however, the
informants described album covers as a representation
of the music and an aesthetic art form that when owned
can be used to identify oneself towards other people
and when seen serve as a cue for memories.
4.2 Evolution of album cover meanings
This study was framed only around artifacts relating to
audio storage in order to focus on album covers. By
audio storage we refer to techniques and formats used
to store audio with the goal to reproduce the audio
later using audio signal processing to something that
resembles the original [24]. Due to this focus, the time
scope of our study has been from 1870s up to 2006.
Table 3 provides a summary of our analysis of audio

storage and their carried meanings. In total a number
of 25 audio storage artifacts, categorized into 12
categories, were described for meanings. This resulted
in a list of 7 meanings (Left hand column). These are:
(1) Record Label Marketing, referring to the way that
record companies used an album cover to market their
brand; (2) Packaging & Protection of the Recordings,
referring to the way in which album covers were a
package for and protected the recordings; (3) Event
Support, referring to what kind of role album covers
played in supporting music events; (4) The Ability to
Identify, referring to how album covers supported
finding/recognizing music (from a collection); (5) Artist
Branding, referring to the ways in which the artists
branded themselves by means of album covers; (6)
45
Compilation Albums referring to ways in which a

bundle of tracks known as an album came to existence;
and finally (7) Self Artwork, referring to how album
covers meant as a platform for personal expression for
self made compilations for example.
In order to indicate how the artifacts carry and make
these meanings available in culture for people, we have
also provided an overview of meaning inscriptions for
each category (Bottom left corner of the table). As an
example for the category of the phonograph cylinder
we can see that meaning (4) The Ability to Identify for
example, has evolved as following: phonograph cylinders
were packaged and protected in cardboard tube boxes
with printed logos of the record company on both the
cardboard tube boxes and the cylinders themselves.
At first, phonograph cylinders could only be identified
through a small paper insert with written information
about the content of the phonograph cylinder, inside
of the package. Later, people started to write this
information on the labels by hand. This influenced the
record labels to decide to stamp this information on
the top lid of the phonograph cylinders. At a certain
moment, the new packaging included a printed label on
the lid, which could also be cut out to use for specific
phonograph cylinder collection storage furniture. These
early artifacts more or less initiated the rise of the first
album cover that came out in 1909 [25].
Once the meanings of album covers were described
from all relevant audio storage in the history of
recorded music culture, we performed an analysis of
how these meanings evolve over time. More specifically,
we looked at the presence of meanings in culture
related to the rise and endurance (existence) and
discontinuation (sometimes perish) of some audio storage
artifacts in time. Table 4 summarizes this analysis, which
provides interesting insights. The evolution of meanings
can one by one be explained as following:
(1) Record Label Marketing: The first audio storage
artifact, the phonograph cylinder, was a technological
invention by Thomas Alva Edison also known from the
Edison Records. When analyzing the evolution of the
Record Label Marketing we see a decrease in the role
of record labels, especially after the introduction of
compact cassette, compact disc, and most influential
of all, the MP3.
46
(2) Packaging & Protection of Recordings: The

packaging and protection of recordings was a meaning
present from the beginning of audio storage history, i.e.
the phonograph cylinders, but became more and more
present through the introduction of the 8-track in 1960,
the compact cassette in 1963, and the compact disc in
1979. This meaning started to decrease in product form
with the introduction of MP3 in 1995, which managed
to dominate the compact disc as an audio storage
artifact. Nowadays, this meaning, in product form, is
only carried by digital content storage devices such as
USB keys or compact discs, and to the extent that they
exist, some other classical audio storage artifacts. This
meaning however, has migrated from products form to
online service forms like Myspace.
(3) Event Support: Events were probably one of the
first places where music captured on audio storage
was sold. While this already took place during the
phonograph cylinder era, with the invention of
gramophone record in 1895 and later the invention of
the album cover in 1909, event support as a meaning
started to increase its presence in culture. This increase
of meaning presence was even more after the invention
of vinyl records that became widely spread and popular
in the society. Nowadays, album covers are still used to
support events. They often even match event posters,
tickets, etc. with regard to their visual identity.
(4) The Ability to Identify: Due to differences in
packaging inherent to technological innovation this
meaning has had ups and downs throughout the
history. Record packaging for example, provided
a much larger surface for album art compared to
Phonograph Cylinders, compact cassettes, or compact
discs. Vinyl records, due to a much better endurance
than Gramophone Records, became more and more
accessible for people and present in the culture.
People have been using album covers for years to
browse through music collections and identify music.
Album covers have also been used as artifacts for
self-expression towards other people. The dominance
of compact cassette decreased the presence of this
meaning from a higher visual sense but the invention of
compact disc and related artist booklets contributed to
an increase of this meaning presence again. From 1995
on, when MP3 was invented, there has been a radical
change in the inscription of identification, from a mainly
visual medium to a mainly textual medium. Inventions

like cover flow introduced in 2006 by Apple Computers
Inc., re-introduced some of the visual aspects of album
covers in digital spaces [Cover Flow, 2006].
(5) Artist Branding: This meaning more or less came
to existence after the invention of album cover in 1909.
Later it increased presence due to the introduction of
the vinyl records but really gained importance after
the invention of the compact cassette and especially
compact disc and related booklet. With the rise of
Internet and invention of the MP3, artist branding
started to migrate as a meaning from the compact disc
booklets into artist websites and social media websites
such as Myspace that focus on music [26]. Nowadays
artist branding is almost an indispensible meaning in
recorded music culture.
(6) Compilation Albums: The word compilation comes
from the era that a couple of records were literary bound
by an album. The invention of reel-to-reel and the increase of space for audio content on storages contributed
to the creation of compiled albums with multiple songs.
(7) Self Artwork: The introduction of compact cassette
in 1963 was the beginning of Self Artwork as a meaning
related to album covers. From then on people could
easily record a compilation on a blank compact cassette,
which had a template including boxes for writing down
the tracks and a surface for artwork that could be
created by people themselves. The introduction of the
compact disk in 1979 continued to support this meaning
presence along with the introduction of the MP3 in
1995. A couple of years after that though, Self Artwork
became less present as a meaning due to the decrease
of tangible audio storage and an increase of digital audio
storage formats that do not really support it. Nowadays
people seldom create Self Artwork for their digital
music content and Self Artwork has remained limited
to tangible audio storages.
In general what we can learn from the analysis shown
in table 4 is that both Record Label Marketing and
Packaging & Protection of Recordings as meanings
are becoming less present while Artist Branding,
Compilation Albums, and Event Support meanings of
album covers are becoming more present in culture.
Self Artwork and The Ability to Identify are also
interesting to look at since both of them are less

present, probably inherent with Packaging & Protection
of Recordings since the audio storages of today have
mainly been dominated by the MP3, which is a digital
audio storage.
4.3 Designers reflection
In this section we provide a summary of the designers
reflection based on the earlier results (Section 4.1 and
4.2). Looking at the rise of the album cover phenomena
from the very beginning, brought us back to 1870s when
the phonograph cylinder was introduced as the first
audio storage artifact ever, that radically changed the
culture of music experience. This radical innovation
of meanings brought about change, which was both
appreciated, and not appreciated at all by people during
that time. A more positive received new meaning
carried by phonograph cylinders was for example, the
possibility that people could suddenly listen to music
from the other side of the world without traveling to
see a live event. A less positive received meaning carried
by the phonograph cylinders was the separation of the
music from its artist. In fact, back then, many artists
were objecting to this meaning. When looking at the
evolution of album cover meanings from a birds eye
perspective what can be seen is that album covers came
to existence due to the separation of artist from its
music. In fact, the album cover throughout the years
has become like a stage for the artist to somehow
have a presence next to the recorded (and playing)
music. The large Record covers, the compact cassette
and the compact disc booklets, along with the ID3
tag of the MP3 and the Artist Website are all artifacts
with inscriptions that support this meaning in various
ways causing it to be sometimes more and sometimes
less present in culture. This conclusion can also be
triangulated with the conclusions of our informant
meaning exploration study using social media, as Artist/
Brand Expression was mentioned as a meaning of album
covers by most informants. But what the album cover
has also been throughout these years is a way for people
(the audience) to display and browse through their
collections in both individual and in social settings. This
is however, is a meaning that has been less facilitated
due to the digitalization of audio storage. Our current
(2006) audio systems unfortunately are all based on text
browsing and have neglected album covers; an artifact,
which has been carrying many interesting meanings in
47
Table 3. Audio storage, the described meanings, and their inscriptions per category
the culture of music experience. Moreover, an artifact

of which its development (meaning innovation) has not
been utilized yet.
In sum, we can say that the meaning of album covers on
a more deep level, i.e. a conclusion based on an analysis
48
of over a century and three decades, actually is a

medium that affords interaction between the artist and
the audience. What if, we can envision a future audio
system that is designed based on this understanding? In
the following section we present NAVA, a home audio
system that we designed based on this vision.
Table 4. Album cover evolution and described meanings.

High intensity green = more present. Low intensity green = less present. Green frame = influential artifact.
49
4.4 Design-driven innovation: interactive

album covers
In this section we first introduce a radical home audio
system, which was designed based on the evolution
of recorded music culture meanings. In specific, we
introduce two design-driven innovations (radical
meaning innovations) for album covers, integrated in this
system. Both innovations were driven by our study of
the evolution of album cover meanings.
Figure 2. The NAVA social home audio system (left);
What follows is a presentation of the results by

answering the why and what questions based on
our evolution study. The how question, i.e. how the
meanings were inscribed, will not be discussed in the
present paper.
NAVA (in Farsi nava means tone/tune; it is also a
principal mode of Iranian traditional music) is a home
audio system in the form of a coffee table. Digital
(music) content can be uploaded wirelessly and
displayed on the NAVA, which provides an inviting
interaction that affords a social music experience
(Figure 2).
QR code for a video of the NAVA (right)
Figure 3. Example of a dynamic and interactive album cover:

50 Cent interactive album cover (left); The NAVAs multi-user
This then newly designed home audio system was

quite radical for its time, and to some extent, can still
be considered radical innovation of meanings (Figure
1). Below we describe two new meanings that we
introduced in the NAVA social music table. Both of
these meanings are related to album cover meanings:
(1) Intuitive multi-user digital music browsing and
playing: The NAVA is designed in such a way that more
than one user can operate it. Moreover, it allows for
multi user-, and therefore social-, interaction. The
NAVA allows for a rich visual and almost tangible
interaction with digital (music) content. People can sit
around the NAVA and experience music on different
levels by interacting with the digital (music) content on
its touch screen display. (Figure 3). Furthermore, people
can share their digital (music) content with others by
uploading it to the NAVA where it can be browsed and
played intuitively, and experienced socially.
(2) Interactive album covers for music context
experience: Next to digital music content (digital
audio), music context (information that one used to
find in CD booklets and can now find on the Internet)
50
interface (right)
is represented by means of dynamic and interactive

album covers. This enables fans to stay in touch and up
to date with the artists, and artists to regularly express
themselves and interact with their audience. Since the
album or track content with regards to both music
content and music context is dynamic and changing, the
experience is always unique (Figure 3).
Both meanings are based on innovating the meaning of
album covers from physical or digital static artwork,
which is designed, sometimes printed, and sold, to
digital and dynamic artwork which can continuously
change and be updated from the artist side and
interacted with from the audience side providing
services since the NAVA social music table is connected
to the internet. Although Apple Computer Inc. has
introduced Cover Flow in 2006 and Microsoft Inc. has
introduced the Surface Table in 2007, this is a meaning
innovation, which is to date (2013) a potential radical
innovation of meaning for the recorded music culture or
more specific the home audio system market [25, 27].
5 Conclusion and discussion

We have shown how meanings of a product/service can
more deeply and thoroughly be understood. This was
done using script analysis theory to describe meanings
from predecessor artifacts evolved over time and
through a meaning evolution exploration in which one
can investigate how the presence of meanings carried
by artifact in a certain culture changes inherently with
those artifacts over time.
Also supported by our previous work [9], we conclude
that the discovery of these more deep and thorough
meanings can support a more radical and purposeful
design-driven innovation strategy: (1) One could
easier select which existing meanings to preserve, reintroduce, or dispose; and (2) Which new meaning to
introduce next to the existing meanings in such a way
that one creates design-driven innovations that are truly
sustainable (in the sense of people, planet, and profit;
i.e. innovations that develop our culture and society)
and have a higher chance of adoption [9].
A weakness of our current research approach is that
the description and analysis of album cover meanings
evolution was only done by one person, the first author
of this paper. The social media meaning exploration
study shows that such description and analysis studies
can benefit from a more holistic approach. The reason
for this is twofold: First, the meaning description and
meaning evolution analysis can be done by more than
one person and preferably through the involvement
of actors or interpreters related to the target culture
for innovation. Multiple expert perspectives can enrich
the study and increase its quality [30, 31]. Besides,
sometimes a one-man job is not even possible due to
the complexity of the project and/or lack of expertise in
a certain project context (target culture for innovation)
[31]. Second, one should not forget that all actants,
human or non-human (artifacts), are entangled in the
world [32]. Moreover, they form a sociocultural regime
that one could see as a products service system or
ecosystem. This stance, i.e. knowing that an artifact
is part of a bigger landscape, could have a significant
change of how the meanings of an artifact could be
defined and perceived [33, 34].
a product service system perspective. Moreover, how

designers can work in co-operation with interpreters
(human actors who can contribute to the meaning,
description, creation, inscription, and realization of
design outcomes) during early developmental stages of
projects to benefit from objectivity and their resources
for operationalizing innovation concepts.
While our focus within this paper has been more on
the meta level and from an innovation design strategy
perspective, we would also like to explore how existing
and new meanings can be inscribed into products/
services. Moreover, in this paper we have focused on
the what and why questions regarding meanings
(which meanings and why) while we can also focus on
the how question regarding meanings (how can radical
new meanings be inscribed with combination of existing
and past meanings) [28, 9].
Furthermore, we believe that our approach of
investigating the overall established meanings of a
product, service, or phenomena in the here and now as
a result of history and evolution, also has potential for
dealing with societal challenges a.k.a. wicked problems
(e.g. aging). Therefore, we are pondering how this
approach can be used for addressing societal challenges
and also used for the development of phenomena in
sociocultural regimes.
7 Acknowledgements
Our thanks go to: Sietske Klooster (project coach of
the NAVA social music table); Steven Kyffin, Anton
Oguzhan Andrews, Robert Kortenoeven, Judith Peeten,
Luc Geurts, Richard Appleby, and Kees Overbeeke
(all project experts); Bart van Eden (prototyping expert);
and last but certainly not least Roel Bomers, Pardis Baha,
Gabriele Tempesta, Heather Daam, and Laura van Geel
(all for their support in reviewing this paper).
This research was funded within the Creative Industry
Scientific Program (CRISP). CRISP is supported by the
Dutch Ministry of Education, Culture, and Science.
8 References
1. Akrich M 1992, The De-Scription of Technical Objects,
in Bijker WE & Law J (eds), Shaping Technology/Building
Society: Studies in Sociotechnical Change, MIT Press,
Cambridge, MA, USA, pp. 205-224.
6 Future Work
We plan to continue our research by exploring
how meanings can be described and inscribed from
2. Akrich M & Latour B 1992, A Summary of a Convenient

Vocabulary for the Semiotics of Human and Nonhuman
Assemblies, in Bijker WE & Law J (eds), Shaping Technology/
51
Building Society: Studies in Sociotechnical Change, MIT Press,

Cambridge, MA, USA, pp. 259.
3. Conway H 1987, Design History - A Students Handbook.
HarperCollins Academic, London, UK.
4. Heskett J 2002, Toothpicks & Logos: Design in Everyday Life.
Oxford University Press, New York, NY, USA.
A hermeneutic framework to look at radical innovation of

meanings, Industrial Marketing Management, vol. 42, pp. 86-95.
19. Norman DA & Verganti R (submitted) Incremental and
Radical Innovation: Design Research versus Technology and
Meaning Change, Design Issues, viewed 7 August 2013: http://
jnd.org/dn.mss/Norman%20%26%20Verganti.%20Design%20
5. Lloyd PA & Snelders, HMJJ 2003, What was Philippe Starck
Research%20%26%20Innovation-18%20Mar%202012.pdf
thinking of?, Design Studies, vol. 24, no. 3, pp. 237-253.
20. Yin RK 1994, Case Study Research: Design and Methods, 2nd
6. Fallan K 2008, Describing Design: Appropriating Script

Analysis to Design History, Design Issues, vol. 24, no. 4, pp. 61-75.
7. Fallan K 2010, Design History: Understanding Theory and
Method, English edn, Berg Publishers, Oxford, UK.
8. Verganti R 2009, Design-Driven Innovation - Changing the
Rules of Competition by Radically Innovating what Things
Mean, Harvard Business Press, Boston, USA.
9. Baha SE, Lu Y, Brombacher AC & van Mensvoort, KM
2012, Most Advanced Yet Acceptable, but dont forget.
Proceedings of the 9th Biannual Conference on Engineering
Design and Product Development (NordDesign 2012), Center
for Industrial Production, Aalborg University and The Design
Society, University of Strathclyde, Alborg, Denmark, pp. 51-58.
10. llgaard G 1999, A Super-Elliptical Moment in the Cultural
Form of the Table: A Case Study of a Danish Table, Journal
of Design History, vol. 12, no. 2, pp. 143-157.
11. McCracken G 1986, Culture and Consumption: A theoretical
account of the structure and movement of the cultural
UK/New Delhi, India.

21. Schn D 1983, The Reflective Practitioner: How Professionals
Think In Action, Basic Books, New York, NY, USA.
2 2. Frayling C 1993, Research in Art and Design, Royal Collage of
Art Research Papers, vol. 1, no. 1, pp. 1-5.
23. Koskinen I, Zimmerman J, Binder T, Redstrm J & Wensveen S
2011, Design Research Through Practice: From The Lab, Field and
Showroom, 1st edn, Morgan Kaufmann, Burlington, MA, USA.
24. Sound Recording and Reproduction, viewed 9 July 2013,
http://en.wikipedia.org/wiki/Audio_storage
2 5. Schoenherr S 2005, Recording Technology History,
University of San Diego, Archived from the original on 29
March 2007, Retrieved 9 July 2013.
26. Cover Flow 2006, viewed 9 July 2013, http://en.wikipedia.org/
wiki/Cover_Flow
27. Myspace 2003, viewed 9 July 2013, https://myspace.com/
28. Marriott M 2007, Touch Screen in a Table Is the Latest
meaning of consumer goods, Journal of Consumer Research,
Wrinkle in Computers, New York Times, Technology
vol. 13, no. 1, pp. 71-84.
Section, New York, NY, USA.
12. Kopytoff I 1986, The Cultural Biography of Things:

Commoditization as Process, in Appadurai A (ed), The
Social Life of Things: Commodities in Cultural Perspective,
Cambridge University Press, Cambridge, UK, pp. 64-91.
13. Geels FW 2004, From Sectoral Systems of Innovation to
Socio-Technical Systems: Insights about Dynamics and
Change from Sociology and Institutional Theory, Research
Policy, vol. 33, no. 6-7, pp. 897-920.
14. Geels FW 2005, Technological Transitions and System
29. Latour B 2005, Reassembling the Social - An Introduction to ActorNetwork-Theory, Oxford University Press, New York, NY, USA.
3 0. Sturkenboom N, Baha SE, Lu Y & Tempesta G 2013,
Using Social Media for Asynchronous Collaboration
within Collaborative Networks, Proceedings of the 3rd
Participatory Innovation Conference (PIN-C 2013), 18-20
June 2013, Lahti, Finland.
31. Baha, SE, Sturkenboom N, Lu Y & Raijmakers B 2013, Using
Design to Initiate Collaborative Networks, Proceedings
Innovations: A Co-evolutionary and Socio-technical Analysis,
of the 2nd Cambridge Academic Design Management
Edward Elgar Publishing, Cheltenham/Camberley, UK/
Conference (CADMC 2013), 4-5 September 2013,
Northampton, USA.
15. McLuhan M 1969, The Playboy Interview: Marshall McLuhan,
Playboy Magazine, March.
16. McLuhan M & Gordon WT 2003, Understanding Media: The
Extensions of Man, Critical edn, Gingko Press, Cheltenham/
Camberley, UK/Northampton, MA, USA.
17. Aristotle 384-322 BC, viewed 9 July 2013, http://en.wikipedia.
org/wiki/Aristotle
18. Verganti R & berg 2013, Interpreting and envisioning 52
edn, Sage Publications, Thousand Oaks, CA, USA/London,
University of Cambridge, Cambridge, UK.

32. Hodder I 2012, Entangled: An Archaeology of the
Relationships between Humans and Things, 1st edn, WileyBlackwell, Hoboken, NJ, USA.
33. Hecht S & Colin K 2005, Product as Landscape: Volume two,
Industrial Facility, London, UK.
34. Verkerk MJ, Hoogland, J, van der Stoep, J & de Vries MJ 2007,
Denken, Ontwerpen, Maken - Basisboek Techniek filosofie,
Uitgeverij Boom, Amsterdam, The Netherlands.
Evelien van de Garde-Perik1, Federico Trevia 2 , Adam Henriksson3, Luc Geurts3, Helle Ullerup3
1
Eindhoven University of Technology, e.m.v.d.garde@tue.nl
Delft University of Technology
Philips Design Innovation, Eindhoven
Getting a GRIP at the Design of a

Nature Inspired Relaxation Space for
Work-Related Stress
Abstract
1 Introduction
This paper presents the work that has been conducted

as part of the GRIP project about work-related stress
in combination with the nature inspired design conso
rtium. After 1,5 years of extensive research on the
GRIP project including contextual user research,
gathering stress insights through probes and data
visualisation, and co-creation with stress experts and
employees, it was finally decided to design an ambient
space that enables employees to relax. The design of
the relaxation space was inspired by nature and offers
employees an adaptive environment that reacts to their
presence by creating a personal environment, varying
in size, sounsdscape and animated light. The environ
ment stimulates paced breathing, meditation and helps
employees to become more aware and in control of
their personal response to stressors and relaxation.
The design has been evaluated by 23 experts on the
basis of one time use of the space. The expert evalu
ations resulted in very positive responses regarding the
low effort required and the high quality of the relaxation
experience provided. We are currently collecting feed
back from employees taking part in a prolonged end user
evaluation to explore the value of small adaptations in
our design in order to further improve the relaxation
experience.
The work described in this paper is conducted in light

of two collaborations, namely the GRIP project, and the
Nature Inspired Design consortium, and aims to address
work-related stress through design. In this section first
relevant background information about the project
collaborations will be provided, followed by a short
introduction to the domain of work-related stress,
and the peculiarities that need to be considered when
designing within this domain.
1.1 Project collaborations
The Dutch government funded the GRIP project, as
part of a larger program (CRISP) focusing on the design
of Product Service Systems (PSS). The aim of CRISP is
to develop new markets for the creative industry by
stimulating research on the role of design in industry.
GRIP deals with flexibility versus control in the design
of PSS for work-related stress (Badke-Schaub &
Snelders, 2011). The GRIP project is a collaboration
between Philips Design Innovation, the Design Academy
Eindhoven, and TU Delft and Eindhoven University of
Technology. GRIP starts from the premise that in the
case of PSS, the control of designers over processes and
outcomes is reduced.
The Nature Inspired Design Consortium unites multiple
companies, among which are Philips and Interface, and
the Industrial Design department of the TU Delft in
53
performing research on the practice of Nature Inspired

Design (NID). The aim of the consortium is to come
to better and proven effective NID methods. Design
inspiration through nature can take various forms such
as biomimicry or Lifes Principles. Biomimicry can be
defined as imitating or taking inspiration from natures
forms and processes to solve problems for humans
(Benyus, 1997). Lifes principles are successful principles
of nature that provide the parameters, conditions,
and requirements to function more sustainably. Lifes
Principles can be used to develop selfsustaining ventures
and technologies that, like nature, maximize benefits
with minimal effort and negative impact. (Patel &
Mehta, 2011).
This paper presents our experiences within the GRIP
project and the Nature Inspired Design Consortium,
and discusses the development of a nature inspired
prototype, namely a relaxation space. The relaxation
space offers a platform for designers and service
providers to create new PSS in the future.
1.2 Work-related stress
The concept of stress has been introduced first by Hans
Selye who defined stress as a non-specific response of
the organism to any pressure or demand (Selye, 1956).
Later, Selye (1974) made a distinction between eustress
and distress. If an effective coping strategy can be
found, and if the necessary resources are present, this
will result in positive stress, called Eustress. This state
of positive stress brings an individual in an optimum
state to perform. When the individual is not able to
cope effectively the stress becomes chronic and the
individual will experience distress (Selye, 1974). The
influence of the environment on the evaluation of stress
is pointed out by Lazarus & Folkman who define stress
as: a particular relationship between a person and the
environment that is appraised by the person as taxing
or exceeding his or her resource and endangering his or
her wellbeing (Lazarus & Folkman, 1984).
Some literature targets work-related stress in particular.
Le Blanc et al. (2003) identified the main stressors
occurring in an office environment and divided them
in the following four categories: job content, working
conditions, employment conditions and social relations
at work. Job content includes among others work over/
underload, complex or monotonous work, excess
54
of responsibilities, conflicting/ambiguous demands.

Working conditions include factors such as noise,
vibrations, lighting, temperature, posture, or lack of
protective devices. Employment conditions include
shift work, low wage, poor career perspective, job
flexibility/insecurity. Social relations at work include
poor leadership, low social support, liberties, and
discrimination Le Blanc et al. (2003). A model that
explains the experienced pressure from work is the
job demands and job resources model (Bakker et al.,
2004). Job demands refer to physical, psychological,
social or organizational aspects of the job that require
physical or psychological effort and costs (e.g., work
pressure, role overload, emotional demands, and
poor environment conditions). Job resources refer to
physical, psychological, social or organizational aspects
of the job that are functional in achieving work goals,
reduce job demands and the associated physiological
and psychological cost, or stimulate personal growth
and development.
Stress or tension can be effective and healthy, and may
help you to perform better, for example when giving a
presentation. However, stress can become ineffective
and harmful if it continues over a prolonged period
of time (Petri & Bouman, 2009). Without sufficient
moments of relaxation to release tension, the stress will
build up over time. If you push your body by constantly
being in an alert state and do not give it a chance to
recover, a time may come when you wont be able to
continue (Petri & Bouman, 2009). In this paper we
explore various design directions and opportunities that
could help working people to balance between an alert
state and recovery.
1.3 Designing for work-related stress
One of the roles of designers is to think about the
relationship between human and technology and to
designate it: give meaning to it (Krippendorf, 1989;
Verganti, 2009). Thus, the role of the designer is to give
new meaning to technology. While technology has the
capability to expand upon the possibilities of human
beings, this is not achieved in a neutral way. Technology
changes human perception.
Krippendorff & Butter (1984) indicate that designers can
demystify complex technology, improve the interaction
between artifacts and their users, and enhance oppor-
Fig. 1. GRIP Service Model v6, designers (white) and all stakeholders (black) alternate in taking up a central role in 3 different stages:
design for analysis (reframing), design for research (probing), design for support (prototyping)
tunities for self-expression through product semantics.

Meanings emerge in human interaction with objects
(Krippendorff & Butter, 1984). By formulating ideas
about the object, and cognitively placing it into (real or
imagined) contexts, people allow themself to formulate
an understanding of the object, or in other words:
Meaning is a cognitively constructed relationship
(Krippendorf, 1989)
The role of the designer is different for PSS design
compared to traditional product design (see also
Raijmakers et al., 2012; Van de Garde-Perik et al., 2013).
In PSS development multiple stakeholders can play
a role and each can offer different values to the PSS
development. In our project targeting work-related
stress PSS, we discovered already at the start that there
were a large number of stress combating or relaxation
techniques that were available through various channels
(books, websites) and service providers (psychologist,
mindfulness therapists, wellness resorts).
One of the first activities within the GRIP project
was an expert day were industry stress experts with
backgrounds in psychology, technology and occupational
health, and a worker with experience of burnout
met with designers to discuss and explore the topic
of stress at work. Furthermore it enabled us to gain
and share knowledge with industry professionals, and
to start building a network of expertise. The invited
stress experts indicated that there is currently a lack

of accessible tools that visualise the effects of stress.
Thus, many of these existing techniques lack the use
of technology, while Philips had the opportunity to
make use of various technologies to monitor human
physiological processes.
The main conclusion of the expert day was that people
need to be aware of their body signals or stress levels,
but also need to be motivated, in control, responsible,
and active to deal with stress. In addition, it became
apparent that there is a strong taboo on discussing
stress at work, and that the image of stress might be in
need of rebranding. Therefore, within this project we
took an open innovation approach together with various
experts and explored the potential of a combination
between existing stress techniques, and state of the art
technology, while using research and design expertise.
2 Choosing a design direction for

work-related stress
Before choosing a final direction for our nature inspired
design prototype we performed contextual research
into work-related stress. The three stages of the GRIP
service model were followed in this process: reframing,
probing and prototyping (See Fig. 1; For a more detailed
description see Raijmakers et al., 2012; Van de GardePerik et al., 2013).
55
2.1 Reframing work related stress

Currently, many organizations feel more responsibility
to take care of their employees, e.g. to lower burnout
rate, to let people perform optimally, to take care
of employee well being. Within the GRIP project,
we partnered with a mental health care institution
in Eindhoven (GGZE), to co-develop a PSS targeting
work-related stress. We chose for a mental healthcare
organization because we learned from the expert day
that especially employees in a caring job such as in
health care or education tend to take up more work
than they can deal with or find it hard to say no in their
job, because they are working with people who need
their help (mental health clients, or students). Besides,
the GGZE is particularly open to employee wellbeing,
because of their Planetree vision that good healthcare
can only be provided through by a healthy organization
(for more information, see http://planetree.org/).
As a first step in getting familiar with the GGZE
organisation and work routines, we observed and
interviewed health care workers in their own environ
ment to spot potential problems or causes of stress that
we could target and monitor. As designers we wanted
to better understand their work pressure and stressors,
and understand the differences between various
approaches or strategies that employees have regarding
work-life balance and stress relief. It was agreed with
the department of Ambulant Care that we could
shadow several GGZE employees during a working day
to observe the overall organisations culture of stress.
Furthermore, we collected more stress insights by
having employees monitor their work patterns through
three different stress data collection probes: a manual
registration of breathing patterns, a GRIP-booklet
to collect various stress measurements, and a public
relaxation measure.
Firstly, the manual registration of breathing pattern
was inspired through different channels, for example
by various expert presentations, existing technologies
such as emWave (http://www.heartmathstore.com/),
stresseraser (http://stresseraser.com/), and the book
by OHare & Blase (2010). Participants were asked
to register their breathing for a period of 2 minutes
by drawing a wave pattern on a paper roll (see Fig 2).
Secondly, the GRIP-booklet of stress measurements
was inspired by diaries as often used in cultural probes
56
Fig. 2. One of the probes used to collect stress insights via

breathing patterns. Instruction (top) and samples (bottom)
of manual breathing registration on paper (upwards line
when inhaling, downwards when exhaling).
studies (Mattelmki, 2006). Each page of the GRIP

booklet consisted of a number of fields where the
following data could be entered: date and time of the
registration, a comment field, the Self-Assessment
Manikin (Bradley & Lang, 1994: positive-negative
valence, high-low arousal and low-high dominance),
the six item State Trait Anxiety Index, and a field to
enter the result of the Azumio Stress Check (http://
www.azumio.com/apps/stress-check/). Thirdly, the
public relaxation measure asked questions concerning
relaxation and break taking behavior at work. The
relaxation measure encompassed reading a question
and giving an answer by putting an ice stick in the
appropriate vase (see Fig. 3).
Multiple workshops were organised with researchers,
designers, GGZE employees and managers, as well
as stress experts to discuss and visualize workrelated stress from various perspectives. Together
we discovered large individual differences between
employees; in terms of work strategy and perceived
obstacles/difficulties, in preferences regarding the
use of particular data probes, but also in actual stress
measurements (i.e. level of stress). For some people the
mere reflection on the personal situation was sufficient,
whereas others were in need of the data being visualised
and interpreted for them. Some people indicated
to want evidence of the data probe before using it,
whereas others where open to using it without any
Fig. 3. One of the three probes used to collect stress insights
Fig. 5. Example Paced Breathing: Rhythm Pad 3D render (top)
on break taking behavior, namely the public relaxation measure
and prototype (bottom) including feedback on Heart Rate
used in canteen of GGZE.
Variability TU/e student project by Sergej Lojko
Fig. 4. Example Personal Balance: Reco, smartphone relaxation

coach app TU/e student project by Christoph Lukkien
evidence or explanation. It became clear that extreme

scores and deviations are potentially interesting and
easy to investigate in more detail. Furthermore, in order
for self-assessment to become successful there needs to
be motivation, but also some sort of routine (i.e. fixed
times during the day; every time you return to your
desk, at the start of a team meeting). The environment
was another important factor that turned out to have
an influence on perceived work load (i.e. different
locations of the GGZE were perceived differently; open
plan offices resulted in most disturbances from visitors
or noisy colleagues).
2.2 Probing work-related stress solutions

More than 50 ideas and probes targeting work-related
stress were developed on the basis of data visualization
and co-creation events with all stakeholders involved.
After receiving feedback from stress experts and
GGZE employees these ideas were eventually reduced
to 3 directions for prototypes of work-related stress
solutions: Personal Balance, Paced Breathing and
Ambient Experience (see Figures 4-6). Over 25 prototypes have been made in total, both by students of
Eindhoven University of Technology and by Philips
Design Innovation. Below, we present a selection of
these prototypes, aimed at making work-related stress
less of a taboo topic for people, encouraging people
to treat stress in a more lighthearted fashion, while
respecting that people differ in what might help them.
Note that there are overlapping elements between
the prototypes, since all of them require monitoring
some kind of parameter (physiological stress levels,
consecutive working times, keyboard hits, emotions etc.).
The Personal Balance solutions aim to help people by
providing insight into this measured parameter (an
example is shown in Figure 4). This is a personal wellbeing solution that helps a person to see their daily
stress and the way they cope with it. By giving feedback
of previous stress events and the current stress level,
a person gets motivated to recuperate, by going for
a walk, or relaxing for a while.
57
Fig. 6. Example Ambient Experience: Initial concept sketch and

prototype (left). Zen Pebble, Intelligent Aroma Diffuser. Emotion
detecting software is used to release aroma at an appropriate
moment through Zen Pebbles (right) TU/e student project by
Sherry Hui Wang
The Paced Breathing solutions are centred on helping

people to regulate their breathing rhythm and conse
quently to influence other relevant parameters (see
Figure 5 for an example). Paced breathing is intended to
re-establish an optimal connection between the mind
and the body. People are sometimes not aware of their
mental state while at work. As tension goes up, the
head is overflowing with things that need attention or
require decision-making. People get disconnected from
thoughts, feelings and bodily responses, and unaware of
their well-being.
Finally, the Ambient Experience solutions are based on
reactive environments that provide a more public (and
thus social) type of feedback to people (see Figure 6). By
tickling the senses and offering different types of mental
triggers, an environment can stimulate you to become
more aware of your body and mind. Subtle fragrances,
sounds, haptics and visuals can be (sub)conscious cues
for people in the environment to take a moment for
themselves, and nudge people into different norms and
values about relaxation at work.
2.3 Co-creating a work-related stress solution
We intended to build and evaluate a nature inspired
prototype for work-related stress that is informed
by insights from all three directions at Philips Design
Innovation, in collaboration with multiple experts.
58
This means that experts in various domains are actively

involved in the design process, and give intermediate
feedback to our design decisions and input for future
decisions. As such, Philips has chosen a flexible
approach for the development of the prototype, based
on the experiences in the GRIP project and the NID
consortium. The final design activities of Philips Design
Innovation are a powerful push towards re-branding the
notion of work-related stress, following some of the
probing directions we discussed above.
The projects aim of the prototype building was to
generate expert and user feedback. The purpose of the
actual design was to help people to cope with workrelated stress during work time, both as prevention
of stress building up and treatment of existing stress.
Furthermore, we wished to place the demonstrator in
a public space to raise awareness regarding the topic
of stress, and hopefully to change group dynamics
within the workplace. Therefore, the space should build
on the strengths of nature, yet fit into the working
environment. It should be an accepted and attractive
place to retreat oneself during work time, it should not
be stigmatizing in any way but inviting by its appearance.
A pressure cooker was conducted to build further
on the experiences in the GRIP project and the NID
consortium. Within this pressure cooker we generated
more detailed ideas for our relaxation space, created
a rough first prototype and applied body storming as a
means to explore various use scenarios and obtain quick
feedback from Philips employees. From this pressure
cooker we learned among others that the space should
allow oneself to be shielded of from the work place
and provide a safe and private environment that would

fit both individual and group use. The insights from
the pressure cooker were later complemented with
additional internal research and employee interviews
within Philips Design Innovation.
Originally the idea was to have two related prototypes,
namely one prototype at peoples workplace to trigger
people to take a break. And the other prototype would
be the place where people can go to have their break,
a relaxing session or a peaceful moment away from
the usual work place and activities. However, after
long deliberation it was decided to offer a communal
experience instead of a personal treatment. A personal
measurement or trigger to go to the relaxation space,
could lead to stigmatizing people (being too stressed
or incapable of managing your work), or even resulting
in people avoiding to use the triggering function (e.g.
such as frequently happens after installing RSI software).
Therefore, the personal stress measurement or trigger,
was not incorporated in the final design. Instead, we
aimed to increase awareness within the community
regarding the need for healthy and balanced working
behaviour by intrinsically motivating people and
encouraging positive social support among colleagues
within the working environment.
The prototype of the relaxation space had to be build
within two months and should fit to the context and
employees of the Philips Design building at the High Tech
Campus in Eindhoven. The design of the space would
be a nature inspired ambient environment, supporting
individuals and groups of people to regain their personal
balance. Besides, it should support several relaxation
techniques, such as paced breathing, mindfulness,
meditation, regaining energy, and power napping.
In order to achieve these qualities, a mix between
technology (mainly lights, sound, enclosing curtains)
and natural aspects (materials, feeling of a natural
environment, aroma therapy etc.) was strived for.
In our design of the relaxation space we wanted to
build on existing knowledge in the field of stress, as
well as rely on natures qualities (see Figure 7). Nature
has a high level of attraction to people, it offers many
experiences through various senses, and it has relaxing
qualities for people even though we have limited
control over it (e.g. we cannot control the weather).
Fig. 7. Inspiration from nature and design
Nevertheless, we can have some control over our

experience in nature by choosing when, where and
how to go (e.g. winter or summer, forest or seaside,
alone or with others). Taking into account the fact that
people who suffer from negative stress have difficulties
making decisions, we wanted to create a similar yet
different experience to nature with our relaxation
space. The relaxation space should offer always changing
experiences, have a relaxing quality yet offer limited
control. Thus, as in biomimicry and in line with Lifes
principles we wanted to take inspiration from natures
forms and processes to enable people to easily balance
relaxation and work; maximizing the benefits for
employees with minimal effort. The aim was to create
an ambient environment that will help people to feel at
ease, and let go of some control.
The relaxation space could be considered as a gateway
from the work environment into a different world, with
the following features:
It should be a welcoming space with an inviting
ambiance, appealing to peoples curiosity. The space
should appear to be alive and inviting even if nobody is
using it (like with nature there is always activity which
could potentially catch someones attention).
59
Physically away from the work space-environment but

still be nearby/out of sight from working people to pro
vide a shield from the open office working environment.
It should offer a relaxing, calming experience through
intuitive interaction, and provide focus and breathing
guidance through enclosure, music therapy and
chromo therapy (in max. 15 min.)
The space should enable people to tune the
experience to their own preferences and strategies
for relaxation in an easy way (e.g. individual vs. group
use, size of space, moment of day, position)
3 Implementation & evaluation

of a relaxation space
To enhance choosing the final design direction for the
relaxation space three different scale models were built.
Then, both a full scale paper prototype and a working
prototype of the relaxation space were developed. Each
of these stages will be discussed below.
Fig. 8. Small scale model for relaxation space (4*4 cells)
3.1 Prototyping a relaxation space

After building three different scale models, the rough
set up of the final design was chosen (see scale model
in Figure 8). The final design would be a relaxation
space that was shielded of from the environment
through curtains that could move separately from one
another to adapt the size of the space to the number
of inhabitants. The original scale model was designed as
4 by 4 cells, allowing many different configurations (e.g.
4 single cell configurations, 2 simultaneously used 2*2
configurations, a 3*3 and single cell configuration). The
next intermediate stage, was a low-fidelity prototype
that was built in the basement of the Philips Design
building (see Figure 9). The paper prototype consisted
of just 2 cells, and had paper curtains, a computer to
provide sound experience, and one single light source
from the ceiling above each cell. Nevertheless, the
paper prototype enabled us and invited experts to
experience what being in the relaxation space could be
like. Both the scale model and the low-fidelity prototype
were presented to experts in the stress domain, to get
intermediate feedback regarding our design.
We arranged meetings with various experts and
made visits to 2 location of the GGZE where special
attention is put into the design of rooms and buildings
to make clients feel at ease. These visits and meetings
60
Fig. 9. Full scale low-fidelity prototype of relaxation space (1*2 cells)
served as a great inspiration to our design process.

For example, we discovered that many experts we
have consulted use nature in their treatments in some
form (e.g, use of images, or going for a walk outside in
nature). The overall sensation of nature around oneself
can help to achieve a positive result (i.e. changing in
space, differences in temperature, wind, smells, etc.).
According to our experts nature works calming, but
it can also give problems (e.g. if you dont like certain
weather types; or if you suffer from winter depression,
spring exhaustion, or allergies). Therefore, we needed
to decide how closely we wanted to get inspired
through nature. Taking into account all feedback and
insights obtained so far, we decided to create an indoor
space with the strength of nature, yet fitted to an office
environment. As such, we wanted to create a space with
its own specific qualities next to having the opportunity
to go out in the real nature. As in real nature, the relax
ation space should offer a rich, multisensory experience,
with a rhythm of its own (daily or seasonal).
Furthermore, it was important that we provided a space
that is not work-related (to enable people to step out
of the work context). A cozy environment or the feeling
of safety can be truly important. Also, having the feeling
of owning a particular spot, or having control over the
situation are influencing factors that can make people
change their mind. Furthermore, in the research phase
we have seen that people sometime feel the lack of
control over their working environment (especially in
open plan work spaces).
Also the posture during meditation and relaxation
is very important; you should not be distracted by
physical discomfort. People are not supposed to fall
asleep when meditating, but that one should stay alert,
while mind and body should be relaxed. Following
specific indications can help people to stop their mind
from working. This is a skill that can be trained. Once
you know how to do it, you can do it without help.
Furthermore, from stress experts we learned that
relaxation is personal; what works for someone may
not work for another.
3.2 Full scale Implementation
Because of restrictions in the amount of available
time and resources, it was decided to build a smaller
configuration for our working prototype than the
original 4*4 configuration. The smaller configuration

allowed fewer possibilities for simultaneous use of
the relaxation space, but still it allowed us to explore
the overall concept underlying the space. The final
relaxation space is a configuration of 2*2 cells, with
Interface flooring, 12 curtains allowing the separation
of the 4 different quadrants, a ceiling with 16 individual
light sources, and an ambient soundscape.
Initially, the relaxation space invites passers by to
come in through the attraction of a white pulsating
light coming from the ceiling. Once people enter
the space, the relaxation space responds to peoples
presence. The curtains lower, the soundscape is being
generated, and the lights turn into a colored spectrum.
Depending on peoples preferences they can either sit
down or lay down, using the comfortable repositional
cushions provided with the space. Besides the ceiling
light, also light objects could be used inside the space
to create a different dynamic to the space. People
have implicit control over the relaxation space. The
curtains automatically adapt to peoples position within
the space. In addition, people can choose to bring in
furniture or light objects to further adapt the space
to their own relaxation preferences.
The relaxation space has been evaluated by 23
experts from various fields (see Figure 10), i.e. ambient
experience design, interaction design, relaxation (service
providers & researchers). This evaluation proved that
the space is very well suitable to speed up the relax
ation process. People experienced with yoga and medi
tation indicated the relaxation space enables them to
get an even quicker experience than when relying on
traditional yoga and meditation procedures. People with
less experience in relaxation (e.g. interaction designers),
did not really try to use the space for their own benefit,
but stated on the basis of exploring the relaxation space
that It could work. Perhaps the context of the expert
evaluation (they were asked to professionally evaluate
the relaxation space in the presence of a designer and
design researcher) did not fully enable them to use the
space for relaxation purposes. Therefore, in the next
stage of evaluation more emphasis was put on the need
for users to surrender to the space. The relaxation
space does not provide a magical cure to stress, but
people need to play and an active role (as was already
discovered during the first expert day).
61
situations and meanings of the relaxation space emerged

over time. For example, some people made use of
the relaxation space to get out of sight of colleagues
and enjoy a moment for oneself (see Figure 10), while
there were also large groups of people that used the
relaxation space during lunch time for a group mediation
with approximately 20 people (see Figure 11). The
pace of the lights suggests a breathing rhythm to users
and can provide support in paying attention to ones
breathing pace. The soundscape is generally appreciated
and supports the relaxation experience (e.g., the
sounds are soft and contribute to thinking less and
feeling/ experiencing more). Furthermore, based on
positive responses from various industries it could
be that (some parts of) our design become of meaning
in different application areas e.g. mental healthcare,
or hospitals.
Fig. 10. Individual use of relaxation space occurred
in expert evaluations and prolonged user evaluation
4 Conclusion
This paper illustrates how open innovation together
with multiple experts can be achieved in a relatively
new area for design (i.e. the field of work-related
stress). Our open and flexible approach has proven
beneficial to all stakeholders involved in PSS design.
Furthermore, the adaptive environment enables multiple
forms of interaction and hence can carry different
meanings depending on its use.
We have discovered that the relaxation space that was
designed, implemented in an office environment and
evaluated enables people to let go of control and allows
them to open up to a relaxing experience. People who
already practice yoga or meditation indicate that they
are capable to achieve a quicker and deeper effect by
being in the relaxation space.
Fig. 11. Shared use of relaxation space for

meditation purposes
Currently, we are further investigating the meaning

of the relaxation space for employees from Philips
Design over a prolonged period of interaction, namely
three weeks. Simultaneously, we are exploring small
adaptations in our design to improve the relaxation
experience for users. While the investigation is not
yet completed, we can already see that different use
62
5 Acknowledgements
This project was funded by the Dutch Government
via CRISP and supported by the Nature Inspired
Design consortium. This work could not have been
done without the inspiring input from all internal and
external GRIP & NID partners. Therefore, we would
like to thank everyone for their contributions to work
described in this paper. Special thanks to: TU/e (Dirk
Snelders, Christoph Lukkien, Sergej Lojko, Sherry Hui
Wang), Philips Design Innovation (Geert Christiaansen,
Anja Janssen, Tom Djadjadiningrat), DAE (Mike Thomp

son), Philips Research (Martin Ouwerkerk, Joyce Weste
rink), Interface (for providing the subfloor and carpet
tiles used in our relaxation space), GGZE (Erik Kuijpers,
Spike Ebbing, Marian van Engelenburg), Hans van Os,
Margot van de Ven, and Trimbos Institute (Linda Bolier).
vitality and stress resistance Thema, Publisher of Schouten

Global, Zaltbommel.
Raijmakers, B., Thompson, M. & Van de Garde-Perik, E.M.
(2012). New goals for design, new roles for designers? paper
presented at the Cumulus Conference in Helsinki, June 2012.
http://cumulushelsinki2012.org/cumulushelsinki2012.org/wpcontent/uploads/2012
/05/New-goals-for-design-new-roles-for-designers.pdf
6 References
Selye, H. The Stress of Life, New York, McGraw-Hill, 1956.
Badke-Schaub, P. and Snelders, D. (2011) GRIP. Project proposal
Selye, H. Stress without distress. Philadelphia: Lippincott
by project leaders. Retrieved 02 20, 2012, from: http://www.

crispplatform.nl/grip/grip
Bakker, A.B. Demerouti, E., & Verbeke, W. Using the job
demands-resources model to predict burnout and
Williams & Wilkins, 1974.

Verganti, R. Design Driven Innovation Changing the Rules
of Competition by Radically Innovating what Things Mean.
Bosto
performance. Human Resource Management; Spring 2004;

43, 1; pp. 83
Benyus, J. (1997). Biomimicry - Innovation Inspired by Nature.
New York, Harper Collins Publishers.
Bradley, M.M., and Lang, P.J. (1994) Measuring emotion: The
self-assessment manikin and the semantic differential. Journal
of Behavior Therapy and Experimental Psychiatry, vol. 25, no.
1, pp. 49-59.
Garde-Perik, van de, E.M., Snelders, H.M.J.J., Thompson, M.
(2013). Getting a GRIP on work-related stress: Designing
services with users and other committed stakeholders.
Paper accepted for the IASDR conference in Tokyo,
September 2013.
Krippendorff, K. and Butter, R.. 1984. Product Semantics:
Exploring the Symbolic Qualities of Form in Innovation,
Innovation, Spring 1984, 49.
Krippendorff, K.. On the Essential Contexts of Artifacts or on
the Proposition That "Design Is Making Sense (Of Things)".
Design Issues. Vol. 5, No. 2 (Spring, 1989), pp. 9-39
Mattelmki, T. (2006) Design Probes. Doctoral Dissertation University of Art and Design Helsinki.
Lazarus, R.S. & Folkman, S. Stress, appraisal and coping, New
York, Springer Publishing Company, 1984.
Le Blanc, P., de Jonge, J. & Schaufeli, W. (2003). Job stress and
health. In Nick Chmiel (ed.), Introduction to Work and
Organisational Psychology, a European perspective. Oxford:
Blackwell (pp.148-178)
OHare, D., and Blase, K. (2010) Slanker met je hartritme:
In negen weken afvallen met je hart als dirigent. Kosmos,
Utrecht.
Patel, S., and Mehta, K. (2011) Lifes Principles as a Framework
for Designing Successful Social Enterprises, Journal of Social
Entrepreneurship, 2:2, 218-230.
Petri, C., and Bouman, J, 2009. Busy, busy, busy: How to create
63
Christoph Bartneck
HIT Lab NZ, University of Canterbury Private Bag 4800, 8410 Christchurch, New Zealand
christoph.bartneck@canterbury.ac.nz
http://hitlabnz.org
Robots in the Theatre and the Media
Abstract
Robots frequently feature in the movies and occasionally
in the theatre. Their presence in the media is of major
importance on how users interact with them. Due
to the scarcity of robots in everyday life, most of the
users expectations and interpretations about robots
stem from the media. This paper provides a framework
of robots in the media that will help robot interaction
designers to assess preconceptions about robots.
Keywords
Robots, HRI, media, theatre
1 Introduction
Robots have triggered the imaginations of writers and
directors for many years. Robots have been featured in
all types of artistic expressions, such as books, movies,
theatre plays and computer games. The importance of
the media in forming our understanding of the world
cannot be underestimated, in particular in situations
in which access to real experience is limited. We are
at an interesting point in time where on the one hand
more and more robots enter our everyday lives, but
on the other hand, almost all our knowledge about
robots stems from the media. This tension between the
expectations fuelled from SciFi and the actual abilities
of the robots can result in negative experiences. It is
therefore important for us to know about how the
media has portrayed robots.
64
We would like to focus on robots being used in the

theatre since they better represent the current state
of robotics. Computer graphics can nowadays visualize
almost anything and hence depiction of robots in movies
can be more fantastic. Movies can show us robots that
use anti gravity to float around, but there is little use for
such a vision for the actual future of robotics. Robots
that have to work on stage are constrained to the
current state of the art in robotics and are therefore
closer to what robots will be like in the close future.
While this form of real-time acting introduces a healthy
dose of reality, it also constrains the distribution of the
performance. The audience needs to be present in the
theatre to experience a play. This temporal and physical
constraint of the performance limits the number of
people able to see the show. Movies, on the other hand,
can be distributed on disks or through the internet and
can be viewed at home at any desired moment. Theatre
plays are, as a matter of fact, often video-recorded to
document the event and to make the play available for
review, discussion and debate. We believe that although
the theatre is the preferred source for our study, we
need to expand it to include movies and other media
since far more people will have seen a movie with a
robot than a theatre play with one.
In the field of HRI, there has also been a long discussion
on whether video recordings of robots can be used as
a replacement for live human-robot interaction. While
Woods et al argued that the two interaction styles were

broadly equivalent [1], Bainbridge et al concluded that
participants had a more positive experience interacting
with robots than with a video representation [2].
Powers et al also found large attitudinal differences
between participants interacting with a collocated robot
in comparison to a remote robot [3]. These results
strengthen the importance of considering robots in the
theatre over robots appearing in movies.
We have to acknowledge that it was not possible for
us to consider every robot mentioned in every book,
film, computer game or theatre play. Their number
vastly exceeds our limited capacity for processing. But
an exhaustive review is, in our view, not even necessary.
We believe that we can still draw some valid conclusions
from a representative sample of robots in the media.
We also do not claim that any of the conclusions drawn
in this paper are scientifically valid by the standards of
the natural sciences. We are filtering and interpreting
the available data and we cannot exclude the possibility
that our personal knowledge and experience biases the
results. But we shall not shy away from the task because
of this limitation. Most of the work in the humanities
relies on interpretation and the arguments we bring
forward are hopefully still of interest to the HRI
community. Cynthia Breazeal has already argued that
theatre is an interesting test bed for HRI [4].
The goal of this paper is to reflect on how robots
have been portrayed in the media, which leads us
into a definition of a framework to describe common
patterns of human-robot interaction in the media, and
this in turn gives us clues on the expectations that
future users have towards their robots.
2 HRI as theatre
The research field of human-robot interaction (HRI)
investigates how humans and robots interact with
each other. This multidisciplinary field includes three
large sub fields. Firstly, it includes the developments
of robotic technology that are targeted to be useful
for the interaction. Secondly, it includes a creative field
in which daring new ideas are explored and artistic
installations are developed. Lastly, it includes studies
that are targeted at understanding human reaction
towards robots. For this purpose, experiments are
conducted in which robots act in front of human users,

as not all actions of a robot can be produced as a result
of computations. Due to the limitations of artificial
intelligence, researchers often have to fall back on the
Wizard-of-Oz method. In this method, an experimenter
observes the interaction from a hidden location and
remotely controls the robot accordingly. This is a
mild form of deception, since it allows the robot to
appear more intelligent than it really is. More elaborate
deceptions are occasionally necessary to test certain
aspects of HRI, such as the study of embarrassment [5]
or of the users hesitation to kill a robot [6].
Experiments with users in the field of HRI have the
character of theatre plays, if not busking. The actions
of the robots often follow scripts and occasionally
the robots are even able to improvise based on their
autonomous decisions. The users play the role of an
involuntary actor that is called up onto the stage from
the audience, similar to what is practised in busker
festivals or comedy shows. While the busker (aka
robot) has some understanding of the structure and goal
of the act, the users are not always fully informed about
what they may do or not do. They mainly try to comply
with the expectations of the experimenter outlined
in the description of the experiment. But even such
descriptions can be intentionally misleading to achieve
a certain deception. The users then have to fall back on
their own experiences and ideas of robotics, which are
based on the depiction of robots in the media. What
we experience in laboratory experiments today is what
future users will experience at home when interacting
with their home robots.
The performing arts have been experimenting and
testing effective methods of persuasive interaction for
centuries and we agree with Guy Hoffman who argued
that we can learn from their insights [7].
3 Short overview of robots in the media

Robots have been portrayed in the media for many
years. Although general stories of artificial human
beings, such as Golem in Jewish folklore, have been
around for hundreds of years, it is Karel Capek who
arguably formed the word Robot with his theatre play
R.U.R. - Rossums Universal Robots that premiered
in 1921. The relationship between humans as puppets
65
in the theatre has also been explored by Oskar

Schlemmer in his Triadisches Ballett [8].There are
some archetypes in that storytelling about robots that
we would like to reflect upon.
3.1 Robots want to be humans
In many stories, robots are portrayed as wanting to be
human, despite their superiority in many aspects, such
as strength or computational power.
This desire to become human is the central story line
for Isaac Asimovs The Bicentennial Man in which a
robot named Andrew Martin is following a lifelong plan
to become a fully recognized man [9]. The book was
the base on which the movie of the same name was
released in 1999, featuring Robin Williams as Andrew
Martin. Besides becoming physically more human-like,
Andrew Martin also fights a legal battle to gain full legal
status. He is even prepared to accept mortality to gain
this status.
Other robots, such as the replicant Rachael in the
movie Blade Runner, based on the book by Philip
K. Dick [10] are not even aware of the fact that they
are robots. The same holds true for the some of the
human-like cylons in the 2004 TV series Battlestar
Galactica. Not being aware of the nature of your own
existence is possibly the highest form of human-likeness.
A prime example of a robotic character that is aware of
its robotic nature is Mr. Data from the TV series Star
Trek The Next Generation (see figure 1) played by
actor Brent Spiner.
Fig. 1. Mr. Data played by Brent Spiner.
66
Mr. Data is stronger, has more computational powers

and does not need sleep, nutrition or oxygen. Still, his
character is set up to have the desire to become more
human-like. Mr. Datas main difference from humans is
his lack of emotions.
Steven Spielbergs movie A.I. based on Brian Aldisss
short story Super Toys Last All Summer Long [11]
accepts the main premise that robots lack emotions,
and hence Professor Allen Hobby, played by William
Hurt, builds the robot David that does possess the
ability to love.
These three archetypical examples are only the tip
of the iceberg. The appeal of this storyline stems, in
our opinion, from our inferiority complex. Already a
hundred years ago we were building machines more
powerful than us, and there has also been major
progress in the area of artificial intelligence. On May 11,
1997, the Deep Blue computer won the first chess
match against a world champion. In 2011, the computer
Watson won the game quiz show Jeopardy. Both
computers were developed by the IBM Company. In
light of this progress, it is easy to imagine how robots
in the future might be strong and intelligent. Humans
would be therefore be reduced to an inferior position.
By giving robots the desire to be human-like, we rescue
our self esteem. We flatter ourselves by making robots
wanting to be like us.
Rational problem-solving and calculations have been the
prime focus of the development of computers and hence
it comes as no surprise that SciFi authors consider
emotions to be a feature that all robots would lack.
It would be a crucial desire of robots to feel emotions.
It has been shown that humans require emotions to
act intelligently but the same does not hold true for
machines. Moreover, several computational systems of
emotions have already been successfully implemented.
The computer programs implementing the so called
OCC Model of Emotions [12, 13] are prime examples.
Equipping robots with emotions to finally make them
human is therefore an archetypical story line.
But we are not only flattering ourselves. Our need for
appreciation goes beyond that. We want to be creators
of life and create machines that are not only in our
image, but that even admire us. Professor Allen Hobby,
creator of the emotion robot, puts it like this in the first

minutes of the film A.I. In the beginning, didnt God
create Adam to love him? We are afraid of entities that
are stronger and smarter than us and hence we make
them to want to be like us and love us. It feels much
better that way.
3.2 Robots will kill all humans
Another archetypical story line is the robotic uprising.
In short, humanity builds intelligent and strong robots,
then the robots decide to take over the world and
enslave or kill all humans. Going back to the example
of Mr. Data, he has a brother named Lore that
possesses an emotion chip. Lore follows this typical
path of not wanting to be like a human but instead
wanting to enslave humanity. Other popular examples
are The Terminator (1984), the cylons on the 2004
TV series Battlestar Galactica, the Machines in the
movie The Matrix (1999) and the robots in the 2004
movie iRobot. The latter is based on the book of the
same name by Isaac Asimov. Asimov coined the term
Frankenstein Complex to describe this archetypical
storyline.
This archetype is built on two assumptions. Firstly,
that robots resemble humans. The robots have been
designed to look and think and act like their creators.
However, they exceed their creators in intelligence
and power. Secondly, once they interact with the
now inferior human species, they will act exactly
as humans have historically acted when they have
encountered so called inferior species. The example
of the Spanish Inquisitors and the German Nazis
provide horrifying examples on how one superior
race enslaved and killed an inferior one. And since
robots resemble humans, they would also be likely to
enslave and kill humans. It can be argued that even today
the developed countries take advantage of the less
developed ones, but we certainly have widely overcome
slavery and genocide. It still occasionally takes place, but
is condemned by the international community.
It is time that we overcame this superficial view that
all superior robots would have nothing better to do
than to enslave and kill humans. Several science fiction
authors have already proposed future scenarios in which
superior robots quietly and positively influence human
society. In Isaac Asimovs Prelude to Foundation, he
describes a robotic First Minister that keeps the empire

on the right track. Interestingly, the First Minister hides
his robotic nature. He is a very human-like looking robot,
but needs to resort to various strategies to blend in.
He does, for example, eat food, despite the fact that he
cannot digest it, collecting it in a pouch that can be emptied
later. Here we have a scenario in which a superior being
helps the human society behind the scenes.
4 Framework for performing robots

The goal of our framework is to provide a high level
categorisation of human-robot interaction. First we will
discuss the actual implementations of performing robots
before reflecting on different types of interaction themes.
4.1 System Architecture
On a high level of abstraction, theatrical robots can
be categorised by two factors. The Locus of Control
factor describes where the control unit for the robot is
located, and can either be inside the robot or outside of
it. The Control Entity factor can either be human or the
robot itself.
Locus Of Control
Control Entity Inside
Outside
Human
Guy in a suit
Puppeteering
Robot
Autonomous
Robotdrone
Table 1. System Architecture
With this matrix in place (see table 1) we can now

illustrate the four quadrants with some examples.
The Guy in a Suit quadrant is populated with robots
that are operated by a human placed inside the robot.
Possibly the most well known example might be The Tin
Man from the movie The Wizard of Oz played by Jack
Haley (see figure 2 or 3CPO from the movie Star Wars.
One of the difficulties of this approach is that the robot
needs to have roughly the shape and size of a human.
George Lucas cast Kenny Baker, who is 112 cm tall, to
operate the R2-D2 robot in the movie Star Wars. This
allowed him to give R2-D2 a distinctly different shape.
Lucas possibly got this idea from Douglas Trumbulls
1972 film Silent Running, where double-amputee Mark
Persons acts inside Drone 1 (Dewey) (see figure 3).
67
robotic dance-like practices there were the participants

of the Daft Punk Aibo Dance Competition in 2005
organised by Sony, and the many dances of the Nao
robot, such as in Michael Jacksons famous Thriller
dance. While there are many examples of dancing robots,
there are very few that are built to dance together with
humans. The best such dancing robot is the Partner
Ballroom Dance Robot (PBDR) that allows humans to
dance in the classical ballroom style with the robot [14].
5 Theme types
Fig. 2. Jack Haley as the Tin Man in The Wizard of Oz
Now that we have a first overview of robots in the

media, we may now proceed to define our research
question: What is the role of robots in theatre and
in what direction can we take it in the future? From
a conceptual point of view, robot theatre plays are
typically of four types. They can either focus on the
similarities or the differences between humans and
robots, and in terms of either their body or their mind
(see table 2). Dixon supports this view by stating that
artists explore the deep-seated fears and fascinations
associated with machine embodiment in relation to
two distinct themes: the humanisation of machines
and the dehumanisation of humans [15]. We extend
his framework by dividing these two themes into the
physical embodiment and the mind. We thereby define
four types of themes (see table 2).
Fig. 3. Mark Persons inside of the robot Dewey.
For movies, it is no longer necessary to use the Guy

in a Suit method, since modern Computer Generated
Graphics (CGI) can easily create realistically moving
robots. But for a theatre play, this approach is still
the most common methodsince it barely requires any
technology. All the motions and behaviors are directly
controlled and performed by the actor. The illusion of
a robot can even be created by movements alone. In the
popular robot dance style, dancers create the illusion
of being a robot by very abruptly stopping movements
(dimestop) and by constraining the degree of freedom
of their body. Both methods imitate typical movements
of robots that have electrical motors.
While humans try to dance like robots, robots,
ironically, try to improve their dancing skills to become
more human-like. Of the more well-known examples of
68
Mind
Body
Similar
Different
Similar
Type I
Type II
Different
Type III
Type IV
Table 2. Topics of HRI in the theatre
These four types of topics can of course be mixed.

If we take the example of Mr.Data from above, he looks
very much like a human which sets our expectations
accordingly (Type II). It then appears dramatic and
surprising if Mr. Data is able to enter the vacuum
of space without taking any damage. In the movie
Prometheus, the android David, played by Michael
Fassbender, is wearing a space suit when walking on
an alien planet. Wearing this suit does not serve a
functional purpose since David does not require air
at all. The following dialog emerges:
charlie holloway: David, why are you wearing

a suit, man?
david: I beg your pardon?
charlie holloway: You dont breathe, remember?
So, why wear the suit? david: I was designed like this,
because you people are more comfortable interacting
with your own kind. If I didnt wear the suit, it would
defeat the purpose.
Again, the apparently human embodiment sets our
expectations and when a difference to humans is
displayed, it surprises the audience. Godfried-Willem
Raes with his robot orchestra takes a different
approach. He emphasises the equality of robots and
humans in his theatrical performances (Type I).
He argues that:
If these robots conceal nothing, it is fairly self-evident that
when their functioning is made dependent on human input
and interaction, this human input is also provided naked.
The naked human in confrontation with the naked machine
reveals the simple fact that humans, too, are actually
machines, albeit fundamentally more refined and efficient
machines than our musical robots. [16]
An example of Type III could be Johnny Five from the
1986 movie Short Circuit. Although Johnny Five has a
distinctly robotic body he expresses human emotions,
which suggests that his mind is similar to humans.
This contrast between robots being similar or different
to humans is certainly one of the most often-used story
elements of robots in the theatre and movies. But this
view on human robot relationships is superficial. It is
like admiring the eye shape or coloring of someone from
a culture that you have never encountered before. At
the beginning, this might be fascinating, but soon enough
we understand that there is much more to this person
than his eyes or color. The four types of robot theatre
plays described above can also be applied to humanhuman relations.
superior robots act as badly as humans have done when

encountering inferior beings? We may ask ourselves
why these questions are so persistent in the media.
The most obvious answer is that stories need to have
a conflict to generate tension. A fictional world in
which everybody is happily living together is unlikely
to capture the attention of the audience. Pitching evil
robots against good humans does not only serve the
purpose of creating a conflict, but also triggers an ingroup effect. We humans feel that we need to defend
our species against the out-group robots. This division
can then be challenged by introducing robots that are
indistinguishable from humans, such as in Battlestar
Galactica. This creates a great uncertainty and this
creates tension. We wonder, is the character a robot
or a human? And if so, is he good or evil?
The matching of evil robots and good humans is most
persistent in Western culture. Robots are extremely
popular in the Japanese media but here we can observe
a more nuanced balance. Robots such as Atomic
Boy are good characters that help the humans against
other evil humans. These diverse scenarios break the
Terminator Pattern and allow us to consider the
relationship between humans and robots in more depth.
The research on HRI cannot shy away from the represen
tation of robots in the media and the elicitation of the
associated fears since this is what the current users are
aware of. It might be useful for HRI experiments to take
away some of the ambiguity when confronting users
with real robots. It should be clear that the robot is
a machine, and what its abilities and intentions are.
6 Conclusions
But we do not only need to take past exposures to

robots into account, we also need to present our users
with positive and realistic visions of the future. Notable
exceptions from the gloomy visions in the media are
in the TV series Futurama by Matt Groening and
the Movie Robot and Frank by Jake Schreier. In both
shows, a vision of the future in which humans and
robots live peacefully side by side is depicted. More-
over, they even become friends. Thinking about how
we develop our relationships with robots is one of
the most important problems in HRI.
The story archetypes and theme types proposed

appear frequently in the media. They address our inner
fears and fascinations when interacting with robots.
Are we like robots? Are robots like us? And if so, will
But there is also a positive vision for the performing

arts in human-robot interaction. Home theatre systems
can now become what their name promises. Imagine a
69
future in which you do not download the movie, but in

which you download the theatre script into your robots.
You can then either watch the robots perform the play
or even join in. It is important to notice that robotics
has in the past and still is today being used to automate
tasks that we do not want to perform ourselves.
Industrial robots, for example, were introduced to
relieve us of difficult and repetitive manual labor. There
is little use in automating tasks that we actually enjoy
doing. This does not mean that there is no place for
robots in the theatre at all. Plays that actually deal with
robots should of course be cast with robots.
8. Scheper, D., Schlemmer, O.: Oskar Schlemmer - das
Acknowledgments
I would like to thank Ryan Reynolds for a vivid
discussion on the topic.
13. Ortony, A. In: On making believable emotional agents
Triadische Ballett und die Bauhausbhne. Akadademie der

Knste, Berlin (1988)
9. Asimov, I.: The Bicentennial man and other stories. [book
club edn. Doubleday science fiction. Doubleday, Garden City,
N. Y. (1976)
10. Dick, P.K.: Blade runner : (Do androids dream of electric sheep).
25th anniversary edn. Ballantine Books, New York (2007)
11. Aldiss, B.W.: Supertoys last all summer long : and other
stories of future time. St. Martins Griffin (2001)
12. Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of
Emotions. Cambridge University Press, Cambridge (1988)
RefMgr field[1]: Book, Whole RefMgr field[8]: Not in File.
believable. MIT Press, Cambridge (2003)
14. Kosuge, K., Takeda, T., Hirata, Y., Endo, M., Nomura, M.,
Sakai, K., Koizumi, M., Oconogi, T.: Partner ballroom dance
robot-pbdr. SICE Journal of Control, Measurement, and
References
1. Woods, S.N., Walters, M.L., Koay, K.L., Dautenhahn, K.:
Methodological issues in hri: A comparison of live and video-
to nature, and camping about. TDR (1988-) 48(4) (2004)
based methods in robot to human approach direction trials.
pp. 1546
In: The 15th IEEE International Symposium on Robot and

Human Interactive Communication. (2006) 5158
2. Bainbridge, W., Hart, J., Kim, E., Scassellati, B.: The benefits
of interactions with physically present robots overvideodisplayed agents. International Journal of Social Robotics 3(1)
(2011) 4152
3. Powers, A., Kiesler, S., Fussell, S., Torrey, C.: Comparing a
computer agent with a humanoid robot. In: 2nd ACM/IEEE
International Conference on Human-Robot Interaction.
(2007) 145152
4. Breazeal, C., Brooks, A., Gray, J., Hancher, M., Kidd, C.,
McBean, J., Stiehl, D., Strickon, J.: Interactive robot theatre.
In: Intelligent Robots and Systems, 2003. (IROS 2003).
Proceedings. 2003 IEEE/RSJ International Conference on.
Volume 4. (oct. 2003) 3648 3655 vol.3
5. Bartneck, C., Bleeker, T., Bun, J., Fens, P., Riet, L.: The
influence of robot anthropomorphism on the feelings of
embarrassment when interacting with robots. Paladyn Journal of Behavioral Robotics 1(2) (2010) 109115
6. Bartneck, C., Hoek, M.v.d., Mubin, O., Mahmud, A.A.: daisy,
daisy, give me your answer do! - switching off a robot. In:
2nd ACM/IEEE International Conference on Human-Robot
Interaction, ACM Press 217 222
7. Hoffman, G.: On stage: Robots as performers. In: RSS 2011
Workshop on Human- Robot Interaction: Perspectives and
Contributions to Robotics from the Human Sciences. (2011)
70
System Integration 1 (2011) 7480

15. Dixon, S.: Metal performance humanizing robots, returning
16. Raes, G.W.: Naked (2009)
Sara Colombo, Lucia Rampino

Design Department, Politecnico di Milano, Milano, Italy
(sara.colombo, lucia.rampino)@polimi.it
Beyond Screens.
Exploring product dynamic features
as communication means
Abstract
1 Introduction
Product semantics has been studying how product

sensory features convey implicit messages to users for
decades. However, thanks to technology advances,
traditional static product features (such as its shape,
texture, colour, smell) are becoming dynamic, able to
change actively over time. These new properties might
be employed to communicate dynamic information to
users. Exploring how this sensory communication can
be performed, and the advantages and limits connected
to dynamic products, is the aim of this paper. To achieve
the goal, three studies have been conducted: a case
studies analysis, a design workshop and interviews
with users. Results show that dynamic products can
communicate messages to users and can also enhance
the user experience, by making the communication
both more effective and engaging. However, some
difficulties have been highlighted, mainly related to the
designers ability to control those dynamic features.
The work gives an overview of the new category of
dynamic products, together with their advantages and
limits, when design theory still lacks the knowledge to
support practice; it thus constitutes a starting point
in the exploration of an emerging scenario for the
communication from products to users.
Artefacts have the ability to communicate to users

by means of different channels. One is that of the
product sensory features (like its shape, texture or
colour), which convey messages to users through an
implicit language - the product language [1, 2]. The
meanings transferred by this channel are intrinsic to the
product, casted in its matter by designers to remain
(almost completely) static during the product life, even
if apt to different interpretations. For instance, the
shape, material or motor sound of a sport car might be
associated to the idea of speed, luxury and social power,
and designers have to control such product features in
order to convey the intended messages to users.
Keywords
Dynamic products, communication, senses, user
experience, product design
The other media is that of information displays, in the

form of digital screens, which transmit messages mainly
connected to phenomena that are extrinsic to the
physical product [3].
These two layers of information have remained
separated for some time, since screens and displays
were being added to the physical substrates of products
as external and independent elements. Until something
has changed. The digital and physical world have been
gradually merged in a unique and complex layer of
bits and matter [4], making it always more difficult to
distinct the virtual from the real. Such a process opens
new scenarios for product design, since the massive
71
invasion of technology into objects affects not only the

functions and performances of products, but also their
aesthetics, sensory properties and meanings, bringing
forth new interaction modalities and new media for
communicating to users.
The following paragraphs will briefly present the main
characteristics of the sensory and digital communication
performed by products, to finally introduce a peculiar
novel kind of communication, which is the focus of the
present work.
1.1 Sensory communication
Sensory product features (e.g. shape, texture, sound,
temperature, smell, etc.) convey messages to users, who
perceive, interpret and associate meanings to them.
The communication performed by product sensory
features can be divided into two layers. The first
layer of communication is static and related to the
product itself. Through the product language [2], this
layer conveys to users intrinsic messages, which are
related to what product semantics has defined as:
description, expression, exhortation and identification
[5]. Different scholars have proved that the aesthetic
features of artefacts (shape, colour, material, etc.) have
semantic functions themselves, since they affect the
interpretation of the product, such as its character or
personality, affordances, or gender associations [2] [6].
As Crilly states, semantic interpretations and symbolic
associations are triggered by the physical qualities of
artefacts, which are often understood as the visual
qualities of products, but can also be related to other
sensory features, like the tactile or auditory ones [5].
The second layer concerns the natural changes these
sensory features can undergo. For instance, a change
in the temperature of the cup one is holding may tell
that the tea just poured in is very hot, or a change
in the shape of a carpet may tell us that someone
has trampled it. In this case, we (consciously or not)
interpret products as detectors and indicators of
changes of the environment. These changes can also
result from something occurred inside the product.
For instance, the sound of a washing machine motor
reveals it is working, the temperature of a computer
tells us if it is on or off, and so on. In all these cases, we
receive information about situations that are external
(or internal) to the products, which naturally modify the
72
physical and sensory features of the objects. Thus, we

are accustomed to receive and elaborate these dynamic
messages while interacting with products. However,
while the first layer of communication (the product
static appearance) is designed and controlled by the
designer, the second one follows instead natural and
physical laws. They differ one from another because of
the content of the communication (intrinsic and fixed
in the first case, extrinsic and dynamic in the second
one) and the possibility to control them.
1.2 Digital communication
Another form of communication from products to
users is the digital communication. We intend by digital
communication the transmission of messages coming
from the world of digital information, which relies
mostly on interfaces.
Environments, spaces, products and people are more
and more connected or integrated with sensors,
technology and smart systems, which are able to
produce a lot of data that have to be communicated
to users. Products are becoming mediators between
the information such intelligent systems want to
convey and the user. The easiest way to do that is
to add digital interfaces to products, in the form of
screens. As Krippendorff and Butter state, products
can convey messages related to external situation
through the media of information displays (interfaces).
These messages have a dynamic nature, since they
refer to changing situations and events, and adopt an
alphanumeric, iconic and graphic language [3].
Nevertheless, digital screens are not the only way for
users to have access to the digital world. During the
last decades, disciplines such as interaction design and
HCI have been trying to reconnect the digital to the
physical world. Under the umbrella name of tangible
interactions, many approaches have been developed in
order to find new ways to bridge tangible products and
virtual bits, starting from the assumption that traditional
digital interfaces need to be redefined [7]. Indeed, their
main limit lies in the constraints of the two-dimensional
screens, which lack the complexity of the sensory and
physical experience we have while interacting with
the real world [8]. Some examples of these directions
are the works of Ishii and Ullmer (tangible media)
[9] and Van Campenhout et al. (tangible interactions)
[10]. Other approaches to the question can be found in

ubiquitous computing and calm technology [11, 12], and
ambient interfaces [13]. Some authors also explored the
aesthetics of technology [14], the sense of presence and
expression in technological artefacts [15], and new ways
to address form and materiality in interaction design [16].
However, none of them specifically analyses how
products can communicate dynamic messages to
users without relying on traditional interfaces. We
believe product design can fruitfully face this issue,
as technology is always more affecting the design of
artefacts by opening new scenarios for the manipulation
of product physical features.
1.3 Hybrid communication
One of the scenarios technology is opening concerns
the transformation of the product appearance. The
traditional static appearance of objects (intended as
the ensemble of visual, tactile, auditory and olfactory
features), can now become dynamic, for it can be
automatically transformed (in an active and independent
manner) by electronic controllers, devices and smart
materials. The colour, the shape, the tactile properties
(like texture and temperature) of products, the sound
emitted or the smell released by artefacts, can change
over time during the product life, in a reversible manner.
theoretical literature on this issue is very scarce, and

we could not find specific studies aimed at generating
theoretical knowledge around this topic. That results
in leaving a gap in the (product) design literature and
in the designers education. For the reasons explained
so far, we believe this field can be full of possibilities, in
particular from the user experience point of view, and
thus worth exploring.
Fig. 1. Ambient Orb by Ambient devices. The colour of this lamp

changes to inform about the weather conditions. The change
in colour can also show other kinds of digital data, according to
users preferences.
2 Aims
That creates a new means of communication, since,
as we previously argued, a change in the sensory
features of products can tell something about internal
or external situations. Although this change is usually
natural, because determined by transformations in the
physical environment, what is new is that now it can be
provoked and controlled by intelligent components in
the product. Products showing dynamic appearances
(from now on, dynamic products) can thus convey
messages similar to those transmitted by interfaces,
but through a different media: changes in the sensory
features. For instance, weather conditions may be
conveyed by a digital screen, with numbers and verbal
language, but they can also be communicated by a light
changing in colour according to the temperature (Fig. 1).
The possibility to communicate through dynamic
products is the topic this work wants to investigate.
Design researchers have developed many examples of
these products in the form of research prototypes, but
The present work aims to explore if dynamic products

(intended as products that show an active and
reversible change in their sensory features) can be a
valid alternative to interfaces in the communication
of messages from products to users. In particular, the
paper focuses on the analysis of the limits and potentials
connected to the use of dynamic products, both from
the designers and the users points of view.
In order to find out if and why it would be convenient
to adopt dynamic products for the communication of
messages to users, an analysis of the state of the art has
been performed as first step. Then, a test with a group
of designers has been set up, and finally qualitative
interviews with users have been performed. As a result
of these investigations, we extracted a number of
insights related to the advantages and disadvantages that
can be encountered both in the design and in the use of
dynamic products, which constitute the results and the
main findings of this work.
73
3 Methodology
The investigation of the field of dynamic products has
been performed by the authors through three different
studies, each one adopting diverse perspectives and
methods, but all aimed at exploring, from a particular
point of view, different aspects of the analysed field.
The present work introduces and critically discusses the
three research experimentations, in order to create an
overview able to describe the potentials and limits of
dynamic products by different perspectives. The next
paragraphs present such studies, by synthetically explaining
their approaches, their processes, and the results useful
to fulfil the purpose of this work. The research methods
and procedures of the Studies 1 and 2 are presented to
a very concise extent, since exhaustive details can be
found in related works published by the authors.
3.1 Study 1. A framework for dynamic products
The first study followed a theoretical approach and
aimed to shed some light on the issue of dynamic
products, by creating a general descriptive framework
for this category of artefacts. In particular, its goal was
to analyse the role of different sensory modalities in the
communication of messages.
It consisted in a case studies analysis, performed
through the collection, selection and examination of 48
samples of dynamic products (chosen among concepts,
prototypes and commercial products on the basis of
their novelty factor. Indeed, we discarded products that
adopt standardized dynamic signals, such as common
warning lights or sound alarms embedded in appliances).
This process was part of a previous study conducted
by the authors, in which a descriptive framework for
dynamic products has been proposed as result [17].
The framework bases on the identification of three
criteria useful for the description and categorization
of dynamic products: the source of the message, the
aim of the message, and the receiver of the message
(i.e. the stimulated sensory modality).
The message source. From the samples collection and
analysis, it emerged that dynamic products can convey
messages coming from three main sources:
The product itself. Dynamic products can tell some thing about their internal physical state (like temperature),
their functions, the action they are performing, their
energy consumptions, and so on. (Fig. 2)
74
The environment. Dynamic products can give informa

tion about states of the environment (e.g. the tempe
rature of a room, the level of pollution in the air, the
energy consumption), which they are indicators of. (Fig. 3)
A person. Dynamic products can be used in order to
convey messages coming from a person. For instance,
they can be employed to communicate the inner states
of the person (like emotions) but more often they are
employed in distance communication, to replace the
communication based on verbal language. (Fig. 4)
Fig. 2. One Kettle by Vessel Design. The kettle changes its own
surface when the water boils
Fig. 3. Flower lamp by Interactive Institute Swedish ICT.

The lamp opens and closes to communicate the electricity
consumption of the household.
The message aim. The second criterion used to

analyse and describe dynamic products is the message
they convey. Since the messages transmitted by the
samples vary a lot, it was not relevant to categorize
them according to the content of the communication.
The parameter we identified to describe the message
concerns the aim of the communication. Indeed,
dynamic products can convey messages directed to:
Sharing some data or knowledge to the user (for
instance the emotion a person is feeling, Fig. 5, or the
temperature of a room). In this case we defined this
message cognitive, or data-aimed
Fig. 4. HugShirt by CuteCircuite. This shirt makes people
Fig. 6. I-Dration by Cambridge Consultants.
send hugs over distance, by reproducing the sensation
The bottle tells the user when s/he needs to drink
of a hug on the body that is wearing it.
water, by emitting a flashing blue light
Exhorting the user to take a short-term action

(for instance, encouraging the user to drink water,
when s/he is dehydrated, Fig. 6). In this case we de fined the message exhortative, or action-aimed
The message receiver. The third criterion we
considered is the message receiver, that is the users
senses. The four main sensory modalities at stake in
the interaction with objects are sight, touch, sound and
smell. Dynamic products convey messages by means of
transformations in their appearance, which activate one
(or more) of these modalities. For instance, changes in
colour, shape, or light stimulate sight, while changes in
temperature or vibration activates touch. The possible
product transformations, grouped by the modality
they activate, were summarized and represented by
a sensory map (Fig. 7).
The roles of senses. The identification of the three
criteria let us better understand in which situations
dynamic products can be employed to communicate
to users and what kinds of messages they can convey.
Moreover, we deepened the analysis by investigating
which senses dynamic products address and what trans
formations of the product appearance may be designed,
and we summarized them in the sensory map.
Fig. 5. Bubelle by Philips Design.

The dress recognizes the persons emotions and
reproduces them through different colours of the lights
75
Fig. 7. The sensory map collects the possible transformations

of products sensory features. The changing stimuli are grouped
according to the sensory modality they activate.
Starting from the classification made so far, we also

tried to understand if different senses played a particular
role in specific situations. In other words, we checked
if certain senses or stimuli were adopted with more
frequency to convey messages coming from a specific
source, or with a specific aim. Since this was the case,
we tried to analyse the reasons why it happens [18].
The results we obtained show that all the senses can
be addressed by dynamic products, and that each
sense may have specific roles in the communication
of different messages. Indeed, they seem to have
particular features and meanings, which make them
better suit specific situations. As an example, in our
product sample, touch is often adopted to communicate
messages coming from a person, while it is rarely
chosen in the communication of messages coming from
environment or product. Moreover, specific tactile
stimuli are adopted to convey messages coming from
76
person, like temperature and pressure changes. A

different tactile stimulus, vibration, is only adopted to
convey messages coming from the environment, and
always, in the collected samples, with an exhortative
aim (to induce the user to rapidly react). Thus, touch
seems to be suitable to convey the idea of intimacy, in
human-human interaction, but also to alert the user
by vibration. In addition, visual changes came out to be
the most used stimuli in every source category, thus
confirming the dominance of this modality both in the
human perception, and, especially, in the designers
attitudes [19]. To conclude, selecting a sense instead
of another is never a neutral choice. On the contrary,
such a choice may affect the kind of interaction and the
experience generated, as we will better explain in the
discussion paragraph.
Results. This study confirmed that dynamic sensory
features of products can be adopted to communicate
messages to users to a new extent, which constitute
an alternative to traditional interfaces. It demonstrated
that all sensory modalities can be employed in order to
convey bits of information even the most overcome by
designers, such as smell.
Finally, it supported the comprehension of the category

of dynamic products, by creating a descriptive frame
work for it, and by highlighting the different roles of
senses in the communication of messages coming from
specific sources and with specific aims (for more details
about the research process and results, see [17-18]).
3.2 Study 2. Designers perspective
The second study about dynamic products was aimed
both at investigating the difficulties designers can
encounter in the design of such products, and at
testing how the design process of such artefacts can
be supported and improved [20].
It consisted of a design workshop performed with
14 design students in the end of their master education
in Design&Engineering at Politecnico di Milano, in
collaboration with a worldwide company producing
home appliances. The aim of the workshop was to
create more pleasant experiences for users during the
interaction with a domestic appliance, in particular a
dishwasher. The workshop had the duration of one
week, during which designers had to develop one or
more innovative concepts for a dishwasher, aimed at
involving the user on the sensorial level. They could
focus on the redesign of both the dynamic communi
cative elements and the static sensory features, in
order to delight users senses. They were free to design
for any sensory modality, with no constraints but
maintaining the product archetype (the external shape
and the main functions of the dishwasher).
The workshop was useful for us to understand the
attitudes of designers, their propensity for the design
of static rather than dynamic product features, and the
difficulties they encountered in designing the latter.
We provided the designers with the sensory map (see
Fig. 7), to be used as a supporting tool in the generation
of new product dynamic features. We also performed
a one-hour presentation about dynamic products, to
explain how changes in sensory features could be used
to communicate to users.
Process. The 14 designers had been working in pairs
for one week, led by the authors research group and
by the companys design and marketing team. At the
end of the design process, the designers produced 12
concepts, which showed both static and dynamic stimuli.

The authors analysed the concepts in order to evaluate
if designers preferred to focus on static rather than
dynamic stimuli, and to see which senses the concepts
containing dynamic stimuli addressed.
After the workshop, the designers were asked to
complete a questionnaire aimed at investigating the
difficulties they encountered in the design of dynamic
products and at collecting their opinions about the
usefulness of the sensory map and the lecture on
dynamic products.
Workshop results. Many innovative solutions emerged
on the side of both static and dynamic stimuli.
Dynamic stimuli were mostly used to communicate
the progression of the washing cycle: the designers
tried to replace the visual screens with more appealing
elements, by using lights or colours to indicate the cycle
steps. For instance in Light concept, the designers
placed a LED strip in the top front part of the appliance
that changes its colour to indicate the progress of the
washing cycle (Fig. 8).
Fig. 8. Light by M. Broggio and M. Spotti. The colour of the light

changes according to the current phase of the washing cycle.
Even if the concept of dynamic stimuli is not well

established in product design, half of the concepts
(6 out of 12) included dynamic sensory elements.
However, designers preferred to focus on the static
features of products, mostly visual and tactile.
The analysis of the resulting concepts showed some
critical issues related to the design of dynamic sensory
features. Even though students were free to choose
the sense to address in order to create new engaging
interactions, most of them focused on the design of
visual dynamic stimuli. They mainly comprised changes
in light (both in light intensity and in light colour).
77
Beyond sight, tactile stimuli were investigated in two

concepts, with temperature and shape transformations.
Finally, one concept explored smell, by designing a
dishwasher which communicates the end of its activity
by releasing a particular fragrance in the environment.
Questionnaire results. Feedback from the designers
was collected through an online questionnaire, made
of both open and close questions, in order to assess
the usefulness of the theoretical and practical tools.
The first part of the questionnaire investigated the
designers prior knowledge on the topic of static and
dynamic sensory stimuli. Most of them had never
heard about this distinction and had never been taught
to consider it while designing. That reveals a lack in
education about this issue, which also explains why
designers were more inclined to design for the static
aspects of product sensory appearance. Nevertheless,
none of them declared they found it difficult to apply
the notions of dynamic stimuli in their design concepts.
In the second part of the questionnaire, designers
were asked to give their opinion about the provided
knowledge and the sensory map. Answers showed
a general positive evaluation about the sensory map:
it was considered a valuable basis to design dynamic
products with enhanced sensory experiences.
However, some designers argued that the map was
too detailed and therefore confusing because it offered
too many possibilities.
The designers affirmed that they would not have
considered sensorial or emotional aspects in designing
the household appliance if they had not been pushed by
the lecture given by authors, nor would they have taken
into account to design dynamic features to communicate
with senses different from view.
They evaluated the lecture and the sensory map very
useful to guide their design activity towards unconven
tional and more emotional interactions with products.
Authors insights. An important element to stress
is that designers were constrained by their lack of
knowledge about the available technologies that can
support the design of dynamic features. Feasibility
was one of their concerns, and they found it difficult
to design those dynamic features they were not able
78
to fully develop. For this reason, they mainly used

light transformations. In our view, light was employed
in many concepts because of its dynamic behaviour,
which easily convey information and feedback about
the product status and ongoing work. Nevertheless,
they avoided sound (another available technology, easy
to implement), because they considered it too little
innovative to convey messages coming from products,
and unpleasant in many cases. Moreover, they had no
knowledge about sound design.
In synthesis, properties connected to the physical and
tangible aspects of products, that is visual and tactile
features, were the most adopted. It is important to
stress that the participants were students with little
industry experience, and more experienced designers
might have produced more heterogeneous solutions;
however, it was the scope of the study to analyse the
designers skills and knowledge provided by design
education programmes.
3.3 Study 3. Users perspective
The last study was aimed at exploring the users
reaction to dynamic products. Indeed, in analysing the
potentials and limits related to the adoption of this new
kind of communication, it is necessary to consider the
effects it has on the final users.
We decided to investigate the users reactions to
dynamic products by testing the product samples we
selected for Study 1. Since most of the samples were
not commercial products, but concepts or prototypes,
Fig. 9. Example of a card used in the interview. It shows a picture
of the product, together with its description. The concept
shown in the card is Scent of time by Hyun Choi.
it was not practicable to perform a test on the real

objects. We decided to set up semi-structured qualitative
interviews supported by pictures and brief explanations
for every product [21]. The bits of information related to
each product were summarized into a card (Fig. 9).
It is necessary to underline that a test performed just
through visual images instead of real products, especially
while investigating the sensory experience, presents
many limits. Nevertheless, this is just a first step in the
exploration of users reactions to dynamic products,
whose aim is not to evaluate the single products, but
to extract suggestions, insights and general impressions
about this kind of artefacts.
The aim of the interviews was to assess both the
perceived communication effectiveness and the quality
of the experiences elicited by dynamic products.
Process. As many products adopted similar stimuli to
convey the same kind of message, we decided to reduce
the sample of products to test. Indeed, the reduction of
the sample size did not compromise the reliability of the
study, since the interviews did not have a quantitative,
but a qualitative approach. In order to have a good
overview of the users reactions to these products,
we considered necessary to select the products to
test maintaining the highest variety among the samples,
on the basis of the three criteria identified in Study 1.
For this reason, we selected for each source category
(product, environment and person) both products
conveying exhortative and cognitive messages. Within
each subcategory (e.g. products conveying exhortative
messages coming from environment) we picked only one
product for each different sensory stimulus adopted.
At the end of the process, 15 dynamic products were
selected for the interviews (Fig. 2 to 6 and Fig. 10 were
all part of the sample).
Five users were involved in the interviews, both males
and females from an age of 19 to 60. Each user was
asked to analyse three products.
The interview structure consisted of two sections.
The first one investigated the communicative aspects
of dynamic products (efficiency of the communication,
comprehensibility and intuitiveness of the message,
consistency of the activated sensory modality to the
context of use, the source and the aim of the message);

the second one focused on the emotional impact of the
products (level of involvement, interest towards the
product, elicited emotions).
Each single interview comprised three sessions on three
different products. Every session lasted about 15-20
minutes. During each session, the user was shown the
card of the product, and a brief explanation of how
the product works was accomplished by the authors.
Subsequently, the interview was performed, following
the track composed of 10 questions, 5 related to the
first section (focused on communication), 5 to the second
one (focused on experience). The questions led to an
open discussion between the interviewer and the user.
The entire interviews were recorded and subsequently
transcribed by the investigators. Memos were written
during the interviews, to note additional details.
The recordings and memo were analysed through the
methods used in Grounded Theory, with the aim to
highlight significant findings. We followed an approach
based on Strauss and Corbins model, already used by
previous studies in the field of user experience [22].
The findings were extracted through a coding process
aimed at identifying recurrent concepts which were
then clustered into categories based upon similarities.
This helped us highlight and connect recurring themes.
Indeed, our purpose was to identify insights related to
the user experience with dynamic products.
Results. This section presents the results of the inter
views, according to the purpose of the paper, that is
to stress which are, in the final users opinions, the
difficulties and the advantages (pros and cons) founded
in the use of dynamic products. For this reason, not
all the findings emerged by the interviews analysis
are reported, but only the elements that express the
positive and negative aspects of dynamic products, as
far as communication effectiveness and user experience
are concerned.
Communication effectiveness. Users reported some
difficulties connected to the communication performed
by dynamic products. The first one concerns the intuiti
veness of the message: in many dynamic products, the link
between the stimulus and the content of the message is
not direct, and the message has to be explained to the
user in order to be understood. For instance, the user
79
has to know in advance that if the pattern on the tiles

of the shower disappears, too much hot water is being
used (Fig. 10). On the contrary, if a writing appeared
on the wall, telling: you are using too much hot water,
the content of the message would be self-explaining.
When the message was not explained clearly, users
interpreted the content of the communication very
subjectively, and gave it different meanings. However,
in some few cases, the message was self-explaining
and more objective even when transmitted only by
a sensation. For instance, the users perceived the
message conveyed by the Hug Shirt, which reproduces
the sensation of a hug to whom is wearing it (see Fig. 4),
as very intuitive. Moreover, once the message had been
explained, it resulted clear and effective in the most
of cases, especially when related to the intensity of
something (e.g. the electricity consumption, the level
of pollution in the air, the temper ature), where the
gradient of the information can be represented by
a gradual change in the stimulus (e.g. the intensity of
a colour, the density of a texture), not achievable with
the same effectiveness by verbal language.
The second difficulty regards the simplification of the
message. Indeed, sensations are less detailed and precise
than verbal language, and sometimes this vagueness
represents a problem for the user. For instance, this
element emerged in reaction to a lamp that communicates
the external temperature through a change in the colour
of the light (see Fig. 1). In fact, the user considered
valuable the immediacy of the sensation, but he would
have also appreciated the presence of numbers telling
the exact temperature, to be consulted when needed, as
an added function to the product. However, the lack of
details, which is intrinsic to this kind of communication
by sensations, was also evaluated as positive in some
cases, because it requires less effort to get the message.
An example is I-dration (see Fig. 6), a bottle that tells the
user when s/he needs to drink water while doing fitness,
by emitting a flashing blue light. The user reported it is very
useful to get the message through a flashing light, since the
interpretation of the sensation is effortful, differently from
reading something while you are doing physical activity.
The users also reported some advantages related to
dynamic products. One is immediacy, which is still
connected to simplification. In fact, according to the
users, sensations stimulate the senses in a more direct
way and are more rapid to interpret, compared to
80
verbal language of traditional interfaces. In many cases,

during the interviews, they defined the messages
conveyed by sensations as direct and immediate.
Sensations are also more evident, since they can reach
the user from far and one does not have to look for the
information actively. For instance, sound and smell reach
the users senses also when the stimulus is distant, and
they are difficult to avoid, thus being very effective in
particular contexts (e.g. when the information is urgent,
or when one is busy and not able to pay attention to
the message spontaneously). Users have evaluated this
aspect both as positive and negative, since the sensation
can also distract or disturb the person, if not designed
in the right way.
The context of use came out to be very important in the
communication of messages through sensations. In fact,
in some cases, the users evaluated the choice of one
sensory modality as wrong, with respect to the
context or scenario of use of the product. For instance,
if the product has to remind the user to take a pill, a small
LED light is not considered suitable to communicate the
message, since the user will easily miss the stimulus if
s/he does not keep on looking at the product.
The last consideration concerns the exhortation power
of some sensations. It came out that specific stimuli have
a great alarming power. In particular, users interpreted
vibration, sound and flashing light as conveying urgent
information, which require an immediate action. They
felt the products were asking them to do something,
and they were given a strong motivation towards action.
In other cases, when the message wanted to be exhortative in the intention of the designer, but it was conveyed
by a change in the colour of the product, the information
was interpreted as the opposite of urgent and alarming,
in fact it was evaluated as suggesting or inviting.
Experience pleasantness. With respect to the experience,
the users were asked to describe their impressions
towards the dynamic products in terms of elicited
emotions, engagement, pleasantness of the communi
cation and aesthetics. We will focus here on the elicited
emotions, making some brief considerations about the
engagement and the communication pleasantness.
To assess the users emotional responses, the
participants were shown a list of 14 words expressing
engagement: the pleasantness of the communication

was rated very high, mostly with 4 or 5 point (high and
very high). Two products were evaluated as medium,
and one as low.
Fig. 10. Disappearing Pattern Tiles by Interactive Institute.

The pattern on the tiles disappears when the user consumes
too much hot water for a shower.
emotions, among which they had to select the most

appropriate ones to describe their emotional reactions
to the products. The list, based on Desmets 14
emotions elicited by products [23], reported 7 positive
and 7 negative emotions. Results show that users cited
31 positive emotions to describe their experience, while
they reported negative emotions only 8 times (plus one
case in which the user felt indifferent towards one product).
The most cited positive emotions were fascination
(11 times) and pleasant surprise (8 times), followed
by admiration, desire, amusement, and inspiration.
Negative emotions were almost never related to the
stimulus used to communicate, but resulted from the
evaluation of the aesthetics of the product, or the
content of the message conveyed (e.g. a user reported
dissatisfaction because the product was attractive but
it communicated the level of pollution in the air, and
she was not interested in receiving that message). Only
in one case, boredom and delusion were expressed in
reaction to the use of (a pleasant tweeting) sound by
a kettle, defined not new.
The users were also asked to rate their perceived level
of attraction and engagement towards the products
(from 1 to 5, where 1 was very low and 5 was very
high). The answers showed high ratings (equal or above
3) in all the cases but two. These negative responses
were, as for emotions, connected to the little interest
towards the message conveyed, and to the lack of
innovation in the transmission of the message (the
use of sound in the kitchen appliance).
Finally, the pleasantness of the stimulus employed
to communicate was evaluated, again on a 5-point
scale. This time, answers were more positive than for
An interesting element that was reported by the users is

the concept of discretion. Dynamic products commu
nicate by a non-explicit language, since they do not
employ verbal communication. Thus, they are perceived
as more discreet in the communication of messages.
This is particularly valid for messages coming from
persons (human-human communication), where the use
of sensations is much appreciated, it being a less direct
language, more suitable for subjective interpretations.
Dynamic products vs interfaces. The users were also
asked if they would prefer to receive the same message
through a verbal or numerical language. In none of the
cases, the users expressed the intention to replace
the communication dynamic products performed with
a traditional interface. In fact, they stated that the
experience with dynamic products was more engaging
and pleasant and the message more immediate, evident
and discrete. In more than half of the cases, they argued
that they would appreciate a more complete or detailed
message, by incorporating or associating an interface to
the dynamic product. Nevertheless, they would never
replace the direct sensation with the interface. This is
a very interesting result, for it highlights that users are
ready to lose some details, in favour of more immediate
sensations, less precise but more surprising; in other
words, in favour of the experiences the products create.
4 Discussion
The results of the three studies have been critically
analysed, in order to highlight and summarize advantages
and limits of employing dynamic sensory features in the
communication of messages from products to users.
4.1 Advantages
The advantages of dynamic products are mostly
connected to the efficiency of the communication and
to the user experience. The main benefits from the
designers point of view reside in the fact that this new
product category is still unexplored and thus represents
and interesting area for the development of innovative
projects. Moreover, the product dynamic features
81
represent a new material for design, which enlarge

the possibilities and horizons of the designers activity.
Benefits and potentials connected to the use of dynamic
products are summarized below.
Immediacy (of the message). The message conveyed
by sensations is more simple and immediate. It means
that the communication can be more direct and can
better suit some situations, for instance, when the user
is under physical or psychological stress or s/he does
not want to pay too much attention in decoding the
message. Moreover, sometimes users are overloaded
by information, even when it is not necessary: some
messages do not convey complex information, and can
be more easily and rapidly understood by sensations,
instead of difficult cognitive elaborations.
Evidence (of the stimulus). The user does not have to
look for the information, but in many cases the signal is
self-evident and reaches the user in a more straight way
(with sound and smell, but also with touch if the product
is wearable) and with less effort. This is valid also for
what are called ambient interfaces: they are present in
the environment as constant reminders, and are easily
perceivable by the user. Both evidence and immediacy
may be connected to the concept of calm technology
[12], which Weiser and Brown describe as the ability of
technological products to address the periphery of our
sensory system, thus avoiding information overload.
Exhortative power. Some sensations came out to
be very effective in alerting and alarming the users.
In particular vibration, sound and flashing light were
assessed by users as exhortative stimuli. The ability
to gain the users attention and to activate them can
be exploited in order to convey urgent messages, or
to make the communication more effective when it
requires the user to act in a short time.
Discretion. Communicating by dynamic products
requires a less explicit language than the alphanumerical
ones. This element has been judged as extremely
positive in specific situations, when the user wanted to
be informed about something, but in a subtle and hidden
way. Moreover, relying on more discreet communicative
modalities was highly appreciated in the human-human
communication. Indeed, changes in the light, colour,
smell or temperature of an object can communicate,
82
in a more gentle and implicit way, emotional states,

moods, or just that your partner is thinking of you.
Fascination, surprise, engagement. Thanks to their
unusual way of communicating, dynamic products elicit
positive emotional reactions in the user. The emotional
response can be described in terms of fascination
and pleasant surprise. Engagement is also part of the
experience generated by dynamic products, thus
confirming the idea that communicating by sensations
can positively affect the users experience.
Variety. The broad range of dynamic stimuli gives the
designers the opportunity to choose the best sensory
channel in every situation. Since senses have their
specificities, each of them can better suit particular
contexts. For instance, touch is connected to the idea
of intimacy [24], sound has an alarming power and
smell is strongly linked to memories and emotions
[25]. Moreover, within each sensory modality, different
stimuli have different meanings (e.g. vibration is used
to alert, while temperature and pressure to convey the
idea of affection). Choosing the more suitable stimulus
can improve the communication, by making it both
more effective and more pleasant.
4.2 Difficulties
The difficulties and constraints connected to dynamic
products are reported below. We specified in brackets
if the difficulty concerns the user rather than the designer.
Subjectivity (user). Dynamic products rely on sensations
instead of verbal language to communicate to users. This
makes the message apt to the users subjective interpretation, since it is not explicit. The message can thus
result ambiguous and the content of the information can
be easily misunderstood, if not explained.
Simplification (user). The messages conveyed by
sensations result simplified, since complex data are
difficult to transmit by basic sensory transformations,
in an effective way. If this can be judged an advantage
in some cases, it also represents a limit, since accuracy
is reduced and the user can just get approximate data.
Complex information came out to be better conveyed
by the verbal or numeric language.
Sight dominance (designer). Study 2 demonstrated
that designers are used to handle the visual and tactile
qualities of products, which are their tangible features.

Controlling more intangible elements, such as sound
and smell properties, seems to be a more difficult task:
in the experiment, none of the concepts focused on
sound, while smell was investigated in only one project.
Moreover, the fact that vision was the most investigated
sense, confirms again the dominance of this modality in
the approach of designers [19].
Educational gaps (designer). The questionnaire in
Study 2 provided some evidence that designers are
not taught how to consider product dynamic features
as a media to communicate to users. Knowledge on
this topic is very limited, and the natural inclination
of designers is to design for the static sensory features
of products. Even when explicitly required to design
for dynamic features, designers found it hard to explore
different sensory modalities (they focused on sight,
and mostly with light changes).
Technical feasibility (designer). Another difficult that
designers can encounter in the development of dynamic
products is their technical feasibility. Indeed, as Study 1
demonstrates, the dynamic products developed so far
in the design practice are mostly in form of concepts
and prototypes, rather than commercial products.
This for two reasons: on the one side, technologies
supporting the functioning of dynamic products are still
little developed or far from being easily implementable
in commercial products (also because of their cost).
On the other side, designers often do not have enough
knowledge about the technical possibilities that are
being developed to support dynamic features. The
most available technologies on the market for dynamic
products implementation, which can be found in
commercial artefacts, are thermo-chromic inks, lights
and auditory signals.
Context dependency (designer). The design of
dynamic products must take into account the context
and the scenario of use. It means that it is not sufficient
to identify the most suitable sensory modality to
activate, and the most appropriate stimulus, consistent
to the message to convey. The scenario in which the
interaction will take place plays a fundamental role
in the design of the sensory communication, since
senses have objective constraints, which must be
carefully considered. As an example, sight should not
be addressed if the object is designed to be hold in a

pocket. It may seem a banal consideration, but not few
products, among the collected samples, showed a lack
of attention on this aspect.
4.3 Limits of the study
The findings we propose in the discussion cannot
be considered exhaustive, as they present limitations
connected to the research process methodology.
As far as the designers perspective is concerned, it is
important to remind that the results are based on an
experiment with graduated design students, with little
industrial experience. Their educational background,
which results from the industrial design programme in
one specific university, may be different from the one of
designers coming from different contexts. However, our
interest was to investigate how product designers having
no specific education in user experience or interaction
design would address the design of dynamic products.
With respect to the users perspective, methodological
limits lie mainly in the fact that not real, but hypothetical
designs were tested. Users were asked to discuss
dynamic products by observing their images and
descriptions, instead of by interacting with them. This
represents the main limit of the study, also because the
quality of the sensory experience cannot be investigated
through such a process. Although this approach is not
sufficient to explore all the aspects of the experience
elicited by dynamic products, still it gave interesting
insights about the level of acceptance of these products,
the perceived effectiveness and emotional values.
5 Conclusions
This work presents and analyses three studies aimed
at investigating, from different points of view, the
possibility to communicate by dynamic products.
The studies adopted different perspectives, in order
to identify advantages and difficulties in both the
design and the perception of this kind of products.
Results emerged by this first analysis show that the
advantages are mostly connected to the user experience,
which can be enhanced and enriched by the communi
cation performed by dynamic sensory features of
products. On the contrary, difficulties can be found
mainly on the designers side: his education, knowledge
and inclinations do not support an exhaustive exploration
of different sensory channels. Moreover, technical
83
feasibility and the scarce availability of solutions on the

market, seem to reinforce the constraints of designers.
The picture this investigation returns aims to give an
initial reading of the current situation in the develop
ment, diffusion and users acceptance level of dynamic
products. Again, the intention of this analysis is not to
be exhaustive, but to constitute a first attempt to shed
some light on this emerging area of design. Further
studies are necessary to better comprehend the limits,
potentials and implications connected to dynamic
products. However, the goal of this work is also to
encourage the design community to open a debate on
such themes, as the practice of design is developing
solutions in this direction, while theory still lacks the
corpus of knowledge able to support the designers work.
11. Weiser, M. (1993). Hot topics-ubiquitous computing.

Computer, 26(10), 71-72.
12. Weiser, M., & Brown, J. S. (1997). The coming age of calm tech
nology. In Beyond calculation (pp. 75-85). Springer New York.
13. Gross, T. (2003). Ambient Interfaces: Design Challenges and
Recommendations. In Proceedings of the 10th International
Conference on Human-Computer Interaction HCII, Crete
(pp. 68-72). Lawrence Erlbaum, Hillsdale, NJ.
14. Gaver, B., & Martin, H. (2000, April). Alternatives: exploring
information appliances through conceptual design proposals.
In Proceedings of the SIGCHI conference on Human factors
in computing systems (pp. 209-216). ACM.
15. Hallns, L., & Redstrm, J. (2002). From use to presence: on
the expressions and aesthetics of everyday computational
things. ACM Transactions on Computer-Human Interaction
(TOCHI), 9(2), 106-124.
16. Jung, H., & Stolterman, E. (2011). Form and materiality
References
in interaction design: a new approach to HCI. In CHI11
1. Krippendorff, K. (1989). On the essential contexts of artifacts
Extended Abstracts on Human Factors in Computing
or on the proposition that" design is making sense

(of things)". Design Issues, 5(2), 9-39.
2. Krippendorff, K. (2006). The semantic turn: A new
foundation for design. Boca Raton: Taylor & Francis.
3. Krippendorff, K., & Butter, R. (1984). Product Semantics:
Exploring the Symbolic Qualities of Form. University of
Pennsylvania press.
4. Wiberg, M., & Robles, E. (2010). Computational
compositions: Aesthetics, materials, and interaction design.
International Journal of Design, 4(2), 65-76.
5. Crilly, N., Moultrie, J., & Clarkson, P. J. (2004). Seeing things:
17. Colombo, S., Bergamaschi, S., Rampino, L. (2013). What

Are You Telling Me? How Objects Communicate Through
Dynamic Features. In Proceedings of xCoAx Conference,
Computation, Communication, Aesthetics and X, Bergamo.
18. Colombo, S., Rampino, L., (2013) Information embodiment:
how products communicate through dynamic sensory
features. In Proceedings of DPPI Conference. ACM. (in press)
19. Hekkert, P. (2006). Design aesthetics: principles of pleasure
in design. Psychology science, 48(2), 157.
20. Colombo, S., Gorno, R., Bergamaschi, S. (2013). Enhancing
consumer response to the visual domain in product design.
product sensory experience: cultural tools for design
Design studies, 25(6), 547-577.
education. In Proceedings of 15th International Conference
6. Ehrnberger, K., Rsnen, M., & Ilstedt, S. (2012).Visualising

gender norms in design: Meet the mega hurricane mixer and
the drill dolphia. International Journal of Design, 6(3)
7. Interaction Design Foundation, The Encyclopedia of HumanComputer Interaction, http://www.interaction-design.org/
encyclopedia/industrial_design.html
8. Holman, D., & Vertegaal, R. (2008). Organic user
interfaces: designing computers in any way, shape, or form.
Communications of the ACM, 51(6), 48-55.
9. Ishii, H., & Ullmer, B. (1997, March). Tangible bits: towards
seamless interfaces between people, bits and atoms. In
Proceedings of the ACM SIGCHI Conference on Human
factors in computing systems (pp. 234-241). ACM.
10. Van Campenhout, L. D. E., Frens, J. W., Overbeeke, C. J.,
84
Systems (pp. 399-408). ACM.
on Engineering and Product Design Education, Dublin. In press.

21. Drever, Eric. Using Semi-Structured Interviews in Small-Scale
Research. A Teachers Guide. Edinburgh: Scottish Council for
Research in Education, 2003.
22. Khambete, P., & Athavankar, U. (2010). Grounded Theory:
An Effective Method for User Experience Design Research.
Design Thoughts, 11-24.
23. Desmet, P. (2005). Measuring emotion: Development and
application of an instrument to measure emotional responses
to products. In Funology (pp. 111-123). Springer Netherlands.
24. Gallace, A., & Spence, C. (2010). The science of interpersonal
touch: an overview. Neuro-science & Biobehavioral Reviews,
34(2), 246-259.
25. Herz, R. S. (1998). Are Odors the Best Cues to Memory?
Standaert, A., & Peremans, H. (2013). Physical interaction in a
A Cross-Modal Comparison of Associative Memory Stimuli.
dematerialized world. International Journal of Design, 7(1), 1-18.
Annals of the New York Academy of Sciences, 855(1)
Heather Rene Barker

California State University, Long Beach, USA
heather.barker@csulb.edu
Hive-Design for Big Data
Abstract
An epistomologic framework, this paper draws from
social sciences, media theory, philosophy, business,
and the practice of design to develop a comprehensive
knowledge construct deciphering the role of the
designer in an age of Big Data. From this framework,
areas of design research and practice are defined.
Practice methods for the designer as an executive
member of the business strategy team are framed
in terms of efficiency and value. The possibilities for
implementing crowd-sourced subjective and multi/
media data, interpreted by the vast numbers of social
media users at a scale even greater than big data, are
explored - hivedesign for big data. It is argued that
designers are best qualified to perceive, interpret and
contextualize phenomena such as mass customization,
the quantified-self and wearable interfaces in an effort
to achieve an optimized and rich whole data synthesis.
The paper asserts:
Big Data is quantifiable but ultimately without much
meaning.
Soft Data, four times as big as Big Data, requires
human minds to interpret.
Data without Design is insufficient to make rich
decisions
Whole Data Research integrates structured and non/
structured data and is the ideal information construct
to generate the most valuable knowledge
Social Media Ontology describes identity and
existence as mediated through the agency of social

media interfaces, presence and status
Interfacing and the Data-fied Self is the phenomena
of self-monitoring and engaging in biological/technical
interfacing through wearable technology
Mass Customization and Device Topology describes
creating value and longevity for the platform and
tools by empowering individuals to personalize the
performance of his/her device.
Keywords
Design optimization, design strategy, device topology,
hivedesign, whole data
Introduction: Design is Optimization

Those fortunate enough to collaborate across
disciplines may have heard well-meaning team members
ask, What exactly is design? A colleague from
Engineering, somewhat self-consciously, admitted
he thought it was the designers job to tell him if our
(his) final product should be square... or round.
He was genuinely struggling to understand what design
could offer early-on in project development. What
exactly can a designer contribute? I told him, Design
is optimization; designers interpret vast amounts of
data on materials, production methods, technology,
economic conditions, cultural, ethical, historical and
environmental issues, aesthetics, performative systems,
85
and ergonomics. Designers contextualize that infor

mation and assimilate it to create an optimized result.
After being initially confounded, he then knew what to
expect and became an enthusiastic and demanding partner.
With the explosion of data now available due to the
social media revolution, intuitively, many people are
asking a similar question. Can a designer contribute
to the Big Data mlange other than just designing
content? As more and more data becomes available,
it is becoming difficult to communicate larger amounts
of more nuanced data in a way that can be effectively
absorbed by the receiver. Words and text alone are
no longer up to the task. People are supplementing
abbreviated word exchanges with images, video, music,
emoticons, and memes to quickly convey a richer
meaning. Therefore, in this era of Social Media weve
moved from word to image in order to facilitate greater
efficiencies in swarm communication.
Even hard data has become affected by expectations
of more affective information. It is no longer sufficient
to present hard data through traditional means; those
who aspire to communicate their findings are expected
to do so through infographics. As design researchers
work to quantify and qualify the multifarious meanings
in the data swarm; many models of thought developed
by linguists and behaviorists prove to be useful
structuring tools (Sausaure, Baskin, Meisel, Saussy
2011). Interpersonal and public communication has
come from being primarily gestural and audible to
super-gestural and super-audible. In our interconnected
world, imparting meaning demands creating engaging
content and further stimulating the higher perceptual
senses. Like sensing an approaching storm, we are aware
of the energy of social media, just beneath reliable
perception - much in the way some animals sense
acoustic pressures beneath the frequency of audible
sound. Analogous to infrasound, this energy can cover
long distances and is not susceptible to interference.
We can sense energy without knowing what it is.
Perhaps this is an indication that our bodies are adapting
to the overwhelming flow of data.
Why HiveDesign?
Social Mediators = Mass Customizers
Enormous energies have been invested in an effort
to quantify the Big Data produced by our current
86
state of technological information generation. There

are ongoing heroic efforts to find ways to fabricate
intelligence from data. Those efforts have been
concentrating on easily quantified hard (or structured)
data. The real challenge has been to garner intelligence
from the soft (or unstructured) data. That data
includes not only the subjective data of text messages,
images, reviews, rankings, videos, music, and memes
but also the ways in which this data is shared (IBM
2013). There is an epistemic culture to our social
media information exchanges. Per Knorr-Cetina, an
"amalgam of arrangements and mechanisms - bonded
through affinity, necessity and historical coincidence which in a given field, make up how we know what we
know (Knorr-Cetina 1999). Evaluating the information
artifacts containing situational, cultural, symbolic and
aesthetic content is key to discovering meaning in our
mediated information exchanges. Designers have a
unique skill set enabling them to interpret and translate
data relative to the social media epistemic culture.
Envisioning social media networks as hives provides
the analogous framework to address the component
constituents, ubiquitous communication, fluidity and
unity of purpose (Burns, Stalker 1961). Every individual
engaged in social media has made choices as to the
ways in which they want to represent themselves.
Those choices are design choices. Every person with a
profile page is aware of the personal affect they wish to
convey. The community encourages individualization and
sharing. The agency provided through a personalized
interface transforms social mediators into mass
customizers. A shared proprioceptive sense of the
social media system provides an awareness of the spatial
relationship amongst the various components and the
sense of balance achieved when all parts are functioning
correctly. The mass customizers serve to expand the
power of the social media hive by collecting, creating
and distributing content that is not only personal but
also political, commercial and entertaining. We have
reached a point where all actions in society must deal
with the social media hive. Everyone is a hivedesigner
with a keen awareness of the cultural and contextual
implications their actions may have.
The HiveDesign Manifesto:
We are searching for a new intelligence from the Big
Data of the internet of everything.1 We are witness to
Refers to the Cisco

campaign by Goodby,
Silverstein & Partners
the revolutionary impact of the Big Data/New Media

Synthesis. We hope to discern a data of desire from
the posts, tweets, pins, reviews, etc. We customize,
create and share our data experience. We define our
identity through our choice of device and tools (apps).
We hive as a result of our desire for interaction and
knowledge. We quantify ourselves through wearables
and visualize that data to improve self-diagnostics,
challenge and motivate ourselves. We interface every
where. We create and recreate our identity. Our
agent is social media and that agency provides a space
of reflection and exchange. We engage in processes
of assimilation, interpretation, and creation in all we
share. Our presence can be felt everywhere. We
initiate experiences, invite input, create desire, generate
meaning and knowledge to affect our shared spaces of
commerce, communication, entertainment and politics.
We want you to like us, so we try hard to be likeable.
We want you to look at us, so we try hard to be goodlooking. HiveDesign is emergent, collective and agile.
HiveDesign is always on, does not sleep, possesses
the adaptability and intelligence of a swarm and is in
constant motion. HiveDesign collectively possesses
a higher consciousness. HiveDesigners are distinct,
egocentric, sentient individuals working collectively
in hypertime and heterospace by virtue of our shared
devices, spaces & tools, if not a shared agenda. The
behavior and particularities of the hive cannot be
predicted by the behavior and particularities of the
individual hivers. It is impossible to observe the hive
without engaging it. The hive is inevitable and you are
likely part of it. Welcome.
20% is not Big Data
Big Data is about going wide and thinking fast. It is
holds quantity over quality and has achieved superior
results in doing so. It is logical: processors analyzing
data; similar systems correlating and resolving metadata just as fast as the advances in technology will let
them. This works well for data that can be tagged or
put into a database or other structure. Information that
can be described numerically or has limited variables of
associations (like language) are the Big Data hotspots
at the moment. Big Data through Analytics becomes
Meta Data: the data about data, highly abstract and
impersonal and beyond the scale of the individual.
...90% of the data in the world today has been created in

the last two years alone. This data comes from everywhere:
sensors used to gather climate information, posts to social
media sites, digital pictures and videos, purchase transaction
records, and cell phone GPS signals to name a few. This data
is big data (IBM 2013).
The real windfall comes as the tools of analytics to
interpret that big data fall into place. The predictive
power, potential for more efficient allocation of
resources and access to previously unrevealed markets
is stunning. Google is the most visible player in the
analytics game with advanced translators and attempts
at predicting epidemics before official institutions
are able to do the same (Google 2013). They quickly
generate the structured data for analysis. A Health
Department must consolidate reports after illness has
been tested for, confirmed and reported. This type of
data processing is done by humans. It is good, quality
data and it is inherently deep and slow. Consequence:
fewer lives saved due to identifying the outbreak so late.
This Big Data Analytics revolution is a game-changer on
a societal scale. A rupture appears in the revolutionary
bubble once one realizes that the Big Data so often
lauded is actually only the structured 20% of data and
major decisions are being made on such incomplete data
analysis. What about the other 80% of data?
So while structured data is the 200lb. gorilla in
the room, it is dwarfed by unstructured data the
unseemly, difficult 800lb. gorilla that ought to be tended
to. That unstructured data includes messy, text-heavy
information and emotionally driven information in the
form of images, video, reviews, etc. In our social media
era, that information is disseminated and translates
as Posts and Re-posts, Pins, Shares, Likes and Chats.
It seems the ultimate prize remains for those able
to decipher the unstructured data that 80% of
everything: the most personal, subjective and powerful
information upon which we act. The value of emotional
data in marketing is widely recognized. So, What is
emotion? It is the modification of neural activity that
animates and focuses mental activity. (Wilson 1998)
The strategic function of emotionally enhanced data
is beginning to be adopted by those most driven by
innovation (Muller 2003). Animating and focusing
mental activity in strategic decisions is the difference
between those who innovate and those who do not.
87
Emotional content and meaning must be deciphered;

intent must be revealed. That will require going deep
and thinking slow (Kahneman 2011).
In June of 2012, there was a lot of buzz around research
results coming out of Googles X Labs. In an effort
to simulate human interpretive learning, they exposed
a network of processors to millions of images and
then parsed the data to reveal that the computer
could recognize human faces, forms and... cats. As
extraordinary as that is... it doesnt help us much. We
can recognize cats all by ourselves. What we cannot
do is, based on one image (say the Barcelona Chair in
the Barcelona Pavilion) find all other images on the net
that are related to it, not by name or tag but by visual
information. Even new things unrelated to content could
be revealed: proportion, image density, composition,
contrast, etc. This tool could further help in the
associative/interpretive exercise and push creative
thought further by augmenting the brains ability to
access the shared database of the global internet.
Heres where Google could help, although someone
else may be better poised to do so (perhaps Adobe will
be the leader here). There are relationships amongst
images that even most designers cannot recognize.
Those quantitative relationships could reveal additional
correlations. Advances in image parsing could allow us
to browse visual data based on metrics including, for
example, shape affinity, image density, edge definition,
text regions, hue, saturation, exposure, structure,
resolution and perhaps hundreds of other qualities.
A long overdue tool, the image search engine will let
us browse images, not descriptions of images but an
image/metrics based browser. The results would deliver
discriminative probabilities. Every image would have an
algorithmic fingerprint that would then be compared to
other images algorithmic fingerprints to discriminate
amongst the datasets and return likeness values and
probabilities. The recent technological possibility to
share images and movies has changed everything. As
we move from word to image, this tool is desperately
needed. A parametric, algorithmic description of the
selected images is put through all images that share the
same or similar data set for results. Quantifiable images
will return expected and unexpected results and
reveal similarities that we understand perceptually but
cannot quantify. Ideally that image data would take on
88
the additional search data every time it was selected as

correct by an interpretive human, further adding to
and refining the descriptive image data. This process
continues, eliciting a seamless human interpretation/
data processing synthesis could be initiated indefinitely.
Each iteration reveals new meaning and redefines the
meaning of the previous subset. We are engaging in
what Pierce called infinite semiosis a digital mimicry
of infinite semiosis (Eco 1996). The revelations await.
It is expected of Big Datas meta-info that there must
be a mega meaning, deep meaning. That is exactly what
Google is trying to achieve with its cat recognition
software (Dean 2012). As humans, we are excellent
at going deep and thinking slowly. That may sound
like quite a handicap in the face of extraordinary
technology; but even a human baby can identify a cat in
less than three days. In fact, a baby can identify millions
of cats by having only ever seen one as opposed to
having to see millions of cats in order to identify one.
This fact is telling; it appears the best analytic tools for
interpreting the unstructured 80% are human minds.
Crucial to creating a meaningful Big Data field from
which to make informed decisions will be the synthesis
of machine and human analytics. Per Cukier and MayerSchoenberger, Behavioral economics has shown that
humans are conditioned to see causes even where none
exist. So we need to be particularly on guard to prevent
our cognitive biases from deluding us; sometimes,
we just have to let the data speak. (Cukier, MayerSchoenberger 2013) It seems humans are notorious for
seeing meaning where there is none and machines are
notorious for seeing no meaning where there is plenty.
While structured data may correlate with change, nonstructured (meaningful) data creates change. Getting at
that meaning is everything.
Revealing the Meaning of 80% of Everything
Business typically looks for relative meaning: meaning
as it relates to buying patterns, entertainment choices,
politics or economics: ...one of the most important
factors in social activity is meaning and change in
meaning (Osgood 1975) There are many who focus
on the business of meaning. Criminologists, Analysts of
all types and those engaged in taking polls and marketing
all share a common desire to predict behaviors. There
are new measurements of meaning emerging. The latest
tools though, are still far from being able to decipher
and interpret meaning. So the question remains, how

can we get to the meaning of 80% of everything? If the
meaning of a sign has been defined as a representational
mediation process, a complex reaction divisible into
some unknown but infinite number of components.
(Charles Osgood, The measurement of meaning p. 31),
then designers are uniquely suited to interpret the data
of meaning.
the structure/system. It bounces around as an anomaly

without the associative meanings of other ideas to
ground it. But systems shift & opportunities reveal
themselves through inevitable changes in society and
time. Designers can intervene in the system and design
the groundwork of associative meaning structures,
anticipating the introduction of the represented idea
and improving the likelihood of timely acceptance.
Let us define Design simply as a process that gives form

or representation to ideas. That definition presupposes
three things: ideas process representation. Design
though, is much more complex than that definition
suggests due to the richness and the implementation
of the process. Ideas are informed by data. That data
is perceived, translated and interpreted for meaning
to become an idea. Ideas must be abstracted and
contextualized to become comprehensible. Once
an idea is comprehended a constructive skill set is
necessary to create a representation. Once an idea is
represented, meanings can be gleaned from it. From
those meanings, knowledge or understanding can be
achieved. From that knowledge, we are able to make
choices and decisions:
data perception translation
interpretation/meaning ideas abstraction
contextualization comprehension/meaning
applied skills representation meaning
knowledge/understanding choice.
Humans process perceptual sensory input through

abstraction, reduction, association and synthesis. We
often call this thinking. Thinking assumes varied and
highly developed sensors, massive processing power,
a phenomenal database of stored reference data and a
capacity for articulating the result of the process. The
right brain and subjective ordering processes happen so
quickly and on so many levels simultaneously that it is
impossible to quantify the process. Even more difficult
if the goal is to synthesize what millions are thinking.
Yet, as in affinity diagramming (Ulrich 2003) if we can
extract the most prominent ideas from the group, take
them out of the mind and engage in a simulated version
of thinking by delineating the most prominent ideas we
have a brainstorming session. Representing ideas visually
is crucial to achieving the emergent qualities similar to
thinking. Therefore, the representation of ideas and
the structure/process of articulating them shape the
potential for emergent concepts in a group setting.
The desire to discover order and facilitate change is
often what draws a group to the creative process.
Because the process has a structure, the degree of
affinity between the emergent results and the initial
condition can be steered based on the parameters of
the process. Through the designed creative process,
subtly-progressive, or radically-disruptive change can
be achieved. Now what if that process can act in a
billion minds simultaneously?
This process is called ideation and is assumed to spend

most of its time in the brain as a cognitive process
a creative cognitive process. But how do we know it
exists? Skinner cites Bloomfield, ...the only evidence for
the mental processes is the linguistic process (Skinner
1948). I suggest, as we move from text to image in our
communications, the image, review, ranking, video,
or meme is also evidence of mental processes. The
cognitive events that occur during these processes
require skills to externalize, articulate and introduce
into a social semiotic system (Coles 1999). Once these
representations have taken their places in our social
fabric they are either afforded acceptance or not. If a
product is deemed ahead of its time, usually that means
the structure of the receiving environment recognizes
the value of the product but social, economic or cultural
zeitgeist, does not afford a place for it within the social
fabric. There was not a place for that idea to enter into
We discover is that the definition of Design should

look more like this: data process ideas process
representation. Within each process phase there
exists the work of resolving for degrees of meaning. The
component elements of meaning are assimilated through
the advancement process of design and the meaning
gets richer as it evolves towards conclusion (Eco 1994).
Once a design is concluded, the representation takes
on additional/other meanings as it is introduced into
the system and adopted by society. Those meanings
89
are dynamic and decisions based on them affect society

and culture. The conceptual framework for creating
knowledge grows out of culture (Cetina 2009) and
knowledge gained is reintroduced, and so on... forever.
What though, is that initial data that catalyzes the design
process? Stimuli taken in through sensory integration,
is the data of perception. In order to assign meaning
to data, we must first be able to perceive it. The
working definition of Perception as the organization,
identification, and interpretation of sensory information
in order to represent and understand the environment
(Schacter 2009) will serve to frame the skills of per
ception. We can perceive much more than external
stimuli and as this data becomes information, the
process of giving it meaning creates associations with
other sensory input. Much of the input to the brain
does not come from the outside world but from internal
body sensors that monitor the state of respiration,
heartbeat, digestion, and other physiological activities.
The flood of gut feeling that results is blended with
rational thought... (Wilson 1998). There is no meaning
without the synthesis of stimulus data with emotional,
contextual and intuitive data.
The Big Analytics of structured data are attempting to
squeeze the gut feeling out of decision making. The
danger there is that without the stimulus and guidance
of emotion, rational thought slows and disintegrates
(Wilson 1998). Rational decision-making based on the
numbers is valuable and especially important for the
maintain & grow progression of business but deadly
for the lucrative and catalytic and discover & innovate
core innovation needs the gut. The ...five discovery
skills of disruptive innovators: questioning, observing,
networking, experimenting, associating (Gregersen,
Dyer 2011). cannot be sufficiently achieved through
Big Analytics. Decisions are most fruitful when the
numbers are combined with intuitive wisdom gained
through experience (Benjamin 1985).
In is generally agreed that intuition is a type of know
ledge (Bohm, Peat 2000). For many, it becomes
unsettling to recognize this knowledge exists; and it
can be acquired stealthily. The product of perception
and experience, intuition is often seen as something
somehow supernatural (Feuerstein 2008), or worse
feminine. Big Analytics argues that intuition is nothing
90
upon which to be basing business decisions. Great

minds have indicated the contrary: "The workings of
intuition transcend those of the intellect, and as is well
known, innovation is often a triumph of intuition over
logic." (Holton 1997)
Intuition is knowledge based on highly developed skills
of perception. Persons described as very perceptive are
also often described as highly creative. Jung described
the highly communicative and intuitive as likely to be
entrepreneurs or revolutionaries ( Jung 1971). The
designer is constantly absorbing meaningful data from
the environment, sifting through it, recognizing similar
information subsets in that data and revealing potential
for innovation based on meaning. This perceptive process
happens extremely quickly and many perceptive occur
simultaneously. The process itself is interpretive.
Regardless of whether perceptive data is experienced
directly or transmitted through social media, those
experienced in processing sensory data are able to
that process to the point it becomes automatic and
subconscious. Active thought can then be dedicated to
higher-level analysis of sensory information. That is what
distinguishes the trained designer from the novice designer.
An example: All data perceived and represented has a
design. There is always some order, even if we cannot
define exactly what it is. Why, for example do Tara
Donovans works get adopted to represent elusive
concepts like strategy (HBR 2013)? Her work has an
obvious material order to it. The order is immediately
recognizable, although the overall form is unclear or
at least open to interpretation. Perhaps we just want
it to have meaning/association where there is none or
perhaps there is plenty. Humans tend want to order
information. If that be the case, and all humans are
doing it... it must be worth designing for.
Data without Design
Big Data does not have a plan. It is just data data without,
context, without emotion, without intuition, without
culture, without meaning, without design. How do we
look at this data? How do we find what it could tell us?
Just as we have structured and unstructured data; we also
have structured and unstructured observations of data.
The analysis of Big Data is undifferentiated. It is looking
at everything in the hopes of discerning patterns or
behaviors. Once particular behavioral patterns are

exposed, a more differentiated structural analysis can
begin. It is the difference between looking at particular
behaviors to get answers to particular questions and
taking in a wide set of behaviors to perhaps reveal a
pattern that could suggest the question.
The tools for parsing this data are woefully lacking.
The motivation for those persons analyzing big data is
to get objective information. Its the data of desire they
want to get away from. Intuition is unreliable. Removing
gut feel from the decision making process and basing
decisions on hard data provides reassurance and a way
to shift responsibility away from the decision maker.
But doesnt innovation come from the gut? Arent we
always trying to add emotion to convey more accurate
meaning? Be it the richer information of images, reviews,
video, music, emoticons, or memes shared in lieu of, or
in addition to text, the desire to convey deeper meaning
is pervasive. Adapting to posting multi-sensory data in
order to convey ideas as we move from the word to
the image has been made easy through social media.
Analyzing data without interpreting emotional content
is, at best, a stab in the dark at meaning. If we need
structured data to drive our decisions, strategic moves
and innovations, how is it that the most innovative new
communication platforms, tools and liaison structures,
came into being. Where is the data set driving the
development of the Iphone, Amazon, Facebook, Twitter
or Pinterest? It seems making decisions based on the
gut feel of unstructured data is not only a good idea
it is a game changer on a global scale. To the designer
though, unstructured data is an oxymoron.
Big Data needs a strategy. Big Analysis requires fewer
resources as processing and retrieving data becomes
less expensive. Leveraging Big Data has become the
biggest game in town and in order to stay competitive,
business must play. Deciding what big data should tell
you is half the battle and the point where a creative
strategy must come into place. Creative Strategy
benefits from Think Fast, Then Slow (Kahneman 2011).
Big Data can think fast but cannot go deep; the samples
at hand are very broad and undistinguished. Specific
questions about specific groups cannot be answered.
The data set is too wide and shallow. You may though,
by thinking fast, discover areas worth going deep into
and re-specify your data collection efforts to get
to deep meaning. Going deep creates value.

Designers have mastered the tools for going deep.
To design is to get into the details and simulate,
assimilate, digest, decipher, and ingest massive amounts
of data in the process of reaching a solution. Once
we datafy things, we can transform their purpose and
turn the information into new forms of value. (Cukier,
Mayer-Schoenberger 2013) Both quantified and qualified
datafication are necessary to maximize value potential.
As complex as the design process may seem, most
of it occurs under radar and there are few discernible
steps in the path from inception to production. The
way a tool looks and its ease of use determines its
survival. By getting designers on board at the incipient
strategy stage, business can improve products and the
likelihood of user acceptance. If business proceeds
without including designers early in the process, they
will likely spend resources trying to fix their product,
network or interface along the way. Fixing can be
expensive and sometimes businesses make the curve
through re-branding and redesigning. But more often,
the consequence of not having designers on board
early is a super-expensive, awkward... something or
other... without a place in the world. Designers are the
professionals who can transition the undifferentiated
data feed through the cultural semiotic hive and create
new structures, representations, forms and products
for business and commerce. It is critical to success to
design as you go.
Whole Data Research with Design
Big Data feigns satisfaction with quantitative metho
dologies to analyze data. The complex analytics now
possible allow relationships amongst relationships of
data to be analyzed and correlation amongst numerous
phenomena to be revealed; all without understanding
causality and while disregarding 80% of all data out
there. The whole data research (including both
structured and non-structured data) emerging in Design
is changing perceptions of design research from soft
science to solid science. Advanced technology tools
enable the augmentation of quantitative operations with
descriptive qualitative elements to enhance the meaning
of the whole data set. Always focused on behavioral
meaning in social relationships; the qualitative research,
in order to measure itself, is developing quantitative
artifacts as markers and indicators.
91
Apps have allowed people to act as agents of social

interaction, define value systems, modify behavior.
They also have, unlike big data, a welcome, defined
scope which provides empirical data to shape the
soft data. Apps can be monitored while in use, on the
move and within their geographical context. Analyzing
App data allows researchers to get intimate proximity
to those being studied without being intrusive. This
new modality of Whole Data Research is providing a
yet unheard-of proximity of study. As a parameter of
good research we engage in 100% natural observation
direct and immediate contrasted to a selected test
group in an artificial environment. What might be the
value of such market research?
The most notable advantage is the opportunity to study
the way particular environments affects behaviors.
We now have a variable of place and movement
contextual data to add to the mix. An ever more
complex and differentiated whole data set emerges.
Through app research, many kinds of observational
studies are possible simultaneously (cohort, selfreporting diary studies, ethnographic, mechanical,
photographic, disguised, randomized, etc.) which lends
to an efficiency in data analysis and doing more with
less. It is possible to get very individualized information
and draw and redraw test groups based on ever more
distinguishing parameters. Example: What are the
patterns of apparelpurchasing do followers of a DIY
Pinterest board who also use Tumblr and Twitter,
have the Etsy, Dwell, Cnn, Amazon, Mint.com apps, a
walkmeter and at least two preschool learning games
installed and post images of kids on facebook? Where
can I find them in order to direct market my new line of
hand-made organic hemp accessories? How much will
they likely pay for my product? This form of research
takes advantage of directly applicable and quantifiable
demographics. Traditional demographic categories are
nullified. The action of customizing devices with apps
achieves a higher level of individualization for each
user. Demographic subsets defined along traditional
categories no longer satisfactorily indicates behavior.
The data of race, gender or class is now largely
irrelevant in the face of the richness of the more telling
data. The Whole Data combination of Structured and
Unstructured data analysis is necessary to reach the
most meaningful answers.
92
An unprecedented penetration of communication and

entertainment technologies has changed the nature
of social relationships and interactions. Movement
and proximity once strong indicators of relationships
are being redefined. The assertion of social spatiality
shatters the traditional dualism [of physical and mental
space] and forces a major reinterpretation of the
materiality of space, time, and being... ...not only are
the spaces of nature and cognition incorporated into
the social production of spatiality, they are significantly
transformed in the process (Soja 1989). One may sit
next to the same individuals every day on the train but
feel his/her primary community consists of people with
whom one has chosen to interface. Perhaps it is a group
sharing results of the same marathon training app or
parents who choose to home-school or even people
with unusual hobbies.
Space has, through social media, crossed that anti
cipated threshold where the spaces our bodies inhabit
are more or less interchangeable as our social spaces
become more defined (Tuan 1977). Through the ability
to customize interface aesthetics and tools to reflect
desires; personal space is now personal interface.
Home is still where the heart is; and the heart is
with the device. Interpretation of this hyper-personal
space requires cultural understanding. This is where
computers again fail... so far. It is another example of
the necessity for collaboration between computerized
big data analysis and the interpretive analysis of cultural
researchers. Setting up a design research strategy
to facilitate an overall data strategy is essential and
identifying the appropriate tactics and techniques to get
the best results is conditional to success (Zenger 2013).
A bottom-up (constructive) processing of sensory
information: lines define regions, regions define shapes,
shapes suggest forms is largely programmable and
we should allow the computers to do it. It is the topdown (deconstructive) processing of expectations
and associations based on experiences that we must
still leave to the humans. Experience presupposes
knowledge. Designers act as intermediaries and
interpreters of unstructured data. Design innovation
is knowledge often based on the unstructured data,
which if we believe IBM and others, is 80% of all data.
One could say the lions share is embraced by designers
and derided by those trying to teach computers

to decipher the noisy text and emotional content
of images. Design knowledge is attained through
perceptive experience. The richest data is culturally
and sub-culturally specific, dynamic, catalytic and flies
in the face of parametric big data in terms of impact
and potential for innovation. Attempts to structure the
unstructured data through text analytics or tagging with
meta-data are attempts to create a perception machine.
Those savvy enough to seize the opportunities revealed
through whole design research to gain design knowledge
will be best poised to innovate.
Big Social Media Ontology
How agile we are in social networks in America has
much to do with cultural bias, technological user
literacy, communication effectiveness, access to fast
networks, age and to some extent gender. It has
surprisingly little to do with income, education level,
or race. In the US nearly 70% of those on the internet
are users of social media. (Duggan, Brenner, Pew 2013).
In the face of such proliferation, it merits the question,
what is the nature of being/reality in social media? What
is our social media ontology? The question is of course
philosophical and the relevance of the answer lies in our
ability to use that answer to direct whole data research.
It is fitting to frame the philosophical question as
an existential one. In social media we see the key
components of the individual as the thinking, perceiving,
experiencing center in the quest for knowledge and
meaning in an absurd world (Crowell 2004). Most
relevant to question of being in social media is
the concept of Agency. Social media interfaces are
mediated spaces (Thompson 1995). Those spaces
are contextual avatars, representing the self and the
context surrounding ones self. For many it is the edited
version of themselves; and for many (often much to
their chagrin) it is the unedited version of themselves.
This agency gives the individual a moment to think
before acting. It is an opportunity for self-reflection and
to anticipate how the mediated public interface will be
interpreted by friends and society at large.
Creating the agency through images and shared
information is a design process. Social media allows
the individual to deliberately act covertly or overtly in
order to promote a public perception, image or status
(Vygotsky, Cole 1978). There exists a split awareness

when designing a personal social media interface. The
awareness of the complete self and all the information
and artifacts that could that describe that self; and the
curated interface that has been selected to represent
the most advantageous information and artifacts. There
is a potential stress of disconnect between the complete
self and the curated self. In summary, then, covert
behavior is behavior which (1) is effective upon the
speaker himself and (2) avoids the consequences of the
overt form. But a considerable reinforcement survives
in verbal behavior at the covert level because may be
his own listener and may reinforce his own behavior in
many ways... In general then, when talking to oneself, it is
unnecessary to talk aloud and easier not to (Skinner 1948).
There were once distinct spaces for commerce,
entertainment, communication and politics and
distinct behaviors appropriate to those spaces.
Through social media, commerce, networking and both
image and written communication travel along the same
networks (Lefebvre 1991). To limit one is to limit them
all. To heavily curate one demands curating the others
in order to achieve consistency and credibility. It is a
Sisyphus battle to try and censor political exchanges
from the top down while promoting commerce and
entertainment. Social media may provide a commercial
motivation for more exchanges and less censorship.
This is a double edged sword and revolutions have been
coordinated through Twitter. It would be easier to
change your politics/policies/ruler than to change the
potential for Twitter to facilitate a protest.
Social media facilitated revolutions can be also
revolutions for good: improving health, protecting
the environment, etc. Whether for consumption or
conservation; social media serves all masters equally.
So the question is, who can most affect society and
identity? What is the most convincing/appealing
argument? What does that argument look like? What
are the tools of getting buy-in/sympathy/emotion and
ac-tion? How do you create/design desire? How do you
keep it from going radical? Stability comes through the
sheer vastness of social media. Having many divergent
opinions rather than one strong voice insures that
society remain fragmented and peaceful and busy
customizing their commerce, entertainment, political
and communication experience.
93
The Data-fied Self: Interfacing in Social Space

Be it Nike Training, Mind Training, Fitbit or the UP
self-monitoring bracelet made by VC backed Jawbone,
tools and apps to link personal diagnostics (heart rate,
sleep patterns, weight, exercise routine) to a mobile
device are prolific. According to the Pew Internet &
American Life Poll, 60 % of Americans track health
related data (Pew Research 2012). This is an example
of the quantified-self movement (Wolf 2009); a hereto
unprecedented connection of the personal biology
through the interface of clothing/accessories and apps
to a mobile device. The apps and devices are linked to
social media and metrics are shared with the intent of
modifying behavior through feedback, reminders and
social pressure.
The once radically progressive technology of connecting
a functioning pacemaker directly to the phone jack to
send diagnostics, now seems antiquated. In the 21st
Century, wearables send diagnostics remotely and in
real time and are worn more or less visibly. Even the big
players have recognized this booming market and are
getting in on the game (Kharif 2013). That internet of
everything or "Total Information Awareness (Reaves
2002)" we keep hearing about includes sleep rhythms,
blood sugar levels, frequency and intensity of physical
activity. The self-monitoring and self-diagnostic offerings
are growing and will be the next step in the seamless
integration of individual and internet. For designers,
the question posed is how to design the experience
of being watched? What is the experience interface,
the experience of alerts, alarms and feedback? How
do we design the motivators of behavior modification;
the reward experience, the sharing experience? In
order to answer these questions designers must engage
in processes that facilitate acting as agents of the
user relative to a product, idea or service. Breaking
down the user experience into micro-moments or
selecting a wide variety of users and depicting their
interaction through agency becomes a strong tool in
the strategy development tool box. This is agent based
design through the UX (user experience) scenario
process: a user-based design process. ...Combining
big data, behavioral targetingmobile devices and
GPS... [as a research methodology, designers can set
up a framework to get direct access to behaviors],
...application developers can design more effective
94
operant conditioning environments and keep us in

virtual skinner boxes as long as we have a smart phone
in our pockets (Davidow 2013). The data collection can
occur over long periods of time in a non-invasive manner.
Designing for numerous interactions in a day with social
media requires a different approach that traditional
design of objects or artifacts. Design must be conceived
as perpetual and participatory. The experience of the
same interface changes in the space of moments. The
instant and interface of desire must be designed. The
user experience becomes the roadmap for a design
strategy. All touchpoints are analyzed, interpreted,
designed and tested through numerous scenarios to
insure a responsive interactive, customized experience.
This approach is common in marketing and advertising;
a focus on behavioral phenomena to create an
experience around a product. For the experience of
personal monitoring to affect behavior, the microincremental moments of desire must be anticipated.
What does it feel like to like, pin, tweet, post, view
etc.? Is there a tangible or perceptual usefulness in
the experience of quantifying oneself? Do the benefits
outweigh the vulnerabilities that come with sharing
such personal information? Does the user find the
tools for doing so - valuable? In a time where exhibiting
hyper-extroverted qualities is seen as the new ideal
personality type (Cain 2012) everything is spontaneous,
everyone is watching, sharing and commenting. Social
media further swells the potential for being outgoing
or, more positively, provides a place to be extroverted
for those who are otherwise quiet.
Mass Customization & Device Topology
As we wear our devices and sync them to our physical
bodies and social minds they become not just an
extension of ourselves but a physical part of our
perceptive apparatus. We are aware of the proximity
of our devices at all times and are conscious when
they are not behaving predictably; or worse, lost. We
can customize our phones to vibrate, sing, or play a
tune depending on the caller or the appropriateness
of device use at a particular time or place. We have
a proprioceptive sense of the physical position and
movement of our devices as well as its language of
communication: it reminds, rings and speaks. We have
taught it to do those things according to our own
preferences. Through acoustic and visual modifications
as well as performative modifications, through adding

apps, layer upon layer; every device becomes one of a
kind and specific to its user. Such mass customization of
a widely-adopted tool is unprecedented in history.
The proliferation of smart devices is largely due to
the combination of fetish and utility. Owning such a
device is gratifying in that it satisfies the desire for
beauty, represents status and justifies ownership
based on functionality. The devices particularly serve
the individual who has invested large amounts of time
customizing and carefully defining the parameters of its
utility: purchase a platform curate its use (MonroyHernandez, Resnick 2010). As the users engage in the
process of customization of his/her smart devices, they
consider personal preferences as well as geographical
and situational factors. The assemblage of apps on any
persons device define the user as much as the user defines
the tools. By selecting particular apps the user chooses to
interact with them to behave in a particular manner.
The intricacies of an individuals personal experience
though is becoming ever more particular through the
selection and customization of interfaces aesthetics,
tools and media sources. The simple act of choosing a
selection of particular news feeds causes realities to
differentiate. Reality and truth, not just across cultures
but across the table can vary radically. Not that long
ago, the news was presented in thirty minutes. Now the
expectation of information is much higher and users of
media thirst for many different types of unstructured
data to create their own truth about any event.
Achieving comprehension of any news event is much
more differentiated and biased. We can choose to view
information in the way we want to have it. Having the
facts reinforce an opinion has always been desirable
and now it is achievable through customized, selected
information interfaces. Since experience is a type of
knowledge and we can customize our experiences; the
person doing the knowing can be customized. Designing
the experience is designing the person = Behavioral
Change. The designed experience has become a
commodity (Pine, Gilmore 1999). We now design for
behavioral change.
The power of customization assures longevity.
The adaptability of the platform and the ability to build
upon previously selected tools while removing ones that
do not work, sustain interest and add value to

the device. The state of the platform/tools union
at any given time is Device Topology.
Doreen Masseys layers of investment refer to a
changing spatial structure defined by layers of invest
ment, and therefore activity in building etc. through
time. As each layer is put into place, it interacts in
a sort of fluid terrain model with the layers before
(beneath) and becomes ever more differentiated and
particular with each layer over time. These layers of
investment were explored in relation to division of
labor but interesting insights can be drawn when using
this analogy to describe the platform and the investment
in geographic and situation-specific apps added to
serve the functions desired on an individual level. Like
in Masseys discussion, the apps and the platform have
a mutual determination each giving and shaping the
others meaning (Massey 1984).
Especially interesting is the impact of local personal,
geographical and situational forces is considered as well
as the relevance of non-local, external but relevant
forces meta forces. At the scale of each individual,
mass-customization is performed relevant to his/her
value system. The value of a persons value system is
very high, particularly when trying to predict, elicit,
or modify behavior the great desire of commerce,
politics, and entertainment. The adaptability of the
purchase a platform curate its use service/product
insures the longevity since it breaks down the demo
graphic market definitions. Everyone is the market.
Whether you are a field journalist for CNN, a teacher
in a community preschool or an aspiring Olympic
athlete, this tool was made especially for you and
you cannot do without it. You make it meaningful.
Emergence of the HiveDesign
To initiate discussion on the emergence of hive-design,
it would be beneficial restate the premises thus far:
Big Data is a vast collection of quantifiable but meaning
less data parsed by computers to reveal correlations.
Soft/unquantifiable data is four times as big as
Big Data and requires human minds to interpret.
Data without Design is insufficient to make rich decisions
Whole Data Research integrates structured and non/
structured data and is the ideal information construct
to generate the most valuable knowledge
95
Social Media Ontology describes identity and

existence as mediated through the agency of social
media interfaces, presence and status
Interfacing and the Datafied Self is the phenomena
of self-monitoring and engaging in biological/technical
interfacing through wearable technology
Mass Customization and Device Topology describes
creating value and longevity for the platform and
tools by empowering individuals to personalize the
performance of his/her device
For the purposes of describing the phenomena
of Hive design, we refer to major aspects of
Jeffrey Goldsteins description of emergence:
...the arising of novel and coherent structures, patterns,
and properties during the process of self-organization in
complex systems. Emergent phenomena are conceptualized as occurring on the macro level, in contrast to
the micro-level components and processes out of which
they arise. Certain interrelated, common properties
identify phenomena as emergent:
radical novelty the phenomena cannot be predicted
from the behavior of micro-level components
coherence emergents appear as integrated wholes
maintaining a sense of identity over time
emergents act on the macro level above the micro
level of the associated components that engendered
the emergence
emergents are dynamic and evolve in a complex
system over time
emergent phenomena reveal themselves
(Goldstein 1999)

HiveDesign is an emergent community of co-creators
simultaneously working in a hyper-temporal (exchanges
happening pervasively and concurrently) and heterotemporal (perception of time varies relative to place)
environmentally responsive space/time construct
engendered by social media (Sha 2011). Individual
creators make up the hive and may view themselves
as fiercely independent. HiveDesign is open source,
adaptive design and is practiced by both trained and
novice designers (Florida 2002). The hive of designers
is largely democratic and intuitively identifies its
leaders yet may deny being led. HiveDesign holds little
allegiance and is fluid and reactive.
96
There is meaning in that movement (fluidity). How

present a user is and where he/she can be found
contributes to the profile and status of the user. The
questions as to an individuals presence are common:
Are you on Facebook, Twitter, Tumblr, Pintrest,
LinkedIn, or Google+? The answers to those questions
define social media presence. Within the hive of social
media, users either move between these spaces or are
on them all simultaneously. If a user has been deemed
inactive, social media will send email or text messages
to motivate users to share what you know or update
your status; which brings us to the importance of status.
Status is central to all complex mammal societies,
humanity included. To say that people generally seek
status, whether by rank, class, or wealth, is to sum up
a large part of the catalogue of human social behavior
(Wilson 1998). The agency of social media provides the
novice and trained designer the opportunity to reflect
on status and take action through re-assembling content
and presence. ...Given that ones own character and
worth will be partly read from the relations that one
keeps, a desirable relationship may be displayed in
proper and populous places for the purpose of establishing ones own value (Goffman 1971). Recently, a
student confided, Im so worried about my friend,
she changed her Facebook page and everything. She
obviously feared that there had been an existential
rupture in her friend; a fear that the change in social
media status was so fundamental that it could change
the nature of their friendship. That was five years ago
and it has become expected to change your status in
order to be actively engaging with your social network.
The individual designers of the hive consistently and
intuitively conduct design research through experiments
in expectation. Social media platforms allow users
to try-out different versions of the self through the
agency of the interface. Users can create and recreate
themselves through changing profile pictures, adding
and deleting friends, changing feed subscriptions and
using different apps. If the curated content of the
interface agent doesnt get the desired results; the user
can rebrand him or herself as often as he or she may
choose. The effect of rapidly revamping our customized
agents is the depletion of the value of personal data.
Even the most revealing information can be made
irrelevant/obsolete at any moment in the manner
of that was so five minutes ago. The correlation

between value and meaning also indicates that the
possibility of rapid reinvention also strips the data of
meaning. The process of making identity includes the
informational purge we can reinvent ourselves by
changing our apps. This process is adaptive customization.
Followers of Saussure and Pierce can agree that
meaning is derived through relationships. Our new
media tools enable Piercian unlimited/infinite semiosis
(which implies the deepening of meaning). Eco refers
to Hermetic Drift or Neoplastic Growth to describe
...the uncontrolled ability to shift from meaning to
meaning, from similarity to similarity, from a connection
to another. (Eco 1994). Meaning changes and evolves
in, and out of, this situation. Through sharing, repinning
and reposting blogs, images, video, reviews, articles,
quotes and memes through social media, the hive of
designers is effectively engaging in both infinite semiosis
and Hermetic drift. For example, by quantifying the
interpretation of images through adding hashtags (#) as
images are shared, more and more information can be
associated with an image.
Infinite semiotic growth is what we experience with
the pinning, posting, tagging and sharing now possible
through our apps, devices and social networks. Every
individual that passes the information along takes it
out of context and makes new associations... a kind of
hive interpretation, hive design. Depending on cultural
background, experiences and community, meaning can
be something very different from one interpreter to
another. If though we are sharing the same tools and
acting in the same hive, our common tools and common
platforms provide common experiences and more
common meanings that allow design to reach a broader
audience, on a superficial level. If one of the interpreters
in the hive finds a share irrelevant, it will not be
passed on and dies en route unless someone else picks
it up and shares it further (Asur, Huberman, Szabo,
Wang 2011). The hashtags: #Taj, #India, #Beautiful
repinned as #white, #my dreamhouse, #vacation
again repined as #mausoleum, #architecture, #stone,
and again #tourist trap, #Bollywood, #bright. This
is HiveDesign engaged in Pierces infinite semiosis:
repinning and hashtaging to deeper meaning. Typically,
there is little interest in the original meaning of the
catalyst image (Barthes 1977). For some though,
greater information can be gleaned by delving deeper

into meaning through correlating new images and the
plethora of associated hashtags to develop a richer
overall comprehension/knowledge. These new social
media sharing parameters are an unexpected and
astoundingly deep cradle of information available to
design research.
Strategic Co-opting of HiveDesign
Benefitting from hivedesign requires skills in crowdsourcing. Enlisting hive-designers to contribute
interpretations from the 80% of unstructured data in
order to add to the Big Data mix and approach whole
data evaluation as a basis for decision-making is the big
idea. Recognizing that soft data has profound value and
making the choice to leverage the meaning of that data
is the key to innovation. Interactions in social media are
always making meaning, consciously or not. Part of a
successful Whole Data Strategy includes a method for
harvesting knowledge from the field of social media. The
sheer volume of quantification through interpretation
being carried out by the novice and trained designers
in social space demands attention. Experienced design
executives/professionals are uniquely qualified to
develop the soft data interpretation strategy as a
collaborator in the strategy development process.
These design professionals can identify what is rare,
distinctive, well-represented and valuable as well as
design the modalities to enlist and engage hivedesigners
in order to steer more focused results. Hivedesigners
can provide answers to such questions as, Why do we
click, follow, or share? What are the indices that signal
a pattern? How are ideas categorized thematically
and who are the thematic neighbors (competitors and
collaborators)? Hivedesigners provide opportunities to
covertly experiment with small scale implementations
of new strategies, prepare & manipulate visual content.
Hivedesigners can source content and decipher bias of
reviews from amazon or yelp. The Return on Investment
is impossible to quantify.
Once patterns and structures have been identified from
the hivedesign data, a design strategy team engages
in Dynamic Semiotic Analysis to decipher meaning
relative to identified goals. The dynamic semiotic
analysis process happens in the midst of the activity
it is studying and evaluates meaning based on microcomponents of time, place, culture, interface, aesthetic
97
and experience. A design strategy team can, based on

the results of the soft data research provide orientation
in an ever evolving social media and interactive world.
Designers can quickly share this information through
data visualization and agent simulation in order to
heighten institutional intelligence by letting all players
in on where the strategy is heading. It can improve the
entire organizations strategy sophistication by providing
the opportunity to experience information rather than
to simply see it.
Bohm, D., & Peat, F. D. (2000). Science, order, and creativity.

London [u.a.: Routledge.
Burns, T., & Stalker, G. M. (1961). The management of innovation.
[London]: Tavistock Publications. 19-36
Cain, S. (2012). Quiet: The power of introverts in a world that
cant stop talking. New York, NY: Crown.
Crowell, Steven, C. (2004, August 23). Existentialism. Stanford
University. Retrieved from http://plato.stanford.edu/entries/
existentialism/
Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences
make knowledge. Cambridge, MA: Harvard University
By engaging a larger body of the organization and

having decision makers become party to the analytics;
deciding which questions to ask and evaluating risk
assessment are seamlessly integrated into the overall
strategy. Through a durable soft-data strategy built on
an aesthetic, temporal and cultural semiotic knowledge
base, more (and less) desirable experience paths can
been designed with the intent to guide choice.
Press., p.1
Cetina, K. K. (2009). The synthetic situation: Interactionism for a
global world. Symbolic Interaction, 32(1), 61-87. doi: 10.1525/
si.2009.32.1.61
Coles, A. (1999). The optic of Walter Benjamin. London: Black
Dog Pub.
Cukier, K., & Mayer-Schoenberger, V. (2013, May/June). The rise
of big data. Foreign Affairs, 92(3), 28-40.
Davidow, B. (n.d.). Skinner marketing: Were the rats, and
In the 1930s BR Skinner developed the concept of

operant conditioning... we now know how to design
cue, activity and reward systems to more effectively
leverage our brain chemistry and program human
behavior (Davidow 2013). Once highly controversial,
the practice is de rigueur eighty years later and it has
been shown that consumers are not adverse to this type
of native advertising provided the content provides
a valuable experience. Users recognize that they are
affected by their experiences and choose to be affected
by appealing experiences. It is for the designer to create
that appealing experience. Designing the experience
is designing the person. This is the added value behind
design. Organizations with a well-developed whole data
and creative content strategy will be adept at influencing
behavior and will be the innovators of the future.
facebook likes are the reward: Our internet handlers

are using operant conditioning to modify our behavior.
The Atlantic. Retrieved June 10, 2013, from http://www.
theatlantic.com/technology/archive/2013/06/skinnermarketing-were-the-rats-and-facebook-likes-are-thereward/276613/
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q.,
... Ng, A. Y. (2012). Large scale distributed deep networks.
[Scholarly project]. Retrieved June 15, 2013, from http://
static.googleusercontent.com/external_content/untrusted_
dlcp/research.google.com/en/us/archive/large_deep_
networks_nips2012.pdf
Duggan, M., Brenner, J., & Pew Internet. (2013, February 14). The
Demographics of Social Media Users. Pew Research Centers
Internet & American Life Project. Retrieved from http://
www.pewinternet.org/Reports/2013/Social-media-users.aspx
Eco, U. (1994). The limits of interpretation. Bloomington, IN:
Indiana University Press.
References
Asur, A., Huberman, B. A., Szabo, G., Wang, C., & Hewlett
Packard. (2011, February 8).Trends in social media :
"The Author and His Inter-preters" Retrieved from http://

www.themodernword.com/eco/eco_author.html
Persistence and decay [Scholarly project]. In Twitter Trends.
1996 lecture at The Italian Academy for Advanced Studies in America.
Retrieved June 15, 2013, from http://www.hpl.hp.com/
Feuerstein, R., Feuerstein, R. S., & Falik, L. H. (2010). Beyond
research/scl/papers/trends/trends_web.pdf
Barthes, R., & Heath, S. (1977). Image, music, text. New York,
NY: Hill and Wang. 79-148
Benjamin, W., & Arendt, H. (1985). Illuminations: [essays and
reflections]. New York, NY: Stocken. 83-109
98
Eco, U. (1996). Eco - "The Author and his Interpreters" Eco -
smarter: Mediated learning and the brains capacity for

change. New York, NY: Teachers College Press. P.79
Florida, R. (2002). Rise of the Creative Class. Washington
Monthly. Retrieved June 22, 2013, from http://www.
washingtonmonthly.com/features/2001/0205.florida.html
Goffman, E. (1959). The presentation of self in everyday life.

Garden City, NY: Doubleday.
Goffman, E. (1971). Relations in public; microstudies of the public
order. New York, NY: Basic Books.
Goldstein, J. (1999). Emergence as a construct: History
and issues. Emergence, 1(1), 49-72. doi: 10.1207/
s15327000em0101_4
Google. (2013). Flu Trends. Google Technolgoy for Social
Impact. Retrieved June 15, 2013, from http://www.google.
org/flutrends/
Gregersen, H., & Dyer, J. (2011, October 20). The worlds most
Harvard Business School Press.

Reaves, J. (2002, November 19). Homeland security: A primer.
Time.com. Retrieved from http://www.time.com/time/nation/
article/0,8599,391161,00.html
Ryall, M. D. (2013, June). The new dynamics of competition:
An emerging science for modeling strategic moves. Harvard
Business Review, 91(6), 80-87.
Saussure, F. D., Baskin, W., Meisel, P., & Saussy, H. (2011). Course
in general linguistics. New York, NY: Columbia University
Press. 79-100
Schacter, D. L., Gilbert, D. T., & Wegner, D. M. (2009).
innovative growth companies. Forbes. Retrieved from http://
Psychology. New York, NY: Worth. Sha, X. (2011, August
www.forbes.com/sites/innovatorsdna/2011/10/20/the-
02). Recherche-Creation. Lecture presented at University
worlds-most-innovative-growth-companies/print/
of California Humanities Research Institute (UCHRI) Seminar
Holton, G. J., & Elkana, Y. (1997). Albert Einstein: Historical
in Experimental Critical Theory, Section VII, ReWired:
and cultural perspectives : The centennial symposium in
[Asian+TechnoScience+Area] studies in University of Hawaii,
Jerusalem. Mineola, NY: Dover Publications. p. 97
Manoa.
IBM Industries & Solutions. (n.d.). Apply new analytics tools
Skinner, B. (1948). Verbal Behavior. Lecture presented at William
to reveal new opportunities. IBM Analytics IT Business
James Lectures in Harvard University, Cambridge. Retrieved
Intelligence. Retrieved June 15, 2013, from http://www.
from http://www.bfskinner.org/bfskinner/PDFBooks_files/
ibm.com/smarterplanet/us/en/business_analytics/article/
it_business_intelligence.html
IBM. (n.d.). What is big data? - bringing big data to the enterprise.
IBM Big Data. Retrieved June 15, 2013, from http://www-01.
ibm.com/software/data/bigdata/
Jung, C. G. (1971). Psychological types. Princeton, NJ: Princeton
University Press.
Kahneman, D. (2011). Thinking, fast and slow. New York, NY:
Farrar, Straus and Giroux.
Kharif, O. (2013, May 13). Google Glass Woos Developers to
$6 Billion Wearable Market: Tech. Business: Washington
Post Business Page, Business News. Retrieved from
http://washpost.bloomberg.com/Story?docId=1376MKYNK86K50Y901-26R5750SOCLE5EM85KL4O84TE6
Lefebvre, H. (1991). The production of space. Oxford, OX, UK:
Blackwell.
Massey, D. B. (1984). Spatial divisions of labor: Social structures
and the geography of production. New York, NY: Methuen.
Monroy-Hernandez, A., & Resnick, M. (n.d.). Making projects,
William%20James%20Lectures.pdf
Soja, E. W. (1989). Postmodern geographies: The reassertion
of space in critical social theory. London: Verso. P. 120
Thompson, J. B. (1995). The media and modernity: A social
theory of the media. Stanford, CA: Stanford University Press.
119-148
Tuan, Y. (1977). Space and place: The perspective of experience.
Minneapolis, MN: University of Minnesota Press. 67-84
Ulrich, K. (2003, October). KJ diagrams. The Wharton School,
University of Pennsylvania. Retrieved from http://opim.
wharton.upenn.edu/~ulrich/documents/ulrich-KJdiagrams.pdf
Affinity Diagrams/Methodology Acknowledge: Shoji Shiba
methods taught at MIT in the mid 1990s.
Vygotski, L. S., & Cole, M. (1978). Mind in society: The
development of higher psychological processes. Cambridge,
MA: Harvard University Press. 61-63
Wilson, E. O. (1998). Consilience: The unity of knowledge.
New York, NY: Knopf. 111-116
Wolf, G. (2009, June 22). Know Thyself: Tracking every
making friends: Online community as catalyst for interactive
facet of life, from sleep to mood to pain, 24/7/365.
media creation. In K. Brennan (Author). doi: 10.1002/yd.377
Wired.com. Retrieved June 15, 2013, from http://www.
Muller, J. Z. (2003). The mind and the market: Capitalism in

Western thought. New York, NY: Anchor Books. 395-399
Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1975). The
measurement of meaning. Urbana-chicago-london, IL:
wired.com/medtech/health/magazine/17-07/lbnp_
knowthyself?currentPage=all
Zenger, T. (2013, June). What is the theory of your firm?
Harvard Business Review, 91(6), 72-78.
University of Illinois press.

Pine, B. J., & Gilmore, J. H. (1999). The experience economy:
Work is theatre & every business a stage. Boston, MA:
99
Dr. Colin M. Kennedy

School of Design, Victoria University of Wellington, New Zealand
Colin.kennedy@vuw.ac.nz
Designing to Scale: Social Relations Design
Abstract
1 Introduction
Social relations and networks have changed so rapidly, and

to such an extent, that we all see, experience, and interact with our communities in different ways and on different
scales. Within such a haphazard environment, social
designs objective to create positive change is unquestionably
commendable. However, such change is problematic to
sustain without a coherent methodology and strategy
and it can be argued that social design, as an explicit
approach, is still in the process of developing these. This
article suggests that by promoting a design focus at the
scale of social relations, both in terms of objectives
(reducing social disparity) and strategy (focusing on social
relations), social design can make a more effective and
sustainable community contribution. To examine the
potential of this approach, the article introduces a conceptual process model to help visualize how inequality
operates. Using this model, examples of successful
social design projects are considered to explore how
social relations may have been positively impacted in the
various contexts. Due to the complexity of the concept
of inequality, the issue of working at simultaneous scales
(external and internal to the community) is recognized as
an important factor in a social relations design approach,
as is a commitment to place-specific sustainable change.
Simmons [2011, 3] observes that [d]esigners frequently

describe themselves as problem-solvers. We apply
our creative talents to finding new and appropriately
innovative solutions. What if part of the problem is
finding the right problem? Globalization has cultivated
and complicated our social networks and relations to
such an extent that we all see and interact with different
problems at different scales. Within such a haphazard
environment, social designs objective to create positive
change is unquestionably commendable. However, such
change is problematic to sustain without a coherent
methodology and strategy and it can be argued that
social design, as an explicit approach, is still in the
process of developing these.
Keywords
Social design, social relations, inequality, scale,
community design, context
100
This article suggests that by creating a focus, both

in terms of objectives and strategy, social design can
make a more effective contribution. The objective is
the disruption of social inequality and the strategy
is a concentration at the scale of uneven social
relations. The scope of this article does not allow for
a comprehensive discussion of methodology, especially
of co-design, which is seen as an integral element (see
Sanders and Stappers (2008) and DiSalvo et al (2011),
but it does make reference to the various tools that
are often utilized in social design. As all social relations
operate within an instrumental context, this approach
will also need to consider this external scale as well.
This concern of operating at simultaneous scales is

a challenge that is discussed later in the article.
The first section introduces the characteristics
(objectives and tools) of social design from a selection
of contemporary design literature. The intention is
not to attempt a definition of the concept but rather
to establish a broad understanding of what it is. The
following section introduces a brief conceptual process
model of inequality to propose that a social relations
design approach should isolate the mechanisms of
the process to create discrete points of focus. Finally,
project examples highlight elements of social relations
design within them in an attempt to illustrate its
potential contribution.
2 What is Social Design?

Any attempt to categorize social (or socially responsible)
design can be argued to be futile as the scope of design
that influences the social sphere is both dynamic and
multidimensional. As Faud-Luke [2009, 1-2] suggests,
[i]t seems design is difficult to pin down and is every
whereDesign is manifest in all facets of contemporary
life. Consequently, this paper makes no attempt to
define the concept but rather suggests a new and
particular social design focus (social relations design)
that can produce positive, sustainable and efficient
outcomes. However, it is useful to first comment on
the characteristics attributed to the term of social
design. These characteristics can be viewed in two
categories: objectives and tools.
2.1 Objectives
Social Design is about innovative approaches driven by
a commitment to positive change, social equality and
environmental sustainability. It aims for transformational
structural and systemic change that tackles the root
causes of social marginalization and environmental
deterioration through disruptive design thinking. Ingrid
Burkett [2012], from the Centre of Social Impact,
proposes that the focus,
Is on the design of products that benefit people (for example,
the design of water purifiers for people living without potable
water); or services (for example, designing more inclusive
financial services); or processes (for example, designing parti
cipatory decision-making processes inside organizations).
Put simply, at the heart of social design is the pursuit for

a better world. Responsible design seeks solutions and
improvements to enhance peoples lives and create a
sustainable society by applying design principles to social
issues. Faud-Luke [2009, 49] appreciates the extent of
the undertaking and comments,
There is a growing need for new design heroes and heroines
to provide some guidance to meet the enormity of the scale
of the environmental, social and economic crises in the
global, and regional/local, economies.
Shea [2012, 7] agrees and appeals for a shift in thinking,
saying that [s]ocial design defines a new kind of
designer. It needs to be expansively conceived beyond
trained designers to include end users and social
participants. Social design cannot be a subspecialty of
the design profession. This is in line with Browns belief
that design has become too important to be left to
designers: innovation has become nothing less than a
survival strategy. [2010, 7]
Potentially social design could be criticized as
having competing characteristics. On the one hand
the approach is understood to be responsible and
constructive, working with communities to achieve
shared goals. Conversely, it is also discussed in terms of
design activism, which Faud-Luke [2009] often describes
as the deliberate generation of a counter-narrative to
create disruption and massive change. These two faces
of social design need not cancel each other however, as
the struggle to achieve sustainable and positive change
within the context of contemporary challenges often
demands interruptive approaches. As Berman [2009, 1]
concludes,
Designers have an essential social responsibility because
design is at the core of the worlds largest challengesand
solutions. Designers create so much of the world we live in,
the things we consume, and the expectations we seek to
fulfill. They shape what we see, what we use, and what we
waste. Designers have enormous potential to influence how
we engage with our world, and how we envision our future.
2.2 Tools
Many of the design tools associated with the approach
cannot exclusively be claimed by social designers. They
are used by design and development practitioners
101
among others but what is unique to social design

is the convergence of these methods. The resultant
combination is a response to Browns appeal: [w]hat
we need now are new choices [2010, 3].
Social design innovation needs to be accessible and
viable, so that generated ideas can be implemented
by (and with) non-designers for the greatest impact.
This can be achieved through a culture and process of
storytelling and exploration or, in different situations,
through observation, experience and research. This
level of preparation is essential and is what IDEOs
Human Centered Design Toolkit labels as the hearing
phase of the process. This refers to the concepts of
facilitating and co-creation (or participatory or codesign), which have been long been discussed within
development contexts. As Harris [2008, 703] says,
[a]lthough the rhetoric of partnership has been
widely adopted, its implementation ranges from
instrumental arrangements that perpetuate unequal
power relationships and change little in practice, to
authentic relationships based on maturity and trust.
Escobar [2012, 18] sees these authentic relationships
as essential to any developmental interaction because
we are always immersed in a network of interactions
which are at every instant the result of our biological
and cultural histories. We necessarily co-create the
world with others. Such a bottom-up perspective
requires that the interaction to be context-specific
(or place-based) to ensure that the design outputs
have people and location at the core of the empathetic
collaboration, what Escobar [ibid, 6] calls bringing
people back into situations. This idea is also evident
within the user-experience interface between design
and human involvement, which Brown [2010, 94] labels
the customer journey.
Having outlined these characteristics, this article now
suggests that a new social relations design approach, by
using the same tools as outlined previously, can prove
efficient and effective in achieving positive change. The
focus is on the uneven social relations that result from
the perpetuation of socio-economic and environmental
disparity. The following sections explore how social
relations design is able to focus on the isolated
mechanisms that drive the process of inequality, leaving
designers more capable of disrupting the process that
creates disparity.
102
3 How Can Social Relations Design Impact

Inequality?
This article argues that inequality needs to be a key
focus for social design. To do this efficiently and
effectively it is now proposed that a new approach,
social relations design, be considered. The following
sections consider the nature of inequality and social
relations, and how these relationships can be made
uneven as a result of the process of inequality. Finally,
the notion of a social relations design is introduced
and discussed.
As with social design, the notion of inequality is
somewhat amorphous and consequently complicated
to define. It is easier to answer the question, inequality
of what? The disparity of income and disparity is
demonstrably connected to inequality in contexts such
as education, well-being, housing and employment.
Within these contexts are uneven social relations of, for
example, gender, ethnicity and class that are the units of
comparison. For the purpose of this article social relations
are understood as relationships between individuals within
a group, between groups, or between an individual and a
group. The relations vary and are, as described by Marx
(n.d., 92) below, connected to a context.
[S]ocial relations are just as much produced by men as
linen, flax, etc. Social relations are closely bound up with
productive forces. In acquiring new productive forces men
change their mode of production; and in changing their mode
of production, in changing the way of earning their living,
they change all their social relations. The hand-mill gives you
society with the feudal lord; the steam-mill society with the
industrial capitalist.
These social relations of production, as outlined in the
Marxian sense above, refer to how people are related
to the sphere of production as social classes and in
socio-economic dependencies between themselves.
The processes of inequality operate and are sustained
within various contexts, not exclusively within a
production framework. Social relationships also form
in environmental, structural (for example, education
and access to healthcare) and cultural contexts.
Partly as a consequence of globalization, these
contextual spaces have become simultaneously smaller
and larger (e.g. Google Earth and the Skype platform,
which offers localizations in over sixty different

languages). Consequently, social relations have been
transformed and these changes are evident in societys
widening socio-economic disparity. It is suggested in this
article that design can make a significant difference at
the community scale of these uneven social relations.
Inequality can be argued to operate as a contextual
process and is a consequence of internal and external
pressures, which can be understood as a complex
whole [Cardoso, 1979]. This perspective requires
a design response to the challenge of operating on
simultaneous scales. As Jin Yong Kim (2012), President
of the World Bank Group commented at the last
Clinton Global Initiatives Annual Meeting, [d]esigning
for impact, I think, means taking seriously this issue of
scale. Thinking about scale from the beginning is key.
Design needs to rise to the challenge of engaging with
both the contexts in which the relations occur and also
the relations themselves.
Socio-economic disparity is too complex a process to
be solved by one overarching design approach, as there
are various processes of inequalities, often culturally
specific, operating at different scales that reinforce each
other. The UN [2005, 136] support his, commenting
that, as useful as community-based policy targeting
can be for reducing inequality, it should not become
a substitute for universal coverage. Consequently
a situation of inequality needs to be understood
externally, as well as from within. As such, a social
relations design approach requires a simultaneous-scale
approach to disrupt ongoing processes and create room
for contextually-specific positive change. To achieve
this it is necessary for a designer to articulate and
visualize the mechanisms that play a role in creating
and perpetuating inequality.
3.1 The Process of Inequality
For social designers to understand how place-specific
inequality operates it is imperative to recognise social
and cultural entities (e.g. an ethnic group or a socioeconomic class) as existing within a matrix of social
relations. Uneven relationships are unique to the
context, for example a refugee community negotiating
with a local council to access appropriate healthcare
or lower socio-economic status communities living in
highly-polluted environments due to nearby industry.
de Ferranti et al (2004, 7) suggest that a pretense to

an exhaustive discussion of the causes of inequality is
thus a shortcut to failure and while this section in no
way seeks to construct a definitive understanding of the
causes of inequality, it presents a simplistic conceptual
process model (Figure 1) as a framework for a social
design approach.
Position
Uneven integration
(power and prejudice barriers)
Participation
Fig. 1. A conceptual model of the process of socioeconomic inequality
This process is understood from a structuralist

perspective and is not designed to be operational
but has been created to visualise the mechanisms
that play a role in creating and perpetuating inequality.
It is based upon the approach presented by Greig et
al (2007, 13): [i]nequalities are not simply carefully
constructed measurements scales but complex webs
of dynamic social relations that privilege some while
constraining the life chances of others. To allow the
consideration of the interplay between external and
internal factors the model can be argued to sit within
an external environment (made up of geo-political,
historical, economic, socio-cultural and environmental
aspects) and the model itself can be seen as the
internal process.
To elucidate this process model further, a case from the
next section can also be used here. Within the context
of well-being, it has been documented that Latina
women suffer the highest rates of cervical cancer of any
main racial or ethnic group in the United States. This
is exacerbated by the statistic that many do not attend
their annual Pap tests examinations. This highlights
a lack of participation that, when compared to other
ethnic groups, clearly indicates a social disparity. By
using the process model in Figure 1, certain questions
arise why in particular Latina women (position) and
what social obstacles prevent a fuller participation
(barriers)? These are the questions that a social designer
investigates to separate out the design foci, and which
will be discussed in the following examples.
103
4 Social Relations Design

It has been suggested that uneven social relations are
the result of the process of contextual inequality, but
it can also be argued that they are the engine of the
process as well, contributing to inequality traps.
A social relations design approach therefore focuses
on social relations to effectively disrupt the process
and can position itself to operate at the local scale,
using community knowledge and networks to achieve
a social interaction that is specific and collaborative.
As contextual space also significantly influences the
nature of the relationships, this externality similarly
requires attention. Contextual space challenges design,
and challenges generate innovation.
This section draws from the previous discussion
on the process of inequality to highlight some of
the disadvantages that might be focused upon and
introduces some successful examples of social design
where this is already taking place. These disadvantages
include access to quality health services, vulnerable
living conditions in isolated or polluted environments,
or cultural obstacles for full community engagement.
Although these examples do not explicitly have a social
relations focus, the comparable elements of the projects
demonstrate the potential for the approach.
4.1 Access to Essential and Quality Health
Services
VisionSpring. On their website (www.visionspring.
org) VisionSpring proclaims that 703 million people
could have their vision restored with a pair of glasses,
and 90% of those living with uncorrected vision live in
the developing world. It is proposed that this situation
results in an annual loss of US $202 billion to the global
economy. Subsequently VisionSpring supplies affordable
glasses to regions within the developing world, targeting
communities that have limited access to eye care. The
company organizes vision campaigns, which usually take
place at a school, church, or community center over
the course of a day, and includes a temporary exam
room and a pop up shop that sells glasses and protective
sunglasses. These campaigns are operated by local,
trained people, who educate communities about the
importance of eye care and the benefits of corrected
vision. These vision entrepreneurs are often women,
disrupting and eliminating context-specific barriers to
an equal gender social participation.
104
If, as VisionSpring suggests on the website, glasses

can increase productivity by 35%, then there is the
potential to increase individual monthly income by 20%.
Based on their conservative estimates for the average
daily income of their customers, it is calculated that
the annual increase in earning potential for each pair
of glasses sold (multiplied by the two-year lifespan of
a pair of glasses) has already resulted in an economic
impact of US $269 million. The website continues
to suggest that recent research conducted in China
determined that correcting vision in primary school
students is the equivalent of an additional four to six
months of schooling. The community outcomes of
these operations are significant: not only do they allow
workers and students alike to participate more fully in
the world around them, but the uneven social relations
(rural vs. urban or socio-economic access to quality
resources) are also improved.
An integral component of this approach is a pricing
strategy that addresses the need for glasses while
creating viable businesses at the same time. VisionSpring
sells frame options ranging from basic and affordable,
to higher priced styles for those who can afford them.
The income generated through the sale of these
higher margin products enables the resource-intensive
operations required to reach more remote locations
with limited access to eye care.
This is a good example of social design using the process
model presented in Figure 1. Within the context of wellbeing, it established the spatial and economic position of
the disadvantaged communities, identified the barriers
of integration (gender and lack of knowledge) and
consequently facilitated participation. By operating at
simultaneous scales, the project looked at the external
context (what product was unavailable) and at the scale
of social relations, which were preventing certain groups
from fully participating.
Es Tiempo.
Hispanic women have the highest incidence and mortality
rates for cervical cancer of any major racial or ethnic group
in the United States. While case studies show that the best
way to prevent and detect cervical cancer is with annual Pap
tests, many Hispanic women do not go to the gynaecologist
regularly. [Alterkruse, 2010, cited in Shea, 2012, 90]
Students from the Art Centre College of Designs

Designmatters department used this context to design a
campaign focused on the Los Angeles Latina community
concerning the importance of annual Pap tests. The
approach was place-based and participatory, working
with the local community in focus groups to uncover
the attitudinal and logistical barriers to regular checkups. They had predicted that it would concern a lack
of education regarding Pap tests but in fact it emerged
that it was more about understanding the importance of
an annual check-up. The research also showed that the
group did not want to feel targeted and so the approach
avoided being too direct, using personal language and
familiar imagery (for example, on bus shelter maps
and on the paper that covers the beds in the clinic
examination room.) Shea [2012, 83] suggests this level
of empathy is vital:
Instead of focusing only on a communitys shortcomings,
chart both its strengths (local language, style, skills) and
challenges (literacy levels, drug and crime problems) and
use that list as a guide through the project. Take inspirations
from your interactions with community members and find
ways to create an emotional tie with the general public by
representing them with dignity.
The campaign realised the need to build trust between
the women and the doctors and so focused on the
positive aspects of prevention rather than the negative
consequences of inattention. This included creating a
relationship with VISA card to help the women cover
the time they needed to be absent from work to go to
the doctor, and clinic kits to make the exam experience
less distressing, which included a blanket, gown stickers
and slippers. Ultimately the Es Tiempo campaign
targeted the economic, gender and cultural barriers that
created uneven integration, resulting in a greater access
to vital health resources.
4.2 Access to a Safe and Healthy Environment
Aquaduct. The World Health Organizations 2013 annual
compilation of health-related data noted that, despite
increased global usage of improved drinking sources
since 2010, progress has been irregular in different
regions. One in eight people still do not have access to
clean drinking water. This equates to almost a billion
people, of which 37% live in Sub-Saharan Africa (WHO,
2010). Water-related diseases kill thousands of people
each day. Additionally, water sources in some remote

areas can be a significant distance from habitation,
requiring women or children to walk carrying heavy
water vessels every day.
To contend with this major issue IDEO designed a tool
to secure clean drinking water. The Aquaduct is a pedalpowered vehicle that transports, filters, and stores
water, and addresses the two principal water challenges
in the developing world: cleanliness and transportation.
The Aquaduct is designed to enable a person to sanitize
and transport water simultaneously, potentially lessening
the physical strain of the task and freeing up more time
for work, education, or family. As the bike is pedaled,
a pump attached draws water from a holding tank, and
passes it through a filter, to a smaller, clean tank. It can
filter while the bike is in motion or when stationary.
The clean tank is suitable for home storage and use.
Presently it is not a viable solution due to production
costs and resilience, but it does raise awareness around
the water issues in developing countries and, as such,
demonstrates how social relations can be improved by
giving access to basic needs. The women and children
who have to walk considerable distances to get the
water could potentially be able to gain paid employment
or attend school as a result of this invention, allowing
them an equality of opportunity with others who have
easier access to water sources.
This project focused on an issue that involved a similar
position across many communities, that being a
location with inadequate access to potable water.
The barrier of insufficient resources is still being worked
through to create the opportunity of full participation
of individuals in society, be it in children being able to
attend more school or through improved community
health.
4.3 Access to Full Community Engagement
projectOPEN. This project from the University of
California Los Angeles (UCLA) focused on the issue
of homelessness in Santa Monica, California, seeking
to empower the homeless community by removing
the barriers it faced to accessing important knowledge
and amenities. The research group considered the
various causes of homelessness and met with over forty
associated organizations. The aim was to set up a robust
105
network among these organizations and facilitate better

communication between them, while ensuring that the
homeless people were fully educated on their rights and
what resources were available to them.
The outcome was a services resource guide called
project OPEN that opened out into a large map that
identified valuable services, such as doctors offices,
shelters, social services and food banks with information
such as opening hours, contact info etc. On the back
of the poster there was a reference to legal rights and
safety tips. What is particularly interesting about this
example is that the Santa Monicas city government
withdrew their offer to pay for and distribute the map
when they discovered that the legal rights were printed
on the back of the resource. It was they feared that this
would increase levels of homelessness. Ultimately the
map was distributed through a different printer. This
points to the importance of keeping the design process
transparent, meeting with the community, clients, and
funders frequently to keep everyone on the same page.
[Shea, 2012, 67] It is also a display of how uneven social
relations can prevent access to knowledge that can
remove barriers to full participation.
The success of the initiative has seen it expand to
include other cities, including the development of
making a tri-lingual map in English, Spanish and Chinese.
This is a good example of how social relations design
can operate over time, building on an initial venture
to develop into a larger project with a greater impact
potential. It is useful when discussing scale to consider
time and the sustainability of projects. Social relations
design should not focus solely on one-off projects but
aim for long-term commitments, which this project
has the opportunity to achieve. Overall, this project
demonstrates how the process model in Figure 1 can
assist social designers in identifying the key mechanisms
in a situation of inequality. It clearly shows how the
barriers of prejudice and power were able maintain the
disadvantaged position of the homeless community,
preventing full participation.
5 Conclusions
The notion of social design is both vibrant and complex.
It is characterized by a no fear of failure type of
exploration and disruption, thorough observation and
106
preparation, and a participatory, empathetic approach

to co-creation. It is place-based and is committed to
sustainability and positive change. This paper has made
no notable attempt to define the concept but has
rather suggested that a new social design focus might
be capable of producing even more positive, effective
and efficient social outcomes. This new focus has been
called social relations design and seeks to disrupt the
process of inequality by focusing on uneven social
relations in various contexts. The process of inequality,
as understood through a conceptual process model,
operates within a complex whole of externalities and
internal relations and consequently requires a design
strategy that can work at different scales simultaneously.
It has been argued here that a social relations design
is capable of taking on this challenge and therefore
causing significant disruption to on-going disparity
traps and processes.
The examples used, loosely categorized into access
to services, environment and engagement, illustrate
the potential for an approach that is place-based
and focused on the specific relations and issues of a
context. Individuals and groups in communities, and
communities themselves, often struggle in a network
of uneven, dependent social relations and design
thinking has significant potential to remove one or
two of these barriers to create greater participation
and equality of opportunity.
The main challenges for social designers interested
in this focus are working with a community to isolate
particular uneven social relations within a context
and doing this in such a way that is non-invasive or
directional. This requires considerable preparation and
observation to map a matrix of social relations in the
community before discussing which of the relationships
can be considered dependent or uneven, and therefore
contributing to inequality. Once the community
or community group has identified specific social
relations, then the design space has been created and
is ready for the initial stages of a co-design project.
It is vital that the design space has been selected by
the community. This methodology, although not fully
developed in the article, is recognized here as having
significant potential for further discussion within a social
relations design context.
The issue of scale has been raised in this article,

notably regarding the simultaneous operation of
external and internal influences. However, it was also
briefly suggested that a temporal scale also needs to
be considered. This refers to the need for a sustained
and sustainable commitment to the project. Inequality,
as discussed, is complex and persistent and it requires
a concerted effort to disrupt its processes. Design is
capable of doing this but not through one-off, band-aid
projects. Social relations design needs to be proactive,
flexible and committed as communities are dynamic
entities, vulnerable to external pressures. And more
than anything a social relation focus approach needs
to be unshakably contextual and designed to scale.
Kim, Jin Yong . 2012. Clinton Global Initiatives Annual Meeting,

23-25 September 2012, New York. http://www.smartplanet.
com/blog/report/design-a-key-strategy-for-large-scale-socialinnovation/121 [Accessed 28 June 2013]
Marx, Karl. n.d. The Poverty of Philosophy. International
Publishers: New York, USA
Sanders, E.B.N and Jan Stappers, P. (2008) Co-creation and the
new landscapes of design Co-design: International Journal of
CoCreation in Design and the Arts. 4 (1)
Shea, Andrew. 2012. Designing for Social Change: Strategies for
Community-Based Graphic Design. Princeton Architectural
Press: New York
Simmons, Christopher. 2011. Just Design: Socially Conscious
Design for Critical Causes. How Books: Ohio, USA
United Nations. 2005. The Inequality Predicament: Report on
the World Social Situation. New York. Sales No. E.05.IV.5
References
Berman, David. 2009. Do Good Design: How designers can
change the world. New Riders: Berkeley, USA
Brown, Tim. 2009. Change by Design: How Design Thinking
Wilkinson, Richard and Kate Pickett. 2009. The Spirit Level:

Why more equal societies almost always do better. London:
Penguin
World Health Organisation. 2010. Progress on Sanitation
Transforms Organization and Inspires Innovation.
and Drinking Water. WHO/UNICEF Joint Monitoring
HarperCollins: New York
Programme for Water Supply and Sanitation. Available at
Burkitt, Ingrid. 2012. What is Social Design? The Centre for
www.wssinfo.org/ [Accessed 28 June 2013]
Social Impact, [internet] 11 January 2012. Available at: https://

secure.csi.edu.au/site/Home/Blog.aspx?defaultblog=https://
blog.csi.edu.au/2012/01/what-is-social-design/ [Accessed 14
June 2013]
Cardoso, Fernando Henrique. 1979. Dependency and
Development in Latin America. Berkeley: University of
California Press
de Ferranti, David, Guillermo E. Perry, Francisco H. G. Ferreira
and Michael Walton. 2004. Inequality in Latin America:
Breaking with History? Washington DC: The International
Bank for Reconstruction and Development/ The World Bank
DiSalvo, C. et al. (2011) The Collective Articulation of Issues
as Design Practice Co-design: International Journal of
CoCreation in Design and the Arts. 7 (3/4)
Escobar, Arturo. 2013. Notes on the Ontology of Design [work
in progress]
Faud-Luke, Alistair. 2009. Design Activism: Beautiful Strangeness
for a Sustainable World. Earthscan: London, UK.
Greig, Alastair, David Hulme and Mark Turner. 2007. Challenging
Global Inequality: Development Theory and Practice in the
21st Century. Hampshire / New York: Palgrave MacMillan
Harris, Vandra. 2008. Mediators or Partners? Practitioner
Perspectives or Partnership in Development Practice, Vol.
18, No.6 (Nov.,2008), pp. 701 - 712
107
Sophie Brenny and Jun Hu

Department of Industrial Design, Eindhoven University of Technology
s.m.j.brenny@student.tue.nl, j.hu@tue.nl
Social Connectedness and Inclusion by

Digital Augmentation in Public Spaces
Abstract
With several design iterations we explored the
ways of increasing peoples feeling of inclusion and
connectedness in a public space using interactive
public art installations as media. The study used the
Social Connectedness Revised Scale and the pictorial
Inclusion of Community in Self as measurement tools
and a university cafeteria as the context. Experiments
were carried out to investigate the possible difference
among several types of interaction elements, examining
the influence on peoples perceived sense of inclusion
and connectedness to others who were present in the
same space. Subsequent interviews were conducted
to support the findings of the questionnaires and to
shed light on some of the results. The results showed
a notable difference when the physical space was
augmented with interactive digital content.
Keywords
Social connectedness, social inclusion, public spaces,
interactive public arts
1 Introduction
Cities are evolving at a rapid pace. A problem of this
rapid evolution is that people may feel less and less
connected to their city as they may find it challenging
to keep up with all the changes. In public spaces
people usually do not spend much time. A way to
108
engage people in interacting more with their city

is through public art installations where onlookers
change their participative roles from spectator to actor
by influencing the art piece in their own way [1-4].
Improving quality of life are issues that many growing
cities must deal with. People want a better quality of
life. To increase the quality, cities are turning in part
to strategies to heighten the sense of inclusion and
connectedness of their citizens. Making spaces inclusive
instead of exclusive is a trend that is growing worldwide,
particularly in city planning for newly evolving Chinese
cities [5, 6]. The design context for the concept is public
spaces across a city. On these locations a network of
the installation Leave your mark would be placed
to allow people to freely express themselves on a
blackboard. This would enable them to augment the
public space in which they find themselves with a digital
added element connecting several locations of the city
together. This approach is to help people feel more
connected with and included in the space they are in.
We started the project by asking ourselves the question
how to design a public digital art installation with digital
augmentation to physical spaces to increase peoples
feeling of inclusion and connectedness? To answer this
question, taking into account our initial concept of leaving
their marks, the first step was to determine whether,
if people were given tools in the form of markers and
papers in a public space, they would use them. If this
was not the case, the whole project could be stopped at

that stage. From this question we derived a hypothesis:
H1: When giving people a designated place to express
themselves with tools to do so (pen and paper for
example), they will not do so.
If H1 can be rejected, we could take the concept
further. Buildings in a public space consist of a mix of
geometric shapes. When creating an interactive public
art installation system, geometric shapes are thus truly
important and should be taken into account to fit the
surroundings. A few more hypotheses were then derived:
H2: People will not use the geometric shapes for their
forms of expression.
H3: People will not use the geometric shapes for their
forms of expression when hints to this option are given.
H4: Even when giving people hints to the possibility of
using geometric shapes, their sense of connectedness
is not different from when only using a plain blackboard
with nothing added.
We introduced a blackboard in later experiments
for people to leave their marks. A blackboard is a
medium that has a low threshold, and that everyone
knows how to interact with. The blackboard was
thus made to explore if such types of expressions
do, or do not, influence peoples feelings of inclusion
and connectedness. The digital element that will be

discussed later in this paper is an addition to this
exploration. To study peoples sense of connectedness
and inclusion with the blackboard, four more
hypotheses were proposed:
H5: Peoples sense of connectedness to the space and
to others in the same space will not be increased by
letting them express themselves in a designed way with
the blackboard in the public space.
H6: Projecting a feed from one blackboard on another,
allowing people to see others drawing, etc. will not
influence their sense of connectedness.
H7: Letting people express themselves in a controlled
manner with the designed blackboard in the public space
will not influence peoples sense of inclusion.
H8: Projecting a feed from one blackboard to another,
allowing people to see others drawing will have no
influence on their feeling of inclusion.
We will first introduce some related existing concepts
and will then describe the proposed concept. This
will be followed by the initial explorations conducted
in line with the above mentioned research question
and hypotheses. The tools used will subsequently
be explained, followed by an explanation of the final
experiment, including the setup, participants and
procedure. The results will then be presented and
discussed and finalized with conclusions.
2 Related work
Many people have seen graffiti put on a wall or building
or even very elaborate chalk drawings on the ground of
a public space in their city. People expressing themselves
in creative ways in the public space can thus been seen
as an event that has some history. But next to these
unorganized, often illegal forms of expressions there are
also designs or organized events that give individuals in
the space the chance to express their individuality and
share their thoughts with their community. Flash mobs
are such an example.
Fig. 1. (a) a public art installation in Cape Town, South Africa [7];
(b) the Obliteration room [8]; (c) Community Chalkboard [9];
(d) Before I Die [10]
Infecting the city is a small festival in Cape Town,

South Africa (Fig. 1(a)). Initiated by the Africa Centre,
the festivals goal is an endeavor to kindle the inter
connectedness of the people in the city through artistic
expressions, as they say making the public space
public [7].
109
Japanese artist Yayoi Kusama created the Obliteration

room (Fig. 1(b)). Her white living room is personalized
by every visitor to the space who is provided with
a colored sticker dot that they can choose to put
wherever they wish in the space. This means that
every dot is the expression of the visitors experience
in that space [8].
Community Chalkboards created by the Company Site
works in Charlottesville Virginia (Fig. 1(c)), in the United
States consists of a public chalkboard. It represents a
memorial for the first amendment where anyone can
share their thoughts and opinions [9].
Candy Chang started the Before I Die project after
losing a loved one (Fig. 1(d)). Before I Die started in
New Orleans. It is a chalk board with a grid created
with the start of the sentence Before I Die... with
room for people to fill in the rest of the sentence.
Anyone who walks past this chalkboard can pick up
a piece of chalk and write down what they want to do
before dying. The inspiring project has since expanded
to 189 other cities across the globe [10].
Research that studies the effects of intrinsic and
extrinsic motivation, in some cases is closely related
to the topic at hand [11, 12]. Research conducted in the
field of inclusion and connectedness has mostly been
conducted on a purely digital or purely physical field.
Such research used tools like blogs and mobile phones
for the concepts. An example is the work by Shuk Ying
Ho who studied the effects of location personalization
on individuals intention to use mobile services [13].
This research differentiates itself by combining the
digital and the physical while using tools for measuring
inclusion and connectedness for public art installations
in a public space. Next the concept Leave you mark
is briefly described
3 Leave your mark

People were provided with an opportunity to interact in
a playful way by drawing and leaving their mark behind
on the public space (Fig. 2). The opportunity gave them
a chance to express themselves by playfully triggering
and intrinsically motivating them. The concept involves
projection mapping and the use of digitally augmented
110
Fig. 2. Blackboard concept
blackboards to create a public art growing system in a

city. The goal of the concept is to increase feelings of
inclusion and connectedness of the citizens of the city
to each other and to the public space they are in.
Aspect 1:
Imagine, a blackboard situated on a building in a city.
A person walks by, grabs a piece of chalk and starts
drawing or writing on it, leaving their mark. The
blackboard projects the drawing or written words of
those who left their mark onto the building, in other
words augmenting the physical world in a digital way.
Aspect 2:
A more connected aspect was also added to the
concept. In some locations the blackboard will be
provided with a camera. The feed of this camera will
be projected onto a blackboard in another location in
the city. If a person walks by this secondary location,
they could possibly see someone, a complete stranger,
leaving his or her mark on that first blackboard. All of
this is carried out using video feed. This creates the
result that people see individuals in another location
walking by or sharing on the blackboard of this other
location where the first individual is not actually him
or herself.
Positive
Neutral
Negative Total
Drawing
27.2%
31.6%
2.6%
61.4%
Text
18.4%
13.2%
7.0%
38.6%
Total
45.6%
44.8%
9.6%
100%
Table 1. Percentage of elements shared
something in a public space or not (H1). An initial

exploration was carried out for this purpose. Eight
pieces of A0 papers were hung in various spaces at
Eindhoven University of Technology (Fig. 3 (a)). Each
large paper had 1 or 2 colored markers attached
alongside the papers.
(a)
As a trigger, each paper also had the words: Leave your

mark, draw, write, do whatever you like. They were
left hanging for 1 week on average.
(b)
Fig. 3. Exploration results
4 Explorations
4.1 Exploration 1
The first step into the development of the concept
was to see if people would be triggered to share
Using the cut out shape
itself
Using the inside of the cut out

shape
Positive Neutral Negative Total
Positive Neutral Negative Total
Drawing 22.9%
25.7%
0%
48.5%
15.8%
7.2%
0%
22.9%
Text
7.2%
7.2%
1.4%
15.8%
7.2%
5.7%
0%
12.9%
Total
30%
32.9%
1.4%
64.3%
22.8%
12.9%
0%
35.87%
64.3%
percent of total
percent of total
35.7%
Combined total
70
Combined percentage
100%
Table 2. Elements shared using the outside of the shape or the inside of the shape
In total 144 things were shared (Table 1). Seventy of

these were drawings and 44 were texts. We could
speculate from this that people are more inclined to
draw than write. This meant that the final concept
should be open to both possibilities. 37 of these (32.5%)
were on the papers put in the auditorium close to the
cafeteria of the auditorium in the university. Partially
due to this result, this location would later be chosen
for the subsequent tests as it was the most elaborated
paper. Daily visits to each location showed that once
the high threshold of the first drawing or two was
shared, the threshold seemed to be lowered as more
people shared in a shorter time span. This indicated
that, given time, people would be triggered to partici
pate through the participation of others.
The first paper and pen based exploration showed
that people appeared to be triggered by the presence
of paper and pens to draw or write something on the
sheets of paper hung around campus. Public spaces are
constrained by the shapes and forms of the buildings in
that space. This was seen as an interesting opportunity
to use the final concept to augment the physical space.
But if constraints are given, such as paper with cut out
shapes, would individuals use them?
4.2 Exploration 2
The same format was used as for the first exploration.
111
8 papers with pens were hung in the same 8 locations.

The difference was that these papers had cut out geometric shapes in them (Fig. 3(b)). The idea behind the
geometric shapes is that they are representative of the
windows and doors of a building.
The total number of shared marks was 70 this time
(20 texts and 50 drawings). The number of drawings
that ended up using the inside of the cut out shapes
was of 45 (35.7%). Peoples use of the cut out shapes
was 25 (64.3%). See Table 2.
This allows for speculation that people can be triggered
to use geometric shapes for the use of their drawings
and shared content. Though because of the nature of
the prototype used in the final experiment, however,
individuals will not be able to draw in the holes.
5 Experiment
The cafeteria of the auditorium at the Eindhoven
University of Technology was chosen as the location
for the experiment. It was the location with the most
results during the pen and paper exploration. It is a
social interaction location that resembles the squares in
cities with restaurants and cafes in interaction, attitude
and frame of mind.
Depending on courses and days of the week, it is
uncommon for the same group of people to pass
through the cafeteria of the auditorium every day.
Enough participants and the same profile of them
allowed for both between group tests that has less
learning effect.
The tests were conducted over lunch hours, between
11.30 AM and 1.45PM on Tuesdays and Fridays.
Depending on the tests, the tools used were the paper
questionnaires (SCS-R, ICS), the handmade blackboards
and chalk (Fig. 1(a)); a projector and a pre-filmed movie
of people drawings on the blackboard on a computer
(Fig. 4(b)).
5.1 Instruments
Social Connectedness Scale Revised
The Social Connectedness Scale Revised (SCS-R) scale
[14] was used. This 20-declaration scale asks people
to grade from 1 to 6 depending on whether they
112
Fig. 4a + 4b. Experiment settings
respectively strongly agree or strongly disagree with the

statement made. High internal reliability of the scale at a
degree of a=0.86-0.89 across the scale gives us enough
confidence in using this scale in the experiment.
Inclusion of Community in Self scale
The pictorial Inclusion of Community in Self (ICS)
scale [15] was also used in the experiment. The
scale is composed of 6 pictorial representations of
two circles (one representing community and the
other representing the self). Each of the pictorial
representations varies from its neighbor by increasing
the intersection surface. An increase in this intersection
shows a closer sense of inclusion to the city and people
in their environment.
Interviews
Interviews were conducted on dates and times that
were suitable for the interviewees. Interviewees were
either interviewed at their workspace or in the audi
torium cafeteria depending on what was more suitable
for the interviewees. Interviews lasted an average of
thirty minutes and varied from fifteen to forty-five
minutes. Ten people in total were interviewed, two from
each test. The purpose of the interviews was to gather
valuable, qualitative information that might go unsaid

through the self-examinations tests conducted first.
5.2 Participants
100 students participated in the experiment (20 partici
pants for each test). They were from varied ethnic
backgrounds and level of education accomplished at the
time of the test (Bachelors students, Masters Students,
PhD students) and varied in age between 18 and 33.
The average age was 22.31. Of the 100 participants,
61 were men and 39 were women.
context for the results of the follow up data. The

control tests were conducted on Tuesday May 14th
2013 and Friday May 18th 2013. The 20 participating
people were asked to only fill in the same questionnaire
that included the SCR-scale and the ICS-scale.
Test A
Test A was conducted on May 22th 2013. Two
blackboards were placed on opposite sides of the
vending machine close to the tunnel leading to the main
building of the cafeteria of the auditorium (Fig. 5).
5.3 Procedure
After each test, they were personally asked to fill in the
questionnaire that included the above-mentioned scales.
Participants were informed about the purpose of the
form. The form requested participants to fill in their
personal email address if they agreed to be contacted
for further interviewing. No reward of any kind was
provided to the test participants.
For every test group, two people would be selected for
interviews in order to gather qualitatively useful data,
one with a higher score on the Social Connectedness
Revised scale and one with a lower final score.
5.4 Data analysis
The results from the scales used in the control tests,
test A, B and C (to be described in detail later in this
section) were evaluated according to their described
methodologies, calculating both scale and item
mediums as well as the standard deviations for each.
To determine whether the difference in means is
statistically significant, a one tailed ANOVA test was
conducted. The analysis is a one-way because there are
more than 3 groups (tests) with participants that are
in no way related to each other aside from visiting the
same location. Bonferroni is chosen for these reasons
as the post-hoc settings for the one-way ANOVA test.
Bonferroni test is also suitable when small numbers of
pairwise comparisons are being analyzed.
5.5 Test settings and conditions
Two control tests
Two control tests were conducted. These control tests
were carried out in order to have a valid baseline with
respect to the level of inclusion and connectedness
within the campus in the cafeteria of the auditorium
during lunchtime. This allowed for a comparable control
Fig. 5. Test A setting
Both blackboards were identical and were provided

with colored chalk. The words Leave your mark were
printed and hung above the blackboards. People who
chose to come and share something were asked to fill
in the same questionnaire that the individuals in the
control test had filled in. Because most participants
ended up only seeing and experiencing one of the two
blackboards, the second one was found to be excessive
for subsequent tests.
Fig. 6. Test B setting

113
Test B
Test B setting is similar to Test A, but with an added
digital aspect. It was conducted on Tuesday the 28th
of May 2013. A previously filmed 2-hour footage of
people occasionally coming to draw something was
projected on the board (Fig. 6). People who chose to
come and share something were asked to fill in the
same questionnaire that the individuals in the control
test and test A filled in.
5.6 Results
Observations about the use of the geometric shapes
In test A, 2 people used the geometric shapes as basis
or inspiration for their drawings. In test B, 6 people
used the geometric shapes as basis or inspiration for
their drawings. In test C, 15 people used the geometric
shapes as basis or inspiration for their drawings.
Examples are shown in Fig. 8.
Social Connectedness Scale Revised
There was no statistical significance between the two
control tests. This shows the reliability of the scale used
in the context. Consequently, the results of the two
control tests were merged into one.
Fig. 7. Test C setting
Test C
The final test was carried out on Friday the 31rst of May
2013. Similar to test B, a projector was used once more
in this setting. In this test, however, hints to the use
of the geometric shapes were projected. That is, how
people could use the shapes to incorporate them into
their drawings instead of drawing around them (Fig. 7).
This test would show whether or not it would be
necessary, or even useful, to have this constraint of
geometric shapes based on the public space in the final
real life version of the concept. As already stated the
geometric shapes are representative of windows and
doors of a building. This final test was thus to fine-tune
the concept and to see if people would use these shapes
or not.
Fig. 8. (a) Using the shape as basis; (b) Using the shape as
inspiration
114
The mean result for the control test was 54.90 with a
standard deviation of 7.78 and a standard error of 1.23.
This compared to the total mean of test A that was of
60.40 with a standard deviation of 6.35 and a standard
error of 1.42. The mean result for the test B on the SCR
scale was 68.75. The standard deviation is 6.53. There is
a standard error of 1.46. Test C had an average of 60.75 with
a standard deviation of 7.29 and an error margin of 1.63.
Table 3. Bonferroni SPSS results of the SCS-R scale across the tests
The statistical significant variances between the control

tests, test A, B and C (Table 3) show a significant
improvement in SCS-R scores (p=0.037, 0.000, 0.022
respectively). The result also shows significant increase
in SCS-R scores of B over A (p = 0.002) and B over
C (p=0.002). There is no significant difference found
between A and C.
Inclusion of Community in Self scale
The data of the control test was a mean of 3.90 with a
standard deviation of 0.84 and a standard error margin
of 0.13. Test A had a mean of 4.50 with a standard
deviation of 0.60 and a standard error margin of 0.14.
For test B, the mean was of 4.85 with a standard
Table 4. Bonferroni SPSS results of the ICS scale across the tests
deviation of 0.67 and a standard error margin of 0.15.

Test C had an average of 4.55 with a standard deviation
of 0.83 and a standard Error of 0.18.
Based on the one-way ANOVA calculation conducted
through SPSS the statistical significance of this variation in
mean between the control tests and the other tests (A, B
and C was significant (p=0.031 0.000, 0.015 respectively).
When looking at the scores in the ICS scale of B over
A, B over C, and A over C, there was no significant
difference found.
Fig. 9. Number of people using the shapes for their drawings per test
rejected. This is because, without hints, even though

people used the shapes, only very few people did so.
H3: People will not use the geometric shapes for their forms
of expression when hints to this option are given.
For test C, there was a hint projected on the black
boards for people. In this case more individuals used
the geometric shapes as inspiration or for part of their
drawings in test C than when compared to tests A and
B. In the case of test B more than double the number of
people did so and, when compared to test A (Fig. 9).
6 Discussion
The study presented is exploratory and aimed at
answering the research question which emerged
during the early project phase. The research question
was decomposed into smaller ones with which the
hypotheses were set.
The tests were conducted in a more controlled manner
in a smaller environment than the city that the concept
is aimed at. It should be stated that the validity of the
tests is limited by the experimental environment. The
following discussions are based on the assumption that
the results concluded from these experiments would be
also valid for the city environment.
6.1 Use of geometric shapes
H2: People will not use the geometric shapes for their forms
of expression.
The results of both the second experiment and of the
final experiment show that when there was no real hint
to use the geometric shapes a few individuals did use
them as basis or inspiration for their drawings. Based
on only these observations, hypothesis 2 cannot be fully
Observing the statistical analysis based on SCS-R

and ICS when comparing test C to A and B, one
may conclude that providing hints on drawing along
the geometric shapes does not contribute to the
participants feeling of social connectedness and
inclusion. In Fig. 9, however, it is also evident that the
participants did follow the hints and geometric shapes
that were taken into account by others when interacting
with the blackboard.
When looking at the proportions, it can be inferred that
H3 is negated as more people did end up using these
shapes. Based on these conditions, it can be stated that
there is a relevance to keeping the use of the geometric
shapes in the final version of the concept.
H4: Even when giving people hints to the possibility of
using geometric shapes, their sense of connectedness is
not different from when only using a plain blackboard with
nothing added.
Based on the results of test A over C for the SCS-R
scale, it can be supposed that adding the hint only adds
115
to people using the geometric shapes as stated earlier in

this discussion but not the feeling of connectedness. In
other words, it appears that the individuals in test C and
A share a similar sense of connectedness and inclusion.
This can mean that H4 can be accepted.
Between B and C: The significant difference for B over

C shows that interacting with other people in real
time with digital augmentation has a stronger effect
than when digital augmentation is only about the hints
regarding geometric shapes.
6.2 SCS-R scale

When comparing the mean of the tests, it appears that
there is a strong sense of connectedness increase when
comparing the control test to the other tests.
H6: Projecting a feed from one blackboard on another,

allowing people to see others drawing, etc. will not influence
their sense of connectedness.
When comparing B to A and B to C, the results allow
speculation that H6 can be rejected. The influence of
the added digital feed seems to have had a very big
influence on peoples feeling of connectedness. This
result also confirms the result from other studies [3].
This aspect was further explored more closely in the
interviews.
The results show a significant statistical difference

between the control test and test A. This could mean
that people had a higher sense of connectedness in the
space with other people in the same space because of
the presence of the blackboard and their interaction
with it. The similar significant difference is also found
when comparing Test B and Test C to the control test.
As the Hawthorne effect [16] suggests, any interference
or change to an environment of the participant that
gets their common attention will have an impact on
their feeling of connectedness. We have to agree that
it is not entirely clear whether it was the design of the
blackboard that made a difference. Test A, however,
shows that a difference can be made; moreover it is
used as a reference for test B and C in which we pay
more attention to digital augmentation. We may also
have the similar discussions when comparing Test B
and C to the control test.
H5: Peoples sense of connectedness to the space - and to
others in the same space - will not be increased by letting
them express themselves in a designed way with the
blackboard in the public space.
The results from the various tests SCS-R numbers
contradict H5. It can be interpreted to mean that the
H5 is voided by these results, seemingly showing that
there is an increase in the feeling of connectedness
of people when letting them express themselves in a
public space. This finding led the researcher to ponder
whether there are differences in the results that are
significant between test A, B and C.
Between A and B: B appears to have a significant
difference over A. Interacting with other people in real
time with digital augmentation seems to have a strong
impact on social connectedness.
116
The results show that there is no significant difference

between A and C. It would seem that the digital
augmentation does not have an influence on the feeling
of social connectedness. This suggests that, in these
cases, the physical interaction with the environment
can be replaced with well-designed digital interaction,
without harming the participants feeling of social
connectedness.
One could suggest that this lack of difference between
tests A to C as opposed to the presence of the difference
between the control test and the other tests is due to
the Hawthorne effect [16]. This aspect would need to
be examined more closely in a future experiment.
6.3 ICS scale
H7: Letting people express themselves in a controlled
manner with the designed blackboard in the public space
will not influence peoples sense of inclusion.
When comparing test A, B and C to the results of the
control test, the participants in these tests appear
to have felt more included in the community than
those in the control tests. This means that H7 can
be, statistically speaking, disproven. It can thus be
postulated that through the act of participating in
the test in any way, even just by leaving their mark,
people felt an increase in their sense of inclusion in
the community.
H8: Projecting a feed from one blackboard to another,
allowing people to see others drawing will have no influence
on their feeling of inclusion.
When comparing the results of test A to those of test

B, the difference is not significant. The same applies for
the results of test A and C. Furthermore, this equally
applies for the comparison of test B and C. This shows
that H8 was not rejected.
control group interviewees were all similar as well: As

I said I felt like part of the blackboard system. I did not
feel excluded. We seemed to really be working together
to create something even though we did not know each
other. This was a sentiment echoed by all.
Due to the difference present in the SCS-R scale but

lacking in the results of ICS scale, it could be suspected
that the sensibility and validity of the ICS scale could be
brought into question as a tool set for such tests.
Participants in the interviews from test A and C said

that they felt more included when other people had
extended their own drawings. The revisiting fact, or
seeing the final results from the test, influenced how
people felt afterwards. Here it can be asked whether
seeing ones work elaborated on during the next visit
to the public space influences the feeling of inclusion.
It could also be interesting to explore whether the
beamed feed of other people drawing or if the
extensions of a drawing by others is influential on
people feelings of inclusion.
6.4 Interviews
When asked to describe a moment where they felt
particularly included, 9 out of the 10 people interviewed
described stories where they got to know strangers
due to either unexpected events (for example in the
Netherlands, something going wrong with a train) or
due to the setting (for instance a festival). From this
information it would appear that the event of connecting
to people one did not know beforehand really influences
their feelings of inclusion.
In the cases where someone else had expanded the
drawing of the interviewee, responses were strong and
similar. As one person said, I like it! I like the shape!
It means, I feel like I made something for someone. I
wanted to keep going, I was being helpful to someone
else. People felt they helped each other and like they
became part of the systems by collaborating.
When people saw others draw, both in real and with
projected ones, the responses were as follows: In a way
we were making something together. If they were drawing
on the same board at the same time it influenced me
more. We were doing it together and not by myself. I felt
like we were drawing together. When I drew I saw cat
whiskers already drawn on the board. They were inviting
and felt open to me. I also saw a smiley... I felt open and
happy, the drawings made me laugh. I drew something
and it was funny because suddenly a persona appeared
on the beamer and started drawing something similar
to my drawing. That made me feel happy. The existing
drawing influenced people by apparently lowering the
threshold and inviting others to participate. They stated
that they felt happy when seeing others draw.
When asked how they felt seeing their drawing having
become part of a system the response of the 6 non-
Intrinsic motivations are founded in our innate human

nature. Our ingrained features push humans to be
driven by curiosity and to participate in social activities
[12]. It became apparent through the participants
reactions in the interviews that they were triggered
by their curiosity and their natural intrinsic motivation
to participate in this social activity. One individual, for
example, stated, I was thinking of what everyone else
put on the blackboard. (What they wrote or drew).
I looked at how what I left would compare to what
others put. It influenced the colors and size and location
I chose. But it didnt influence what I drew. I liked it.
7 Conclusions
The beauty of the concept Leave your mark lies in
its simplicity and the ease for people to step in and out
creating a reasonably low threshold. It seems to be an
appropriate solution to design a public art installation
with digital augmentation to the physical world to
increase people feeling of inclusion and connectedness.
In this specific case we can speculate from the answers
that the presence of a prototype of the concept
influenced peoples sense of connectedness and
inclusion. The lack of difference found between tests
A, B and C in the ICS scale is unlikely to be caused by
the Hawthorne effect. This is a point that should be
explored further. Specifically as it related to the results
of the Inclusion of Community in Self scale where no
117
significant differences were found between the tests.

Seeing there was such a significant difference in the
SCS_R scale, which was lacking in the ICS scale, this
could bring into question the validity and effectiveness
of the ICS scale or it is simply because the social
connectedness and inclusion are not coupled concepts.
8. Jobson, C., This is What Happens When You Give Thousands

of Stickers to Thousands of Kids. 2012: Retrieved June 6,
2013 from Colossal: Art and visual ingenuity: http://www.
thisiscolossal.com/2012/01/yayoi-kusama-obiliteration-room/.
9. Johnson, P., Things I Wish I Saw on the Playground:
Community Chalkboards. 2012: Retrieved June 6, 2013
from Play Scapes: http://www.play-scapes.com/play-design/
The results of the Social Connectedness Revised scale

however were more noticeable. Using the designed
concept Leave your mark as a form of public art
system could help citizens of that city feel more
connected. It can be concluded that when there was the
added digital feed of other people drawings, people felt
a much higher sense of connectedness. This was also
confirmed by the results of the interviews.
contemporary-design/things-i-wish-i-saw-on-the-playgroundcommunity-chalkboards/.
10. Chang, C., Before I Die. 2011: Retrieved June 6, 2013 from
Before I Die: http://beforeidie.cc/.
11. Teo, T.S., V.K. Lim, and R.Y. Lai, Intrinsic and extrinsic
motivation in Internet usage. Omega, 1999. 27(1): p. 25-37.
12. Ryan, R.M. and E.L. Deci, Intrinsic and extrinsic motivations:
Classic definitions and new directions. Contemporary
educational psychology, 2000. 25(1): p. 54-67.
Given that this was an exploratory test conducted in

controlled settings away from the real final set up, there
are limitations with regard to the results. It is important
to note that this test should be conducted again with
a real setup in a public space. This is necessary to
determine if the results will coincide with those found
here. This exploratory research project could be
seen as an interesting start to inspire future work and
explorations in public digital art installations with digital
augmentation to public spaces in cities.
13. Ho, S.Y., The effects of location personalization on individuals

intention to use mobile services. Decision Support Systems,
2012. 53(4): p. 802-812.
14. Lee, R.M., M. Draper, and S. Lee, Social connectedness,
dysfunctional interpersonal behaviors, and psychological
distress: Testing a mediator model. Journal of Counseling
Psychology, 2001. 48(3): p. 310.
15. Mashek, D., L.W. Cannaday, and J.P. Tangney, Inclusion of
community in self scale: A single item pictorial measure
of community connectedness. Journal of Community
Psychology, 2007. 35(2): p. 257-275.
16. Adair, J.G., The Hawthorne effect: A reconsideration of the
References
methodological artifact. Journal of applied psychology, 1984.
1. Hu, J., et al., Attractiveness of an Interactive Public Art
69(2): p. 334.
Installation, in Distributed, Ambient, and Pervasive

Interactions. 2013, Springer. p. 430-438.
2. Hu, J., et al., Paticipatory Public Media Arts for Social Creativity,
in Culture and Computing 2013. 2013: Kyoto, Japan.
3. Le, D., M. Funk, and J. Hu, Blobulous: Computers As Social
Actors, in Experiencing Interactivity in Public Spaces (EIPS),
CHI13. 2013: Paris. p. 62-66.
4. Wang, F., J. Hu, and M. Rauterberg, New Carriers, Media
and Forms of Public Digital Arts, in Culture and Computing.
2012, Springer-Verlag Berlin Heidelberg: Hangzhou, China. p.
83-93.
5. Mitchell, W.J., Designing the digital city, in Digital cities. 2000,
Springer. p. 1-6.
6. Vince, G. Chinas eco-cities: Sustainable urban living in Tianjin.
Future: Smart Planet (BBC), 3 May 2012.
7. Africa Centre, Public Arts Festival Cape Town City Center.
2013: Retrieved June 6, 2013 from Infecting the City: http://
www.infectingthecity.com/.
118
Loe Feijs
Department of Industrial Design, Eindhoven University of Technology, The Netherlands
l.m.g.feijs@tue.nl
Multi-tasking and Arduino:

Why and How?
Abstract
In this article I argue that it is important to develop
experiential prototypes which have multi-tasking
capabilities. At the same time I show that for embedded
prototype software based on the popular Arduino
platform this is not too difficult. The approach is
explained and illustrated using technical examples
practical and hands-on, down to the code level. At the
same time a few helpful notations for designing and
documenting the software are introduced and illustrated
by the same examples. Finally a few case studies of the
technical approach are listed.
Keywords
Prototyping, specification languages, parallelism,
multitasking, experiential, models
1 Introduction
In the word, in which we live, there is a lot of
parallelism. Around us many things happen and they
happen more or less simultaneously. Zooming in to
the detailed behavior and the internal working of the
objects around us we see more parallelism. If we zoom
out to our environment and consider what happens at
some distance even more parallelism becomes visible.
As human beings, we cope with this aspect of the
worlds complexity remarkably well. We are able to
perform multitasking (Fig. 1).
Fig. 1. Humans are good in multitasking: seeing, navigating,

communicating, feeling temperature etc.
So if we are designing artifacts which display some form

of intelligence, it is natural to assume that they can
perform multitasking as well. For ambient intelligent
environments, this is usually the case indeed, which is no
surprise when they are equipped with powerful sensors
such as cameras and data from mobile devices; these
are all connected to networks of servers which run in
the background with lots of parallelism anyhow. Modern
computers, including PCs, smart phones and Internet
servers are equipped with Operating Systems which
allow for many parallel processes on a single computer.
Even small stand-alone embedded processors can have
operating systems, usually marketed as RTOS (real time
operating system). Technically these are appropriate,
but somehow they are considered high-threshold for
programming quick experiential prototypes. For this
reason interaction designers and industrial designers
119
prefer systems such as MAX, Processing (for PC

and MAC platforms) and Arduino (for embedded
applications). Arduino does not come with an operating
system. It works with the same theatre metaphor as
the Processing environment which is meant for creative
work on PC. There are two functions: setup()
prepares the stage and loop() defines the actions.
Arduinos are used as control devices, in combination
with sensors in of a huge variety of experimental
art installations, experiential prototypes and
experimental research vehicles. See [1] and [2]. Some
of these systems sense light conditions, others detect
sounds, temperature differences, bodily parameters,
movements and so on. Their actuators include motors,
loudspeakers, lamps, lasers, fluid valves and servo
motors, voice coils, memory metals, just to name a few
of many options.
For many of these sensors and actuators there is a
software library and examples how to connect, read and
write these sensors. Usually setup() defines which
device is connected to which input or output pins. And
usually loop() contains a few read, write, and delay
actions. The actions depend on the inputs received so
a few if-else or switch-case statements are thrown in
and a few variables to keep track of what goes on in the
external world.
All this is well, but when combining a few sensors and
actuators the programs suddenly become much harder
to design. During the delay needed for one task, the
Arduino is unreceptive for the signals of other sensors.
All the nested if-else and switch case statements
soon blow up the loop() to a multi-page debugging
nightmare. Lowering the projects ambition level and
adjusting the planning to allow for more programming
and debugging time are typical decisions I witnessed in
student projects.
As a way out, some of the designers work with
interrupts, allowing external events to launch interrupt
service routines which briefly suspend the ongoing task.
No doubt this is a powerful technique, but it requires
a much deeper knowledge of the processor and its
hardware. Often the service routines are written in
assembly language and manipulate the processors
timer registers.
120
In this article I describe an approach to avoid these

difficulties - the only price being a bit of discipline to
adhere to an orderly structure. At the same time the
approach offers very nice opportunities to document
the software at a higher level than the code itself. In
particular, I propose State Transition Diagrams (STD,
sometimes called Finite State Machines) as a helpful
way of explaining and documenting the logic of decision
making (normally implicitly coded in if-else and switch
case). And I propose Data Flow Diagrams (DFD) as
a desirable way of giving a high-level overview of the
parallel tasks going on and how they communicate via
shared variables.
The notations of STD and DFD are not new:
they were developed in software engineering and
telecommunication industry more than two decades
ago. They are among the simplest and yet most powerful
notations of software engineering. Yet they appear not
to have been handed down with the other nice things
which were adopted in Arduino culture.
The next sections contain examples illustrating the
proposed principles. The nice property of the approach
is that it is compositional: the loop() does not become
exponentially more complex. A loop() with two
parallel tasks contains 2+1 statements. A loop() with
twenty parallel tasks contains 20+1 statements.
2 State Transition Diagrams

The example of this section serves mainly to introduce
the STD notation. The system is the type of light
encountered in the corridors of hotels: the guest pushes
a button and the light goes on. After ten seconds, the
light goes off automatically. The system therefore has
two states: ON and OFF, represented by the circles in
the diagram of Fig. 2.
Fig. 2. State
Transition Diagram
with two states.
One of these states is labeled to be the initial state.

The states themselves are an abstraction and the
text balloons tell about the resulting actions in the
real world. The transitions from one state to the next
are represented by arrows. Each arrow is labeled by a
condition or an external action which tells when the
transition happens. The transition from state OFF to
state ON happens when the hotel guest has pushed
the button. The transition from state ON to state
OFF happens when a timer set to 10,000ms goes off.
Typical experiential prototypes cope with experiences
ranging from a few milliseconds to many seconds or
minutes, which is why time is specified in ms throughout this article.
The STD is a high-level description of the behavior
which is helpful for communication and documentation
purposes. The same behavior will be coded in a
programming language, for example the C++ dialect
of Arduino, but this tends to be harder for stakeholders who are not familiar with such language.
This is the code:
void LEDstep()
int button;
button = digitalRead(buttonPin);
switch (state){
case OFF:
if (button == HIGH){
state = ON;
timer = 1000; //10000ms
analogWrite(ledPin,255);
} break;
case ON:
if (timer <= 0){
state = OFF;
analogWrite(ledPin,0);
} else timer--;
}
}
This LEDstep can be viewed as a process which runs
forever. When it writes 255 to the ledPin, the external
light will be switched on (assuming LED light). The step
is to be called every 10ms which guarantees that the
timing works right. The rest of the code now follows
the theatre metaphor.
void setup() {
pinMode(ledPin, OUTPUT);
pinMode(buttonPin, INPUT);
}
void loop() {
LEDstep();
delay(10);//ms
}
The state transition diagrams are not restricted to the
on and off states, but can be used for a wide variety of
configurations up to tens or even hundreds of states
(although I would not recommend hundreds because
then it would probably be difficult to understand). As an
example, consider a more sophisticated version of the
hotel corridor light in which the light does not simply
shut off at once, but dims smoothly during two seconds
(so the guest has a few seconds to find the light button
again). See Fig. 3. The implementation is done along the
same lines as before.
Fig. 3. State Transition Diagram with three states.
STDs have been used in engineering for a long time, see

for example [3]. Harel extended them with hierarchy
into a formalism called Statecharts [4], which now is
also part of UML [5].
3 Data Flow Diagrams

Now the example is extended by introducing a second
process. Consider a system where the light must go on
whenever there is a sudden change in the external light
condition. This could be used in an exhibition where
the visitor approaching an object on display would
create a shadow which would be detected to initiate a
121
demonstration sequence of the object on display. In this

case the demonstration sequence is just a LED going
on. A practical sensor for doing this is an LDR, a lightdependent resistor.
Of course it could be tried to include the statements
reading the LDR directly into the State Transition Dia
gram, but I argue that it is much better to have a clear
separation of concerns: there are two processes now:
The LDR process which reads the LDR and finds out
whether there is a sudden change in light intensitity;
The LED process which drives the external output, for
example the same LED as before.
Note that also the LDR process has one private variable,
called avg, for average. That will be explained later. Data
Flow Diagrams have been made popular by Yourdon [6]
Remark: computer scientists have invented powerful
alternatives for inter-process communication, such as
message queues, sockets and semaphores. Although
these are valuable and even indispensible for certain
complex engineering problems, the simplicity of shared
variables is a great asset during the development of
experiential prototypes and 4D sketching [7].
4 How to implement multitasking

Parallel processes is a powerful concept. It is essentially
multitasking. But when designing the software of
the prototype some communication between these
processes has to be provided. I like to use shared
variables for that.
In the Data Flow Diagram of Fig. 4 there are two
processes, each represented by a circle. So depending
on the type of diagram, a circle may be either a state
(in an STD) or a process (in an DFD). Here in Fig. 4,
it is a process. The variables are indicated by the boxlike structures. A variable is like a small storage bin
to memorize a number, a text string, or a truth value,
or even a whole sequence of those. Usually there is
information flowing back and forth between processes
and variables. The information flows are represented
by arrows. For example LEDstep writes its state into
a variable called state. And LEDstep also reads and
writes its timer. These could be called private variables
(of LEDstep). LDRstep signals the occurrence of an
event by putting the truthvalue true in a shared variable
called LDRevent. This event in turn is read by the LED
process.
Fig. 4. Data Flow Diagram with two processes and four variables.
122
Continuing the example I shall demonstrate how

these parallel processes are implemented effortlessly.
The initialization of the variables determines how the
multitasking system will begin its active life. Already
when the variable is declared in Arduino language it can
be given a begin value:
int state = OFF;
int timer = 0;
boolean LDRevent = false;
The LED process is still almost the same as the one of
Section 2. The only additions are about resetting the
event flag (i.e. the variable LDRevent). The implemen
tation follows the Arduino theatre metaphor again:
void setup() {
pinMode(ledPin, OUTPUT);
pinMode(ldrPin, INPUT);
avg = (float)analogRead(ldrPin);
}
void loop() {
LDRstep();
LEDstep();
delay(10);//ms
}
For all practical purposes, this gives us perfect
multitasking. To be honest, it is not really true
parallelism: technically it is so-called interleaving
parallelism. Considering the consecutive calls of delay
as an "idle process", the three processes alternate
quickly as follows: LDRstep is active during about 0.01
milliseconds (or perhaps even less), then LEDstep is

active during about 0.01 milliseconds (or perhaps even
less) and then the "idle process" takes 9.98 milliseconds.
And this goes on forever in an infinite loop. So why do I
call this "perfect multitasking for all practical purposes?"
This is because for the human observers of this type of
experiential prototype we cannot perceive such quick
switching between processes anyhow (like in a movie
theatre where 24 images per second give an impression
of continuous motion from individual images). Moreover,
our bodies are not fast enough to make movements which
would go undetected by the sensor process. The light
condition is checked every 10ms, which is fast enough.
Some readers may be worried by the delay statements
and the time wasted by the idle process, so I ought
to clarify my position about it: typical embedded
processors are fast enough to drive our experiential
prototypes and still have sufficient time left. For an
Arduino Uno, running at 16Mhz, the processor can
spend most of its time in an idle process. The repeated
calls of delay(10) constitute this idle process, which
runs in the background in parallel with the LEDstep()
process. In the early days of computing, and sometimes
even still today, idle processor time is considered to be
a valuable resource not to be wasted. Mathematicians
may use it to calculate digits of , for example [8].
In industrial design I am quite happy to waste the
processors time in the repeated application of a
delay(10) statement. Speed of 4D sketching and peace
of mind regarding programming problems are my first
priority. Things only become (seriously) more difficult
when an Arduino has to do audio or video processing.
5 Adaptivity
One of the most important reasons for wanting
experiential prototypes to do multitasking is that they
have to know what is going on in their environment.
Just like we, humans, a smart system, even a not so-verysmart artificial system needs to continuously monitor its
environment. This need becomes more urgent when
multiple modalities are involved: sensing light, sensing
movement, and so on. I shall illustrate this by an
example: detecting the presence of humans by an LDR.
This is a cheap alternative for camera-image analysis,
capacitive sensors, ultrasound distance radars etc.
The obvious idea is to choose a suitable threshold value
and test whether the voltage coming out of the LDR

goes below or above that threshold.
We humans have an amazing adaptivity with respect to
changing illumination levels: we can already see a bit at
0.002 lux on a moonless clear night sky with airglow and
already much more at 0.271.0 lux when there is a full
moon on a clear night. We work in 320500 lux office
lighting and amazingly we are not completely overloaded
and saturated at 32,000130,000 lux in direct sunlight
(source: [9]). Precisely because we are so good in
adapting, we hardly notice the huge differences around
us consciously. That why it is tempting to assume that
a fixed threshold in a technical prototype, though
not perfect, may be not such a bad idea after all. In
practice however, a fixed threshold works miserably and
unreliably unless we exert full control over the external
light conditions. But wouldnt it be great to equip the
experiential prototype with adaptivity?
In this section the earlier example of the LDR-based
event detection is worked out in more detail and
equipped with an elementary form of adaptivity. The
idea is simple: if I would know the average LDR voltage
level, then I could use this average as a threshold. The
LDR can be connected from +5V to pin A0 with a pulldown resistor of 22k to GND. The incoming voltages
should be averaged: that is why our process LDRstep
has a private variable called avg. It is a floating point
number, so it can be calculated with some precision (not
rounding to the nearest integer). This is how it works:
if we have 2000 values, V0, V1, V2, , V1998, V1999,
say then we all know how to calculate the average: AVG
= (V0 + V1 + V2 + + V1998 + V1999)/2000. Now it
is a bad idea trying to store 2000 sensor values inside
an embedded processor. An Arduino Uno has only 2k
RAM. But I can calculate it in steps keeping track of a
"current approximation" of the average.
void LDRstep(){
int v = analogRead(ldrPin);
avg = 0.9995 * avg
+ 0.0005 * (float)v;
float delta = 20.0;
float dif = (float)v - avg;
if (abs(dif) > delta)
LDRevent = true;
}
123
So in each step, the new average is mostly equal to

the old average, and adjusted a little bit by taking the
most recently read value into account. The ideas of
"mostly equal" and "a little bit" are made precise by
weighing factors of 0.9995 and 0.0005, which are found
as 1999/2000 and 1/2000, respectively. The effect is that
the more recent values contribute most and that very
old values are faded out. Technically, it is a so-called
exponentially weighted moving average. In this case
the time constant is 20s because this is 2000 times
10ms (the time between sensor samples). It has some
resemblance to the idea of a moving time window, but
formally it is more like a low-pass filter. And it is very
memory-friendly: this one floating point number average
occupies only 4 bytes.
summarize the architecture of this system by another

DFD in Fig. 6.
Fig. 6. Data Flow Diagram with three processes and five
Although the software is adaptive now, the hardware

is still rigid: one LDR can only give a Voltage range
sampled and analog-digital converted to a range of
0-1023 values, so in darkness the Arduino reads 0 or 1
and in full sunlight the LDR is saturated and the Arduino
reads a value near 1023. An alternative would be to have
a kind of shutter, like the pupil of our human eyes. After
all, for us humans, it is not only the brain which adapts
to varying lighting conditions, we do have special sensor
"hardware" (the pupil). The hardware for my Arduino
example consists of two voltage dividers with one LDR
each, but with different resistors.
variables.
The loop function of the system is still extremely

simple: just one more step added. The system now
has three processes and one idle process.
void loop() {
LDR0step();
LDR1step();
LEDstep();
delay(10);//ms
}
The two LDR processes are almost identical, except
for their input pins and private variables. There is no
extra complexity involved in adding this new parallel
task. Even if I would add ten more light sensors, ten
temperature sensors and ten moisture sensors, the
software would keep its simplicity (I would run out
of analog hardware pins long before running out of
software resources).
Fig. 5. Sensor circuitry for two light dependent resistors.
Now the multitasking capability developed in the

previous section comes in handy. Another copy of the
LDR process is all that is needed. Each process can
have its own average calculation and event detection.
So, when the first LDR is saturated, the other is still
sensitive. Or the other way around: when the second
LDR does not sense anything yet, the first is already
acting. Rather than giving more program code, I
124
Similar background tasks can make a system adaptive

with respect to temperature, fluid levels, a users heart
rate and so on. Also matters of movement in 3D space
give rise to background tasks such as calibration and
navigation. A systems current position and orientation
can be obtained by calculation from sensor data
by methods like dead reckoning (developed in ship
navigation when sailors had sextants and ship logs only
before John Harrison invented the chronometer).
6 Fine grained timing control

It is tempting to assume that the proposed approach
only works with processes and timers which are rather
slow, such as the 10 and 20 seconds of the examples
so far. Fortunately the approach can work with finegrained time control as well, without any problem. I
shall demonstrate how to control a servo motor, as task
which requires subtle timing in the sub-millisecond range.
As an intermezzo, let me explain how a servo motor
must be handled. A servo motor needs commands
which are repeated 50 times per second. Each
"command" has to be in the form of a pulse whose
width is around 1.5ms. A pulse of 0.8ms means "turn
completely left" and 2.2ms means "go to the 180
degrees right position". Obviously 1.5ms tells the servo
to go to the central position. In this way any position
between 0 and 180 degrees is possible.
I describe the software for a system with two servos.
It is a kind of medical-inspired artistic installation where
one servo moves as a simulated heart beat and the
other a simulated breathing. I show the DFD first in Fig. 7.
void breathServoStep(){
th = ;//between 800 and 2200
digitalWrite(servoPin,HIGH);
delayMicroseconds(th);
digitalWrite(servoPin,LOW);
}
void idleStep(){
delayMicroseconds(10000 - th);
delayMicroseconds(10000 - tb);
}
void loop(){
breathServoStep();
heartServoStep();
idleStep();
}
In this way I can even control the servo with an
accuracy of 0.1 degree (vs. 1.0 degree for standard
servo.h).
7 Outlook
variables.
My claim is that the proposed way of organizing

the embedded software in experiential prototypes
facilitates highly multitasking systems. Moreover these
systems are much more robust and reliable than
traditional naive approaches. For example the adaptivity
added in Section 4 eliminates the need for hidden
calibration potentiometers. It happened that prototypes
which worked fine in the design studio fail at the crucial
moment of the demo. I am sorry, but I swear, yesterday
it did work is a story I heard often. Too often. The
main reason (apart from unexcusable issues like loose
wires and empty batteries) is that the prototype is not
adaptive.
It is well known that the breath modulates the heart

rate, a phenomenon known as respiratory sinus
arrhythmia (RSA). This is modelled by the breath
variable between the breath servo process and the
heart servo process. Moreover the idle process has
access to the fine grained time information tb and th of
the servo commands. The code fragment shows how
this boils down to the implementation level is:
Nobody has the evil intention to lead unexperienced

design students into the dead-end street of programs
which are complicated, mono-tasking and unreliable.
Typical Arduino code examples are attractive and
low threshold. Yet there is danger. Let me clarify the
paradox by showing an example about a servo, which
comes with the usual Arduino environment (without
the comment):
Fig. 7. Data Flow Diagram with three processes and three
125
void loop()
{
for(pos = 0; pos < 180; pos += 1)
{
myservo.write(pos);
delay(15);
}
for(pos = 180; pos>=1; pos-=1)
{
myservo.write(pos);
delay(15);
}
}
In principle this is a good program. Yet there is an
opportunity for misery to begin when this loop is taken
as a template for an extension, include more sensors
and actuators. Soon the main loop may contain all the
delays, and a cocktail of nested if-else and switch-case
statements for a variety of multiplied state transition
diagrams. Despite a lot of apparent code complexity,
multitasking is still difficult: the Arduino does one thing
after the other. The approach outlined in this article
prevents the paradox from happening in the first place.
At the same time abstract thinking, precision and docu
mentation get better by the proposed STD and DFD.
In TU/e ID I have seen (and sometimes contributed to)
succesful examples of good multitasking in areas ranging
from fashion to medical simulation. A few examples:
The "Close-to-you" concept demonstrator and
research test tool by Sibrecht Bouwstra of TU/e ID
where a visual and haptic information display reanimates a (prematurely born) babys heart rate and
breathing. Multitasking, STD and DFD were all used.
The example of section 4 is inspired by Sibrechts work.
Perceptive objects by Eva Deckers such as PeP,
PeP+ and PeR, capable of sensing their environment,
even perceiving presence of a person and his or her
activity. The objects have a moving light body which
emerges from a multitude of LEDs and are capable of
perceptive interaction or even perceptual crossing.
The objects are highly adaptive. DFD notation was
used succesfully to manage complexity. Eva was
awarded a Cum Laude for her PhD [10].
Drapely-o-lightment, an innovative skirt in high-tech
fashion design by Marina Toeters and me. Drapelyo-lightment is about OLEDs and their embedding
126
in soft fabrics. An Arduino senses the presence of

other persons and feeds several (state-dependent)
light patterns into the OLEDs. Drapely-o-lightenment
was shown at Architextiles in Tilburg, Place-it in
Berlin, Gouden Geesten in Utrecht, Pretty Smart
Textiles Ronse, Belgium, the pop-up fashion show
of Amsterdam Fashion week, Smart Textiles Salon
2013 Gent Belgium, TEDx Brainport, and the Bridges
Mathematical Art Galleries Enschede [11].
The multitasking approach explained here is not new,
and has been used succesfully in traditional engineering
projects outside the design community [12], where it
is called "synchronous multitasking". It is popular in
industrial automation because the programs tend to be
more deterministic than programs using the interruptbased multitasking of Windows and Linux. The deter
minism allows for better testing and debugging. The
main contribution of this article to make such lost
knowledge available to the design community again and
translate it to our Arduino environment. In the multi
tasking approach, tasks can be put together in a multi
tasking way effortlessly: the adpative LDRs, the heart
and breath servos etc. could be blended into a single
program as simple as:
void loop(){
LDR0step();
LDR1step();
LEDstep();
navigationStep();
heartServoStep();
breathServoStep();
// more
idleStep();
}
If the reader thinks that the examples are small and
simple-minded: yes, that is the big advantage.
References
1. Banzi, M.: Getting Started with Arduino, OReilly Media, 2008.
2. Arduino, http://www.arduino.cc/
3. Booth, T.: (1967) Sequential Machines and Automata Theory,
John Wiley and Sons, New York
4. Harel, D.: Statecharts, a formalism for complex systems,
Science of computer programming, 8 (1987) 231-274.
5. UML Resource Page, http://www.uml.org/

6. Edward Yourdon, http://yourdon.com/
7. Kyffin, S., Feijs, L., Djajadiningrat, T.: Exploring expression
of form, action and interaction. In: Sloane, A. (Ed.): HomeOriented Informatics and Telematics, pp. 171-192, IFIP
International Federation for Information Processing Volume
178 (2005)
8. Tsz-Wo Sze: The Two Quadrillionth Bit of Pi is 0! Distributed
Computation of Pi with Apache Hadoop, Computing
Research Repository, Vol abs/1008.3171, 2010,http://arxiv.
org/abs/1008.3171
9. Wikipedia: Lux. en.wikipedia.org/wiki/Lux.
10. Deckers, E.: Perceptive qualities in systems of interactive
products. PhD thesis TU/e, May 2013.
11. Feijs, L., Toeters, M.: Drapely-o-lightment, Bridges
Mathematical Art Galleries, gallery.bridgesmathart.org/
exhibitions/2013-bridges-conference/feijs
12. Marchand, E., Rutten, E., Chaumette, F.: From Data-Flow
Task to Multitasking: Applying the Synchronous Approach
to Active Vision in Robotics, IEEE Trans. On Control Systems
Technology, Vol. 5, No. 2, pp 200 216. (1997).
127
Gustavo Ostos Rios, Mathias Funk, Bart Hengeveld, Joep Frens

Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands
g.a.ostos.rios@student.tue.nl, {m.funk, b.j.hengeveld, j.w.frens}@tue.nl
EMjam: Jam with your Emotions
Abstract
During improvisation, musicians express themselves
and their emotions through live music. This project
looks at the relationship between musicians during
music improvisation, the processes of expression
and communication taking place during performance
and possible ways to use specific characteristics of
musicians, such as emotions, to influence a digital
instrument and in consequence improvisation. To
visualize this, a three layer model is described studying
the relationship between band members and the
audience as a system, where emotions, expressivity
and generation of sound give shape to improvisation.
Focus is applied specifically on how individual emotional
arousal can be used as input to control as a group a
musical instrument: EMjam. The instrument is illustrated,
describing the design and implementation as well as the
evaluation of it, to subsequently discuss the results.
Keywords
Music, improvisation, composition, instrument,
performance, emotions, arousal, expressivity, electro
dermal activity, skin conductance.
1 Introduction
When paying attention at a concert it is possible
to see performers expressions; a guitarist playing
a solo and reaching a peak at a certain point of it;
128
a bass guitar player following with his face the lines

played; a drummer making accents with the whole
body; and in addition to this, the audience responding
to the performance. At these moments we can see
that making music is not only playing the instrument.
There are factors that have an important role on
performance; individual expressivity of the emotional
experience through physical gestures allows musicians
to communicate their intentions and express their
emotions towards others. In the same way, interaction during performance has an effect in the arousal
of emotions on performers and listeners [1].
Even though the process of making music can be
individual, meaning that one person plays one
instrument, in a band it is built together by a group
playing different instruments. Hence, group members
influence each other during performance, either in a
physical way through non-verbal communication or
through expression of musicians emotional state.
The work described in this paper focuses on music
improvisation, studying the process that requires
musicians to communicate their intentions and express
their emotions towards band members and audience,
by making use of physical gestures and a computer
generated sound.
While an improvisation session makes use of traditional
instruments to make sound, there are other possibilities
that can add expressivity to the performance, such
as the manipulation of sound through physical gesture
to control digital instruments [2, 3]. Emotions are

an important part of music improvisation. Musicians
play a unique composition that is created in its final
form at the specific moment of improvisation, making
that a single musical design would sound different on
other sessions [4]. Although many musicians draw on
patterns and trademark sequences of their own when
improvising, the composition happens in place, and
musicians can externalize the emotions felt at that time.
The use of emotions in improvisation to control digital
instruments through sensor technology can offer new
opportunities that are yet to be explored in the field.
This project goes beyond the use of physical interfaces and
musicians expressions to control sounds, exploring
the possibilities of using individual physiological
measurements of emotional components, such as arousal,
to be used as an input for a digital musical instrument
controlled by the whole band; giving musicians a second
layer of improvisation and expression.
2 Related work
Even Vocalists use
gestures and other

modalities to better
express their emotions,
apart from using
speech and lyrics.
Musicians communicate emotions by means of musical

instruments in a non-verbal1 way through a variety
of modalities. Expressivity suggests a process in
which musicians externalize their emotions through
expressive gestures [5]. According to Camurri et
al. gestures contain expressive content and convey
information related to the emotional domain such
as: feelings, moods, affect and intensity of emotional
experience; either in an acoustic or visual modality.
Gestures can be expressive not only through physical
gestures of performer, but include also the produced
sound and visual media. Finding new ways of controlling
or influencing music using a performers expressivity
has been addressed in the past [2, 3], introducing a
new layer of interaction in the context of musical
performance. These approaches focus on proposing
new instruments and systems, as well as influencing the
sound generated by traditional instruments through the
use of a musicians physical gesture. Furthermore, new
applications start to focus on the inclusion of emotional
responses through physiological signals for the control
of digital instruments. Related work can be seen on
[6] where physiological signals (electroencephalogram,
electromyogram and heartbeats) are used to control
sound in order to build a biological driven musical
instrument. Another example is the BioMuse system
[7], which uses on-body sensors to measure motion

and physiological signals to determine emotional states
integrated into what is defined as an Integral Music
Controller. The BioMuse is based on a model [8] describing the number of existing interface devices for
controlling a digital musical instrument. Its importance
is the addition and consideration of emotions as a
controlling interface which introduces a new feedback
path within musical performance. Additionally, the
BioMuse system covers three layers of feedback that
can be achieved in musical performance:
Layer 1 represents the internal emotion and thoughts
of the performer.
Layer 2 is the physical interface layer.
Layer 3 represents the creation of music as a
consequence of the gesture.
Here, thoughts and emotions are represented through
physical gestures that are used to control a sound
generator. The resulting sound and interaction create
a direct feedback loop. It also considers emotional
state and physical gestures from the audience and the
possibility to be used to manipulate the generated
sound. However, the model proposed by Knapp and
Cook [8] focuses on individual feedback loops and
interaction with a controller and does not consider the
influence of other band members during performance.
Interaction within the band is important for musicians
and has an influence on improvisation [9]; that is why
it needs to be considered. Looking at the structure of
bands, a band incorporates several feedback loops, e.g.
an individual loop, a band loop and an audience loop;
demanding musicians to be aware of different settings at
the same time [4]. Designing musical instruments for a
band behaving as a system requires a different approach:
understanding the organization of musicians during
music improvisation sets a starting point to develop new
musical instruments for a band behaving as a system. In
consequence, it is possible to propose systems aimed to
create richer interactions that could influence feedback
loops between musicians.
Our approach explores the use of emotional compo
nents, specifically arousal to control a digital musical
instrument as a group, offering band members a tool to
help them reinforce the communication of emotions.
Authors might disagree when defining emotions, but
they do agree on their components. According to
Schachters theory of emotions [10], an emotional state
129
can be seen as the result of the interaction between two

components [11]: physiological arousal and cognition
about the arousing situation. Physiological arousal
determines the intensity of an emotion but not the
quality of the emotion itself. The cognition determines
which emotion is experienced. Both are considered
necessary for the appearance of an emotional state.
Within the study of emotions, several approaches
and definitions can be described. According to Frijda
[12], emotions are elicited by significant events. These
events are of relevant significance to persons when
they touch upon one or more individual concerns or
goals. To Frijda, emotions are then the result from
the interaction of an events actual or anticipated
consequence and the subjects concern [12]. Emotions
emerge in a process including the occurrence of events,
the presence of concerns or goals that make the event
significant to a person and the appraisal of whether concerns are or are not accomplished in the interaction.
Sloboda and Juslin [13] describe how an emotion
being a scientific construct is inferred from three
kinds of evidence: (a) self reports including adjective
checklists, rating scales, questionnaires or free
descriptions; (b) expressive behavior including facial
expressions, vocalizations and body language such as
gestures; and (c) physiological measurements including
heart rate, respiration, skin conductance, muscle
tension, electrocardiogram, blood pressure and
electroencephalograph.
Based on this kind of evidence, Sloboda and Juslin [13]
mention a study, which shows a consensual definition
of emotion based on a review of different authors
definitions. Here, emotions are considered as a
complex set of interactions among subjective and
objective factors [14], including aspects such as
the rise of arousal, generation of cognitive processes,
physiological adjustment to the arousing conditions, and
behavior that is often, but not always, expressive, goal
directed, and adaptive [14].
Moreover, Laurier et al. [15] make a distinction of
works presenting three different ways that have been
used to study emotions in relation to music: (1) The
categorical representation is based on basic emotions
such as happiness, sadness, anger, fear and tenderness.
(2) The multi labeling approach makes use of a wider
set of adjectives to describe categories of emotions. (3)
The dimensional representation models emotions in a
130
two dimensional space. On the approach of dimensional

representation, Russells Model of Affect [16] has
generated the most research [17]. This model represents
affect (an observable expression of emotion) into a two
dimensional space. Here, arousal represents the intensity
of the emotion on a vertical axis (arousal sleep). On
the horizontal axis the valence (pleasure displeasure)
represents the quality of the emotion. One of the
strengths of this model and its use on the music context
is that it represents a simple way of organizing emotions
in terms of their affect appraisals (pleasant or unpleasant)
and physiological reactions (high or low arousal) [17].
Our work focuses on a dimensional representation by
measuring musicians emotional arousal to use data
as a controlling input for a digital musical instrument.
The way musicians experience improvisation gives
place to individual emotional states, which can vary
within the session. Improvisation is a dynamic process
that fluctuates between soft and relaxed compositions
to more active and explosive ones as a fluctuating
wave that rises and falls [18], which can affect also
musicians arousal. In this sense, arousal being a
physiological component can be measured; in order to
relate the intensity of musicians emotional state to the
improvisation session.
Studies have linked music to emotions, showing that
music has the ability to evoke emotions in the listener,
and that the qualities that allow this have been used
by composers and performers to convey emotions to
listeners. According to Zatorre [19] music is capable
of eliciting not only psychological mood changes (e.g.
anxiolytic effect of relaxing music), but also physiological
changes that mirror the changes in mood. Authors
have described the link between emotion and music
proposing important theories such as Cookes melodic
cues theory [20], stating that melodic intervals and
melodic patterns have perceptive characteristics capable
of evoke specific emotions. Other authors (Kivy [21],
Langer [22], Meyer [23] and Justin & Vstfjll [24])
have proposed in the same way theories that are of
importance for the understanding of how music can
elicit emotional responses.
3 Approach
Music improvisation requires musicians to communicate
their intentions to each other and to express their
emotions to the audience on different layers. Even

Fig. 1. Three layers
of interaction during
music improvisation.
Fig. 2. Two layers of

improvisation. Individual
musicians interaction
with their instruments
results in the main
layer of improvisation.
Individual musicians
arousal results in an
underlying improvisation.
though the process of making music is individual, in

a group improvisation is built together by musicians
influencing each other through communication of
intentions and expression of emotions.
With the purpose of explaining the process wherein
musicians communicate and express to each other and
the audience, three layers are explained in the next
paragraphs:
The first layer is represented by the music itself.
Improvisation has a hidden communication [25]
where musicians set what they want listeners to hear.
Individual interaction with their instruments gives as
result an individual generated sound that builds the
group improvisation.
The second layer is represented by the communication
and the expressivity of musicians towards the band
and the audience by means of physical gestures. Here
musicians agree when to start, stop or change aspects
of the music, e.g., the volume. By using body language
such as movements, facial expressions, reactions and
signs, they can show their intentions. These physical
gestures are also part of how musicians express their
emotions during performance.
In addition to this, emotions are an important
component in music improvisation, as musicians
thoughts and feelings on that specific moment are
reflected into sounds. Hence, the third layer is settled
at an emotional level. Musicians experience the music
they make at a point that they could be surprised

or aroused by any musical line exposed to the ears
[26]. Therefore, expressivity is influenced by the
emotional state of musicians and it will also influence
improvisation. Other external influences like the
engagement of the audience with the band might also
have an effect on the response of musicians.
In this way, all three layers become a process where
musicians play together giving shape to improvisation.
The emotions evoked during improvisation have an effect
on the way musicians communicate their intentions and
express their emotions, which will in consequence affect
the sound generated by every band member.
4 EMjam
Understanding all this, makes us believe that different
components of emotions can be used to control an
instrument in the context of music improvisation,
opening new possibilities for a new musical aesthetics
and expression. Could we use emotional arousal
to generate a second layer of improvisation? An
underlying sound generated and controlled by the
band as a system, where one instrument receives
input from every musician. Arousal being an emotional
component is related to improvisation, and its variations
during performance can be measured to control a
digital instrument. EMjam uses these measures to give
musicians an extra level of expression.
4.1 The concept
In the context of band improvisation where musicians
are playing with their own instruments (for instance,
guitar, bass guitar, and drums) to make the main layer of
music (see Fig. 2); emotional arousal during performance
can be used to control a digital musical instrument
creating a second layer of music. This second layer is
independent from the improvisation itself, meaning
that sound coming from the traditional instruments
is not affected directly by this layer (as a sound effect
would do), but it creates a new input. Additionally,
the second layer is defined by the interaction between
band members, in a way that the emotional experience
of musicians while playing define the sound generated.
In this sense, musicians arousal can be reflected
into a second layer of sound and improvisation itself
represents the interface for playing. In this way,
expressivity of the band is reinforced by an individual
input based on their emotional arousal.
131
4.2 Design overview

EMjam consists of a physical interface and a digital
musical instrument.
On the physical interface, three wristbands (fig. 3,
right) hold skin conductance sensors that measure
individual changes in arousal. The sensors are connected
to a pedal box containing the electronics (fig. 3, left)
and holding knobs for the control of keynote and
volume, and a tapping surface for the control of tempo.
Measurements from the sensors are received by an
Arduino microprocessor inside the pedal box and
then sent to an external computer through USB. The
digital musical instrument then converts variations
in measured arousal into MIDI messages to control
specific parameters in relation to rhythm, melody and
harmony of an instrument setup.
The possibility of measuring arousal with a noninvasive and comfortable sensor, to allow musicians
to move freely on stage while improvising, without
constraining physical gestures.
The linear relationship between measurements and
arousal (high measurement high arousal) [27].
The availability of components to build the sensors for
a low price.
To control the instrument, three sensors were used
(one for every musician) and connected to the pedal box
for the control of keynote, volume and tempo. On each
sensor, two copper coins were connected to a capacitor
and a resistor making a low pass filter. We encountered
several problems during the design and implementation
of the sensors as 50Hz frequency was interfering with
the readings. This problem was solved by changing the
capacitor and the resistor which led to a stable reading.
Additionally, the speed and reaction of the readings
were also improved by putting a silver coat on the
coins used as electrodes.
Fig. 3. EMjam. Pedal box (left) for the control of keynote, volume
and tempo. Wristbands (right) holding skin conductance sensors
to measure changes in arousal.
4.3 The physical interface.

Arousal is measured through Electro Dermal Activity
(EDA). EDA refers to the changes of the skins electrical
properties when activated by various brain circuits
[27]. When experiencing emotional arousal, cognitive
workload or physical exertion, the brain sends signals to
the skin to increase the level of sweating. Even though
sweat is not felt on the skin, the pores begin to fill,
increasing electrical conductance of the skin.
The most common measure of EDA is skin conductance,
which captures changes on emotional arousal. It increases
when a person is more aroused: engaged, stressed or
excited; and tends to stay low or drop when a person is
less aroused: disengaged, bored or calm [27].
Even though different sensors where considered (EEG,
EMG and ECG) to measure different components of
emotion, we chose to use skin conductance to measure
musicians EDA in order to control EMjam for the
following reasons:
132
Fig. 4. Physical interface schematics.
4.4 The digital musical instrument.

To process measurements coming from the Arduino
microprocessor, different software is running at the same
time. First, a standard Firmata [28] sketch is used with
the Arduino so it can communicate with other software
on a high level and convert sensor data from analog to
digital. Measurements gained are processed within Pure
Data [29] environment, where digital readings of skin
conductance and changes in volume, keynote and tempo
coming from Arduino (in a scale from 0 to 1), are then
converted into MIDI messages (in a scale from 0 to 127).
Additionally, it is possible to monitor the readings of the
sensors on the pure data sketch. Then, MIDI messages are
coupled to Ableton Live where every sensor is mapped to
different parameters of the instrument allowing musicians
to control them in accordance with their arousal level.
Fig. 5. Physical interface and digital musical instrument.
Parameters of the digital instrument are mapped in a

linear way to each sensor, using the following criteria
[30, 31, 32]:
Start from scratch: the instrument should be designed
entirely trying not to be limited by known and existing
instruments.
Instrument autonomy: how much will the instrument
do without musicians input?
Flexibility: do as much as possible with few knobs as
possible.
Interface optimization: in this case more is not more.
The number of controllers needed to control a setup
reaches a peak at one point, beyond that it becomes
noise.
Control: over important aspects of the track.
Map sensors in a linear way (values from 0 to 127)
to different parameters that are related to rhythm,
melody and harmony.
Open: to make it random yet controllable.
Fig. 6. Digital musical instrument. Measurements are received

in the Pure Data software where are converted into MIDI
messages and sent to Ableton Live to control the instrument.
Based on these, the resulting instrument consists of

the following MIDI Effect Racks or compilation of
effects (fig. 6 and 7):
Rhythm Generator.
The function of the random MIDI generator is to
control the rhythm if the notes triggered. It includes the
effects: one Arpeggiator to affect subdivisions of
the notes, two Velocity controllers to add accents and
cut off the notes and a Note Length controller.
Major Scale.
This Rack defines the scale in which the instrument
will trigger the notes. It includes a Random effect
that triggers a note randomly, a C Major scale which
establishes the notes that will be played and a Chord
effect that makes chords of the notes triggered.
Instrument Rack.
This Rack assigns an instrument to the notes played and
it is a combination of a Muted 1 Pure instrument, which
gives definition to the notes and an All Alone Pad that
gives Echo and ambience to the setup.
This setup is only one example of a mapping of arousal
measurements to sound and other configurations can
result in a different sound. In the same way, other digital
instruments can be design to direct the setup to more
specific music styles.
4.5 Mapping of measures to the digital musical
instrument
Individual effects that are included into an Effect Rack
have several parameters that can be controlled and
help to define the sound. For instance, changes on the
synced rate (see Fig. 7) parameter of the Arpeggiator
will define how many notes will be triggered in a
compass. Parameters are arranged into the categories
based on basic musical concepts of rhythm, melody
and harmony, which are then assigned to every sensor.
In this sense, every musician can control parameters
on a defined category which assure the richness of the
performed improvisation.
The skin conductance sensors were mapped to Ableton
Live parameters in the following way:
Sensor 1: rhythm.
Sensor 2: melody.
Sensor 3: harmony.
Potentiometer 1: control of keynote.
Potentiometer 2: control of general volume.
Button: tap to control tempo.
133
Fig. 7. Mapping of sensors to the digital instrument.
Variations in arousal are mapped in a linear way to

Ableton Live, meaning that an increase or decrease
in individual arousal will have a proportional effect on
the parameters that are assigned to each wristband. In
addition to this, the pedal offers musicians to oppor
tunity to control the keynote in which improvisation will
take place and the volume of the instrument.
By tapping on the tempo controller, musicians can
control the tempo of the sound generated on the
instrument, giving them the opportunity to adapt the
instrument to their own tempo instead of trying to
follow the one settled of the instrument.
5 Evaluation
The final prototype including the physical interface and
the digital instrument was tested with two groups of
musicians at Eindhoven University of Technology. They
were provided with individual skin conductance sensors
to use during improvisation and were asked to interact
with EMjam. The main goal of this study was to detect
patterns in the way EMjam can be used in relation to the
Fig. 8. Evaluation of EMjam during group improvisation.

134
second layer of improvisation, in order to get insight

in the way musicians react and interact with this
second layer. Both sessions were video recorded for
its subsequent analysis through observation, looking
specifically at the engagement of musicians with EMjam,
the effect EMjam had on the interaction with other
musicians, the general effect of having a second layer of
sound and musicians reactions towards it. Notes were
taken on the video recording of sessions and the results
are as follows:
The first group consisted of three participants playing
keyboards, djembe and bass guitar who had never played
or made improvisation together; yet they had done
improvisation in other occasions, so they had some
experience in this activity.
It could be observed that participants were paying
attention to the sound generated by EMjam and the
sound generated was used as inspiration to build a new
improvisation. After improvisation had some structure,
the sound became an ambience layer giving musicians a
space to play further.
It was noticeable that while trying to keep attention to
the generated sound from EMjam, participants were less
conscious of what other musicians were playing. When

they were able to find a line to improvise, attention
could be paid to each other. In this sense, responding to
the sound generated requests attention from musicians
influencing communication between them.
The second group consisted of three participants playing
guitar, djembe and bass guitar that had played together
before, with less experience in improvisation. In this
sense, they improvised on a rock sequence they had
already practiced before. Here, it could be observed
that the digital instrument was used as a background
layer for the improvisation; musicians were playing their
own structure and next to it the EMjam generated sound
was hearable. As they had played together before, it was
noticeable that they had already prepared a basic line
to play, which leaved entirely the sound from the digital
musical instrument as a second layer of improvisation.
Musicians paid more attention to each other instead of
the sound generated trying at times to adapt their own
volume and tempo to the generated sound.
On this, feedback was given from one of the participants
assuring that hearing the changes on the digital instrument
gave them awareness of their emotional arousal, as they
were able to hear the changes coming from EMjam.
6 Discussion
Designing for the band as a group, required an approach
were the whole system needed to be studied. The
abstraction of the components was helpful to establish
a relationship between band members, how they can
influence each other and, in consequence, influence
the generated sound. In this sense, emotions and
expressivity during performance became of importance
for the design of an instrument controlled by the whole
band. The use of emotional components in the field of
music improvisation represents a new feedback path
within music performance. It relates the emotions felt
by musicians at certain points of the session to the
sounds generated, offering a new way of expression.
The design of the digital instrument defines the sound
generated and makes possible its improvement and
its application to more specific music styles such as
blues or jazz.
Generated sound from EMjam goes from slow to
fast patterns showing an increase on the instruments
activity. Changes on these patterns give feedback to

musicians about changes in arousal and the control they
have on the instrument, in this sense, slower patterns
on the generated sound show less control on the
instrument.
7 Conclusions and Future Work

This project demonstrated the possibility of controlling
sound generators through measure and mapping of
individual arousal to a group music improvisation system
in Ableton Live, creating a second layer of improvisation
defined by the band as a group. This new layer can
support musicians offering a tool that helps them by
either helping them to break the ice at the beginning of
the sessions, inspiring them to build up improvisation or
by playing background music as an indicator of arousal.
For future research there are some aspects that need
attention. The calibration of the system needs to set up
a range where musicians can play, making recalibration
possible over time. Additionally, musicians interaction
with EMjam can be improved by developing the interface
further, adding sensors controlled by physical gestures
(e.g. accelerometers) to control parameters such as
volume, keynote and tempo. New features such as
the control of light or visuals can be also included to
reinforce musicians expressivity.
On the other hand, using skin conductance gave an
effective reaction on the digital musical instrument;
nevertheless, it would be interesting to use heartbeat
sensors. Changes on skin conductance can be noticed in
a long period of time; by using frequency of heartbeats
as input for the system, changes on sounds would be
noticed faster and external aspects such as temperature
will have less influence.
For our purposes, emotional arousal was mapped in
a linear way to Ableton Live, meaning that changes
on measurements had a proportional effect on the
parameters assigned. Different way for mapping
measurements could include logarithmic and
exponential functions, to emphasize different parts
of the arousal spectrum.
Finally, the scope of the project can be expanded
towards the inclusion of the audience in the control of
the instrument, by finding ways map their reactions or
engagement on the performance to the digital instrument.
135
References
1. Eerola, T. Modeling Listeners Emotional Response to Music.
2. Helmuth, M., Danard, R., Jung, H., Mishra, A., Agrawal, D.
Water Birds: Compositional Collaboration with Clarinets,
Wireless Sensors, and RTcmix. 26th Annual conference
of the Society for Electro-Acoustic Music. United States.
January 20-22, 2011.
3. Klgel, N., Frie, M., Groh, G., Echtler, F. An Approach to
Collaborative Music Improvisation. Proceedings of the
International Conference on New Interfaces for Musical
Expression. Oslo, Norway, May 30 - June 1, 2011.
4. Hengeveld, B., Frens, J., Funk, M (2013, accepted).
Investigating How To Design For Systems Through Designing
A Group Music Improvisation System.
Proceedings of the IASDR conference 2013, Tokyo, Japan.
5. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R.,
Volpe, G. (2003). Multimodal Analysis of Expressive
of Emotion Definitions, with Suggestions for a Consensual

Definition. Motivation and Emotion Vol. 5, No. 4, p. 355.
15. Laurier, C., Lartillot, O., Eerola, T., Toiviainen, P. (2009).
Exploring Relationships between Audio Features and
Emotion in Music. Frontiers in Human Neuroscience, vol. 3.
16. Russell, J.A. (1980). A Circumplex Model of Affect. Journal
of Personality and Social Psychology, 39, pp. 1161-1178.
17. Thompson, W. F. (2009). Music, Thought, and Feeling.
Understanding the Psychology of Music. Oxford University
Press. New York.
18. Galarraga, E. Artist - Musician. Personal communication.
March 18, 2013.
19. Zatorre, R. (2005). Music, The Food of Neuroscience?.
Nature. vol. 434, no. 7031, p. 312.
20. Cooke, D (1959). The Language of Music. Oxford University
Press. London
21. Kivy, P (1980). The Corded Shell: Reflections on Musical
Gesture in Music and Dance Performances. In: Gestured
Expression. Princeton, University Press. Princeton.
Based Communication in Human Computer Interaction,
22. Langer, S. K. (1957). Phylosophy in a New Key (3rd ed.).
eds. Camurri, A and Volpe, G. 5th International Gesture

Workshop. Italy. April 15-17, 2003
6. Arslan, B., Brouse, A., Castet, J., Lhembre, R., Simon, C.,
Filatriau, J.J., Noirhomme, Q. A Real Time Music Synthesis
New American Library. New York.

23. Meyer, L. B. (1956) Emotion and Meanng in Music. University
of Chicago Press. Chicago
24. Juslin and Vstfjll (2008). Emotional Responses to Music:
Environment driven with Biological Signals. Acoustics,
The Need to Consider Underlying Mechanisms. Behavioral
Speech and Signal Processing (2006). France. May 14-19
and Brain Sciences. Vol 31, pp. 559-621.
7. Knapp, R.B., Cook, P.R. The Integral Music Controller:

Introducing a Direct Emotional Interface to Gestural Control
Of Sound Synthesis. Proceedings of the International
Computer Music Conference (ICMC 2005). Barcelona,
Spain, September 5-9, 2005.
8. Knapp, R.B., Jaimovich, J., Coghlan, N. Measurement of
Motion and Emotion during Musical Performance. 3rd
25. Yee, L. Industrial DesignerMusician. Personal

communication. March 14, 2013.
26. Nio, L. Graphic DesignerMusician. Personal
communication. March 20, 2013
27. Affectiva Inc. (2012). Liberate yourself from the lab: Q sensor
measures EDA in the wild. Affectiva Q Solutions White
Paper. Waltham, USA.
International Conference on Affective Computing and
28. Firmata, available on: http://arduino.cc/en/Reference/Firmata
Intelligent Interaction and Workshops, ACII. Amsterdam,
29. Pduino, available on: http://puredata.info/downloads/pduino
The Netherlands, September 10-12, 2009.
30. Arroyo, A. (2012). Performing Generative Music with Ableton
9. Saavedra, C. Musician. Personal communication. March 14, 2013.
Live [Online Video]. The Ableton Cookbook. Available on:
10. S. Schachter and J. Singer (1962). Cognitive, social and
http://www.anthonyarroyodotcom.com/theabletoncookbook
physiological determinants of emotional state. Psychol. Rev.

69, 379-99.
11. R. Reisenzein, R. (1983). The Schachter Theory of Emotion: Two
Decades Later. Psychological Bulletin Vol. 94, No 2, 239-264.
12. Frijda, N. (1986). The Emotions. Studies in Emotion & Social
Interaction. Cambridge University Press. Cambridge.
13. Juslin, P. N. and Sloboda, J.A. (2001). Psychological
31. Arroyo, A. (2012). Its the Generative Rhythm Generator

[Online Video]. The Ableton Cookbook. Available on:
http://www.anthonyarroyodotcom.com/
theabletoncookbook/ableton-tutorial-2/its-the-generativerhythm-generator/
32. Arroyo, A. (Producer) (2012). Performing Generative
Perspectives on Music and Emotion. In: Music and Emotion:
Music Live with Ableton Live [On-line Video]. The Ableton
Theory and Research, eds. P.N. Juslin & J.A. Sloboda. Oxford
Cookbook. Available on: http://www.anthonyarroyodotcom.
University Press.
com/theabletoncookbook/?s=performing+generative+music
14. Kleinginna, P. and Kleinginna, A. (1981). A Categorized List

136
/?s=performing+generative+music&submit=Search
+live&submit=Search
Christopher G. Henley1, Yaakov Lyubetsky2 , Ajay Ravindran3, Canute J.B. Haroldson4, Mark Baskinger5
1
Industrial Design BFA Candidate, Carnegie Mellon University

Industrial Design BFA Candidate and Human Computer Interaction, Carnegie Mellon University
3
Computer Science BS Candidate, Carnegie Mellon University
4
Mechanical Engineering and Engineering and Public Policy BS Candidate, Carnegie Mellon University
5
School of Design Associate Professor and Human Computer Interaction Institute (HCII) affiliate faculty, Carnegie Mellon University.
2
Project Loci: Haptic Interactions

Influence Situational Awareness
Abstract
Today, technology functions primarily as a means to
provide access or give form to information. Personal
electronic devices allow us to access this information
from almost any place, at any time, yet the most
common way to access this information is through
screen-based interfaces. Project Loci demonstrates the
feasibility of using haptics as a method to convey more
complex information in this case, using a locationaware backpack that provides tactile feedback to the
user to indicate the relative safety level of their location.
This project demonstrates how integrated haptic tech
nology allows people to receive information in a more
intuitive, physical manner beyond the screen.
haptic, non-screen based interfaces. Environmental

awareness, the cornerstone of safety, relies heavily on
the senses, specifically sight and hearing. Both of these
senses are severely diminished while using personal
electronic devices, most of which are screen-based and
provide visual and auditory stimuli. Project Loci uses a
backpack, instead of a screen, as a platform to explore
haptic interactions. Incorporating these interactions
into a backpack removes the users dependence on a
screen, allowing access to complex information through
physical perception of tension and movement. Project
Loci attempts to provide a practical application of
using haptic interactions to create a more intuitive
and beneficial method of experiencing information to
promote safety and contextual awareness.
Keywords
Haptics; haptic technology; interaction design; human
computer interaction
1 Introduction
Haptic interactions hold great potential when applied
to digital systems as they allow additional ways to
experience information. Project Loci was created as a
way to explore this potential by using environmental
awareness to demonstrate an application of haptic
interactions. This project operated in the space of safety
and environmental or situational awareness because it
represents one area that could benefit the most from
2 Conceptual Approach
The ability to convey the safety level of an urban area
was a key driver for the project, requiring the system
to understand and interpret the relative safety level
of a location through crime statistics. Crime statistics
are typically displayed in the form of tables and
graphs showing the types and frequency of crimes in
regions and districts. This information can be found
online through annual police reports issued by most
municipalities as shown in figures 1 and 2. The only way
to interact with this information is to read either the
charts or maps, which is a taxing experience because of
137
the density of the content, and this experience is made

even more demanding if the reader is on the move.
After establishing a method of determining the relative
safety level of an area, a haptic interaction was chosen
by looking into parental attachment, another safety
minded process. It was decided that a tightening grip
would work best as a haptic response because of its
innate and intuitive feel. The tightening interaction
reflected the actions of a child pulling closer to their
parent in an unfamiliar situation or environment.
Because this interaction is deeply rooted in human
psychology, it elicits a visceral response as opposed
to forcing the user to make a conscious association
between an interface and an abstract concept. This
response would be applied to the straps and across
the back of the backpack, providing the most surface
area for contact with the user. The tightening strap
interaction accurately conveys a sense of apprehension,
influences the users posture, and encourages a
heightened sense of awareness.
Fig. 1. The City of Pittsburgh Department of Public Safety

Bureau of Police Annual Report (2010), depicting crimes by
neighborhood (http://apps.pittsburghpa.gov/pghbop/10_Police_
Annual_Report.pdf )
Fig. 2. The City of Pittsburgh Department of Public Safety

Bureau of Police Annual Report (2010), depicting homicide rates
by location (http://apps.pittsburghpa.gov/pghbop/10_Police_
Annual_Report.pdf )
138
The integration of safety level data and information

fidelity were additional considerations for the project.
Statistical information accessed through digital devices
typically has a high level of fidelity; that is to say, most
of what is looked at represents finite amounts in the
form of numerical, textual or visual depiction. As a
result of using the back as the interface between the
device and the user it would not be possible to convey
high information fidelity. Since the back is seldom
used for information processing, any input is quickly
recognized due to the peculiar feeling, and is identified.
The tightening interaction was chosen since it was
determined the relative safety level is more important
to display than the actual statistical safety level, which
has too high a level of information fidelity to be
conveyed through the bag. The tighter the backpack
gripped would indicate to the user that their location
was more dangerous; the looser the backpacks straps
would indicate that their location was relatively safer.
The important interaction for the user would be to
experience the action of tightening or loosening, not
just the tightness or looseness, because the action
would indicate to the user that one area was more
dangerous or less dangerous that the previous area.
More specifically, the change in state of the backpacks
hold on the pedestrian, the more they are aware of the
transition between safe and unsafe areas. The haptic
response within this project requires that the fidelity of
the information be diminished in order to transmit the
information, but in reality considering the context of
the user (a pedestrian in motion) no greater information
fidelity is really necessary to understand the message.
Haptic technology offers the potential to have filtered
information (through necessity as it would be impossible
to convey all of the information found in a screen based
interface through haptics) allowing for the appropriate
information to be accessed and processed faster.
Fig. 3. Internal
framework and layout
of components is as
shown without the
backpacks cover. The
backpack is depicted
unspooled (Left) and
spooling or spooled
(Right).
3 Procedures
3.1 Hardware
The backpack platform for Project Loci is based upon
a 12v 3 amp window motor to create the tightening
action. The straps were secured to the top portion
of the bag and were connected to the motor through
a slot at the bottom of the bag where they were
spooled around a spindle mounted on the motor. This
construction allows for the rotation of the motor to
be transmitted as a linear tightening of the straps.
The bag utilizes the Arduino prototyping platform to
control the bags functions and a 20 channel EM-406A
SiRF GPS receiver with 10 meter positional accuracy
to determine the bags location. The motor is powered
through a 2 channel 4 amp motor shield supported
by the Arduino platform powered by a 12v 4200mAh
NiMH battery pack. The components are mounted onto
an internal framework within the bag as seen in figure 3.
This internal structure is attached to the external shell
constructed from molded ionomer foam with a canvas
outer skin. The full bag and proper bag positioning is
demonstrated in figure 4.
insight as to how the haptic interaction was generally

received, and to provide ideas as to how it could be
improved.
Eight subjects were chosen, with an even distribution
of gender between male and female. Five of the trials
were conducted in the evening around 10pm and the
other three trials were conducted in mid-day hours.
The evening trials were conducted along a high traffic
(pedestrian and vehicle) public urban corridor adjacent
to two university campuses in Pittsburgh, Pennsylvania
whereas the day trials were conducted on the campus
of Carnegie Mellon University. For both sets of trials
fictional crime statistics were used to maximize the
effect of the haptic interaction by increasing the density
of events as shown in tables 1 and 2. The subjects were
not informed of the purpose or interactions of the bag
before testing, and were surveyed before, during, and
after the trial. The subjects were first asked a series
of preliminary questions before the study began. The
backpack was then provided to the subject who was
instructed to wear it for the duration of the trial.
A specific course was plotted and followed for the
trial and the prompters waited for the subjects to
perceive a change in the bag before the second survey
was conducted. Upon the completion of the course,
follow-up questions were asked and the subjects were
informed of the purpose of the bag and the project.
Location
Crime
Level
Longitude
Latitude
Webster Hall
-79.95127
40.44715
Craig St. at Henry St.
-79.94892
40.44636
Craig St. at Filmore St.
-79.94879
40.44527
Craig St. at Forbes Ave.
-79.94867
40.44444
Carnegie Museum of Art
10
-79.94899
40.44376
Table 1. For the evening Craig St. trials five locations were
chosen to be points of crime; locations A, B, C, D, and E.
Fig. 4.
3.2 Testing Procedures

Upon the completion of the bag, field testing was used
to evaluate the effectiveness of the haptic interaction to
the users perception of an area. The results were not
gathered for statistical accuracy, but rather to provide
Location
Crime
Level
Longitude
Latitude
Hunt Library
10
-79.94372
40.44109
Porter Hall
-79.94621
40.44174
Doherty Hall
-79.94431
40.44239
Tepper Hall
-79.94229
40.44104
Table 2. For the day campus trials four locations were chosen
to be points of crime; locations F, G, H, and I.
139
The bags tightening system was programed to be fully

tightened (the motor would run full at power for 1.5
seconds) if the sum of the crime levels within a 125 foot
radius was greater or equal to 25. For every increase
or decrease in crime level, below the 25 threshold, the
system adjust 1/25th the full tightness of the system.
A perceivable change in tightness or looseness could
generally be felt with a 3 to 5 point change in crime level.
4 Results
4.1 Surveys
Before the trials, subjects were surveyed as to what
physical or physiological feelings they usually associated
with danger and safety. 50% of the subjects mentioned
anxiety as one of those feelings they associated with
danger, and 88% of the subjects mentioned calmness
or relaxation when referring to safety. When referring
to the feelings associated with danger the topic of
awareness and tension were mentioned, although with
less frequency.
During the trials, subjects were surveyed, as shown
in figure 5, as to how the tightening or loosening of
the bag, if a perceivable change was felt, made them
feel upon entering a predetermined area. 50% of the
subjects mentioned an increased sense of awareness
or alertness, and 25% of the subjects mentioned
being surprised by the interaction, which is most
likely due to their intentional lack of familiarity with
the project and bag. Subjects were also asked if and
how the change in the straps affects their awareness
of the surrounding environment. 63% of the subjects
described an increased sense of awareness whereas 25%
of the subjects mentioned they had no change in their
environmental awareness level due to the familiarity
they had with the area.
After the trials, subjects were surveyed as to how
knowing the purpose and intention of the bag affect
their perceived feeling of the walk. They were also
asked what they believed the function of the bag was,
knowing how it worked and what it responded to. Half
of the participants mentioned encouraging safety in
dangerous areas. 50% of the subjects when asked if they
felt an emotional change while they walked responded
that they did not and 25% of the subjects responded
that they felt alerted. Across all of the subjects it was
140
agreed that knowledge and familiarity with an area was a

significant indicator of whether or not an area was safe
to be in. When the subjects were asked if safety meant
staying away from dangerous places or staying alert,
88% of the subjects responded that staying alert was the
focus of safety.
4.2 Discussion
Discussion with the subjects revealed a common belief
that situational awareness was the cornerstone of safety.
From this it was also revealed that environmental
familiarity would overrule the effects of the bag since
users would feel more comfortable in areas in which
they are familiar with; as one subject remarked, Im
going to be more aware in an environment that I know.
If I was somewhere Id never been before, thats where
this backpack would be most useful. Surveying the
subjects revealed that although the bag represented a
more visceral interaction the complete purpose of the
tightening interaction did not achieve its full impact
until it was explained. Discussion after the completion
of the tests, trials, and surveys had revealed that
when designing this system there are many different
variables that could affect the users perception of
the interaction. Time of day is one of these important
variables as pedestrians may be more anxious when
walking at night, thus intensifying their perception
of the tightening interaction. The point at which the
bag tightens in relation to the surroundings of the
user is also another important variable that was not
considered. If the straps were only to tighten or loosen
at intersections, a location that is representative of the
opportunity to make a choice, the users actions may
actually be influenced as the interaction may encourage
them to choose a different direction of travel. These are
variables that would be addressed through additional testing.
The project has acted as a catalyst for the discussion of
how digital technology influences common sensibility,
information relevance over time, and the nature of
interactive systems. In regards to common sensibility,
digital technology has the potential to create user
dependence, which on a small scale may not be of great
concern, but when applied to a topic such as being able
to evaluate potentially dangerous situations, it becomes
an issue. Long term use of such a system that relieves
the user of this decision making may create dependence,
ultimately leaving the user without the ability to use
Fig. 5. Subject and prompter during one of the daytime trials
common sensibility to evaluate dangerous situations

or locations. However, it is also possible that long term
use of such a system may stand to reinforce the users
gut-feelings; proving that an area they felt was
potentially dangerous actually was dangerous. In this
way, the system may actually act to subconsciously
condition the user to the characteristics that make
an area dangerous.
The issue of information relevance over time is a major
concern with data in digital products. With the ability
to access and analyze large amounts of information
almost instantaneously, a question is prompted; what is
an appropriate time frame for functionally applying data
such as crime statistics? When determining the danger
level of an area, ultimately what is being determined
is the potential for additional crimes to occur in that
same area. Crime statistics are posted for municipalities
annually, most likely for fiscal purposes, and the annual
reports were used for this project mostly for ease of
access, but this system leaves open the potential for
other methods of information gathering. Real time
gathering of crime data through dedicated resources,
such as community initiatives, or social networking
spaces, is nor outside the realm of possibility and
represent an inevitable adaptation of this design. The
bag prototype proved the feasibility of data mapping in
an integrated haptic system, setting the stage for future
work where they system can utilize more frequently
updated crime statistics sources.
One of the most important aspects of this project is

that it starts to challenge the traditional nature of what
is considered an interactive system or product. The bag
has no buttons, switches, or screens but still remains
interactive because the bag represents an entire system
that receives input, not just a single product. The
system encompasses not only the bag, but also the data
the bag uses and the environment from which the data
is derived. In this system the user interacts with the
bag by moving through the environment. In this way
the system maintains a form of passive interaction since
the bag does require some input in order to receive
an output, but the input is not active in the traditional
sense of buttons, switches, or conscious manipulation.
The system has the ability to provide a wealth of
information to the user without forcing the user to
dedicate their attention to accessing that information.
Through these more subtle interactions the bag starts
to speak to the idea that by using ubiquitous computing,
previously non digital products can provide less
intrusive haptic feedback that is relevant specifically to
that product or that products function. The backpack
reflects this as a physical realization of the concepts of
transience and personal security, the same concepts
embodied in its haptic feedback. As an example, a door
knob could heat and cool to reflect the temperature
outside, thus alerting the user to what clothing should
be chosen to remain comfortable. Computer mice with
scrolling wheels could change their physical resistance
when scrolling to reflect the size of the webpage being
navigated. The backpack serves as a discussion on
how information can be observed and accessed in new
visceral ways.
There are many different potential use cases for a haptic
enabled safety conscious backpack. The most obvious
case was explored through the prototype testing with
students, but other groups such as tourists could also
benefit from the use of this system. With both students
and tourists, the backpack allows the user to feel
more comfortable exploring their surroundings while
discreetly promoting confident decision making. One
of the most interesting potential applications of such
a system is its military applications. The conceptual
space in which the project was originally formatted is
relatively low risk, but the system could easily be shifted
to cater to a high risk environment. Such a system could
indicate to soldiers areas with a high incidence
141
of attacks, or high potential for attacks, quietly and

without forcing them to divert their attention from
their surroundings. For a military application this
system has the benefit of being stealthy, intuitive, and
easily integrated into existing equipment already used
in the field.
5 Conclusion
Innovations in portable screen based interfaces have
allowed us to become incredibly mobile and still have
near instant access to vast amounts of information.
Unfortunately as we gain access to more and more
information on our devices, people become increasingly
distracted and lost in the overwhelming amount of
data that becomes available. This fascination has
demonstrably caused a cultural loss of situational
awareness. Project Loci explores the likely future where
everyday objects have embedded computing power
and communicative properties. Embedded ubiquitous
computing allows these objects to provide a subtle,
intuitive, and active way of accessing and presenting
information, freeing individuals from diverting significant
portions of their attention to handheld devices. Unlike
the passive nature of screen-based interfaces, these
artifacts are able to initiate physical communication
between the device and the user providing for more
intuitive and natural experiences. They allow people
to actively focus on our environment instead of
dividing their attention between their screens and
their surroundings.
By designing haptic interactions that reflect peoples
natural interactions with each other, there is almost no
learning curve to interpreting the information provided.
Leveraging natural and intuitive social interactions allow
for information to be presented in subtle and nuanced
ways, something screen based interfaces fail to do.
Through seemingly simple interactions, information
can be presented with a relatively low fidelity, but
interpreted at a much higher level because of peoples
natural understanding of physical interactions between
one another.
Haptics enable objects to liberate the user from
their screens and enable a wealth of alternative ways
to transmit and interpret information. They dont
require constant interactions initiated by the user
142
and allow the user to be present and attentive to

their surroundings, instead of dissociating us from our
immediate environments like screen-based systems. By
being mentally present and aware of their surroundings,
people are able to pick up on subtle environmental clues
and make more conscious decisions to keep themselves
safe. This liberation translates well to almost all aspect
of our lives. Given the appropriate form factors and
placement in the right context, the fusion of everyday
objects with haptic technology can be seamless,
intuitive, and help create a more aware and confident
world.
Acknowledgments
This project was funded in part by the Undergraduate
Research Office at Carnegie Mellon University. These
results represent the views of the authors and not those
of Carnegie Mellon University.
Dr Edgar Rodrguez Ramrez, Kah Chan

School of Design, Victoria University of Wellington, New Zealand
{edgar.rodriguez, kah.chan}@vuw.ac.nz
Smart interactions for home

healthcare: A semantic shift
Abstract
A semantic shift is happening in the health industry.
Healthcare is moving ever more towards home recovery
and care, while time spent at hospital keeps reducing.
This is beneficial for patients with faster recovery times
and for the health industry through reduced costs.
Home healthcare means that medical devices that
assist people to look after themselves now need to
establish an appropriate communication loop with the
patient. There is no longer a focus on the medical device
communicating with the medical practitioner through
mainly only denotation of meaning. We suggest that the
new communication loop implies that the medical device
can sense information from the patients body, it can
react to the data gathered and it can communicate back
to the patient through denotation and connotation of
meaning: making the information relevant for peoples
everyday lives, addressing pragmatic and hedonic
aspects, and not only through the display of data.
This paper analyses a number of medical devices for
home healthcare. We suggest a set of criteria that
designers can use when designing smart interactions
for empowering patients to take care of their health.
We present a number of designs from the School of
Design, Victoria University of Wellington and assess
them according to our suggested criteria.
Keywords
Smart interactions, healthcare, sensing human data,
semantics, connotation, medical device design,
exergames
1 Introduction
Technology for sensing the human body in the health
industries has largely focused on gathering body
responses relevant to the medical practitioner. With
good reason, the study of human factors in medical
devices has concentrated on making sure that usererror was minimised by the medical practitioners.
Tongson reported for example on the need to incorpo
rate human factors into the design of analgesic pumps,
after a technician entered the wrong setting which
resulted in the tragic death of a patient (1998, p. 1484).
Research on human factors which focused on the
objects interaction with the medical practitioners has
been a necessary field in the design of medical devices.
However, there is now a complex challenge to respond
to worldwide trends that clearly indicate the move from
hospital-based healthcare to home-based healthcare
(see for instance, Davis et al., 2010; Koch, 2006;
Shepperd et al., 2013; Tinetti, Charpentier, Gottschalk,
& Baker, 2012).
143
There are several reasons for the health industry to

move towards home-based healthcare. It has been
well documented that patients tend to improve
faster at home than at hospitals (for instance, see
Eriksson, Lindstrom, & Ekenberg, 2010; Kowalczyk,
2013; Landro, 2010). Depression and infection rates
increase the longer a patient stays at hospital (Husaini
et al., 2013; Prina et al., 2012; Sheeran, Byers, & Bruce,
2010). A Cochrane study revealed that individualised
discharge planning that involves home care brings about
reductions in hospital length of stay and readmission
rates for older people admitted to hospital with a
medical condition (Shepperd et al., 2013, p. 2). Homebased healthcare will also help reduce hospital costs
in space, equipment and staff hours (see for instance,
Feldstein, 2011; Keehan et al., 2008; Koch, 2006; Sheps
et al., 2000). Governments around the world are now
encouraging and supporting initiatives towards homebased healthcare given the benefits for the patients and
for the hospitals.
Home-based healthcare can give a patient independence
that comes with the responsibility of taking care of
their own treatment. Many patients still need relevant
information about their bodies condition to enable
them to make the right decisions and follow the best
treatment. While patients are away from medical
practitioners, they need medical devices that are
focused on gathering body responses relevant to the
patient, so that they can be empowered to follow their
care with the right kind of information that is relevant
for the patient and not unnecessarily technical. The
Qual-comm Tricorder Xprize is an example of the move
towards monitoring and diagnosing peoples health
conditions. The prize intends to give individuals greater
choices in when, where, and how they receive care.
The prize is a US$10 million international competition
due in 2015 to stimulate innovation and integration of
precision diagnostic technologies, making reliable health
diagnoses available directly to health consumers in
their homes (Qualcomm, 2013).
Technology for sensing the human body is rapidly
developing. We can now measure heart rate, breathing
rate, skin conductivity, brainwaves, neuronal activity,
chemical composition of body fluids, muscle activity,
weight, body-fat percentages, among many other
aspects of our physiologies. The quantified self
144
movement (Swan, 2009) is a key indicator of the

progress and ubiquity of this technology; where all
this data is tracked using smart devices and shared and
compared online using available social networks (Swan
2012a, 2012b).
2 Devices shift from talking to the medical

practitioner to talking to the patient
From this point of view and following semantics
terminology, the communication loop between medical
devices and humans is experiencing a paradigmatic shift
where the decoder and interpreter of the information
is changing from the medical practitioner to the patient.
Similarly, the feedback loop for affecting behaviour
change is shrinking: a patient no longer requires explicit
advice from a medical practitioner when they can
monitor the effects of their actions and intrinsically
instigate change in behaviour.
According to the discussion above, we suggest that
medical devices that support home healthcare can have
the potential to address the following three criteria:
Sense human data that is relevant to the patients
condition: what is important to measure?
React to the human data: what should be reacted to
and how?
Inform the patient: what does the patient need to
know, and how should this information be delivered?
We suggest that there is a big opportunity for design
to respond to the necessary semantic change in
medical devices for home healthcare. These devices
have the potential to sense human data in seamless
ways (potentially through wearable devices), digitally
process the data to make it relevant to the patients and
present the data through physical or digital reactions
in appropriate ways to the patients. We suggest that
disciplines such as psychology, health-related disciplines,
engineering, industrial design and media design can work
together to create seamless experiences to empower
patients to take control of their treatments.
This is an area that Human Computer Interaction
(HCI) has started to address. Early work on HCI and
medical devices followed the same pattern of focusing
on the communication between devices and medical
practitioners to reduce errors (for instance, see

Iakovidis, 1998; Obradovich & Woods, 1996). More
recent HCI research has focused on more relevant
gathering of information from the human body. There
have even been calls to ensure that HCI technologies
have a self-evaluating system to assess whether people
perform behavioural changes that improve their health
(Klasnja, Consolvo, & Pratt, 2011).
Digital technologies have the potential to translate raw
human data into relevant experiences for patients. For
instance, videogames that facilitate exercising (exergames) have been used for treating health issues (for
instance, see Gerling, Schild, & Masuch, 2010; Sinclair,
Hingston, & Masek, 2007; Staiano & Calvert, 2011).
Exer-games can be used to help patients engage in
their own treatment at home. This could be further
improved if the exergames offer relevant challenges that
depend on the performance of the patients through the
tracking of relevant human data. For instance, heart rate
could measure how much a patient could be challenged
to perform aerobic physical activity, and increase
resistance through game difficulty appropriately to
customise the recovery rate to the individual patient,
or allow them to engage with their treatment socially
by playing with other patient players (Muller, 2003).
This paper analyses medical devices that target home
treatment where the patient needs to make decisions
based on the information gathered and presented by
the device. The paper then suggests a framework and
a set of criteria for designing medical devices when
it is important to shift the semantic dynamics from
medical practitioner to patient. We present a set of
designs from students at the School of Design, Victoria
University of Wellington, and we analyse them according
to the criteria suggested. Finally, we conclude suggesting
a framework that designers of medical devices may find
useful to make sure that their designs offer a relevant
communication system with the patients.
(FDA) sections on Home Healthcare Medical Devices

and Home Health and Consumer Devices, and design
blogs presenting real and conceptual designs including
designboom, dexigner, core77, mocoloco.
The American Association for Homecare and FDA
highlight the need for devices that help people treat
diabetes (blood glucose monitoring and pumps), physical
rehabilitation following accidents and strokes (exercising
and electronic muscle stimulators), breastpumps and
hearing aids. We present a brief overview and analysis
of four medical devices that address the initial three
criteria of our study: sensing human data, reacting to
human data and informing the patient.
Vitalograph asma-1 PEF meter by CRF health
(figure 1). Vitalograph asma-1 is a spirometer that
measures lung function normally used by patients with
asthma. It communicates with an electronic diary. The
human data measured is lung function. The reaction to
the human data is the collection of the information into
an electronic diary that sends the data to the medical
practitioner. The electronic diary reminds patients
when the next reading needs to occur. The electronic
diary also reminds patients when to take their medicine.
The system intends to be easy to use and encourage
compliance and retention. Its automated nature
prevents manual data transfer mistakes. The
data presented to the patient is numeric and not
translated into a meaningful connotation.
Fig. 1. Vitalograph asma-1. Source http://crfhealth.com/solutions-
3 Analysis of existing devices
services/ecoa-tools/medical-device-integration
We have focused our analysis on devices meant to be

used by the patient, gather data from the human body
and react to the data gathered or present it to the patient.
We performed a search on the American Association
for Homecare, the US Food and Drug Administration
Able-X by Im Able (figure 2). Able-X is a system

for treating stroke, cerebral palsy and other brain and
arm-coordination impairments (Im-Able, 2013). The
system consists of a bar that is held with both hands
145
and moved around. The bar wirelessly connects to an

exergame designed to encourage patients to move both
arms and follow their exercise treatment. The system
follows the principle of using the not affected side of
the body help the affected side recover mobility and
brain coordination. Following our set of criteria, the
system gathers data from the human body through
tracking movement of the arms. It reacts to the data
and presents it to the patient through a screen-based
exergame. The human data is informed to the patient
through relevant media that offers a clear connotation:
movement presented through the videogame. The
patients are encouraged and can therefore take control
of their own treatment.
Fig. 3. Fluenci breastfeeding pump by Jaap Knoester,

Tom Djajadiningrat and Philip Ross.
Fig. 4. Fluenci being used by a participant.
Fig. 2. Able-X by Im-Able.
Fluenci breastfeeding pump by Jaap Knoester, Tom

Djajadiningrat and Philip Ross (figures 3-4). Fluenci is a
breastfeeding pump that mimics many of the stimulating
triggers provided by a baby for a mother who needs
to express milk (Knoester, Djajadiningrat, & Ross,
2012). For instance, the pump is heated to resemble a
baby and expand the mammary ducts. The pump also
mimics a babys suckling behaviour by starting to pump
superficially and quickly to stimulate let down, and
once milk starts flowing (which for the purposes of this
study represents the sensing of human data) the pump
slows down and pumps at a lower frequency. The shape
of the pump conceals the nipples and intends to avoid
connotations of medical equipment.
One of the key factors to consider about Fluenci is
the investigation focused on the experience for the
mother and her relationship with her baby. It means
that the design offers an experience that addresses
pragmatic and hedonic aspects of breastfeeding that
open a communication loop between object and user. In
this case, the consideration of the hedonic experience
146
not only offers a more enjoyable experience, but also

stimulates the production of milk by the mother by
making her feel more relaxed.
The Aid by Egle Ugintaite (figures 5-7). The Aid is
a walking cane that incorporates sensors to monitor
pulse, blood pressure and body temperature through
contact with the hand of the patient. It incorporates
an SOS button that can also be automatically activated
if the person is in trouble. An LCD displays data and
helps with navigation. According to our criteria, the Aid
senses human data, reacts to it by sending an SOS signal
if necessary and informs patients of their health status.
However, the display offers a denotation of the data:
it only gives numeric data that may be hard to relate
to by the patient. It does not offer a connotation that
the patient can interpret according to their everyday
experiences.
The analysis of these existing medical devices suggests
that we can add the following criterion to our study:
Use of tangible or digital interactions that enhance the
experience from cognitive, behavioural or emotional
perspectives through the connotation of information.
We suggest the following four criteria according

to the discussion above:
a) Sensing patient-relevant data
b) Reacting to patient-relevant data
c) Informing the patient
d) Incorporating a digital or tangible interaction that
creates a relevant connotation-based experience
Fig. 5. The Aid by Egle Ugintaite.

Source http://www.designboom.com/technology/the-aid-byegle-ugintaite-fujitsu-design-award-2011-grand-prize/
Fig. 6. The Aid and display of human data for the patient.
Fig. 7. The Aid in use.
While traditional medical devices meant to give harddata to practitioners to help them make decisions,
the semantic shift to communicating with the patient
needs a more human approach. Able-X is an example
of a device that through the use of exer-games not
only facilitates exercising, it also encourages it and
enhances the experience through the challenge of a
videogame that patients can play as part of their exercising. Fluenci exemplifies the opportunity to investigate
human experience and design with it as a starting
point, including pragmatic and emotional aspects in the
design. The communication between the object and the
person is different to what breastfeeding pumps used
to be when they only needed to communicate with the
medical practitioners.
4 Analysis of designs from students at

the School of Design, Victoria University
of Wellington.
The design curriculum at Victoria University of
Wellington is constantly engaging with the forefront
of parametric modelling, digital manufacturing,
and development of products with an integration
of hardware, sensors and software. The unique
programme where Industrial Design students can work
closely with Media Design and Engineering students
fosters an interdisciplinary cohort. We present and
analyse against our proposed criteria a set of designs
from design students for personal healthcare.
Cortex 3D printed cast by Jake Evill (figures 8-10).
The cortex system gathers data from the human body
through two steps. Firstly, it 3D scans the shape of the
broken limb. Secondly, it locates the position of the
fracture. The system reacts to the human body gathered
data through creating a 3D shape that wraps around
the 3D scanned broken limb. The shape also reinforces
the area where the fracture is by increasing the density
of the material applied based on the location of the
fracture. The shape is finally 3D printed and fitted to the
patient. This system allows for a cast that is breathable
and allows access to the skin for washing and scratching.
This system does not offer further information to the
patient for their recovery.
Nexstep leg prosthetics by Cameron Lightfoot
(figure 11). Nexstep is a prosthetic leg that uses
neodymium magnets behind the knee to create force
and movement, allowing the leg to extend, followed
by magnets in front of the knee connecting, locking
the knee and completing the walking motion. A similar
reaction is repeated in the ankle area. Matching polarity
means that the person does not need to swing the leg
as the magnets complete the movement. This design
does not use digital sensors. It relies on sensing human
147
Fig. 8. Cortex 3D printed cast by Jake Evill
Fig. 9. Cortex 3D printed cast process
Fig. 10. Cortex 3D printed cast detail.
Fig. 12. The Revival Vest by James McNab.
Fig. 11. Nexstep prosthetics by Cameron Lightfoot.
data, in this case movement, and reacting mechanically

and automatically to it, giving patients feedback also
in the form of movement. There is no hard data to
understand, only relevant movement to help people
walk again.
The Revival Vest by James McNab (figures 12-13).
The revival vest is a self-inflating life jacket that detects
the breathing rate of free-divers, those who do not
use scuba diving equipment. When the vest detects
that the diver is not breathing, it self-inflates bringing
the diver safely back to the surface. While this is not a
medical device per se, we find it relevant for this study
as it complies with our criteria: it senses human data, it
responds to human data and it informs the user allowing
them to take control of their own care. If the vest needs
to self-inflate, the user has become a patient that needs
medical care.
Fig. 13. The Revival Vest process.
Fig. 14. N-One Intravenous pump by Matt Backler.

148
N-One Intravenous pump by Matt Backler (figure

14). N-One is an intravenous pump that intends to
empower patients to leave the hospital, at least for brief
periods of time, by offering a portable pump that can
be worn on ones arm. The design moves away from a
semantics reflecting hospital environment towards a
more wearable and jewellery-related type of aesthetics.
The pump incorporates an intelligent motor that can
sense blood flow, reacting and giving the right dose to
the patient accordingly.
Design
Sensing
patient
relevant
data
Reacting
to patient
relevant
data
Semantics
inform the
patient
Tangible
interaction
that
enhances
the
experience
Digital
interaction
that
enhances
the
experience
Cortex 3D
printed cast
Yes
Yes
Yes, through
the shape
Yes
No
Nexstep
prosthetics
Yes
Yes
Yes, through
movement
Yes
No
Revival vest
Yes
Yes
Yes, through
behaviour
Yes
No
N-One
Intravenous
pump
Yes
Yes
No
Yes
No
5 Discussion
All of the products included in the study sense human
data in one way or another. Many do it digitally and then
convert the data into relevant information. Others,
like the Nexstep prosthetics, sense the data through
magnets and react to it automatically and mechanically,
without a translation of the data. The increasing
technological power to sense human data will only
allow us to expand the range of applications for home
healthcare.
While some products reacted automatically, others
translated the data into information relevant to the
patients. For instance, the Able-X exergames are a
good example of converting raw data into a meaningful
experience for the patients, who do not need to
understand what the data is, they only need to be
encouraged to perform their exercises.
While a digital interaction seems to be a promising

way to offer relevant information to the patients, very
few medical products seem to include it. Able-X is an
exception, offering a combination of an exercising
device and a set of digital exergames to encourage
and facilitate its use. The possibility to offer a digital
interaction needs to be balanced with its need and the
potential benefits. For instance, the Cortex cast does
not offer a digital interaction to the patients beyond
the scanning of their limbs which would be carried out
by a medical practitioner.
5.1 From denotation for the medical practitioner
to connotation for the patient
Fluenci is a good example of creating a connotationbased experience for the user, in this case the
mother. The device mimics baby-behaviour, making
the experience related to the pragmatic and hedonic
experience of the mother and not only a practical task
of extracting milk.
It is important to note that the semantic meaning of
a connotation that an experience can offer relies on
its underspecification, which is its context-sensitivity
(Dalrymple, 1999, pp. 127130). While considering
the communication between devices and people as a
loop serves us as a starting point to develop this initial
framework of criteria, it is necessary to consider that
multiple loops will play a role in the actual use of the
objects. It is the role of the designers to attempt
to offer connotations that consider the different
contexts in which the objects will be used, as the
intended connotation may change meaning depending
on the context.
There have been important developments into the
design of tangible interactions that focus on the
experience of people beyond a screen or beyond
presenting data. For instance, the Quality in Interaction
group at Technical University of Eindhoven has made
great strides towards the research and design of rich
interactions that focus on the aesthetics of interaction
rather than only on the aesthetics of appearance. If we
consider the aesthetics of interaction in the design of
home healthcare devices that can sense human data and
respond to it, the aesthetics of the interaction can have
a connotation type of approach that offers meaningful
information and interaction to patients.
149
6 Conclusion
Increasing home healthcare and moving away from
hospital-based treatment is a tremendous shift that
brings important positive outcomes, but it also
represents a great challenge for the medical profession
to ensure patients are properly looked after and that
their treatment and healing is indeed better than at a
hospital. From the patients point of view, while being
treated at home may speed recovery and reduce
the risk of depression and infections, it can also be
overwhelming to suddenly need to know how to treat
their condition on their own.
connotation. These interactions can improve peoples

lives and help them improve their health and recovery.
They can also offer more enjoyable experiences through
interactions that involve appropriate reactions to the
human body. We call this framework Smart Interactions.
References
Davis, J. C., Robertson, M. C., Ashe, M. C., Liu-Ambrose, T.,
Khan, K. M., & Marra, C. A. (2010). Does a home-based
strength and balance programme in people aged >80 years
provide the best value for money to prevent falls? A systematic review of economic evaluations of falls prevention inter
We suggest that medical devices can facilitate home

healthcare and offer benefits to patients by offering
them relevant information that respects their under
standing of the world and their own bodies and medical
conditions. In particular and from a semantics point of
view, medical devices meant to be used by the patients
need to offer relevant information that expresses
connotation through items relevant to the patient,
and not only denotation through presenting hard-tounderstand data.
ventions. British Journal of Sports Medicine, 44(2), 8089.

Eriksson, L., Lindstrom, B., & Ekenberg, L. (2010). Patients
experiences of telerehabilitation at home after shoulder joint
replacement. Journal of Telemedicine and Telecare, 17(1),
2530. doi:10.1258/jtt.2010.100317
Feldstein, P. J. (2011). Health care economics. Clifton Park, NY:
Delmar Cengage Learning.
Gerling, K. M., Schild, J., & Masuch, M. (2010). Exergame
Design for Elderly Users: The Case Study of SilverBalance.
In Proceedings of the 7th International Conference
on Advances in Computer Entertainment Technology.
There is a myriad of questions still to be addressed

in this area of design. While we offer a broad initial
framework and set of criteria for designing smart
interactions, designers will need to investigate deep
into specific areas of healthcare and find out the
particular challenges for different conditions and in
different contexts.
Presented at the ACE 10. Retrieved from http://dl.acm.org/

citation.cfm?id=1971650
Husaini, B., Levine, R., Sharp, L., Cain, V., Novotny, M., Hull,
P., ...Moonis, M. (2013). Depression increases stroke
hospitalization cost: An analysis of 17,010 stroke patients in
2008 by race and gender. Stroke Research and Treatment,
2013, 17. doi:10.1155/2013/846732
Iakovidis, I. (1998). Towards personal health record: Current
Medical devices can offer important advantages by

sensing relevant human data to which they react and/
or present to the patients. This can be a process of
empowering patients through the intrinsic reward
of being able to look after themselves in a more
comfortable and familiar setting: their own homes.
situation, obstacles and trends in implementation of

electronic healthcare record in Europe. International Journal
of Medical Informatics, 52(1-3), 105115. doi:10.1016/S13865056(98)00129-4
Im-Able. (2013). Able-X. Retrieved July 31, 2013, from http://
www.im-able.com/nz/product/able-x
Keehan, S., Sisko, A., Truffer, C., Smith, S., Cowan, C., Poisal,
In the bigger scheme of things the framework we

suggest can have multiple applications beyond the
health industries. The interactions between objects and
humans can be enhanced and made smarter if industrial
and media designers work together with mechatronic
and electronic engineers, computer scientists,
psychologists and other experts for sensing human
data, processing it, reacting to it, presenting it to the
person in meaningful ways that involve denotation and
150
J., ...the National Health Expenditure Accounts Projections

Team. (2008). Health spending projections through 2017:
The baby-boom generation is coming to medicare. Health
Affairs, 27(2), w145w155. doi:10.1377/hlthaff.27.2.w145
Klasnja, P., Consolvo, S., & Pratt, W. (2011). How to evaluate
technologies for health behavior change in HCI research (p.
3063). ACM Press. doi:10.1145/1978942.1979396
Knoester, J., Djajadiningrat, T., & Ross, P. (2012). Fluenci: The
expression of expressing. In E. R. Rodrguez Ramrez (Ed.),
Design and semantics of form and movement (pp. 4651).
Sinclair, J., Hingston, P., & Masek, M. (2007). Considerations
Presented at the DesForm 2012: Meaning. Matter. Making.,
for the design of exer-games. In Proceedings of the 5th
Wellington: Philips. Retrieved from http://www.northumbria.
international conference on Computer graphics and
ac.uk/static/5007/despdf/designres/desform201proceedings.pdf
interactive techniques in Australia and Southeast Asia. Pre-
Koch, S. (2006). Home telehealth: Current state and future

trends. International Journal of Medical Informatics, 75(8),
565576. doi:http://dx.doi.org/10.1016/j.ijmedinf.2005.09.002
Kowalczyk, L. (2013). Hospitals reach into patients homes,
deliver meals. The Boston Globe, (June). Retrieved
from http://www.bostonglobe.com/lifestyle/health-
sented at the GRAPHITE 07. doi:10.1145/1321261.1321313

Staiano, A. E., & Calvert, S. L. (2011). Exergames for physical
education courses: physical, social, and cognitive benefits.
Child Development Perspectives, 5(2), 9398. doi:10.1111/
j.1750-8606.2011.00162.x
Swan, M. (2009). Emerging patient-driven health care models:
wellness/2013/06/09/hospitals-reach-into-patients-homes-
An examination of health social networks, consumer
deliver-healthy-smoked-tofu-but-also-bbq-shrimp-scampi/
personalized medicine and quantified self-tracking.
QSqNQiycZUNxa9qbRWzs8I/story.html
International Journal of Environmental Research and Public
Landro, L. (2010). Patient, heal thyself. The Wall Street Journal,

(October). Retrieved from http://online.wsj.com/article/SB10
001424052702304248704575574172720039534.html
Muller, F., Agamanolis, S., & Picard, R. (2003). Exertion Interfaces:
Sports over a Distance for Social Bonding and Fun. In
Health, 6(2), 492525. doi:10.3390/ijerph6020492

Swan, M. (2012a). Health 2050: The realization of personalized
medicine through crowdsourcing, the quantified self, and the
participatory biocitizen. Journal of Personalized Medicine,
2(4), 93118. doi:10.3390/jpm2030093
Proceedings of the SIGCHI confer-ence on Human factors
Swan, M. (2012b). Sensor mania! The internet of things, wearable
in computing systems. Presented at the CHI 03. Retrieved
computing, objective metrics, and the quantified self 2.0.
from http://dl.acm.org/citation.cfm?doid=642611.642709
Journal of Sensor and Actuator Networks, 1(3), 217253.
Obradovich, J. H., & Woods, D. D. (1996). Users as designers:

How people cope with poor HCI design in computer-
doi:10.3390/jsan1030217
Tinetti, M. E., Charpentier, P., Gottschalk, M., & Baker, D. I.
based medical devices. Human Factors: The Journal of the
(2012). Effect of a restorative model of posthospital home
Human Factors and Ergonomics Society, 38(4), 574592.
care on hospital readmissions. Journal of the American
doi:10.1518/001872096778827251
Geriatrics Society, 60(8), 15211526. doi:10.1111/j.1532-
Prina, A. M., Huisman, M., Yeap, B. B., Hankey, G. J., Flicker, L.,
Brayne, C., & Almeida, O. P. (2012). Association between
5415.2012.04060.x
Tongson, N. (1998). Incorporating human factors into the design
depression and hospital outcomes among older men.
of medical devices. The Journal of the American Medical
Canadian Medical Association Journal, 185(2), 117123.
Association, 280(17), 1484.
doi:10.1503/cmaj.121171
Qualcomm. (2013). Qualcomm Tricorder XPRIZE. Qualcomm
Tricorder XPRIZE. Retrieved July 31, 2013, from http://www.
qualcommtricorderxprize.org/
Sheeran, T., Byers, A. L., & Bruce, M. L. (2010). Depression and
increased short-term hospitalization risk among geriatric
patients receiving home health care services. Psychiatric
Services, 61(1), 7880.
Shepperd, S., Lannin, N. A., Clemson, L. M., McCluskey, A.,
Cameron, I. D., & Barras, S. L. (2013). Discharge planning
from hospital to home. In The Cochrane Collaboration & S.
Shepperd (Eds.), Cochrane Database of Systematic Reviews.
Chichester, UK: John Wiley & Sons, Ltd. Retrieved from
http://doi.wiley.com/10.1002/14651858.CD000313.pub4
Sheps, S. B., Reid, R., R., Barer, M. L., Krueger, H., McGrail, K. M.,
Green, B., ...Hertzman, C. (2000). Hospital downsizing and
trends in health care use among elderly people in British
Columbia. Canadian Medical Association Journal, 163(4), 397401.
151
Jing Gu, Yu Zhang, Jun Hu

Design for Elderly with Dementia:

Light, Sound and Movement
Abstract
In this paper we present an interactive lighting and
sound art installation designed for the elderly with
dementia living in a confined area in an elderly care
center. Inspired by lighting and music therapies for
dementia, as well as the social and cultural activities
of healthy elderly, the installation creates sensory
stimulation and natural atmosphere, aiming at bringing
joy and happiness to dementia elderly in a dull corridor.
The technical details are briefly introduced, followed by
evaluation results.
Keywords
Lighting, sound, music, interactive, installation, elderly
care, dementia
1 Introduction
Nowadays, the number of dementia elderly is increasing
and the pressure put on caregiving is demanding.
Research has indicated that the Behavioral and
Psychological Symptoms of Dementia (BPSD) are
associated with the increased burden of care, which
has increased rates of institutionalization of patients
[1]. The dementia elderly who live in the care centers
have limited access to other people and outside
environments. A large part of the life of the dementia
elderly is restricted by the lost cognitive ability and
decayed memories. Such liabilities determine the passive
152
roles dementia elderly play in learning activities. Care

centers thus provide manually organized activities with
clear goals to benefit the recovery of cognitive abilities.
However, the manually organized course-like activities
cannot extend to a larger scale of daily activities.
Emotional and mental wellbeing are not fully taken care
of in such settings. Although dementia elderly suffer
from some cognitive liabilities, they are eager about
happiness and well-being, for which activities in care
centers need to be carefully designed and organized.
One of the local care centers of de Vitalis WoonZorg
Groep in Eindhoven, expressed their interests in
designing a meaningful activity in a corridor through
which the dementia elderly stroll from their bedrooms
to the common room and cafe. Most of the elderly
there are in second to fourth period of disease in their
late eighties. The corridor connects bedrooms and
common room in which the dementia elderly spend
most of their time. The corridor is relatively dark for
dementia people where no natural light sheds in and the
lighting system keep static and constant. The space is
perceived to be dull and boring.
The definition of happiness can be vague and is
extremely personalized feeling related to personal
background and cognitive ability of emotion. Most
researches on dementia define practices that would
have a positive effect on dementia elderlys physical and
physiological performance. However, when it comes

to design, more investigations into contextual and
characteristics of a certain group of people are needed.
This project started with looking into the concept of
meaningful activities for dementia and its connection
to the feeling of happiness. A meaningful activity for
the dementia elderly should allow them to experience
a sense of wellbeing, a sense of belonging and sustained
identity, a sense of continuity of their lifestyle, and their
very sense of self [2,3]. Meaningful activities may lead
to hedonic and eudemonic happiness depending on
whether the activity contributes to a pleasurable life or
a meaningful life [4].
The project continued with getting inspiration from the
lighting and music therapeutic methods for dementia as
well as social and cultural activities of healthy elderly,
in order to design such a meaningful activity for the
dementia elderly at Vitalis. Several concepts were
created and evaluated with experts, leading towards the
final concept that was prototyped and evaluated. Next
in this paper, after a literature review on meaningful
activities for dementia, these design concepts are
presented. The final concept and its prototype are
described in more detail, followed by evaluation results
and the conclusion.
2 Meaningful Activities for Dementia

In therapies for dementia, different methods have
been used in practice. In elderly care institutions, many
different methods are used as well, in order to create
activities that are to certain extent helpful or meaningful
for the dementia elderly.
instrument. Music stimuli may have positive effects on

mood and cognitive functioning, including reduced anxieties,
increased verbal fluency and spatial reasoning [8].
Reminiscence therapy goes beyond one sense, providing
stimulation through multiple sensory channels, such
as sound, movement, smell, flavor, changes in light
and color etc. Doing so could induce vivid and strong
reminiscences [9].
On the website of Opening Knowledge there are many
examples of people applying similar strategies in their
social and cultural activities in protecting themselves
from dementia [10]. Playing piano requires fine motor
skills that improve the circulation to the brain, and
the brain work reduces the risk of dementia (Fig.1).
Music activities would decrease loss of hearing holding
the development of dementia to some extent, and
study shows that daily music-based dance sessions
helps patients with dementia significantly (Fig.2). A
more active lifestyle with physical activity or exercise
helps to prevent a whole number of diseases, including
dementia. Physical exercise may increase temporary
arousal; stimulating cognitive capability (Fig.3). The
results demonstrate the feasibility of achieving higher
levels of well-being and diversity of activity for people
with dementia. Participants benefited regardless of level
of dependency or cognitive impairment. These positive
effects might be brought about by the important role of
activity interventions related to individual interests. The
stimulation of meaningful activities seems to be a powerful
intervention both at home and in institutions. [2]
Lighting has been used in many therapies for dementia.

In general it is found that daily exposure to bright
sunlight is an effective anti-depressive [5]. A prolonged
exposure to daytime lighting improves the stability of
the rest-activity rhythm in the elderly with dementia
[6]. Different colored lights might have different effects,
for example, blue lights help to let body to produce the
feel-good hormone that would contribute to a sense of
wellbeing [7].
Fig. 1: Hungarian-born Pianist Charles Brunner plays the piano
The dementia elderly may still be touched by music

spontaneously be able to sing along with lyrics thought
to be forgotten, or even remember how to play an
to recapture the Guinness World Record for longest keyboard

playing Brunner broke the record of 103 hours and 8 seconds.
(Image from Opening Knowledge)
153
Fig. 2: British DJ Ruth Flowers, 69 years old, mixes music at a

recording studio in Paris. (Image from Opening Knowledge)
Fig. 3: A woman balances the Yue Ya Chan (Crescent Moon

Spade), a traditional Chinese pole weapon, on her head as
she exercises in a park in Xin, Shaanxi province. (Image from
Opening Knowledge)
3 Design Iterations: Lessons Learned from

Early Concepts
The project went through three iterations. First two
iterations resulted in two design concepts (Fig.7 (a-b)).
The concept1 which named Kaleidoshare (Fig.7 (a)) is
an object hung on the wall of the corridor, it encourages
the dementia elderly to become actively engaged
in reliving and sharing their past with others in the
present either verbally or non-verbally. As a means
of psychological support, reminiscence therapy helps
generating a feeling of self-worth and the expression
of individual identity. Based on reminiscence therapy,
reminders such as personal belongings and familiar
music are already confirmed to bring the patients who
impair with cognition great satisfaction. Reminiscence
therapy is a process practiced to evoke personal
memory and experience of the past [12]. When the
elderly play with the old disc-like gadget, commonlyinterested themes such as children, old-fashioned auto,
154
Fig. 10 (a-b): concept1: Kaleidoshare and concept2: Dancing-Ball
scenery would appear on the wall of the corridor.

As most of them have trouble to recognize the
characters in the picture, such way of common
emotional arousal would be preferable to start a short
conversation and response among the elderly who
encounter each other at the corridor. Next to it, when
the elderly rotate the disc, they can feel the speed
of the rotation and the feeling of control under their
fingertips. The Kaleidoscope gives glittering bits and

pieces of life, and thus when the elderly play with the
disc, they can feel actually they are making something
beautiful and meaningful to their lives.
However after evaluating the concept with the experts
and some elderly, it does not seem to be comfortable for
the elderly to stand in front of the wall for a time that
might be too long. If the context is moved to sitting place,
common room would be more suitable for the concept.
The concept 2 is named Dancing-Ball (Fig. 7(b)). The
dancing balls would form a mountain shape according
to the position of the elderly and the peak reflects
the walking manner. The elderly can see that the
movements of the whole system are in accordance with
their own movements. Because of the different pace
and walking manner, everyone would have a different
pattern. When two elders come together, there will
also be a transaction of the wave.
The evaluation of this concept turned out that it was
hard for the dementia elderly to relate the concept
to the things they were familiar to. They were curious
actually about the things they are familiar to but would
like to organize them and work with the concept design
in a different way the concept was too complex for
most of them to comprehend.
Based on the first two concepts, the final concepts
focus is to bring forward something that offer responses
from the surroundings to the elderly, to let them set
free to have a feeling of outside world when walking in
a confined corridor, and to provide a stage to evoke the
liveliness with attention, thinking and responses.
4 The Final Concept

Inspired by the above mentioned methods and the
feedback from the concepts from the first two
iterations, the final design concept is to bring in light
and sound in an installation to help the elderly to get
rid of dull and passive feelings of the confined corridor
area at Vitalis. According to the experiment of the first
iteration, manual facilitate is needed to understand
pictures for the thirds or fourths period of dementia
elderly. Vitalis prefer voluntary activities that dementia
elders could experience on their own when they retour
the corridor habitually. Considering the circulation

function of the corridor, installation should improve the
experience for dementia elderly to pass it by.
The light and sound are designed in a way to reminiscence
the interaction with outdoor nature which is missing
in the confined indoor place. By interacting with the
installation, the dementia elderly perceive the stimuli
that resemble their memories of natural lighting and the
sound of water drops with calm background sound of
a brook, the experience they miss in the care center. It
will be examined in the following test that whether the
installations relive outdoor activities in their previous
years and whether the elderly are actively involved in.
4.1 Concept
The concept is to augment the corridor in the confined
area with the light and sound from the outside world
in response to their movement and touch to help the
dementia elderly get rid of dull and passive timeless
feelings. The concept is based on reminiscence therapy
and implementing lighting and sounds to give stimuli
of memorized events. In the interaction, the dementia
elderly perceive the response from the nature and
they have an active role in making interesting lighting
patterns and the sound of water drops. The interaction
mimics throwing stones in the river with lights blinking
up one by one as the elderly standing in front of the
installation. When several elderly present, flowing lighting
effect and roaring sounds hints the stirring of ripples.
Sound memory as proved in research, has a strong
effect on mood and cognitive functioning until the very
late stage of dementia. The feeling of the natural sound
of water drops is familiar to everyone in their memories
from their childhood when wandering in the wild.
Lighting gives people solid feeling of the elapse of time
and the feeling of liveliness. In a dim corridor where the
dementia elderly would spend hours every day to walk
through, more exposure to light has been proved to be
effective in preventing depression and improving restactivity rhythm.
4.2 Scenario
The design implements the following scenario: Rob is an
elder who suffers at Vitalis. One of the walls of the dull
corridor is now decorated with a row of lights covered
in curved fabrics (Fig. 4).
155
At daytime, glowing blue lights gives a refreshing feeling

for Rob and the others passing by (Fig. 4(a)). During the
night, the light is soothing yellow. The sound of a brook
starts whispering gently, which reminds Rob of the fresh
air and possibly an exciting outdoor travel experience
in his childhood (Fig. 4(b)). Rob hears a chain of water
drop sounds coming with the lights one by one. He is
fascinated and finds it interesting. He is curious about
the lighting and sound effects, and wonders he might
be able to interact with it (Fig. 4(c)). When Rob comes
close, the first light glows brighter. It glows in response
to Robs position (Fig. 4(d)). Rob touches the light and
the touch seems to be sensed. It lights up with solid
color with the sound of a water drop. As he walks faster
and closer, the sound of the whispering brook becomes
louder. The memory of refreshing and playful natural
environment comes back (Fig. 4(e)). There comes the
other elderly person, Marjolijn, walking in the same
direction as Rob. The lights start to flow between
Rob and Marjolijn, and the brook sounds running faster.
Marjolijn walks closer to the lights slowly, and the lights
near her become brighter yet glow slower (Fig. 4(f)).
When Marjolijn walks towards Rob instead, the lights
flow in the manner of joining together.
The lighting augments the encounter of people and the
sounds change as a response to the movements of the
elderly just as what happen outdoors (Fig. 4(g)). We
expect the multi-user interaction would also improve
the feeling of social connectedness [11].
Fig. 4: Scenario
156
4.3 Prototype
The prototype of concept was implemented and tested
with the dementia elderly and the caregiving experts
at Vitalis. The prototype implements the interaction
between two people in the distance, and the effect
of touching. Infrared distance sensors are used to
detect the speed and direction of the movement, and
cap sensors are used to detect touch. When there is
elderly walking closer towards the first light, it becomes
brighter with the background sound of water start
whispering. Lights flashed one by one gradually with
water drop sounds. Once the elder touches the colorful
plastic area, the corresponding lights lit up with water
drop sounds. When there comes another elder, lights
between them would flash with roaming water sound.
4.4 The User test and Results
The final prototype was opened for user test at 3.30pm,
on 20th December 2012 at Vitalis (Fig. 5). Instead of
hanging out in the corridor, the prototype was set on
a long table in the living room where 8 elders with
dementia sitting at another table 2 meters away could
see and experience when others play with the prototype.
However, the prototype was only displayed for a
relatively short period of time. Moreover the actual
concept covered one side walking space in the corridor,
and full size of the concept would have an effect on the
circulation function in the corridor, which was later
discussed with experts from Vitalis.
Fig. 5: The final prototype used in the evaluation
Mood board
Creating mood boards would be a useful way to
reflect the users emotion and attitude towards certain
concepts or products. However creating such a mood
board could be time consuming and, especially for the
dementia elderly, very challenging. Instead, the partici
pants in the evaluation were asked to make choices
from a restricted selection of images for certain
aspects. This supports a more formalized analysis.
On the basis of this prototype, the participants were
asked with the following questions:
- Which image resembles your mood while playing with
the lighting?(Fig. 6(a))
- Which image represents what you are thinking of
while playing with the lighting?(Fig. 6(b))
- Which image can recall the memory you once had
while playing with the lighting?(Fig. 6(c))
The care givers were asked to observe the dementia elderly
also and were asked to answer the following questions:
- Do you think the dementia elderly like the lighting,
and in which ways? And how about Sounds?
- Do they behave relaxed or interested or agitated?
(positive or negative), and in which ways?
- Does the design make the dementia elderly more

active? In which ways?
During the interview, the participants were also asked
to describe why they choose certain pictures in order
to prospect their thinking.
Reflections
In general, the prototype did bring come joy and
happiness to the dementia elderly as expected. The
Elderly loved to keep trying and interacting with
different behaviors of the installation. They loved to
see the glowing and flowing light effects and to hear
the curious water drop sounds after they had touched
the lights. The experts commented it as pleasant;
the elderly love it; they are not bored any more in the
corridor; this provides them with minutes of escape.
During the practice, several behavior patterns are
observed: the elderly keep interact with the blue circles
of the installation. Interaction among multiple elderly
is not active as expected because the elderly cannot
perceive the interaction solely on perceiving sound
volume and blinking rates.
The elders were fond of the interaction with stimuli
such as light and sounds. The installation gave them
stimuli for minutes of attention which allowed them
escape from the reality into another place in a pleasing
way, which was considered by the care givers to be
active and healthy. However, the elderly cannot fully
relates the sounds and lighting to the throwing stones
event when interact with multiple elderly. Thus the
voluntary engagement of multiple elderly is not as active
as that of the only person.
Except for reminiscence on object, it is possible to
relive the events for reminiscence based on lighting
and sounds. Lighting and sounds create a low-fidelity
environment and yet effectively vibrate interaction of
elderlys past activities. In addition to lighting frequency,
the elderly are facilitated by color of lights to grasp
instantly what the scenario is.
5 Conclusion
Fig. 6 (a-c): pictures help to present dementia to prospect their thinking
Dementia elderly in the care house spend hours every

day to pass through the corridor because basically their
footprints are limited to bedrooms, common rooms
157
and corridors. Mostly, they view the corridor as outside

world where they pass through every day many times.
11. D. Le, M. Funk, and J. Hu, Blobulous: Computers As Social
The concept of the lighting and sound installation for

the elderly with dementia was directly inspired by the
therapeutic methods and preventive social and cultural
activities. The prototype of the concept was evaluated
and it achieved the goal of bringing joy and happiness to
the dementia elderly. Dementia elderly are fond of the
stimuli that light and sounds brought up and the natural
atmosphere it creates. It gave them several minutes
escape to other place every time they go through the
corridor. Direct response to the body movements and
respond to touching also attracts them to play with it.
12. The Benevolent Society: Reminiscing Handbook for Those
Actors., in CHI 2013 workshop: Experiencing Interactivity in

Public Spaces (EIPS), Paris, 2013.
However the scale of the prototype is much smaller

than the designed dimension. A more comprehensive
user study with a full scale implementation would make
the claim stronger.
References
1. W. Black, and O. P. Almeida, A systematic review of the
association between the behavioral and Psychological
Symptoms of Dementia and burden of care, Statistics, vol.
1999, no. 1999b, 2001.
2. M. Vernooij-Dassen, Meaningful activities for people with
dementia, 2007.
3. A. Phinney, Family strategies for supporting involvement in
meaningful activity by persons with dementia, Journal of
Family Nursing, vol. 12, no. 1, pp. 80-101, 2006.
4. P. Desmet, "Design for happiness; four ingredients for
designing meaningful activities."
5. P. R. Boyce, Human factors in lighting: CRC Press, 2010.
6. J. Van Hoof, M. Aarts, C. Rense et al., Ambient bright light
in dementia: effects on behaviour and circadian rhythmicity,
Building and Environment, vol. 44, no. 1, pp. 146-155, 2009.
7. L. Volicer, D. G. Harper, B. C. Manning et al., Sundowning
and circadian rhythms in Alzheimers disease, American
Journal of Psychiatry, vol. 158, no. 5, pp. 704-711, 2001.
8. L. Buettner, Therapeutic recreation in the nursing home.
Reinventing a good thing, Journal of gerontological nursing,
vol. 27, no. 5, pp. 8, 2001.
9. H.-F. Hsieh, and J.-J. Wang, Effect of reminiscence therapy
on depression in older adults: a systematic review, Inter
national journal of nursing studies, vol. 40, no. 4, pp. 335, 2003.
10. J. Tulloch, Protect against dementia, http://knowledge.
allianz.com/, 2012.
158
Working with Older Adults,2005
Dominika Potuzakova, Loe M. G. Feijs

Department of Industrial Design, Eindhoven University of Technology, Eindhoven, Netherlands
d.potuzakova@tue.nl, l.m.g.feijs@tue.nl
Oris: Bonding through maternal scent
Abstract
1 Introduction
In this paper, we discuss the conceptual design of an

intimate garment worn by mothers of prematurely
born babies that supports the early and continuous
bonding between the two of them by exposing the
baby to her unique maternal scent. Scientific literature
as well as outcomes from interviews with a mother of
prematurely born baby and neonatal nurses refers to
positive impacts of maternal scent on the psychological
bonding between a mother and her child as well as
positive physiological impacts on the babys wellbeing.
These outcomes led us to develop a concept called
Oris, which through its interactions and functionalities
promotes the bonding through maternal scent. Further,
we discuss the semantic interaction design applied
to the concept as well as metaphorical meaning of a
difficultly envisioned human scent. Further, we propose
the application of smart textile materials for collection,
sustaining and diffusion of a human scent in different
environments. Lastly, we discuss the semantic analysis of
the conceptual design.
1.1 Early Bonding Interaction

Healthy new born babies are able to initiate the first
interaction with their mothers [1, 2]. Due to the
chemical changes in a mothers body during birth, a
hormone oxytocin is released [3, 4] It is a hormone
that stimulates positive caring towards a baby and
emotional bonding with a baby [5]. When a baby is laid
on the mothers bare skin immediately after the birth
(i.e. kangaroo care [6]), the mothers nipples and areola
smells the same as the amniotic fluid covering the baby
[1, 3, 7, 8]. The baby starts to lick the last bits of the
fluid on its hands trying to sustain the environment that
he or she was in for the previous nine months [3]. The
taste and smell of the amniotic fluid makes the baby
move towards the same smelling breast and start to
lick it. The first attempt of suckling from the mothers
breast is initiated.
Keywords
Bonding, maternal scent, semantics, interaction design,
smart textile materials
Unfortunately this early bonding interaction is not

possible when a baby is born prematurely. A pre-term
baby usually suffers severe medical problems that need
immediate attention from medical staff. As mentioned
above the skin-to-skin contact (where the baby is laid
on mothers bare chest) is usually postponed until the
medical condition of the baby is stabilized. The parents
contact with the baby is very limited during that
postponement period. Sometimes parents are allowed
to touch their child, talk, sing to it or smell it. However
159
all this occurs in a constricted environment e.g. through

the walls of an incubator so the baby and the parents
cannot interact with each other freely. The initial
bonding is not developed as substantially as if the baby
was healthy and allowed immediate skin-to-skin contact.
Involving parents to contribute to the care of their
prematurely born baby and reading the infants
behavioural cues increase parents confidence in
caregiving and may facilitate bonding [9]. This may
possibly decrease the chance of later disturbances in
the parent-child relationship [9, 10]. It is therefore
presumed that attempts to facilitate and accomplish
this early parent-child bonding may closely relate
to an improvement of the physiological and mental
development of the prematurely born baby as observed
during the execution of the kangaroo care [11, 6].
Research work of S. Bouwstra et al. [12] further
analyses the parents-to-infant bonding experience in
NICU from the users perspective and proposes the
theoretical design framework focusing on experience
design as the most suitable design approach.
1.2 Axillary Smell Recognition
When a baby lies skin-to-skin on a mother that
mothers scent is acknowledged by the baby. This
recognizable odour comes from the maternal axillary
(armpit) region [13] and neck [14]. In these regions
apocrine sweat glands secret fluid that carries an
individual unique human scent [15]. This scent depends
on three factors: genetics; diet or environmental factors
and influence of outside sources (i.e. lotions, soaps,
perfumes, etc.) [16]. Despite these factors a baby can
distinguish between his mothers and another lactating
mothers odor [13]. Mothers carry their olfactory
signature not only from a breast region but also from
armpit and neck regions. Such characteristic body
odour plays an important role in early child-mother
recognition. Further relevance and importance of
early olfactory stimulation in prematurely born infants
is summarized in M. Croes et al. [17] research work
where they also propose examples of implementing
improvements in the future research and design in
NICU (neonatal intensive care unit) environment.
160
2 User Research
2.1 Maternal scent application practices in
NICU (neonatal intensive care unit)
An observation study followed by an interview was
conducted with a mother of a prematurely born
baby in the Mxima Medical Center Veldhoven, the
Netherlands. Out of all NICU practices, giving a
kangaroo care to a prematurely born baby is one of
the closest and strongest contact moments between
a mother and her child therefore we chose to conduct
the observation study at that moment. The observation
study and interview afterwards was carried out in
order to find out the mothers perception of her child
and any behavioural changes in the child while giving a
kangaroo care as well as at times when she is not with
her baby, especially in relation to a scent perception
of one and another. The mother indicated that she
feels good and relaxed while giving a KC to her child;
especially she stresses the fact that it is her moment
of care. She mentioned that nobody else can give
such a care to her child and she feels that they belong
together and having a strong moment together. While
giving KC, she perceives her baby strongly; she looks
at him and calms him down. She perceives the smell of
her baby the same as her breast milk. Therefore when
she smells breast milk from her clothes at home there
is an immediate mental connection with her child. She
mentioned that her baby likes her hand over his body
and as soon as she takes the hand of, the baby perceives
it negatively and starts crying. She indicated that one of
the hardest moments in NICU is when she is leaving her
baby behind to the care of the nurses. She feels helpless
that she cannot care for him by herself. Further, the
observation showed that as soon as the child was put on
a bare skin of a mother it gradually calmed and relaxed.
The baby was in a close contact with mothers skin,
especially in an area of breast, neck and towards armpits.
Several interviews with neonatal nurses from three
different countries (the Netherlands, U.S. and Czech
Republic) were also repeatedly conducted in order to
gain insight into the caring procedures and practices
related to a scent application in the NICU environment.
The main outcome of these interviews was the
similarity of one particular procedure including the
maternal scent. Across all three cultural practices the
nurses supported the mothers to wear either a piece
of cotton cloth or a t-shirt overnight and leave that
piece of clothing with the baby in the incubator after

their next visit of NICU. The nurses indicated that such
practices are either passed on from the senior nurses
or they have read about the positive implications on
calming the baby or they have experienced the calming
effect on a baby by themselves via their practices. In any
cases, they support this practice and cannot see it as
harm to a baby, quite the opposite.
3 Design ideation
3.1 The embodying of scent interactions
Designing with a scent as one of the sensory perception
implies challenges. Scent as such may, by many, be
perceived as transparent, without one defined shape,
color or any other embodiment. Professional perfumers
may claim the opposite, since their extensive olfactory
expertise is associated with a functional reorganization
of key olfactory and memory brain regions, explaining
their extraordinary ability to imagine odours and create
fragrances [18].
We, as designers usually do not have an extensive
olfactory expertise, on the other hand we possess the
ability of applying inspirational design methodologies
in order to find out about the embodiment as well as
interactions connected with the difficultly imagined
visualizations. We therefore decided to conduct an
Interaction relabeling session [19] with participation of
creative professionals from industrial design and human
technology interaction domains. This methodology
supports the exploration of new interactions by an
inspiration from the mechanical properties of an
introduced object. These interactions are then assigned
onto the conceptual product to be designed, in our
case bonding by scent product. Participants were
also introduced a set of tasks that they had to perform
and imagine while using the introduced object. An
example of tasks was: Imagine: how you would load the
object with a smell. Imagine: how you would preserve as
much scent as possible during transportation. The objects
introduced to the participants were as following: an
umbrella, mechanical wine bottle opener, measuring
tape and vegetable press. One object particularly, the
vegetable press Fig. 1, brought the most interesting
interactions out of all introduced objects. The vegetable
press with its holes at the bottom as well as turning
properties of the mashing part that lead to opening and
closing the holes, inspired the participants to think of

collecting, carrying and releasing the scent through the
holes. Out of about forty ideas, ranging from simple to
sophisticated ones, we selected three very sound ideas
where participants indicated:
1 (The holes could be an) indication of when a scent is
loaded in an object when the holes are closed the smell is
loaded, when they are open the scent is not there anymore.
Directly show where in the object the scent is loaded with
the handle (of the vegetable presser) and the holes moving
into position, position and so on.
2 The scent could be preserved in some sort of capsules
that would fit into the holes. Once the capsules are placed
in the holes, turning of the handle can pop the capsules out
and the scent is released.
3 Preserving a scent could work on the basis of closing
itself in and opening itself out by reacting on for example
temperature, when it is cold outside it closes, when it is
warm it opens.
Fig. 1. Vegetable press used in Interaction relabeling session
4 Concept Design
Building on the observation and interviews with a
mother and neonatal nurses as well as the inspirational
Interaction relabeling session, a concept of an intimate
garment worn by mothers of prematurely born
babies called Oris was developed, Fig. 2. The shapes
bubbles implemented in Oris as well as the
interaction with Oris were designed to communicate
the collection and diffusion of a human scent in a subtle
metaphorical form.
161
Fig. 2. Intimate garment for mothers of prematurely born babies collecting the maternal scent and diffusing it for a baby.
Note two garments in one package and the scent bubbles shrunk inside the garment preserving the scent.
4.1 Semantic design and subtle interactions

of Oris
The unique design and interaction assist to envision
the collection and absorbance of the mothers human
scent and its diffusion for the intended use her baby
smelling it and associating it with a known intimate
scent. In the case of Oris little bubbles show the
fullness by the mothers scent like being blown and
expanded under the amount of scent. Therefore the
bubbles became a strong metaphor for the design
of Oris and serve as the metaphorical collectors
and diffusors of a hardly envisioned human scent.
The mothers scent is collected and diffused only in
a warm environment such as a neighbouring area of
her neck or in an incubator. These bubbles grow in
their shape once exposed to a warm environment

and they shrink once exposed to a cold environment.
This fullness interaction is also strengthened by the
dynamic movement of these bubbles tearing their way
through an opening in a top layer underneath which the
bubbles are embedded. The opening of the top layer
and the bubble subtly popping out indicates that the
mothers scent is ready to be exposed (either collected
or diffused). On the other hand the closing of the top
layer behind a bubble indicates that the mothers scent
is ready to be preserved inside the garment. This very
subtle interaction gives a feedback to a mother that
her invisible human scent is either getting collected
and preserved or diffused. The subtle interactions are
communicated in Fig. 3 below.
Fig. 3. Visuals explaining the interactions of the Oris garment

162
The design of the whole garment (an intimate necklace)

is kept very minimalistic and organic. This underlines the
very subtle interactions as well its intended use for a
natural scent recognition and intimate bonding through
scent. The original colour of the bubbles is warm pink
which should also represent a warm heart of a mother
being given to her beloved newborn. Later in the

development three other colours soft yellow, blue
and green were implemented, Fig. 4. This variety of
colours gives a freedom to the mothers to choose
their colours for their baby i.e. according to the gender
of the baby.
Fig. 4. Oris garments with their distinctive colours
163
4.2 Intended use of Oris by mothers

Mothers of prematurely born babies do not usually
expect a premature baby. A premature baby sometimes
arrives as early as even three months earlier than
expected. These mothers are neither mentally nor
physically prepared to have a preterm baby. They feel
fearful about the critical medical state of their baby and
are very confused. They would love to contribute to
the bringing up of their own child however that is not
possible because their child is incubated. For several
weeks or months the two of them who used to be one
unity are physically and mentally separated.
This is the time when Oris can assist to overcome this
separation. Especially with introducing the mothers
scent that is already well familiar to the foetus. Oris
shall be promoted by NICU nurses because they are
the first assistants that the mothers of prematurely
born babies are in contact with. The full package of Oris
is either purchased at the hospital or at a designated
online shop. The Oris package comes with two Oris
products in a specially designed carrier box. Mothers
can choose which one of the four colours of Oris they
would like to purchase. It is recommended to order
Fig. 5. Process of an intended use of Oris

164
always two different colours for later switching one

coloured product with another one during the use. The
carrier box is also designed in a way that it can keep the
mothers scent for a long period of time. Mother starts
wearing one of the products; lets say the one with
pink bubbles. Her scent is getting loaded in it slowly
and therefore she is suggested to wear it constantly
whenever she feels comfortable. After at least 12 hours
of wearing Oris she brings it to the hospital for her
baby. She places Oris next to her baby close by her or
his head or where convenient. The warm environment
of the incubator helps to diffuse her scent for her
baby. At this moment she can already start wearing the
second Oris (i.e. yellow) she purchased. Her scent is
getting loaded in this second Oris. The next time she
visits her baby she can switch the evaporated pink
Oris for the newly scented yellow Oris. In this way the
mother feels actively contributing to her child health
care, wellbeing and comforting. Moreover the intimate
bonding through scent is initiated and sustained. The
whole scenario of intended use of Oris is displayed
below in Fig. 5.
4.3 Proposed technology of Oris

The technology proposed for Oris has to carry several
properties. It mainly has to be completely harmless to a
human being as well as not interfere with the electrical
devices used in the NICU environment. Further it
should be highly comfortable, soft to look and touch
and appealing to its main users the mothers.
The smart textile material, that we envisage, maintaining
the subtle interaction described above is currently the
state-of-the-art material. It shall be a specially designed
thermal expansion shape memory polymer (SMP) made
into threads and later into a fabric. Shape memory
polymers are smart materials that hold the capability
of changing their shape upon application of an external
stimulus - change in temperature [20]. The concept of
the shape-memory polymers originates from metallic
shape-memory alloys (SMA), used frequently in robotic
field, wearable crafting projects and has also been recently
introduced in a conceptual medical applications such as M.
Croess Family Arising hugging snuggle for prematurely
born babies. The shape memory polymers however surpasses the shape memory alloys in their properties related
to possible industrial applications, especially in its density
and extent of deformation as summarized in Table 1.
The shape memory polymers, as indicated above,

are currently in a stage of ongoing development.
Therefore it is theoretically possible to envision the
future development of this smart material towards our
proposition where the designed thermal expansion shall
reliably operate in a considerably small temperature
range between 20 C (room temperature) and 37 C
(human skin and incubator temperature). In this way
the intended interactions of bubbles expansion and
shrinkage can be advanced. Diagram below, Fig. 6, shows
the envisioned working of the proposed smart material.
On the basis of the information we found it is our
estimation that given time and budget we could build a
prototype and that we are not talking science fiction.
Fig. 6. Envisioned working of the thermally expansive proposed

smart material
Table 1. Summary of the major differences between

SMPs and SMAs [21].
Additionally, the choice of soft fabric materials is

crucial. Not only they should carry an antibacterial
property they should be also susceptible to retaining
a human scent well. This is a very challenging part of
the design however it can be solved by using: bamboo
based fabric that has an antibacterial property; and
100% organic sheep wool that is considered to be very
suitable fibre material for retaining a scent generally,
Fig. 7. This is sustained mainly by the waxy substance
of the sheep wool that well binds with the chemical
compounds of a scent. One of the major benefits of
the proposed technology and soft materials is that it is
passive, non-electronic and therefore avoids difficulties
of interference and batteries as well as does not cause
problems with washing.
Fig. 7. 100% bamboo fabric and 100% organic sheep wool

165
4.4 Evaluation of Oris

The preliminary Oris concept was visualized in a
form of an experience scenario video and evaluated
alongside with two other conceptual designs. The
evaluation that followed after served as a comparison
of the three concepts and affirmation of one that
we followed the development with and finalized as
Oris. We believe, however, that the outcomes of this
preliminary evaluation can serve as an interesting
feedback from the users and experts on the actual Oris
conceptual design. Thus, we decided to mention the
evaluation methodology in short and move onto the
actual evaluation results and feedback from mothers and
medical staff that we approached.
The preliminary Oris concept experience scenario video
was evaluated via an on-line survey with two mothers
of prematurely born babies (based in the Netherlands),
two recent mothers of full-term babies (based in Czech
Republic), one NICU nurse (from Czech Republic) and
three medical staff from Gynecology (from the Netherlands). The same video was also presented offline to
another NICU nurse and NICU neonatologist from
MMC Veldhoven, the Netherlands. The questionnaires
were intended to acquire the open opinions on the
conceptual design as well as to find out the level
of perception towards e.g. connectedness, safety,
invasiveness, usability, active contribution to the care
and acceptance of the conceptual product when in use.
One mother has mentioned in the evaluation of Oris
about the connectedness that: It feels that I can
contribute to make my baby more comfortable. In terms
of acceptance of the product one mother reacted: I
like the idea of using heat to capture and release the scent.
It is something I would use. Love it! One mother was
concerned about the products qualities and therefore
mentioned: I would be concerned about the safety of the
materials. I would only use it if I knew there were no dyes or
chemicals that may harm the baby. In terms of usability
most mothers found wearing Oris on a neck very
natural and as they mentioned it would take little effort
to use this product.
The medical staff has been referring to the connected
ness such as: It (Oris function) gives a feeling of safety
for the baby that it has his/her mother nearby. When
we asked about the active contribution to the care or
166
feeling of involvement of the mother, the medical staff

mentioned: It (Oris) gives the feeling of satisfaction to
help her baby. In terms of invasiveness of the product
in NICU, one nurse answered: I think this (Oris) can
easily become a part of the incubators equipment and
be introduced to the health care in NICU. Few nurses
had similar opinion on questions regarding concerns
about the product: The product could be designed as
a positioning aid, since there is not much space in the
incubator for too many things. and More flexible use, e.g.
as a positioning aid. Last but not least category has been
extracted from the evaluation of the medical staff and
that was the products naturalness. This category has
been supported by the comments on the overall Oris
concept as: Back to nature., The scent is original.
Overall Oris concept was positively evaluated by both
mothers and medical staff. It was very interesting to
find out that the medical staff knew the concept of
bonding through scent very well and therefore could
identify with it immediately. Mothers on the other hand
mentioned that they would gladly wear such a garment
if they knew it helps their child. When Oris concept
was evaluated with a NICU nurse offline she pointed
out that she would like to do an extensive research
about how much the mothers scent can calm the baby
down. Further, a NICU neonatologist debated whether
by designing products that only contain one sensory
modality (in this case the scent) might lead into a kind
of uncoupling of the perception of the mothers whole
package the sound of her heart beat, temperature
and smoothness of her skin and her scent. From the
overall evaluation (when the two other concepts were
also taken into a consideration) it appeared that both
mothers and medical staff prefer a low-tech subtle
design in already high-tech incubators above overengineered products using lights, sensors, batteries etc.
5 Semantic design analysis

From a semantic perspective, the Oris is based
simultaneously on two different approaches to product
semantics. The first perspective we shall call "closed
semantics" (Fig. 8). It is more traditional yet sometimes
a bit unavoidable. It is based on the sender-channelreceiver principle proposed by Shannon for technical
communication [22]. The second perspective we
shall call "open semantics" (Fig. 9). It is more con-
Fig. 8. Closed semantics according the Shannon channel-coding

principles such that the designer encodes messages intended
to be understood by the user.
Fig. 9. Open semantics according Krippendorff s semantic turn

principles such that the designer creates possibilities for the user
to load the product with her own contents and meanings.
temporary and much better fit for products which work

as a tool, a carrier or a medium. This is what Klaus
Krippendorff in his book The Semantic Turn explains
and illustrates [23]. We assume that the distinction
between closed and open semantics is not restricted
to the visual modality or to a form, but that it applies
for all modalities and modality-combinations including
the visual, haptic, auditory, olfactory modalities,
interactions, and combinations of these.
Looking at Oris, its design intentions, its key function
and the application domain we can claim that first of all,
Oris is a carrier in the olfactory domain. It works by
open semantics. Note that it would have been possible
to design for scent by closed semantics: for example
by choosing a scent like patchouli meaning love or
adoration. Or sandalwood for protection, harmony
or wellbeing (source: life-style.iloveindia.com/lounge/
aromatherapy-scent-meanings-5330.html). Yet, that is

not how Oris works: the user uploads her own scent,
which is an absolutely clear case of open semantics. In
fact it is an extension of the purest and most natural
example of personalization: the mothers scent is among
the very few things familiar to the new born child
in its big, strange and overwhelming world (next to
perhaps a few sounds such as the parents voices and
the mothers heart beat). Because of this openness in
the olfactory domain, any other meanings associated
with Oris have to be in the visual, haptic, interactive
domain. The audio domain was not included, mainly to
avoid electronics with all its difficulties of interference,
batteries and so on.
In the (dynamic) visual domain the opening and the
closing of the top layer above expanding and shrinking
bubbles signifies the possibility of the bubbles to contain
scent, to preserve and to protect the scent, and release
it. The semantics is mostly of the closed type, but this
is done to compensate for the complete absence of any
fixed point in the olfactory domain. By metaphor, the
visual domain tells about the open-semantics carrier
olfactory nature of the bubbles. Further, we have
chosen to make the choice of the material (sending a
clear message of love, tenderness, being fragile) as the
closed type of semantics, which has the advantage that
it is fit for the context (either worn by the mother,
or when in the incubator next to the baby) and that
it informs the user (once again) about the intended
application.
But then for the color schemes, the interaction and
the packaging, Oris semantics is very open again. For
example the ritual of adopting two colors, such that
one is being loaded with scent while the other is with
the baby and then the subsequent switching. This is
something the mother (and the father) can invent
themselves. There are other rituals they can invent with
the color scheme, but there are many more rituals and
meanings regarding later usage of the Oris. Where to
keep it, when to wear it, whom to give it later, etc. etc.
6 Discussion and conclusion

A scent is rather difficult to communicate and visualise.
However this primarily developed human sense offers
a powerful function to be a cue for a memory-retrieval
167
[24]. The early bonding between mothers and their

(prematurely born) babies is crucial for the future
physiological as well as mental development of the
babies [11, 6] and it is believed that the early bonding
has positive effects on the mother and child attachment
in later years of the child. Combining the statements
above, we could argue that by exposing the prematurely
born child with the maternal scent has a positive effect
on the bonding between the two of them and therefore
creates a cue for a memory-retrieval in the later years
of the child. The concept (Oris) presented in this paper
is an intimate product with unique semantic features
that assists to envision one of our most subconscious
human senses the human scent. Oris is intended to
be used as a comforter for prematurely born babies
and enabler of the human-to-human bonding at difficult
times of separation, as for example a prematurely
born baby placed in an incubator for a care after a
premature birth.
to other stimuli as her heart beat, warmth of skin, her

voice, etc.? Additionally it might be necessary to find out
how much of a mothers scent should be captured and
diffused for her baby? How often and in which situations
is the mothers scent the most efficient for calming a
baby? How long does it take to capture enough of
human scent to calm a baby? Also what role the fathers
scent plays in comforting a baby? Further designs can
be oriented towards mothers as in to capture a babys
scent for her or his mother to support the expression
of the breast milk. That arises additional questions how
much the scent of a baby can contribute to a more
successful expression of her or his mothers breast milk?
Also it might be appealing to have a look at designing
a comprehensive thought through device based on the
mediated intimacy theory. Moreover as seen, the Oris
concept initiates more and more new questions and
opportunities for further research and design.
The development of such intimate products, especially

products with scent sensory perception, enable the
design and research audience to expand their expertise
into this still unexplored field of scent communication
and visualisation. In this paper we proposed a design
semantic expression of a collection, preserving and
diffusion of a human scent in a context of mother to
baby bonding. As we discussed earlier in the semantic
design analysis paragraph, in closed semantics the
designer decides on what messages to inscribe, in open
semantics the designer creates a vehicle for meaning
leaving flexibility to create meaning by the end user.
We believe that our concept proposition can lead to
a stronger human-to-human attachment, subtle and
poetic communication of a human scent as well as
state-of-the-art smart textile material invitation to
development.
Acknowledgments
The authors would like to thank the mother of a
prematurely born baby and medical staff at MMC
Veldhoven, namely S. Bambang Oetomo and A.
Osagiator, who were involved in the user research
phase of the project and who willingly shared their
experiences. Finding about the scent interactions would
not be possible without a help of industrial design and
human technology interaction colleagues. Last but not
least we would like to thank Misha Croes MSc., a PhD.
candidate at the Eindhoven University of Technology,
whose growing expertise and interest in bonding in the
neonatal intensive care unit through design ignited the
whole project.
References
1. Klaus, M., & Recent, A. (1998). Mother and Infant: Early
As with every development and innovation also

this concept proposition arose new questions and
enquiries for further research and design. To get a solid
affirmation that a mothers scent has the potential to
calm a baby down there needs to be further studies
done. However this arises new questions as i.e. How
to measure stress or calmness of a baby? How to
distinguish a mothers scent from other stimuli so that
the only scent is calming a baby? Is it not confusing the
baby if it is exposed to the mothers scent only and not
168
Emotional Ties. Pediatrics.doi:10.1542/peds.102.5.SE1.1244.

2. Mizuno, K., Mizuno, N., Shinohara, T., & Noda, M. (2004).
Mother infant skin-to-skin contact after delivery results in
early recognition of own mothers milk odour, (13), 16401645. doi:10.1080/08035250410023115.
3. Matthiesen, a S., Ransj-Arvidson, a B., Nissen, E., & UvnsMoberg, K. (2001). Postpartum maternal oxytocin release by
newborns: effects of infant hand massage and sucking. Birth
(Berkeley, Calif.), 28(1), 13-9. Retrieved from http://www.
ncbi.nlm.nih.gov/pubmed/11264623.
4. Nissen E, Lilja G,Widstro s-Moberg K. (1995). Elevation of
17. M. Croes, W. Chen, L.M.G. Feijs, S. Bambang Oetomo,
oxytocin levels early postpartum in women. Acta Obstet
"Olfactory Stimulation in Premature Neonates: the Relevance
Gynecol Scand;74:530533.
of Early Experience", in proceedings of GlobalHealth 2012,
5. Uvnas-Moberg, K. (1998). Oxytocin may mediate the
The First International Conference
benefits of positive social interactions and emotions.
on Global Health Challenges, ISBN: 978-1-61208-243-1,
Psychoneuroendocrinology, 23(8), 819838.
pp. 21-26.
6. Charpak, N., Ruiz, J. G., Zupan, J., Cattaneo, A., Figueroa, Z.,
18. J . Plailly, C. Delon-Martin, and J.-P. Royet, Experience
Tessier, R., Cristo, M., et al. (2005). Kangaroo Mother Care:
induces functional reorganization in brain regions involved in
25 years after. Acta paediatrica (Oslo, Norway: 1992), 94(5),
odor imagery in perfumers., Human brain mapping, vol. 33,
514-22. doi:10.1080/08035250510027381.
7. Marlier, L., Schaal, B., & Soussignan, R. (1998). Neonatal
no. 1, pp. 22434, Jan. 2012.

19. Djajadiningrat, J.P., Gaver, W.W. and Fres, J.W. Interaction
responsiveness to the odor of amniotic and lacteal
relabelling and extreme characters: methods for exploring
fluids: a test of perinatal chemosensory continuity. Child
aesthetic interactions. Proceedings of Designing Interative
development, 69(3), 611-23. Retrieved from http://www.ncbi.
Systems (DIS) 00, ACM Press, 2000.
nlm.nih.gov/pubmed/9680675.
8. Varendi, H., Porter, R. and Winberg, J. (1996), Attractiveness
of amniotic fluid odor: evidence of prenatal olfactory
learning?. Acta Paediatrica, 85: 12231227. doi: 10.1111/
j.1651-2227.1996.tb18233.x.
9. Maguire, C.M, Bruil J., Wit J. M., Walther F. J., Reading
20. A . Lendlein and S. Kelch, Angew. Chem., Int. Ed., 2002, 41,
20342057.
21. C . Liu, H. Qin and P. T. Mather, Review of progress in shape
memory polymers, J. Mater. Chem., 2007, 17, 15431558.
22. Design als Produktsprache Der Offenbacher Ansatz
in Theorie und Praxis, Verlag form, Frankfurt/Main 2000,
preterm infants behavioural cues: An intervention study with
Steffen, Dagmar (editor) with contributions by B. E. Brdek,
parents of premature infants born <32 weeks, Early Human
Jochen Gros and Volker Fischer.
Development, vol. 83, pp. 419-424, 2007.

10. Cronin CM, Shapiro CR, Casiro OG, Cheang MS. The
impact of very low-birth-weight infants on the family is long
lasting. A matched control study. Arch Pediatr Adolesc Med
1995;149:1518.
11. Tessier, R. (2003). Kangaroo Mother Care: A method
for protecting high-risk low-birth-weight and premature
23. K rippendorff, Klaus (2006). The Semantic Turn; A New

Foundation for Design. Boca Raton, London, New York:
Taylor&Francis, CRC Press.
24. D. M. V. Zemke and S. Shoemaker, Scent across a
crowded room: Exploring the effect of ambient scent on
social interactions, International Journal of Hospitality
Management, vol. 26, no. 4, pp. 927940, Dec. 2007.
infants against developmental delay. Infant Behavior

and Development, 26(3), 384-397. doi:10.1016/S01636383(03)00037-7.
12. S . Bouwstra, W. Chen, L.M.G. Feijs, S. Bambang Oetomo,
W.M.J.H. van der Linden,"Designing the parents-to-infant
bonding experience in the NICU", in proceedings of
GlobalHealth 2012, The First International Conference on
Global Health Challenges, ISBN: 978-1-61208-243-1, pp. 7-13.
13. Cernoch, J.M. and R. H. Porter. Recognition of maternal axillary
odors by infants. Child Development 56: 1593-1598, 1985.
14. Schaal, B. Presumed olfactory exchanges between mother
and neonate in humans. In: Ethology and Psychology, J. Le
Camus and J. Cosnier eds. Toulouse: I.E.C., pp 101-110, 1986.
15. S helley W. B. ; Hurley H. J., Jr., (1952). Methods of exploring
human apocrine sweat gland physiology ama Arch Derm
Syphilol. 66(2):156-161.
16. Curran, A. M., Rabin, S. I., and Furton, K. G. (2005). Analysis
of the uniqueness and persistence of human scent. Forensic
Sci. Commun. 7:2.
169
Manon Junggeburth, Luca Giacolini, Tom van Rooij, Bastiaan van Hout, Bart Hengeveld,
Mathias Funk, Joep Frens
{m.w.j.junggeburth,l.giacolini,t.j.a.v.rooij,b.f.k.v.hout} @student.tue.nl, {b.j.hengeveld, m.funk, j.w.frens}@tue.nl
Experio: A Laser-Triggered
Dance Music Generator
Abstract
In the field of Industrial Design we are moving towards
a paradigm of Designing for Systems, which means
designing from a multiple users/multiple technologies
perspective. As this is rather uncharted territory we
explored how to design for systems inspired by fields
with roots in facilitating behavior and personalization;
in this case improvised music. While traditional musical
instruments and performances are played by musicians
with a certain training skill set, participatory musical
instruments are still a challenge as they should allow for
a low entrance threshold, easy and fun operation, and
the possibility to create appealing musical experiences
that go beyond an initial rush. We designed a setting in
which people can express them while being part of a
bigger technology-mediated musical organism, specifically
aimed at the whole rather than the sum of the parts.
The design is called Experio; a system that uses lasertriggered dance floor segments to generate dance
music. A small initial user test has been carried out to
evaluate the possibilities of Experio in a real-life setting.
Keywords
Musical experience, improvisation, free body
expressivity, system.
170
1 Introduction
In Industrial Design, we are moving away from the
traditional paradigms of Designing for Appearance
and Designing for Interaction towards Designing
for Systems. This paradigm breaks with the current
interaction structures of people and technology and
deals with multiple users using multiple technologies
in a highly networked setting. As this design paradigm
is still relatively unexplored we need to explore how
to design for systems, as we experience that we
cannot simply adopt the tools and methods from the
two other paradigms and apply them to the new one.
As an exploration we try to learn lessons from fields
with roots in facilitating group interactions, in this case
improvised music. In improvised music, individual people
performing together are genuinely more than the sum
of the parts; they act as a system [1]. Multiple feedback
loops between musical instruments, players and the
audience complicate this design challenge.
Inspired by this phenomenon, the design challenge
aimed at designing a setting in which people can express
themselves musically, mediated by technology, while
being part of this bigger musical-social organism.
In other words, designing an interactive musical
instrument that was specifically aimed at the whole
rather than the sum of the parts was the final goal of
the project. One can notice a diversion from what already exists in the field of musical instruments, as, for
instance, a keyboard is designed so that it can be played

as a part of a system, such as an orchestra, but it can
also be an experience itself. Instead, the design process
started with defining which benefits are desirable in a
musical system, then the roles of each single input were
given. Such an approach led to the creation of a genuine
musical interaction set.
1.1 The structure of a live performance
As a starting point, we analyzed what a contemporary
musical system involves specifically, in order to be able
to carry it over to a futuristic and experimental stage.
To this purpose, several students of a music academy
who are experienced in rock and popular music were
interviewed. They all agreed on the fact that a live
performance is generally based on four entities: band,
band member, instrument and audience. Each entity
is part of the system shown in Figure 1, which also
illustrates the reciprocal relationships between the
four entities. The missing direct relationship between
the audience and the instrument (shown in grey), in
which the two entities could have a reciprocal positive
influence, could be mediated by technology, thus
framing the challenge in this research project.
Fig. 1. Interactions during a musical live experience.

The grey arrow indicates the missing link.
1.2 Related work

Experio was influenced by several related works.
In the broadest of lines we stand in the tradition of
Louis Theremin, who was arguably the first inventor
of a touch free user interface [2]. In basics however,
Experio is more related to the Music Composition
Game of Chung [3]. First follow the selected beat
with a certain tempo, then create a melody above the
rhythmic passage, and complete the piece of music with
adding a chord layer; the rhythm, melody and triad
chords are interrelated. Experio provides a whole new
environment in which the moderator in the middle
is responsible for the final musical output, mediating

participants live input with the overall design of the
experience. In the study of Gates in DJs perspective
and interaction, DJs were asked about how they like
to work. Timothy Wisdom replied: one good tactic
is to satisfy key dancers who will help raise the
excitement level in a room, causing others to join in
on the dance floor [4]. In Experio, there are also
key dancers who are supported by the system to act
and dance on the platform around the moderator.
The people on the platform will dance and raise the
excitement level in a room, seducing bystanders to join
them. As inspiration for Experio, the laser harp from D.
Rubine and P. McAvinney is used as well. It is one of a
number of generalized musical instruments, which allow
real performance on MIDI-driven systems and also
investigate the possibilities and necessities of interaction
between performer and instrument [5]. Finally, there
is also extensive related research about multi-user
systems and audience participation, for instance, the
work of Jord [6] and Maynes-Aminzade et al. [7].
2 Experio
Experio is a new interactive networking design aimed
at a more active relationship between the audience and
the performer of modern electronic music. The purpose
is to let the people from the crowd have an influence
on the sound, so that they can take on responsibilities
and shape their own experience. Three entities do
interact as a system [8]: the crowd, the crowd musicians
and the moderator. The crowd is a general term for
people who are dancing in a club and enjoy the music.
The crowd musician, also named audience performer,
is a person who challenges herself with a new way of
interacting with the overall music experience, while the
moderator is a professional musician who works with
by the audience performers input. Experio is designed
such that multiple crowd musicians can interact at the
same time.
The Experio setup in specific consists of a big round
platform with the moderator standing in the middle.
The platform consists of several planes made by laser
light, from center to outer rim, like sectors of a pie.
In the center, the moderator selects music content,
which is partly played directly, for instance the beat,
and partly assigned to the different sections of the
171
Fig. 3. Schematic overview of Experio.
circle. Each section of the circle will get a different

function, for instance, a different sound or sound effect.
So, a few people from the audience can step in and
dance in the sections. While dancing, they interrupt
the light with their feet (or any other limb) and this
triggers sensors, which then leads to different musical
output. The moderator mixes the inputs received
from players into the overall sound scape. As an
example, besides playing a drum beat, the moderator
will assign to three different sections three different
interaction possibilities: a lead guitar, a synthesizer and
a reverb effect. He does design the whole music piece
anymore, but he is designing the configuration of the
instrument with which the audience will perform. After
a certain time period the moderator can decide to have
another beat and to change the mapping of sections
and sounds. The final outcome will be a balanced
selection of pleasant, homogenous sounds and more
experimental, daring sounds as inputs, which are turned

into something new by the audience. The challenge for
the moderator is certainly to select not only inputs
that please the audience sonically and will most likely
blend into a pleasurable experience, but also inputs
that trigger and inspire the audience to play with them.
Compared to a common DJ, this gives away parts of
control over the entire experience to the audience with
the desired outcome of more crowd participation in
the creation of an atmosphere. The music performed
with Experio can certainly vary, however, for the first
performances and experiments beat-centric dance and
techno sounds were chosen.
Experio is designed to be quite portable and it can be
placed in many different environments with a clear (semi-)
circular space. Festivals are very suitable for this kind of
experimental music and Experio should add extra value
to such a musical event or even become one in itself.
3 Implementation
Fig. 2. Experio: the final prototype.

172
Experio is built of multiple linked components, which

will be shown in the following (see also Fig. 3 for a global
view). A laser machine projects a line hovering over the
ground and is made visible using a haze machine. The
light produced by the laser is being detected by sensors
on the other side in a circular prop fixed to the ground.
If light falls on the sensors, nothing happens. But as soon
as the light is interrupted, a MIDI signal is being sent to
a music software program and this signal can be mapped

to musical expression, to trigger a certain sound or to
apply a sound effect to a track or pattern. Because the
laser-projected line is based on high speed sweeping,
the light falling on the sensors is not continuous.
This fast flickering is filtered out with software
resulting in one continuous input. The pre-processing
is performed on an Arduino1 chip, which in turn is
connected via a MIDI (Musical Instrument Digital
Interface) to the music software. For the prototype,
Fruity Loops2 was used as a digital audio workstation
to provide the musical output. The current prototypes
Arduino chip is capable of controlling all necessary
processes for this prototype, but in order to monitor
more than the sixteen sensors that are used in the
current model, the sensor inputs on the processor need
to be multiplexed first. A faster processor is necessary
to convert the many fast triggering multiplexed inputs
into a stable midi signal and to keep the delay as low
as possible. Figure 3 shows a schematic overview
that illustrates the workings of Experio, and explains
and details the individual components and how they
influence each other.
4 Evaluation
http://arduino.cc, last
retrieved on August
10, 2013
http://www.image-line
com/documents/
flstudio.html, last
retrieved on August 10,
2013
As a preliminary evaluation and to fine-tune the inter

action with the prototype, we set up an explorative
user test, in which users were asked to behave as
audience performers. Fifteen people took part in the
evaluation, all Industrial Design students; eight of them
had a musical background, seven did not. The user
test started with an intake questionnaire, asking the
participants about their experiences with music and
about their first impressions of the prototype. After
this, the participants were free to interact with the
prototype, experiencing it in order to provide more
specific answers during a debriefing interview.
Although the overall interest of the project lies in the
generation of a new musical experience, the focus of
the interviews was mostly on the ease of use of the
prototype, as in this stage we wanted to find out any
usability issues that may hamper free expression. The
goal was to understand if the connection between the
body movements and the sounds produced was intuitive
enough and how much time it would take to acquire
basic performing skills in Experio.
Most significantly, our interviews showed that there was

a need for a more evident strong and not too complex
beat, which was considered the most important by
the participants in order to dance and improvise to.
Due to an unclear beat in the experiment, almost all
participants moved very unnaturally and had therefore
clear difficulties in looking for a way to dance and
improvise. With this in mind, the role of the moderator
became more interesting: by putting her in charge of the
beat instead of (one of) the participants, the resulting
beatline generally becomes more coherent and allows
the dancers more freedom to express themselves
without having to worry about rhythmic coherence; the
moderator in turn is allowed to compose beats freely as
well as its associated tones as input for the dancers.
Another important observation was that each
participant needed to be able to hear his or her own
input in order to be able to effectively improvise. This
observation was already made in an earlier stage of the
project, but was even more clearly reconfirmed in this
user test, as this prototype had a higher level of realism.
A last significant observation was that input provided
only by the feet was apparently enough to be musically
expressive. One could argue that this might prevent
users from reaching a freebody expressivity, but many
of the interviewed pointed out how difficult it already
is to improvise and contribute to the musical output
with limited possibilities, given the short amount of
time available for understanding the basics of the
interaction. Of course we cannot predict longterm
effects, for example, the extent to which the interaction
remains interesting based on this test, but the signs
are positive. Apart from these fifteen participants,
dozens of other dancers tried out the prototype freely
and informally during two two-hour sessions and the
response was generally enthusiastic.
5 Conclusion and discussion

In this paper we described Experio, a system design,
in which two parties cooperate in creating music.
The moderator provides a rhythmic and tonal building
blocks and the audience shapes the final sound by their
bodily interaction. The intended result goes beyond
being the sum of the two parts alone: in fact we believe
that the final atmosphere based on the collaboration
173
between the parts goes a lot further since both parties

involved co-create a new musical experience based
on reciprocal influence, mediated both technologically
as well as socially. We believe that Experio can be
considered as an interesting way of improvising in a
safe setting, in which people can get familiar with the
instrument in a considerable amount of time.
References
1. Frens, J. W. and Overbeeke, C. J. (2009). Setting the stage
for the design of highly interactive systems, In Proceedings of
IASDR 2009, Seoul, Korea, pp. 1-10.
2. Geiger, C. and Wldecke, B. and Marinos, D. (2011), Design
of a Touchless Multipoint Musical Interface in a Virtual Studio
Environment, ACE11, New York.
3. Rekimoto, J. (2004). Sensing GamePad: Electrostatic Potential
Our preliminary evaluation shows potential to carry

on the project into a more extensive and complete
working prototype, and considering what is already
implemented, a 35 degrees wide Experio (instead of a
full circle), and a laptop used as moderator interface,
the next step would be indeed to design the moderator
interface, a process that will start with interviewing
professional musicians and DJs. The focus will then be
on developing more, better content for the system
as well as finetuning the sounds effects. Apart from
this, more open user tests will be carried out in less
controlled settings, for instance, at public exhibitions
and also regional music festivals. We believe that
these events with their diverse visitors provide great
opportunities to further test and develop deliberately
open systems such as Experio.
sensing for Enhancing Entertainment Oriented Interactions,

Tokyo: Sony Computer Science Laboratories, Inc.
4. Chung, S. and Chen, C. (2011). Improvising on Music
Composition Game, Taiwan: Ling Tung University.
5. Nelson, P. (2009). New Instruments for the performance of
electronic music-introduction, Wales: Routledge.
6. Jord, S. (2005). Multi-user Instruments: Models, Examples
and Promises. In Proceedings of the International Conference
on New Interfaces for Musical Expression, pp. 2326.
7. Maynes-Aminzade, D., Pausch, R., and Seitz, S. (2002).
Techniques for Interactive Audience Participation. In
Proceedings of the 4th IEEE International Conference on
Multi-modal Interfaces (ICMI 02). IEEE Computer Society,
Washington, DC, USA.
8. Hengeveld, B., Frens, J. and Funk, M. (2013). Investigating
How To Design For Systems Through Designing A Group
Music Improvisation System. Proceedings of the IASDR
conference, to be publish
174
Tetske Avontuur, Eveline Brink, Iuliia Malyk, Derec Wu

Technical University of Eindhoven, Eindhoven, The Netherlands
{t.p.avontuur, e.brink, i.malyk, s.wu}@tue.nl
Sentiment Ninja: Sentiment Mining on

Twitter based on Topic and Location
Abstract
Social media have become increasingly popular in
recent years. The amount of data makes it hard to get
an overview of what people are saying. We designed
an app that allows users to get an overview of big
data on Twitter. We allow users to explore opinions
by searching for keywords and allowing them to filter
the results based on location. We believe that this
application might be interesting for many different
target groups such as consumers, companies and
journalists.
Our goal is to provide people with a tool that allows

them to get a clear overview of opinions on Twitter in
an easy manner by searching for keywords and filtering
results based on location.
Our application allows people to query Twitter with
keywords and search for results for a specific location.
After the query, the application returns the Tweets for
the specified location and sorts them according to their
sentiment. The results are then displayed in a pie chart on
a map to provide an overview of the opinions on Twitter
about the queried keyword in the searched location.
Keywords
Sentiment analysis, opinion mining, twitter, map,
visualization, big data, mobile
1 Introduction
In recent years, social media have become a part of
everyday life for many people. Social media allow people
to share thoughts, opinions, information, pictures and
video with virtually anyone in the world. One widely
used social media platform is Twitter. According to the
Statistics Brain [1], there are over 550 million Twitter
users sharing an average of 58 million Tweets every day.
People also use Twitter to search for information: on
average, 2.1 billion different queries are issued each day.
The massive amount of available Tweets makes it hard
for people to get an overview of others opinions.
2 Design and Implementation

In this Section we focus on the design concept and its
implementation. Subsection 2.1 discusses the concept,
Subsection 2.2 provides an overview of the technologies
that we used and Subsection 2.3 shows how we
implemented the design concept into the application.
2.1 Concept
Sentiment Ninja is a mobile application that allows
users to retrieve Tweets from Twitter and sort them
according to their sentiment. Tweets are classified into
one of three categories: positive, negative or neutral.
When users issue a query, the results are visualized in
a pie chart on a map according to the location where
they were sent. Users can specify for which location
175
they want to see results. In addition to pie charts on a

map that provide a general overview of the distribution
of Tweets, the app allows users to examine individual
Tweets in more detail.
We choose to use Twitter for this app because it is a
popular online platform with many active users: sixty
percent of Twitter users log in through a mobile app
on a daily basis. These users are more likely to use
Twitter throughout the day and are more engaged with
the content. They are three times more likely to use it
while shopping and two times more likely to use it while
commuting [2].
2.2 Technologies
We created a prototype for our app using HTML5,
CSS3, Javascript, JQuery Mobile and PHP. The back end
is programmed using PHP, the other technologies are
used to design the front end. We use Google Maps API
to visualize a map and Google Chart API to create the
pie charts. The Twitter API is used to query Twitter.
We use Sentiment140 to classify the Tweets that are
retrieved after a query. We used Phonegap to create
a hybrid app for Android, allowing the app to use the
phones native features.
HTML, CSS, Javascript and JQuery Mobile.
We used the latest web development technologies to
create a single page HTML document for our app. We
chose to use JQuery mobile for the design because it
allows for a faster, more customizable and more flexible
development than other prototyping tools.
Twitter API.
The Twitter API allows us to retrieve Tweets from
Twitter based on user specified keywords and location.
For our prototype, we use a free account which has a
limit of retrieving a hundred Tweets per call.
Classification and Sentiment140.
We use Sentiment 140 to classify the Tweets as either
positive, negative or neutral. This tool was developed
by Stanford computer science graduates. They use a
keyword-based approach that focusses more on recall
and less on precision than other tools [3]. We prefer
a higher recall because Tweets do not always contain
meta-information about location. However, our app has
a heavy focus on location so it is important for us to
176
retrieve as many Tweets as possible. A lower precision

is acceptable for us because we want to provide people
with an general overview of others opinions. Also,
our app allows people to examine individual Tweets.
This way, users can determine the precision of the
classification themselves.
Pie charts and Google Maps API.
Google Maps API is used to create pie charts of the
retrieved Tweets [4]. Pie charts show the distribution of
Tweets according to their sentiment. We chose to show
pie charts because they give a compact overview of the
results to users. Using other visualization techniques,
for example bar charts, would clutter the view as they
take more space on the screen, hiding a larger part
of the map. We use the Google Maps API clustering
algorithm to automatically group Tweets based on a
radius around coordinates on the map [4].
2.3 Interface
When users open the app, they are presented with
a splash screen during the loading of the app on the
phone. Once loaded, people are automatically referred
to the main screen. Here, they see their current
location on the map as can be seen in Figure 1a.
After putting in a query, users are presented with
pie charts that show opinions about the query in the
current location on the map. To gather qualitative and
more specific information about opinions, users can
get more details about individual Tweets. This can be
done in three ways: 1) tapping the pie chart, 2) zooming
in or 3) tapping on the Tweets button in the lower
right corner of the screen. The first two methods show
individual Tweets on the map, as can be seen in Figure 1b.
Fig. 1a. Main screen
Fig. 1b. Zoomed in map with tweet text
177
The third method shows users individual Tweets in a

list, as can be seen in Figure 2a. In this screen, users can
filter Tweets according to their sentiment to see only
the Tweets of one classification. It is also possible to see
an unfiltered list.
On default, the app tries to determine the current location
through the phones gps. If this function is turned off, the
app will use data from the network to approximate the
current location. Users are also allowed to specify their
current location manually in the screen that is shown in
Figure 2b.
From both the map and the Tweet list screens, the user
can query Twitter by tapping on the magnifying glass in
Fig. 2a. Text view of the tweets
178
the top right of the screen. Users are then directed to

the search screen. Here, trending topics are shown and
the user can enter his own keywords in the text bar
at the very top of the screen. After tapping a trending
topic, or tapping the enter-key on the keyboard, the
user is directed back to the previous screen. The search
screen is shown in Figure 3a.
On default, users are presented with the trending topics
in the vicinity of their current location. By tapping the
Change trend location button in the top right of the
screen, users can change the trend location. They can
do this on different levels: worldwide, continent-wide,
country-wide, city-wide or even more local. The change
trend location screen is shown in Figure 3b.
Fig. 2b. Change users location
Fig. 3a. Search for topic screen
Fig. 3b. Change trend location
3 Prospective applications
The Sentiment Ninja application can be used in many
contexts by many different target groups. In the
following Subsections, we focus on the three most
prospective target groups: consumers, companies and
journalists. For consumers, we discuss two different
contexts: exploration of the environment and buying
decisions. Companies can use the app for marketing
purposes and for competitor analysis. For journalists,
the app is suitable in contexts where they are in the field
and want to report the opinions of people on the news.
3.1 Consumers
Consumers can use the app in two contexts:
exploration of the environment and buying decisions.
The former context applies when a user is visiting a
city and wants to find out which places he should visit.
The user can then find opinions about places and events
in the city. Location is highly relevant in this context
because the user wants to explore a specific area. Users
can also query for opinions in the city without specifying
a keyword to gather general opinions and discover
which places are more positive than others.
179
The latter context applies when a user is in a store and

needs to make a buying decision. He can then quickly
find information about the product he is considering to
buy. Location is relevant here because opinions might
differ between countries and cultures.
3.2 Companies
One important goal of commercial companies is to sell
their services or products to customers. Marketing is
a way to gain product awareness which in return might
lead to an increase in sales. When people are not buying
a significant amount of products, companies need to find
out why. They can use Twitter to find out how people
are thinking about their product or service and examine
individual Tweets to get qualitative information about
the reasons of consumers not to buy the product or
service. It might also be the case that a specific product
or service sells well in one country but not in another.
Companies can then find out why this is the case and
make appropriate adjustments to their product or
marketing campaign. For example, in Asia people prefer
blue light while in Europe, people prefer yellow light.
If light bulbs provide a blue light, European people might
complain that the light is too cold. Companies can then
adjust the color of the light.
The app can also be used for competitor analysis.
Companies can easily see how people are feeling about
a competitor brand or product. If required, they can
get examine individual Tweets to find out why
people are having a specific opinion. They can use
this knowledge to distinguish themselves from their
competitors more effectively.
3.3 Journalists
Journalists usually try to present facts and information
objectively. Journalists are usually also interested in
how the general public thinks about a specific news
item. These news items are usually emerging quickly.
However, it can take a lot of time to gather these
opinions. It can also be difficult or dangerous to go out
and talk to people in for example a war situation. With
Sentiment Ninja, they can get an overview of peoples
opinions on the news quickly and easily. Journalists
can then report general opinions, for example which
candidate is in favour during presidential elections, and
specific opinions, for example why a specific candidate
is (im)popular. The location of these opinions can be
180
important. For example, with presidential elections,

it might be interesting to see how people in a specific
region are thinking or how people abroad are feeling
compared to the local population. This is particularly
interesting during potentially controversial events or in
countries with a high level of governmental manipulation.
The journalist can then explain the discrepancy between
officially reported information and information from
Twitter.
4 Discussion
In this Section, we discuss issues with the current design
and provide possible future developments of our app.
4.1 Issues
In this Subsection, we discuss the following issues with
the current design: language, sarcasm and ambiguity,
location, and big data representation.
Language.
Currently, our app only analyzes English Tweets.
This means that many possibly relevant Tweets are
not recognized because they are in a different language.
This problem especially arises when the user is looking
for Tweets in a location where English is not the native
language, although it could also occur in native countries
where tourists might Tweet in their own language.
Another problem here is that users might enter
keywords in their own native language, not English.
These issues are currently hard to resolve since most
algorithms are developed especially for the English
language. Also, providing a keyword in one language
with the goal of searching Tweets in all languages for
this topic is an issue since it is still hard to automatically
translate languages.
Sarcasm and ambiguity recognition.
Although a lot of work has been done in the field of
natural language processing, recognizing sarcasm and
dealing with ambiguity currently still pose a problem
for machine learning algorithms [5]. This can cause
the classifier to annotate Tweets wrongly and poses
a problem to its accuracy.
Location.
Twitter users might protect their privacy by switching off
location tagging. For us, this means that possibly relevant
Tweets might not be retrieved and shown in the app,

resulting in a smaller amount of available data. We try to
compensate for this by using an algorithm that focusses
more on retrieval than on accuracy. To allow users to find
relevant Tweets, we propose to include a function in the
app where users can search for a topic without specifying
a location.
Big data representation.
Our app tries to solve the issue of providing an overview
of big data on a small screen. We do this by providing two
viewpoints separately: a general, quantitative overview in
the form of pie charts and a specific, qualitative overview
by showing individual Tweets. User testing should be done
to examine to what extent this is an appropriate approach
to visualise big data on a small screen.
5 Conclusion
The goal of our app is to allow users to get a clear
overview of opinions on Twitter based on location in a
quick and easy way. We designed the app with three clear
target groups in mind although it could be interesting for
many more people. We did not yet have the opportunity
to conduct extensive user testing to find out to what
extent our app meets its goals. One next step would
be to test our app. Another step is to allow users to
compare different locations more easily and clearly by
developing a web app that is suitable for larger screens.
A next technical development would be to include
algorithms to detect propaganda to prevent companies
and governments from manipulating opinions on Twitter.
References
4.2 Future development
One benefit of creating an app for mobile use is
portability. Opinions on Twitter can be mined from any
place in the world with internet access. One drawback
however is the limited screen size. On a bigger screen,
we could allow users to compare locations in the world
that are far apart or to compare different zoom levels (for
example, compare the general opinion in a country with
the opinion in a city within that country).
We aim to develop a desktop version in the future that
allow this functionality.
1. Twitter Statistics. Statistic Brain. Retrieved July 6, 2013 from

http://www.statisticbrain.com/twitter-statistics/
2. New Compete study: Primary mobile users on Twitter.
Twitter Blog. Retrieved July 6, 2013 from https://blog.twitter.
com/2013/new-compete-study-primary-mobile-users-twitter
3. Sentiment140 Introduction. Sentiment140. Retrieved July 6,
2013 from http://help.sentiment140.com/home
4. Too Many Markers! Google Maps API. Retrieved July 6,
2013 from https://developers.google.com/maps/articles/
toomanymarkers
5. Cleverbot comes very close to passing the Turing Test.
Cleverbot. Retrieved July 7, 2013 from http://www.cleverbot.
Another interesting shape of our app would be to have

a projection of peoples opinions on different surfaces in
public areas. For example, a university could show how its
employees are feeling and a store might provide realtime
information on opinions in the store.
com/human
6. Recognizing Propaganda Techniques and Errors of Faulty
Logic. Cuesta College. Retrieved July 7, 2013 from http://
academic.cuesta.edu/acasupp/as/404.htm
We also aim to improve the technology of the app to

recognize bots and propaganda in Tweets. However,
automatically recognizing bots is becoming increasingly
difficult. Bots getting more advanced and some, like for
example Cleverbot come close to passing the Turing Test
[5] which might make it hard to identify them.
Propaganda might also be an issue as some companies or
governments might try to influence opinion mining results
to their own benefit. Algorithms to automatically detect
propaganda already exist. One existing algorithm works
by looking at specific features of Tweets and is already
sufficiently accurate [6].
181
Cheng Zheng
Department of Industry Design, Zhejiang University, Hangzhou, 310027, China
zzzzzccccc@gmail.com
Visual-Tangram: Tangible Puzzle for

Childrens Intelligence Exploration
Abstract
Tangram is an old Chinese puzzle game mainly for
children intelligence exploration; it is simple and
easy to make but has almost infinite combination
possibilities. In recent years, however, the explosion
of the development of new toys and games has pushed
tangram, the puzzle of Chinese wisdom and culture, into
an awkward situation. Our project sought to make the
game of tangram more visual and interesting through
tangible interaction, trying to inspire children playing
and learning with more enthusiasm, meanwhile help
this old game revive today.
Keywords
Chinese puzzle, tangram, intelligence exploration,
tangible interaction
1 Introduction
Tangram (or Qiqiaoban in Chinese) is a puzzle game
consisting of seven flat shapes(called tans). As an open
game, the objective of the puzzle is to use imagination
to form a specific shape using all seven pieces, which
may not overlap[1]. This gives the game another name
seven boards of skill.
square, that is to say, with a squared thick paper, a knife

and a ruler, anyone can make a tangram himself with ease.
Tangram has been proved effective in intelligence
development. For children from 2 to 6 years old,
tangram helps improving their abilities of hand-eye
coordinating [2], shape cognizing, vision differentiating,
visual memorizing, as well as open-mind-training and
creative thinking [3]. Despite that tangram is so helpful
and has been popular for over 200 years, todays
tangram, together with plenty of traditional puzzle
games, is facing a loss of childrens attention due to
various electronic products like video games, cartoons,
electronic toys, which are deservedly competitive by
visual appearance and rich content.
However, puzzle games, especially dissection puzzles,
are still irreplaceable in helping children with hand-eye
coordinating, abstract-to-visual cognizing and creative
thinking. Based on this we planned to create a new
visual-tangram game more suitable for children by
tangile interaction[4].
2 Concept
Tangram originated from the Ming Dynasty for over
200 years ago. Tangram has 7 pieces, including 5 right
triangles(2 large, 1 medium, 2 small), 1 square and 1
parallelogram. All these pieces can form an exact big
182
Considered that children need others help to complete

cognition of abstract figures, and that children need
more encouragement and achievability in learning [5],
we use a computer to help recognize abstract objects
as a start, then transform into visual objects, and at

last show the objects to children via multimedia e.g.
animation, sound and words.
The hardware of the system includes a PC with and a
web camera (see Figure 1). Operating instructions can
be referred in SCENARIO part. The software of
the system consists of 3 modules: tangram detection
module, object recognition module and multimedia
output module.
Multimedia output
When the object is identified, UI module starts to fetch
objects profile, output static or dynamic animations
with words and explanation voices, as vivid as possible
to impress children (see Figure 3).
Tangram detection
Any shape detection algorithm is appropriate for
identifying tangram shapes, e.g.. Here we use EmguCV
library [6]. Firstly we capture a snapshot from the web
camera, scan the image to get color distribution. Since
different pieces have different colors, we get angular
vertexes of every piece by analyzing each pixel. With
an established coordinate axis, we know the positions
and rotation angles of all tangram pieces. These string
numbers describe a unique tangram figure.
Figure 3. Visual-tangram system in test.
3 Scenario
Figure 1. Architecture of visual-tangram system.
Object recognition
Since any figure could be described as an array of
numbers, we create a property database to store the
corresponding arrays. Figure 2 gives an example of how
to recognize a figure of fox. When the array of the
figure shows a high similarity to the array in database,
the system will judge it as a fox. In our test,
|pri-pr0i|*(|pai-pa0i|+|rai-ra0i|)<10 is considered
as similar.
Figure 2. Any figure can be stored in database.
Lay a piece of white paper on a flat desktop, put

tangram pieces on paper.
Adjust web camera to make sure the paper and
tangram in full view. Run the software.
Play with tangram.
After completing a work, move hands away from the
figure and wait for a second.
The screen shows an animation, telling the child
what he/ she has just completed, and whats funny
about this stuff.
4 Conclusion and future work

In our tests, 15 children aged from 2 to 6 showed great
enthusiasm to visual-tangram system. They couldnt
wait to try any possible combination and expect the
screen show something they had expected. Compared
to virtual puzzle games on PC, this tangible interaction
game is more attractive and easy to operate for
children. Consequently we conclude that the visualtangram is effective in attracting childrens attention and
prolong their time spent on the game.
183
We also notice that children are highly interested in

those games with feedback and interaction. In future
research, we plan to develop more functions to attract
children, for example, encourage them to form a short
story, or even a self-directed cartoon movie with visualtangram system.
Also we realized the insufficiency of this tangible
system the pieces detecting accuracy might be largely
influenced by many factors like light white-balance and
camera image quality. So we will try every possible
means that might work, such as embedding RFIDs into
tangram pieces and put them on a special white boxes
with detect sensors inside.
References
1. http://en.wikipedia.org/wiki/Tangram.
2. Yu Xiuyuan, TianXiaoqiu, Liu Ruomei, Lv Yong. Experiment
on Improvement the Operative Ability of Low-grade Pupils
with Seven-piece Puzzle Training. Exploration of Psychology2
(2001), 1-5.
3. http://zh.wikipedia.org/zhcn/%E4%B8%83%E5%B7%A7%E6%9D%BF.
4. Takahiro Iwata, Tetsuo Yamabe, MikkoPolojarvi. Traditional
Games Meet ICT: A Case Study on Go Game Augmentation.
TEI10 (2010), 237-240.
5. Alissa N. Antle. Embodied Child Computer Interaction:
Why Embodiment Matters. Lifelong Interactions. Mar.+Apr.
(2009). 27-30.
6. Emgu CV free source for shape detection. http://www.emgu.
com/wiki/index.php/Shape_(Triangle,_Rectangle,_Circle,_
Line)_Detection_in_CSharp.
184
Zhiyuan Zheng, Linkai Tao, Loe Feijs, Jun Hu

Eindhoven University of Technology, Den Dolech 2, 5612 AZ Eindhoven, the Netherlands
z.zheng@student.tue.nl, {L.Tao, J.Hu, L.M.G.Feijs}@tue.nl
Two design cases of social network

for parents in the context of
premature birth
Abstract
We present two demonstrators of interactive and
networked systems. The context of both systems is
premature birth, the neonatal intensive care unit and
the social situation of the parents. Both demonstrators
establish a link between the physical and the digital
and both take advantage of contemporary developments
in social computing. The first demonstrator is called
Touchee. The second demonstrator is called NICU-tree.
the NICU, as well as more bonding between parents and

their babies (Chen, Feijs, & Bambang Oetomo, 2010).
Keywords
Social computing, neonatology, interaction.
1 Context: premature birth
T U/e is short for

Eindhoven University
of Technology
M MC is short for
Mxima Medisch
Centrum Veldhoven
When a baby is born after a pregnancy of 37 weeks

or shorter it is said to be premature (Chen, Feijs, &
Bambang Oetomo, 2010). Premature and ill neonates
are to be treated in a special clinical environment,
called the NICU (neonatal intensive care unit). In the
NICU, the neonate is treated according to evidencebased therapies in which modern technology plays
an important role. Continuous monitoring of health
parameters is crucial for premature infants (Ahn & Kim,
2007). However, the equipment tends to separate and
alienate the parents and the baby and interferes with the
holistic care of the newborn (Miles & Holditch-Davis,
1997; Aagaard & Hall, 2008). Researchers at TU/e1 and
MMC2 are contributing to improving the experiences in
Fig. 1. Parents, peers, relatives and friends.
So far, less attention has been given to the relationship

between these parents and their relatives and friends.
Moreover, there is a strong need for this group of parents
to exchange more experiential knowledge, next to medical
knowledge. This opens up opportunities for exploration
by design. In Figure 1 we show the babies, their parents
and this next layer of relatives, friends and peers.
2 Inspiration from social computing

In many countries, there is already some form of
organisation by the parents (and external parties).
185
For example, in The Netherlands, there is the VOC

(Vereniging van Ouders van Couveusekinderen) with
the aim to bring together the experiences of parents
with premature babies. A board, bi-monthly magazines,
a website, a forum and offline gatherings form the
backbone. We believe it will be interesting to see what
social computing could offer additionally. In Figure 2
we present a more elaborate network of relationships
between various relevant stakeholders and institutions.
With fast developing web technologies like SNS (Social
Networking Service), information gets spread with less
time and space limitations.
information and emotion exchange between parents

with premature babies; 2) how to involve relatives in
the recovery process by sharing proper information. We
propose two different concepts, Touchee and NICU-tree.
3.1 Touchee
Fig. 2. Relationship between different institutions
Still, the major objective of present SNSs is to

connect with people who you know already. Yet in
the premature-context, the relevant group of people
is more dynamic, even within a short period of time.
Therefore another method of connecting (new) people
should be proposed to meet this need. Pooling is a
system in which people share information grouped
by their (sub-) cultural values (Straub, Loch, Evaristo,
Karahanna, & Strite, 2002). A similar example can be
found at www.atthepool.com. Besides, in our field
research, we discovered that parents do keep a diary
about their babies together with any meaningful objects
to associate with certain piece of memory. Thus we
take keeping-diary-behaviour as an initial point to start
designing within their natural flow of daily ritual.
3 Design concepts
Two important subjects emerged from previous
investigations: 1) how to stimulate higher level of both
186
Fig. 3. Overview of Touchee interaction components
Touchee embraces both online and offline activities.

There are two major components in the system:
1) a mobile digital device in the NICU room; 2) an
interactive platform in common areas in the hospital
as shown on the left part in figure 3. A cloud system
and a third-party collaboration of materialization
are supporting in the process. Every piece of digital
memory generated in the NICU room by the parents
of pre-term babies is stored in the cloud, which
can be accessed again with personal mobile device
outside NICU room. This data (represented on a
personal mobile device) together with an object from
the hospital that the parents would like to combine
into a holistic recollection, users can place them on
the interactive platform and create a corresponding
atmosphere. All the data will then be transferred to a
third party materialization service where a container
with meaningful shape is realized and delivered directly
to the applicant. In this manner, the combination of

materialization of digital data with tangible-memorial
object form a comprehensive media between the
parents or even at a personal level.
While parents take efforts and spend time in designing
their own gadget, there is much room for them to
come across with other parents where communication
happens. Furthermore, this gadget serves as a trigger
at the beginning of connecting parents that a tangible
object imply much more emotions than digital
information. In this way, parents are bonded in a physical
environment when possible, supporting by the pooling
technology to group similar parents together.
into a physical card that can be attached onto walls in

NICU rooms. The camera of the host machine will track
the location of this card and project a branch between
itself and the card, growing into a tree gradually. At the
same time, this information will be available on those
public displays where other parents/ relatives can scan
the QR code and interact with it, which will be reflected
on the projection in the NICU room.
3.2 NICU-tree
NICU-tree is a diary based socialization system
designed especially for parents with premature babies.
The system can be accessed in three ways: 1) a digital
device in a NICU room with beamer and camera
functionalities; 2) a large public display/ projection
in every hospital throughout Netherlands; and 3) a
personally owned portable digital device. A cloud
system supports all the accessibilities. Each piece of
dairy and related comments has a unique QR code
where users can scan and download corresponding
information. The interaction structure of NICU-tree
is shown as figure 4.
Fig. 5. Setup of NICU-tree for evaluation session
Privacy protection is also well-considered in this

concept that by default, all posted diaries are private.
Only after the user chooses to disclose certain piece of
diary publicly then it will be shown in the public display.
Besides for each of the diaries, parents can choose who
to share with, that only relevant relatives or friends
would receive notification. In all, privacy concern is
among the top issues to tackle in this proposal.
4 Lessons learned
The evaluation session was taken place in MMC where
ten participants from both MMC and RMH3 attended.
Genders were balanced to the degree possible, with
two men and eight women. The babies of three of
them are currently kept in the NICUs in MMC. Five
of them are volunteers working in RMH, among which
one mother had a pre-term baby in January 2012 who
recovered well from illness. Two are experts in the field
of neonatology.
R MH is short for
Fig. 4. Structure of NICU-tree
Ronald McDonald
Huis, ZuidoostBrabant branch to
be specific
Different from writing a diary, parents use their mobile

devices to record moments instantly and upload to the
cloud. Every piece of information uploaded will be made
After evaluating two design concepts with (potential)

parents, the feedbacks confirm the potential of both
systems in terms of facilitating deeper communication
and interaction between parents. On the virtual level,
modern technologies lower the barrier of creating,
storing and sharing information, which is crucial in a
stressful condition. On the tangible level, objects do
187
stimulate a more profound emotional exchange that

helps to build a robust relationship/network between
parents. Therefore it could be meaningful to combine
two systems in the next step.
References
Aagaard, H., & Hall, E. O. (2008). Mothers Experiences of
Having a Preterm Infant in the Neonatal Care Unit: A MetaSynthesis. Journal of Pediatric Nursing, 23(3). doi:10.1016/j.
pedn.2007.02.003
Ahn, Y.-M., & Kim, N.-H. (2007, December). Parental Perception
of Neonates, Parental Stress and Education for NICU
Parents. Asian Nursing Research, 1(3).
Burkhardt, M. E., & Brass, D. J. (1990). Changing Patterns or
Patterns of Change: The Effects of a Change in Technology
on Social Network Structure and Power. Administrative
Science Quarterly, 35(Special Issue: Technology), 104-127.
Chen, W., Feijs, L., & Bambang Oetomo, S. (2010). Neonatal
Monitoring: Current Practice and Future Trends. In
Handbook of Research on Developments in E-Health and
Telemedicine: Technological and Social Perspectives. IGI
Global. doi:10.4018/978-1-61520-670-4.ch045
Miles, M. S., & Holditch-Davis, D. (1997, June). Parenting the
prematurely born child: Pathways of influence. Seminars in
Perinatology, 21(3), 254-266.
Straub, D., Loch, K., Evaristo, R., Karahanna, E., & Strite, M.
(2002). Toward a Theory-Based Measurement of Culture.
Journal of Global Information Management, 10.
188
Yumei Dong1, Xiaohe Luo2 , Xiaoyu Tan3, Miaosen Gong4, Wei Wu5
School of Design, Jiangnan University, 1800 Lihu avenue, Wuxi, China 1,3,4
Nanjing Cynovo Electronic Technology Co.,Ltd, Nanjing, China 2,5
dong.yumei2008@163.com 1
Mobile Fetal Surveillance:

A product service system design
for pregnant women
Abstract
Mobile fetal surveillance service system design is a
cooperative project between DESIS Lab, Jiangnan
University and Nanjing Cynovo Electronic Technology
Co., Ltd. In this project, a fetus-voice meter
integrated with and cloud platform is designed. It
provides fetal surveillance, experts consulting and
social communicating services and makes it available
every time and everywhere for pregnant women and
their family. To some extent, it extends the function
of hospital and remits the problem resulting from
unbalanced healthcare resources.
Keywords
Fetal surveillance, Mobile healthcare, pregnant women,
Product service system
1 Social context
China is a country without enough and balanced health
care resource. Many of the healthcare resources are
distributed in urban areas. (Chuanshu Lu, Xiuli Xie
2007) It is difficult for rural patients to see a doctor
when facing health trouble with the limitation of
healthcare device and service. Healthcare fee is also
a challenge for those people. The function of hospital
should be extended outside.
In addition, the structure of most families in China is
changing into 421family pattern which means there
are 4 grandparents, 2 parents and a child in a family.

(Xun Zhang 2008) Then, the only child has to take
more responsibility for the family. Reducing perinatal
mortality rate and improving the quality of new-born
population is an important mission of woman and child
health care. Keeping pregnant woman and her fetus
healthy become the focus of the whole family. Fetal
surveillance is an important measure to keep pregnant
woman and her fetus healthy. Electronic fetal heart rate
monitoring is commonly used to assess fetal well-being
as a clinical measure. (Hina Gandhi, Lucy Kean 2008).
Keeping concerning the fetus healthy and monitoring
vital signs periodically during ten months pregnancy is a
challenge for pregnant women under the traditional fetal
surveillance in hospital because of body inconvenience.
With the rapid developing of mobile technology and
healthcare, echoscope will be replaced by smart phone;
doctors can check the electrocardiogram for patients
in different areas. It is not science fiction, but an
occuring revolution on healthcare technology, M-health,
combining healthcare with mobile technology, which
changed healthcare service very much, gets rid of the
limitation of time and space (Boroto Hwabamungu,
Quentin Williams 2010). Cloud technology supported
this revolution as well.
2 User research and the concept

After a series of research on pregnant women and their
family, two important findings are highlighted. Firstly
189
pregnant women wanted more interaction with fatherto-be. Secondly, they need more service when they
measured fetus heart rate such as communicating with
other pregnant women and exchange for second-hand
articles. Its found that a single product cant meet the
needs of pregnant women. A product service system
(PSS) (N. Morelli 2003, E. Manzini &C. Vezzoli 2003)
should be build to meet the comprehensive requirement
of pregnant woman.
A concept of mobile fetal surveillance service system
emerged. In this service system, pregnant women use
a device connected with iphone to collect vital
signs. Those dates were delivered to monitoring
cloud-platform as well as the application. Pregnant
women and their families can learn about the fetus
healthy via visualization interface of the application and
cloud platform. Medical care provider on pregnancy
healthcare service analyzed the date from cloud
platform and gave suggestions to relevant pregnant
woman so that they could deal with bad situation
accurately in time. At the meantime, pregnant women
could share the date of vital signs to social network.
A virtual community of pregnant women with the same
concerning would be naturally formed. Pregnant women
in this community would communicate with each other
and solve problems collaboratively, spontaneously and
proactively.
but the relevant and cloud platform were only imple

mented part of features. Details of those three parts of
mobile fetus Surveillance system described as follows.
Fetus-voice meter
Fetus-voice meter consists of body and probe for fetus
heart rate. It needs to be coated with coupling when
using. There is a Bluetooth button which can activate
the connection to the application. In order to create
a nice experience in the process of moving the probe,
a special handle was designed as following picture. The
date of fetus heart rate would be show in a small screen.
Fig. 2. Prototype of fetus-voice meter

(component and using reference image)
The application
If pregnant women want to check the history of the
date and learn about fluctuation of it, a is needed.
This application makes date display and management
possible. It also provides an access of consulting doctors
and sharing dates to social network.
Fig. 1. System map of mobile fetal surveillance service
3 Prototyping
A set of prototypes was made out after determination
of design proposal. It consists of a fetus-voice meter
connected with by Bluetooth, an and a cloud platform.
In this phase, fetus-voice meter could work normally,
190
Fig. 3. Some interface of the application
Cloud platform
Besides functions provided in the application, cloud
platform adds some relevant services such as secondhand article exchange and pregnant women class in order
to meet their extra needs we found in user research.
Mobile fetal surveillance service system was tested in
some pregnant families. It is proved to be effective and
have a more accurate outcome and nicer experience.
Father-to-be and pregnant women have a good
interaction when using it. Something insufficient were
also exposed. Firstly, the size of fetus-voice meter probe
was too big so that it was easy to soil belly of pregnant
women when coated coupling. Secondly, a pregnant
woman cant use body of fetus-voice meter and at the
same time by herself. Without support of her husband,
it was not convenient to use it.
perspective for designers is employed. In this project,

a product service system is designed. It consists of a
mobile device for fetus heart rate measurement, an
and a network for relevant requirement.
Products are able to help to create a harmonious
relationship of humans. The fetus-voice meter in this
project makes father-to-be and pregnant women
join forces and enjoy a good time of communicating
with their fetus. It is also important to point out
that designer should control their utility they put on
products. Influencing but not forcing users is a proper
way to create a harmonious system.
Technology stimulates innovation. Mobile
technology combined with healthcare industry
increases many possibilities for fetal surveillance
revolutionarily. Identifying opportunity of technology
is very important in the new product and service
development.
5 Acknowledgement
We thank to all colleagues and users who give supports
during the project and its partly supported by Ministry
of Education Humanities and Social Science Youth Fund
(11YJC760018).
References:
1. Chuanshu Lu, Xiuli Xie. (2007). Review on Chinas healthcare
reform. Journal of Medicine and philosophy: Humanities and
social medicine. 2,12-14
2. Xun Zhang. (2008). Build a new family pension system
scientifically. Hubei Social Sciences. 5,70-72
3. Hina Gandhi, Lucy Kean. (2008). Electronic fetal heart rate
monitoring. British Journal of Hospital Medicine.69(7), 288-391
4. Boroto Hwabamungu , Quentin Williams (2010).M-Health
adoption and sustainability prognosis from a Care givers
and patients perspective , Proceedings of the 2010 Annual
Research Conference of SAICSIT, 1113 October 2010, Bela
Fig. 4. Scenes of users testing
Bela, South Africa, United Kingdom: ACM Press,123-131

5. N. Morelli(2003) Product-service systems, a perspective shift
4 Discussion and conclusions
for designers: A case study: the design of a telecentre. Design
This project builds a product service system for

pregnant women and their families, the first prototype
works within expectation. there are a lot of work to
be done at next phase including product development
and the whole system optimizing. From concept,
prototyping to testing, what we learned are in follows:
A single product cant meet the comprehensive
requirement of users in many cases. A systematic
Studies.24(1),74-99
6. Manzini, E. & Vezzoli, C. (2003). Product-Service Systems and
Sustainability. Opportuni-ties for sustainable solutions. UNEP.
191
Martijn ten Bhmer, Eunjeong Jeon, Kristi Kuusk

Eindhoven University of Technology, Department of Industrial Design, The Netherlands
{m.t.bhomer, e.jeon, k.kuusk}@tue.nl
Vibe-ing: Designing a smart textile care

tool for the treatment of osteoporosis
Abstract
Vibe-ing is a care tool in the form of a garment, which
invites the body to feel, move, and heal through vibration
therapy. The merino wool garment contains knitted
pockets, equipped with electronic circuit boards that
enable the garment to sense touch and vibrate specific
pressure points on the body. With this design we aim
to inform a multi-disciplinary audience about the oppor
tunities of integrating textile and vibration for health
care applications. We show how new manufacturing
can lead to new possibilities in garment design and the
integration of electronic components. With an example
of dynamic behavior we demonstrate how the vibration
therapy of the garment can be tailored to individual
treatment needs. This design serves as a start. We plan
to further investigate the effects of vibration therapy
combined with textile design and electronics for the
treatment of osteoporosis.
Keywords
Smart textiles, wellbeing, customization, product behavior.
1 Introduction
Vibration has three positive therapeutic applications:
(a) for the improvement of bone density and muscle
strength [1]; (b) for the attenuation of delayed-onset
muscle soreness [2]; and (c) for an increase of the
speed of the blood flow through the body [3]. Vibe-ing
is a care tool in the form of a garment, which invites
192
the body to feel, move, and heal through vibration

therapy. The aim of this design is to contribute to the
treatment of osteoporosis, particularly for women
who are most at risk of developing osteoporosis in the
post-menopause period [1, 3] and therefore supporting
the need for their wellbeing. The textile from which
Vibe-ing was constructed has been produced using a
fully-fashioned knitting machine. This technique allowed
us to create digitally designed, pre-shaped pieces for the
garment. The textile contains pockets in which circuit
boards with sensors (touch sensors) and actuators
(vibration motors) can be placed (shown in Figure 1).
Throughout the textile power and communication lines
are integrated that connect the pockets with each other.
This modular system enabled us to program the exact
areas and the way of stimulation on the body depending
on the specific persons need for rehabilitation and healing.
By developing this prototype we aim to trigger a multidisciplinary audience to consider the opportunities of
integrating textile and vibration for healthcare applications. We hope to bridge the disciplines of fashion,
technology and healthcare. Further, we highlight the
possibilities of integrating textile and electronics using
modern manufacturing techniques. Finally, we will
show how a body-worn decentralized network of
vibration and touch modules enabled us to add a
dynamic behavior to the garment, tailored to the
needs of the person wearing Vibe-ing.
Fig. 1. Left: front of Vibe-ing with two rows of pockets. The

white circles visualize the vibration of the modules. Right: back,
with the pockets containing the vibration and touch modules on
the spine and shoulders. The silver lines are the connection lines
between pockets
2 Textile and technology for healthcare

Relating to the integration of textile and technology for
osteoporosis applications there are some design choices
we would like to emphasize. These design choices show
the challenges within the different levels of the design
process. We choose for merino wool as textile fiber
because of a number of attributes and beneficial properties. These properties include a resistance of chemicals
and dirt, absorbency, insulating capabilities, resilience,
positive tactility [4] and versatility [5]. For the textile
design we used two layered knitting and felting
techniques to produce a textile with a voluminous
shape, soft and bulky surface. This surface invites the
wearer to stroke and touch the fabric and the body. For
the integration of technology we choose for vibration
actuators that are integrated in the textile pockets. This
combination enables the application to have a healing
effect on the muscle density of the people wearing the
garment. In the garment design we placed the pockets

to align with critical pressure points on the body
(Figure 1 shows the specific placement of the modules
on the body of the wearer). For example, the shoulder
area is a difficult location to perform surgery when
fractured. Noninvasive treatment, such as vibration on
this location could increase the bone density and relief
shoulder pain. The lumbar region on the back is often
associated with a sedentary lifestyle, and a source of
chronic pain for many older people.
Vibration could relieve the pain by increasing the blood
flow in this specific area [1]. By rotating the garment
(back to the front, top to the bottom) we can further
stimulate pressure points on the front of the body. This
can offer stimulation of the hips and the ribs. These
two areas are often affected by decreasing bone density
in the post-menopause period [1, 3]. Simultaneously,
flexibility in wearing the garment in different ways
enables different treatments with the same garment,
limiting the amount of textile, electronics and energy. In
conclusion, these aspects merino wool properties, the
knitted soft surface and vibration actuators embedded
in the pockets in the garment could contribute to the
feeling of physical-emotional comfort as a noninvasive
therapeutic treatment of osteoporosis.
193
Fig. 2. Left: the computer interface to design the pockets.

Right: the bottom of the pockets, showing the touch sensitive
fabric, the connection lines and the electronic modules
3 New manufacturing possibilities

Besides showing the potential for healthcare, the design
of Vibe-ing is also an exploration of the possibilities
to bring the manufacturing of textile and digital design
tools closer to each other. Modern manufacturing
techniques, such as fully-fashioned knitting, make it
possible to produce a garment without the need of
additional cutting and sewing (the left picture in Figure
2 shows the digital tools used to design complex
textile complete garments). These techniques will
enable designers to produce in smaller quantities,
customized to the users needs, shape of the body, and
aesthetic preferences, thus reducing waste compared
to traditional mass production of garments [6]. At the
same time, this allows for more complex structures that
make it possible to integrate technological elements into
textiles. This perspective expands the design discourse
in terms of how garment design is more integrated with
body consciousness, which links to a sense of physical
wellbeing. We will illustrate these new possibilities with
examples from Vibe-ing. During the design process the
merino wool was knitted using a jacquard pattern to
create a bulky structure after being felted. The garment
can be customized to different body shapes, while the
placement of the vibration modules can be modified
to the needs of the individual user. The pockets in the
garment contain specially developed modular electronic
circuit boards with actuators and sensors on it (the
modules are shown on the right picture in Figure 2).
Connected by power and communication lines, these
modules provide the therapeutic vibration. The knitted
construction of the pockets consists of a combination of
different yarns. In the bottom of the pocket a mix of a
194
soft conductive yarn and an elastic yarn to function as a

stretchable touch sensitive surface (showed on the right
picture in Figure 2). Because of the conductive yarns
it becomes possible to measure the body capacitance
of the skin touching the fabric, creating a basic touch
sensor. The copper yarns knitted in the textile, provide
the power and communication throughout the garment,
while in the pocket they run freely to make them
accessible for connecting to the circuit board.
4 Behavior of the garment

With this design we would like to show the possibilities
of a body worn network of sensors and actuators
integrated in the garment. We see more and more
examples of intelligent systems consisting of different
nodes, for example in living rooms [7], lighting systems
[8] and textile applications [9]. These make it possible
to extend the functionality of separate products, and
can transform previously static products into products
with a dynamic behavior. In Vibe-ing we distributed
multiple modules in the garment, corresponding to
different locations on the body. With this configuration
it became possible to program the exact areas and form
of stimulation on the body, depending on the specific
persons need for care, rehabilitation, and healing. The
modules in the garment are programmable using the
Arduino platform, enabling designers to quickly adapt
the functionality and behavior of the garment. As
demonstration of this platform we designed an initial
behavior for the garment that is based on a ripple pattern
(similar to a wave in the water or sound travelling
through air). A vibration would start in the pocket
which was touched by the person wearing the garment.
Then, slowly the vibration would transfer to the
surrounding pockets, until it faded away after a certain
period (this dynamic movement of vibration is visualized
by the white circles on the left picture in Figure 1).
5 Further research
7 References
The work we presented in this short paper is still

preliminary and serves as a trigger to open further
research. The garment has been designed as a platform
to explore different behaviors of the vibration motors.
The implemented ripple behavior is one of the
possibilities but has not yet been verified for the
treatment of osteoporosis. Our assumption is that the
behavior of the garment could engage the user in the
treatment process. For example, by guiding the touch
of the person with the vibration motors to specific
pressure points, or by gently reminding the person to
stimulate the pressure points during long inactivity. As
a next step it will be necessary to design these different
behaviors in collaboration with practitioners and endusers, and to validate these behaviors with user tests.
The placement of the pockets is based on the specific
pressure points and has been carefully selected. By
further discussing this placement with experts, and
conducting user studies with the prototype we hope
be able to validate the placement. To treat osteoporosis
effectively we will further investigate existing scientific
and medical implementations to find out more about
the amplitude and frequency characteristics of the
mechanical stimulation. Thereafter, it will be necessary
to monitor this frequency and let the vibration modules
adapt to the right frequency in relation to the location
of the pocket on the body. Because of the fully-fashioned
manufacturing technique it becomes possible to customize
the garment to the preferences of an individual. This
opens up new design possibilities and new business
models that can be explored. To conclude, in further
research it will be crucial to collaborate with medical
experts to evaluate our application of vibration therapy
and extend it further based on knowledge in the medical
field. To explore the possibilities of customization and
the design of a personal care tool it will be necessary to
involve end-users and manufacturing partners to make
sure the garments fits the individual preferences and
body shape and measurements of the person.
1. Verschueren, S.M., Roelants, M., Delecluse, C., Swinnen, S.,

Vanderschueren, D., Boonen, S.: Effect of 6-Month Whole
Body Vibration Training on Hip Density, Muscle Strength, and
Postural Control in Postmenopausal Women: A Randomized
Controlled Pilot Study. Journal of Bone and Mineral Research.
19, 352359 (2003).
2. Lau, W.Y., Nosaka, K.: Effect of Vibration Treatment on
Symptoms Associated with Eccentric Exercise-Induced
Muscle Damage. American Journal of Physical Medicine &
Rehabilitation. 90, 648657 (2011).
3. Klima, R.R., Weigand, A.H., DeLisa, J.A.: Nerve Conduction
Studies and Vibration Perception Thresholds in Diabetic and
Uremic Neuropathy. American Journal of Physical Medicine
& Rehabilitation. 70, 8690 (1991).
4. Jeon, E.: Designing enriched aesthetic interaction for garment
comfort, Doctoral Thesis (2013).
5. Holcombe, B.V.: Wool Technology and Sheep Breeding: The
role of clothing comfort in wool marketing. CSIRO Division
of Textile Physics. 34, 8083 (1986).
6. Larsson, J., Peterson, J., Mattila, H.: The knit on demand
supply chain. AUTEX Research Journal. 12, 6775 (2012).
7. Van der Vlist, B., Niezen, G., Hu, J.: Design semantics of
connections in a smart home environment. Design and
Semantics of Form and Movement (DeSForM 2010). (2010).
8. Magielse, R., Offermans, S.: Future lighting systems. CHI
13 Extended Abstracts on Human Factors in Computing
Systems (CHI EA 13). (2013).
9. Deckers, E., Wensveen, S., Overbeeke, C.J.: PeR: Designing
for perceptive qualities. Presented at the Design and
Semantics of Form and Movement (DeSForM 2010),
Lucerne, Switzerland (2010).
6 Acknowledgements
This work is being carried out as part of the project
Smart Textile Services sponsored by the Dutch
Ministry of Economic Affairs under the CRISP program.
We would like to thank Jesse Asjes from TextielMuseum
TextielLab and Admar Schoonen from Metatronics for
their contributions in realizing Vibe-ing.
195
Shengxiong Zhang, Tiantian Yang, Feng Wang

School of Digital Media, Jiangnan Universitiy, China
zsx.chn@gmail.com, taliayoung@163.com, water6@126.com
Social Blobs, an Interactive Art

Installation in an Urban Public Space
Abstract
In recent years, with the rapid development of
technology and culture, digital media art has arrived
into view. Taking Social Blobs of Interactive Patina
of Culture as an example, this paper begins from the
application of interactive art installations of digital media
arts to urban public space, and introduce how to merge
interactive art installations with public space and the
historical and cultural background of public space.
Keywords
Projection Mapping, Interactive Art Installation,
Interactive Patina of Culture
1. Introduction
Since 21th century, the popularity of computer
technology has changed traditional art forms in
multiple ways. At the right historical moment, digital
arts including interactive art installations appeared.
Representing new audio-visual experience in both
expression forms and interaction forms [1], interactive
art installations cover various subjects including design
arts, computer graphic techniques, media hardware
materials, web of things, etc., even related to biology,
music, and physics. Hence, in order to make complete
design and creation, it requires that the creator had
multi-discipline background knowledge. In recent years,
due to the efforts of artists, designers and architects,
there are numbers of interactive art installations
196
used in global significant festivals and exhibitions like

ParticipArt[2], KALI[3] and Tate (an art museum
in UK). In previous reported works, we found that
technology innovation is focused more. However, the
deeper cultural connotation of the artworks is rarely
concerned. It is a challenge for us to fuse culture and
interactive art installation.
Unlike traditional arts, an interactive art installation,
which is consisted of installation and artistic expression,
is live art controlled by computers. In our design
Social Blobs concerning the subject Interactive Patina
of Culture[4], we use installation to acquire users
action, and then influence users action through artistic
expression method. The philosophy of the unity of
human and computer is merged into the interaction
of the installation, users action is acquired by program,
the gathered data are artistically displayed after being
calculated, and the participants may receive the most
direct experience through projection techniques,
therefore, the fusion impact and exploration of arts,
culture and techniques are really achieved.
2. Design of Social Blobs

The theme of this times workshop is Interactive Patina
of Culture, of which culture is mostly highlighted,
meaning culture is the soul in the interactive design
of urban public space, if the art works do not contain
culture, they will lack ideological connotation. And the
design will be worthless, and feeble even though it exist
in the city. While the word patina means continuousness

and accumulation, that is, each participant may leave
relevant footprint in the works. It is the themes key
point as well as difficult point how to rationally and
perfectly merge patina and culture into interactive
installation design.
Based on the theme of this workshop, designers
designed a set of interactive art installation in public
space for Taicang-the holding location of this workshop.
As the starting point of Zheng Hes seven expeditions
to the western seas[5], Taicang owns plenty of culture
accumulation, and is a natural bridge for Sino-foreign
culture communication. The participants of this
workshop are made up of teachers and students
from Jiangnan University and those from Eindhoven
University of Technology, during the communication,
the creators found enormous cultural differences
between China and Holland. For example, when going
out, Chinese people enjoy walking together with several
friend, and the distance between them may simply equal
that of their personal relationship; while in Holland, the
distance between people cannot just represent their
personal relationship. Based on the extension of this
case, the creators gain lots of design inspiration.
As a design case of urban public space, the importance
of interactive installations site selection is never less
than its content design. Taicang has been a culture
cradle since the ancient times, all sorts of talents
gathered here, so it has a solid culture foundation and
forms its own features, and all these constitute unique
Loudong Culture, leaving age-old and outstanding
culture treasures for people today, of which Zheng
Hes seven expeditions to the western seas is the most
dazzling one. Located in the center of traditional Taicang
urban business circle, and taking Zheng Hes seven
expeditions to the western seas as the whole squares
cultural background, Nangyang (means ocean) Square
Fig. 1. Prototype of Social Blobs and night scene of Nanyang Square
is a shopping plaza which has the largest area, the most

functions, and the most complete industry planning,
therefore, it becomes the first choice for this case.
During the early stage of interactive installation design,
due to Nanyang Squares unique position, the designers
make an in-depth study on the crowd in the square, and
classify them into three categories according to their
behavior characteristics, which are highlighted with
three colors, namely, red, orange, and blue, respectively.
Blue is a kind of cool tone, standing for loneliness,
so the blue color is used to stand for a lonely person
walking in the square. Both orange and red are warm
tone, standing for warmness and friendship, of which
red is more powerful and used to stand for small groups
that own leadership, while orange stands for ordinary
small groups. Every small group is part of the society, so
the creators also use the concept of cells to stand for
every group. For the orange group, the size of relevant
cells is determined by the cohesion among them. In
China, unity means power, so the smaller the distance
between the individuals is, the larger the cell will be,
otherwise, it will be smaller. For the red group, Chinese
leaders usually walk ahead of the group, and the number
of the population following the leaders determines
the size of the red cell, and also stands for the leaders
leadership skills.
As the largest comprehensive square, Nanyang Square
has a round platform and a neat building facade (see
Fig. 1), Projection Mapping technology is adopted to
synchronically cast the crowd features on the round
platform and building facade. To enable the interactive
installation to possess more universal application,
designers further abstract the cells into formal circles,
but reserve their fusible features. When participants
walk on the round platform, circles that have the same
features with the crowd will follow the crowd under
their feet, and the circle features may vary in color, size,
and shape with the changes of the participants distance
and relationship. When two crowds of participants meet
or part with each other, their feature circles may merge
or part with others, the feature circles colors may also
change accordingly. The course of fusion and separation
may the most simply and concretely reflect subtle
changes of the relationship among the participators.
Finally, when the participants walk out of the round
platform, their relevant feature circles will continuously
accumulate on the building facade, new feature circles
will cover old ones, and the long-period feature circles
197
may vanish with the accumulation of time, which

is the so-called patina of culture.
3. Implementation techniques of
Social Blobs
The Social Blobs interactive installation (see Fig. 2) is
made up of three parts, namely, data gathering, data
processing, and graphic output. In data gathering,
sensors are used to obtain the participants position
data, Arduino[6] may obtain data from the sensors
through simulation electric signals, pack and send to
the computer, after the Processing[7] in the computer
obtains the data, it will sort and process the participants
and achieve abstract graphics, then these graphics are
presented to the participants by projector by means of
Projection Mapping.
The current popular Arduino development board
and Processing programming language are adopted in
combination (see Fig. 3) for the main control part of this
interactive installation. Arduino is a handy, convenient
and flexible open source electronics prototyping
platform, including hardwares and softwares (Arduino
IDE). It is widely used by artists, designers and fans.
Arduino may sense the environment through all sorts
of sensors, feedback and influence the environment by
controlling lighting, motors, and other devices. The micro
control units on the electronic board may participate
program response, and connect with the interactive
platform by using Arduino programming language.
Fig. 2. Flow chart of Social Blobs
Fig. 3. Working principle diagram of Social Blobs
With the development of social culture and information

technology, the new art form that uses of interactive
art installations in public spaces will be gradually
popularized and commonized. The workshop of
Interactive Patina of Culture is an active exploration
where interactive installation, public device, and
historical and cultural background of the public space
are merged together. However, as a kind of art form,
interactive art installations should not always pursue
technical innovation, but create distinguished design
works by craftily using reasonable technical means from
the perspective of cultural connotation and meaning.
Reference
1. Wang, F., J. Hu, and M. Rauterberg (2013). New Carriers, Media
and Forms of Public Digital Arts, in Culture and Computing 2012,
Springer-Verlag Berlin Heidelberg: Hangzhou, China. p. 83-93.
4. Discussion
In order to make our work Social Blobs has a strong
public participation, the most instinctive behavior of
human is adopted. The trace left on the square by
participant is displayed as an abstract visual social
interaction. Making participants as the center of the
whole interactive installation, ticking out deliberately
artificial creation, this design creates abstract art
patterns of interpersonal relationship which merge
into cultural fundamental embodiment with the most
natural and simple modes of participants random
wandering. This mode that allows users to design may
make the whole interactive installation full of interest
and unpredictability, and let urban public space more
vivid, full of humanity and cultural connotation. As the
achievement achieved by the Sino-Holland Workshop,
Social Blobs has won spectators praises and panel
judges consistent affirmation.
198
2. Jacucci, G.; Wagner, M.; Wagner, I.; Giaccardi, E.; Annunziato, M.;
Breyer, N.; Hansen, J.; Jo, K.; Ossevoort, S.; Perini, A.; Roussel, N.;
Schuricht, S., "ParticipArt: Exploring participation in interactive
art installations," Mixed and Augmented Reality - Arts, Media,
and Humanities (ISMAR-AMH), 2010 IEEE International
Symposium On , vol., no., pp.3,10, 13-16 Oct. 2010
3. Fischnaller, F., "Kali: the Goddess of the millennium [interactive art
installation]," Virtual Systems and Multimedia, 2001. Proceedings.
Seventh International Conference on , vol., no., pp.545,554, 2001
4. Frens, J., M. Funk, J. Hu, S. Zhang, K. Kang and F. Wang(2013).
Exploring the Concept of Interactive Patina of Culture in the 8th
international conference on Design and Semantics of Form and
Movement (DeSForM), Wuxi, China.
5. Graffe, David A. "Book Review of Zheng He: China and the
Oceans in the Early Ming Dynasty, 14051433". Journal of
Military History. Retrieved November 14, 2012.
6. Arduino: http://www.arduino.cc/
7. Processing: http://processing.org/
Daniel Cermak-Sassenrath
Computer Games and Interaction Design (CGID), IT University of Copenhagen, Denmark
mail@dace.de
Makin Cake and the Meaning in Games
Abstract
Why is it that players can do the most extraordinary
(e.g. violent or physical) acts in games without
hesitation, reluctance or doubt? What do objects and
actions in games mean? This article introduces the
notion of the opacity of play: Play is opaque towards
non-players, its meanings are not transparent. The
article also offers an explanation how people deal with
this, that is, why spectators struggle and players easily
cope with it. The interactive installation Makin Cake
demonstrates the issue by providing an immediate
and provocative experience to players and spectators.
It appears that play is not dependent on or even
interested in the subject matter it plays with. Players
most naturally understand and know that their actions
do not mean anything. In this respect, play acts like a
fun-house mirror into Wonderland, reflecting ordinary
life but giving it its own twist, path, and, finally, meaning,
free and independent from the everyday world.
players are asked to perform quite dodgy things which

are essential to the game play, and without which the
games do not proceed. Stealing an object from each
other (as in many ball games), physically assaulting,
potentionally hurting and injuring each other (as in
many contact sports), or taking off each others people
(as in Chess) are examples from traditional games. In
digital games, digging up a human bone from a graveyard
in Monkey Island 2 (1991) appears relatively harmless
compared to activities in some of todays games (e.g. the
(optional) massacre of unarmed civilians at an airport
in Call of Duty: Modern Warfare 2 (2009) [13] and the
killing of a large portion of the population of the city of
Stratholme in Warcraft 3: Reign of Chaos (2002) [33]).
While occasionally there is some discussion (e.g. on the
baptism sequence in Bioshock Infinite (2013) [22, 7, 10]),
it appears not to be a major problem for the majority of
players to carry out these activities [cf. 34], neither in
traditional games nor in digital games.
Keywords
Installation, play, game, media, transparency, meaning,
swearing, cake, baking, 1950.
There appear to be several possibilites to explain this.

For instance, it can be argued that experienced players of
popular games are desensitised zombies or social freaks
who are unaware and ignorant of their actions; or that
they have become computer-savvy media geeks who are
able to decode games from an analytical distance without
being bothered by their content. Here, a conceptual expla
nation is offered that argues that players most naturally
understand and know perfectly well what their actions
1. Introduction
Why is it that players can do the most extraordinary
(e.g. violent or physical) acts in games without hesitation,
reluctance or doubt? There are many games in which
199
in games mean and how they relate to everyday life.

There appear to be at least three aspects to the
question of meaning in games: What do objects and
actions in games mean? How are these meanings
constructed? By and for whom? The article focuses on
the first one; the interactive installation Makin Cake
demonstrates the issue of meaning of play activities
within and without play by providing an immediate and
provocative experience to players and spectators.
2. The Makin Cake Installation

The Makin Cake installation (Figures 1-2) is a skill-based
competitive single player game with a twist. Players
type-in 1950s cake recipes as fast as they can and get
points for every word. The twist are extra points for
swearing, but there is a time limit, and players might
lose the game if they swear too much.
in each step are displayed at the top of the screen/

projection. Players type in what needs to be done, e.g.
Add flour. They then press return/enter to go to the
next line and the next step. In each step all ingredients
and actions (up to three) have to be typed-in. For each
line players receive a score calculated from the number
and length of the words they entered. Swear words
produce roughly ten times as many points as non-swear
words. Words that are used repeatedly give less points
each time until they do not give any points at all. The
exact points scoring algorithm is purposely not made
transparent to players.
In the game, there are nine different levels or recipes.
The recipes are based on actual vintage cake recipes
from the 1940s, 50 and 60s [21, 3, 5]. They have
different time limits. Recipe 1 (One Egg Cake), for
instance, has a time limit of 90 seconds. When players
succeed in typing-in a complete recipe inside the time
limit, they are informed if their score was high enough
for them to be invited to join the champions hall of
fame. In this case, they can enter their name and have
their picture taken. The photo high score list adds an
interesting layer to the game, i.e. how and if players
stand by their recipes full of swearing; also, players can
play off the props available in the setup (e.g. large spoon,
chefs jacket, apron, hat). The top recipes are displayed
while nobody is playing, revealing a peculiar mix of
baking instructions and shocking language.
Fig. 2. Player and spectators;

high score list entry with
recipe and player photo; player
posing with props for high
score photo
Fig. 1. Title screen, main menu, in-game (screen shots)
To play the game, players select a recipe from the

main menu. Each recipe is devided into nine steps. The
ingredients to be used and the actions to be performed
200
The installation demonstrates a conflict between play

and the everyday world. Players are provoked by a
moment of self-confrontation. The game confronts
people and makes people confront an authentic but
disquieting side of [themselves] [19]. Participants
have to juggle playing the game and performing for

spectators, explaining and potentially justifying their
actions. Spectators are challenged to enter the game
and become players; they switch from the everyday
world to play and back. The setup as an installation
(a 1950s kitchen), where playing the game means also
performing for an audience, invites critical reflection
and discussion.
3. Conceptual Distinction of Play and non-play

For Huizinga [9] and Scheuerl [25], play is conceptually
clearly divided from everyday life. Among the characte
ristics they use to describe play, the freedom and the
secludedness or limitedness of play appear to be the
most relevant for the question at hand.
Huizingas notion of the magic circle describes a place
beyond ordinary life. In the play-state you experience
a protective frame which stands between you and the
real world and its problems, creating an enchanted
zone in which, in the end, you are confident that
no harm can come. [1 qtd. in 26] The boundary of
play which is established in the heads of the players
can coincide with e.g. lines on the ground or special
uniforms, or rather, is articulated or realized through
them. Although this frame is psychological, interestingly
it often has a perceptible physical representation: the
proscenium arch of the theater, the railings around the
park, the boundary line on the cricked pitch, and so on.
But such a frame may also be abstract, such as the rules
governing the game being played. [Ibid.]
4. Play is self-referential
Play is only interested in itself. There are many
examples in which e.g. issues, problems or differences
players have outside of play do not translate into play.
On the possibility of hiring drummer Matt Sorum into
Guns n Roses, Slash recalls: The pay was good and
there were no rules, except for one: all you had to do
was play well. [29] Hot rod and motorbike customizer
Cole Foster states that he does not have any tattoos,
no mobile phone and no iPod, and that he did cry when
Bambi was shot but in the end it was only relevant
what he builds [11].
The meanings play creates by and in itself massively
outweight their import from the outside. Willke [31,
my transl.] describes the autopoetic quality of human
social or personal systems and claims that they
need to be understood as closed systems which create

and maintain themselves, that is, they produce their
specific dynamics not only as a reaction to input from
its surroundings, but primarily through their selforganisation. [23] Maturana [16 qtd. in 20] terms this
a closed causal circular process. It is this particular
formal structure that Maturana and Varela [17] label
autopoiesis (or self-forming) [...]. [20] Arguably this
applies to play even stronger than to other media.
5. Import of Objects and Actions into Play

Play uses activities and objects from ordinary life and
pulls them into play. The everyday meaning something
has might serve as a starting point for people to
understand what something means in play, but this
meaning is only a weak indication of the meaning
something has in play. The question is, [...] which is
to be master thats all. [4] Play can assign meanings
freely and arbitrarily, regardless of what things mean
elsewhere. Game actions refer to actions in the real
world, but because they are taking place in a game,
they are simultaneously quite separate and distinct
from the real world actions they reference. [26] Even
more: When we play a game, we are doing more than
just shuffling signs drawn from the domain of the real
world; instead, we are shifting to another domain of
meaning entirely. [Ibid.] Retter [23] states that the playexternal world only offers material to play, and that play
transforms objects when it moves them into play.
6. The Play Material

Whether the material that is played with is imported
into play, especially manufactured for play or used
exclusively in play, it is only played-with. It is not the
content in ifself which is interesting to play. Play is not
about soccer balls, chess pieces, playing cards, toy
trucks, tennis rackets, or princesses. How all-important
the material might be or become to players, the play
material is not interesting in itself, but only enables play,
and is, essentially, exchangeable.
The activities of play are often quite ordinary, tedious
or trivial. While the actions and objects of play might
appear similar to actions and objects of everyday life,
they lose essential parts of their meaning; play takes
away the need, the necessity, etc. They enter into a
new and different space in which they are assigned new
meanings independend of the ordinary world, by the
201
players, according to the possibilities they offer to play.

Their original meanings might still vaguely hover in the
background, but they are not relevant to play. Playing
on American tanks as after the second Gulf war in Iraq
loses its attraction quite quickly when play takes over
from ordinary life. NBA player Dennis Rodman [24] calls
the basketball the fucking thing, and echos McLuhan
[18] who says that ostensible program content is a
lulling distraction needed to enable the structural form
to get through the barriers of conscious attention.
Plays content is only a tangible articulation of an ideal
space, and is called material props by Dunne and Raby
[6], and texture by Aarseth.
7. Creating Reality
It is not the reflection, the learning or the gain of
insight about reality that drives play, but the creation
of reality. Play is not an abstraction of ordinary life
or a duplicate or a copy of the everyday world. It
applies what Manovich [14] says of computer graphics:
Synthetic computer-generated image is not an
inferior representation of our reality, but a realistic
representation of a different reality. Play is not ordinary
life, but real nonetheless. Who wants to say what it real
and what is not? Following Luhmann, [b]oth non-play
and play are realities, because they are products of a
distinction, a difference that makes a difference. [30]
In play, players encounter a fully-valid reality. Play is
made up of real actions in unreal worlds. Play is neither
a schein reality or an ersatz reality; it is not defined or
legitimized through references into an external reality,
very similar to art which, according to Heidegger, does
not depict or represent the world. It creates a world of
its own. [12; cf. 8]
8. Medial Bleed
Although play does not aim to change the world, the
bleed of effects from play into the play-external world
is often at least tolerated by players in many cases this
by-product of play actually appears to be quite welcome
(for instance, a reputation or fame). Medial overflow
into other media or into everyday life is not limited to
play, and a well-known phenomenon: Hollywood is a
town of fabulators. The people who dwell there create
fictions for a living, fictions that refuse tidily to confine
themselves to the screen, but spill over into the daily
lives of the men and women who regard themselves
as stars in the movies of their own lives. [2] Also,
202
meanings trickle from ordinary life into play. Apparently,

the well-known British train robbers Biggs, Edwards,
Goody, Reynolds, Wilson and Wheater, who took
between 2.3 and 2.6 million pounds from a Royal Mail
train in August 1963, played Monopoly with real money
in their farmhouse hideout after the heist [15].
9. Conclusion
Play is not dependent on or even interested in the
subject matter it plays with. While there is an interplay,
and play material is important to play, it is only playedwith. It does not mean anything, that is, beyond play.
Likewise, players actions do not mean anything. In
this respect, play acts like a fun-house mirror into
Wonderland, reflecting ordinary life but giving it its own
twist, path, and, finally, meaning, free and independent
from the everyday world. The notion of the opacity of
play is proposed to describe this phenomenon.
Players assign meanings to actions and objects which
only depend on the meanings they have or gain in play.
Other media may reference ordinary life to a stronger
degree, or rather, at all. Play is opaque with regard to
meaning, and games are strongly autopoietic. Players
accept all kinds of play actions, because they are empty
[28]. Every child who plays, knows that it plays [9]
and is aware that it is only pretending [9; cf. 26, 27].
Players are different and do things differently [9], and
most naturally step out of the systems of meaning that
surround them in the everyday world, without being
social freaks or media experts. Players understand
play, spectators do not, because they have different
perspectives on what is happening, and different ways
to participate.
This situation is the potential the Makin Cake
installation plays off: If the meaning of a medial text
is always open [32] and is to be determined in an
heterogenous manner by the people involved, and
cannot be predicted, there is potential for conflict.
The installation goes a step further to create a
confrontational situation in which the restraints,
customs, laws, rules, etc. of ordinary life are juxtaposed
with the freedom, emptiness and meaninglessness of
play. Players and spectators have to face the conflicts
that appear between them; but there are also conflicts
to face for the same person having been a spectator
before and being a player now. Makin Cake emphasises
this change of perspective, this step into and out of play;

it confronts players with their own joyful, unreflective,
direct and immediate experience, and provokes people
in their role as spectators.
17. Maturana, H., Varela, F. J.: Autopoiesis and cognition: The

realization of the living. Reidel, Dordrecht (1980)
18. McLuhan, M.: Understanding Media. The Extensions of Man.
Routledge, London (2002)
19. Murray, J.H.: Hamlet on the Holodeck: The Future of
References
1. Apter, M.J.: A Structural-Phenomenology of Play. In: Kerr, J.H.,
Apter, M.J. (eds.): Adult Play: A Reversal Theory Approach,
pp. 1330. Swets and Zeitlinger, Amsterdam (1991)
2. Biskind, P.: Easy Riders, Raging Bulls. How the Sex n Drugs
n Rock n Roll Generation Saved Hollywood, 10th ed.
Bloomsbury, London (1999)
3. Carolyn.: The 1940s Experiment Weight loss blog
losing 100 lb on wartime rations. Blog dated 14/10/2009,
http://1940sexperiment.wordpress.com/2009/10/14/applecrumble (2009)
Narrative in Cyberspace. MIT Press, Cambridge (1997)

20. Myers, D.: Simulation as Play: A Semiotic Analysis. Simulation
& Gaming, Vol. 30, No. 2, June 1999, 14762 (1999)
21. PenVampyre@aol.com.: Homemade Vintage Cake Recipes.
Last update 24/12/2011, http://www.twisted-candy.
com/1950s-cake-recipes.html (2011)
22. Polygon: Valve refunds gamer uncomfortable with forced
baptism in BioShock Infinite, 16/4/2013, http://www.polygon.
com/2013/4/16/4231064/valve-refunds-baptism-bioshockinfinite (2013)
23. Retter, H.: Einfhrung in die Pdagogik des Spiels, ext. ed.
4. Carroll, L.: Alices Adventures in Wonderland, Macmillan (1865)
Institut fr Allgemeine Pdagogik und Technische Bildung der
5. Courier-Mail: Fruit Pavlova wins 1/1/. 13/10/1954, 11, Bris
Technischen Universitt Braunschweig, Abteilung Historisch-
bane, http://trove.nla.gov.au/ndp/ del/article/50624617 (1954)

6. Dunne, A., Raby, F.: Design Noir: The Secret Life of
Electronic Objects. Birkhuser, Basel/Boston/Berlin (2001)
Systematische Pdagogik, Braunschweig (2003)

24. Rodman, D., Keown, T.: Bad as I wanna be. Delacorte Pr.,
New York (1996)
7. Ghost Volta: Bioshock Infinites Forced Baptism Has Sparked
25. Scheuerl, H.: Das Spiel. Untersuchungen ber sein Wesen,
Controversy and Discussion, http://www.forums.ghostvolta.
seine pdagogischen Mglichkeiten und Grenzen, 4./5. ed.
com/discussion/231/bioshock-infinites-forced-baptism-hassparked-controversy-and-discussion/p1 (2013)
8. Heidegger, M.: Poetry, Language, Thought. Harper&Row,
New York (2001)
9. Huizinga, J.: Homo Ludens. A Study of the Play Element in
Culture. Beacon Press, Boston (1955)
10. Irrational Games Forums: The baptism controversy, http://
Julius Beltz, Weinheim (1965)

26. Salen, K., Zimmerman, E.: Rules of Play. Game Design
Fundamentals. Cambridge, MIT Press (2004)
27. Schrck, P.: Wir spielen immer. Kunstforum International. Vol.
178, November 2005Januar 2006, 4857 (2005)
28. Seelen, G., Rost, C.: Pac-Man & Co. Die Welt der
Computerspiele. Rowohlt, Reinbek bei Hamburg (1984)
forums.2kgames.com/showthread.php?234121-The-baptism-
29. Slash, Bozza, A.: Slash, 6. ed. Harper Collins, London (2008)
controversy (2013)
30. Walther, B.K.: Playing and Gaming. Reflections and
11. Klimpke, K., Behlau, D.: Show me your garage. Custombike

3/2013, 649 (2013)
12. Lewis, M., Staehler, T.: Phenomenology. An Introduction.
Continuum, London (2010)
13. Mail Online: Storm over Call of Duty game that allows
players to massacre civilians, 11/11/2009, http://www.
dailymail.co.uk/news/article-1226588/Call-Duty-Politicalstorm-brutal-video-game-allows-killing-civilians-airportmassacre.html (2009)
14. Manovich, L.: The Language of New Media, www.manovich.
net/LNM/Manovich.pdf (2000)
15. Marks, H.: Seor Nice. Straight Life From Wales to South
Classifications. Game Studies. Int. J. Computer Game

Research, Vol. 3, Issue 1, May 2002, http://www.gamestudies.
org/0301/walther (2002)
31. Willke, H.: Systemtheorie, 2nd ed. Fischer, Stuttgart (1987)
32. Winter, R.: Der produktive Zuschauer. Medienaneignung
als kultureller und thetischer Proze. Quintessenz, MMV
Medizin Verlag, Mnchen (1995)
33. WoWWiki: The Culling, http://www.wowwiki.com/The_
Culling
34. WuShogun212: Baptism Controversy in Bioshock Infinite
Man gets Full Refund, 17/4/2013, http://www.youtube.com/
watch?v=Aa6nssPmQoE (2013)
America. Vintage Books, London (2007)

16. Maturana, H.: The neurophysiology of cognition. In: Garvin,
P. (ed.): Cognition: A multiple view, pp. 324. Spartan, New
York (1970)
203
Kai Kang, Tiantian Yang, Feng Wang

School of Digital Media, Jiangnan Universitiy, China
kk4541437@gmail.com, taliayoung@163.com, water6@126.com
Interactive Art Installation for

Creating Sense of Belonging in
an Working Environment
Abstract
Recently we have witnessed increasing interest in
dynamic art forms in public areas using interactive art
installations. We find that funny games are mostly used
as the form of these design works, which often results
in a mismatch of artifacts and culture.
This paper describes the design process of an inter
active installation in the working environment. This
installation is designed to create sense of belonging for
employees in the Taicang industrial zone based on their
needs and local culture. We attempt to find out the
elements which can influence the form of interactive
art installation through this process to remove the
mismatch.
Keywords
Interaction design, working environment, interactive
installation.
1 Introduction
For centuries, design is always about form and function.
But where is the function when it comes to interaction
design. It is a bit different from many other kinds of
design such as industrial design. After the process of
using an industrial product we can say this work of
design is useful. However, can we say a interaction
device is very useful when we have just experienced it?
204
Maybe not. Interaction design is that kind of effort

which combines process with emotion rather than just
colorful digital animation or games for fun. How can we
say an interaction device is useful and how can designers
make it useful? We want to find the answer from this
workshop by exploring the concept of interactive
patina of culture. Culture is one of the factors which
we believe to make the process of interaction unique
and useful. The traditional dynamic arts have much to
offer and it is time to explore how the theories and
techniques from drama, film and opera could contribute
to interaction design [1].
1.1 Taicang industrial zone
We start the project from the case of the Taicang
industrial zone (Fig 1).Newly developed hi-tech zone
like the Taicang industrial zone is very typical for this
project. It is a hi-tech zone 40 kilometers away from the
center of Shanghai. The Taicang industrial zone which
is set up by the Taicang government is built to attract
hi-tech companies, investors and talents to transform
the structure of local economy, creating more working
opportunities. There are a lot of companies in this
hi-tech zone now including scientific institute, media
companies and communication enterprise. Most of them
are newly set up as branches or department offices. The
Taicang industrial zone is still under construction, the
second wave of establishment is about to finish.
standing talents. Of course, employers hope their staff

fit in with the company culture to be more efficient to
gain greater benefits.
Because of the cooperative relationship between the
Taicang government and universities most of the talents
are fresh graduates who travel every day from home
or rental houses to the working place. According to
previous researches and interviews, most employees are
young people who come from other cities or even other
provinces. They are eager to start a new life in Taicang,
not just a career. Although there are lots of companies
in the zone and they can see many familiar faces every
day, they barely talk to each other. They feel lonely
after work and easy to get tired when they are working.
They hope there can be a comfortable zone in their
working environment for them to release pressure and
communicate with others. They showed great interest
if they can escape for a short voyage or role alternative
during the short break of work.
Fig. 1. The Taicang industrial zone
1.2 Stakeholders
What we want to achieve is to put valuable emotion
into interactive physical interface which we believe is
useful and this emotion may come from local culture.
Although employees in this working environment are
the group we want to design for we hope it is not only
useful for employees but other stakeholders. The
usability may be different to different stakeholders.
There are 3 main stakeholders groups: Taicang govern
ment, companies which has set its department office
or decide to invest in the Taicang industrial zone and
employees following by.
The Taicang government builds this zone as a platform
to attract hi-tech companies to boost the local econo
my to create more working opportunities and attract
more talents into this local community. They want it
to be charming and energetic.
These hi-tech companies which are characterized as
innovation, communication and creating values set their
branches here to look for more opportunities to
cooperate with other companies in this zone. They also
want to absorb professional people in Taicang because
the government has ability and resources to
cooperate with many universities to introduce out
However this is the current situation of the Taicang

industrial zone: the construction of the whole zone is
standardized. The working environment is treated as
a tool not an emotional space. Functional zones like
affiliated gyms are still not set up yet, there is no space
for employees to relieve stress and escape for a while
effectively from heavy work, not to mention make friends.
It is essential for government to build its own profile
and create sense of belonging for these newcomers.
2. Create sense of belonging for employees

2.2 User Requirement Analysis
The need of employees can be classified into two levels
according to Maslows hierarchical theory. A shallow
level of need is characterized as their motivation to
escape from their working place for a short while.
This kind of need is physiological like eat and sleep.
It is very common after a long-term heavy work when
they get physical exhausted. They need some time
or some space to relax. Besides, the contradiction
between their passion for a new life style and monotony
of their current situation results in their need for love
and belonging. This is a higher level of need. The tired
they mentioned to is the combination of physical and
psychological. The two needs are consistent with the
need of government and company essentially. What
designers can do is to build the bridge between the
205
stakeholders to create value. If we design something

which can meet the two needs, it is useful for employers
and the government too.
2.2 Idea development
As designers, we want to create a vision of future
through some smart system which can bring a defined
emotion for the users. The emotion of pleasure can
be created by escaping scenario to follow the users
need. Projection as a medium is a good choice to create
this scenario based on current level of technology,
no matter for practical use or demonstration.
Projector can create a multifunctional zone in working
environment, connecting virtual and real world.
However, this is not enough for higher level needs and
fixed scenario means different to different users. The
function of this system does not come from the images
post by the projector but from the moment when users
do something in this virtual environment. In other
words, designers provide a platform for users to define
the usability of the system.
What is the platform? How does it work? This is what
we need to design. The process has to come back to
the need of belonging. Belonging is a close and intimate
relationship to a community. Riger, LeBailly and Gordon
identified four types of community involvement of
feelings [2]: feeling of bond, extent of residential roots,
use of local facilities, and degree of social interaction
with neighbors. In a sense, working environment is a
special community which is fragmented. There need
to be something to reconstruct it and connect
the people in it to create sense of social cohesion of
working together. The more people interact, the more
likely they are to become close [3]. The more positive
the experience and the relationships, the greater the
bond is. To sum up, we need the virtual environment
created by projector to connect different section of
this real world working environment where employees
can interact with their neighbor. Sense of belonging
can be created after some positive experiences which
make them closer. However, users also have the need to
escape. We hope this connection can be a boundary to
make them feel safe and private. People can step out of
the atmosphere anytime if they want.
How can interaction design serve for this? Different
people have different emotions and as time passes
206
by, emotions of a person also change. How can the

interactive system grow itself to intertwine with? We
have to digger deeper and find answer from patina of
local culture to strike the right chord.
2.3 Design and prototype
We want to use an indirect way of metaphor to
integrate the scenario and meaning which has already
accepted by users who have nurtured in Chinese culture
for a long term. Such embedded scenery arouses
emotion of dened scene, by implanting the meeting
moment at train station into the working environment.
Why train station? It means a little different from many
other countries. For example, in western countries
train is one of many forms of transportation to
commute to their work or for a short trip. Train station
is just like a bus station. However, because of the
geography and history aspects, train station relates to
long farewell and reunion especially in the very moment
of approaching the platform in Chinese culture (Fig 2). It
has mentioned above that the young like many people in
the Taicang industrial zone go far away to start their life
and career in a new city, but they can hardly beat their
homesickness. Although they are passionate for their
new life, train station image is still the scenario they can
accept. The culture is sensitive to be alone and unxed
living settlements.
Fig. 2. The moment of meeting someone on the platform means

a lot to Chinese
Considering the users motivation to go out from their

office when they feel exhausted, we want to put them
into a train which moving among colorful views created
by projector (Fig 3). The train windows which are real
are set in the corridors in the office building (Fig 5).
The corridors are set as a railway carriage and the role
of people as employees are transitioned to passengers.

Images projected are continuous visual scenery which
makes user feel like in a train. The speed of users
passing by is designed to trigger the interaction. We
have to admit the pace of life and work is speeding up.
The employees usually follow this rhythm unconsciously
resulting their feeling of fatigue. We want the system
to detect the speed of users passing by. If people are
moving fast doing their work, the continuous scenery
seen from the window is moving fast and getting
blur, which is not that comfortable for people. When
employees slow down or stand right in front of the
window, the views will slow down to the speed in which
people can enjoy the view. The installation delivers the
message take a rest, go out for a while and relive your
tense nerve so that you can work effectively which is
employer want.
of operator, performer, and spectator at any time.

On many occasions, participants are both operating
and performing, and one is also a spectator of actions
of the others.
The design uses an implicit way of interaction
between office workers as the relationships in the
Chinese culture. The windows connect and pinch the
indifference among people in the building naturally.
Fig. 4. Interaction between the employees on different floors
We made a prototype (Fig 5) of our project with some

technical methods to test the usability of our idea.
Due to time constraints, some functions cannot be
totally achieved like speed detect. We believe these
can be finished if we decide to go further when the
result is ideal.
Fig. 3. People like to enjoy the view from the train window
This is the first interaction and it is certainly not

enough. We just let them go out for a while and
escape. Where is sense of belonging? So we need
the second interaction. We set up train windows in
different floors in the office building. Different floors
are like different train stations which are connected
by a moving train. So the working environment
becomes a community. When there are two people
standing in front of the window at a same time, they
can see the train gradually approach the station and
halt. On the platform, there is someone standing in
the station waiting for her (Fig 4). When they meet
each other, they communicate and interact. This kind
of interaction is person to person which is flexible and
emotional. The participant of an interactive public art
installation is more than a passive user. Participating
in creating [4] and interacting [5] with a public art
installation is about transferring roles among the roles
Fig. 5. Prototype
207
3. Conclusion
From the whole process of design thinking and practice,
we find the form of the work of interaction design still
follows the function, no matter the physical interface,
the structure of scenery or the design of interactive
behavior. The function of interactive design which is
mentioned above cannot be judged just by the process
of use. Whether the emotional needs of stakeholders
can be satisfied by the interactive system is the criteria
for function. The employers hope the staff of their
company to be more efficient, which can be satisfied by
the system even without using it. We can say it is useful
for them. Designers just provide the platform for them,
and the way to realize the function is designed by users.
The platform is based on the research and analysis
of the need. Sometimes we need to study the deep
structure of local culture which nurtures the user group
to make the platform unique and cannot be applied in
other countries. After this process, we conclude that
patina of culture is the key to determine the form and
style of this interactive installation. Of course, more
work needs to be done to identify more of these
elements and organize them into a clear structure.
References
1. Zhang, Y., G. Jing, J. Frens, M. Funk, K. Kang, Q. Dong, Y.
Wang, F. Wang and M. Rauterberg (2013), Learning from
traditional dynamic arts: elements for interaction design, in
Culture and Computing 2013, Kyoto, Japan.
2. David W. McMillan and David M. Chavis, Journal of
Community Psychology Volume 14, January 1986 Sense of
Community: A Denition and Theory.
3. Sherif, M., Harvey, O. J., White, B. J., Hood, W. R., & Sherif,
C. W. (1961), Intergroup Conflict and Cooperation: The
Robbers Cave Experiment.
4. Hu, J., et al., Paticipatory Public Media Arts for Social
Creativity, in Culture and Computing. 2013: Kyoto.
5. Wang, F., J. Hu, and M. Rauterberg, New Carriers, Media and
Forms of Public Digital Arts, in Culture and Computing 2012,
Springer-Verlag Berlin Heidelberg: Hangzhou, China.
208
Dan Gao, Hanqi Zhang, Yawei Yin, Yiyi Zhang, Miaosen Gong
School of Design, Jiangnan University, 1800 Lihu avenue, Wuxi, China
zhanghanqiq@126.com
EVGO: A tour service system

with electrical vehicle
Abstract
Evgo is an actual tour service system based on
connecting different scenic spots among Taihu National
Tourist Resort, aiming at lightening the tourists burden
and optimizing touring experience. The system offers
several services, including booking cars and touring
projects online, navigation on the route, finding
companions, sharing information, etc. By offering these,
the information framework of the scenic spots shall
be enriched by the tourists themselves, and to a large
degree, it not only makes the travel more convenient,
but also increasing visiting people by connecting scenic
spots in series. At the same time, electric vehicle (EV)
also fit the theme of travelling green.
Keywords
Electric vehicle; information platform; tourist resort
docking area and extension of the value chain. China

also built a lot of resorts in many places, but the traffic
problem between places, such as, public transportation
construction lag and private cars parking difficulties
have limited the freedom of visitors, which makes a
bad travelling experience. Lacking of means of trans
portation between scenic spots, while choosing public
transportations will restrict the freedom of travel,
tourists feel really inconvenient during travelling, not
to mention the less information about each spot. So
what if each separate scenic spot has its own vehicles,
tourists can arrange their schedule all by themselves
at the same time getting discount through finding
companions via the platform? Moreover, tourists can
synchronize their smart phones with the platform,
sharing touring moments with friends at any time.
2 User research and concept

1 Social context
Wuxi Taihu National Tourist Resort was established
in 1992 by State Council. Besides Lingshan Resort,
there are Taihu Bright Pearl International Grand Hotel
(in 5-star standard), Sanity Lakeside Resort (4-star),
Moon Bay Resort, Taoyuan Resort, Taihu International
Fishing Center, Jinghu Hotel, Yiming Resort and
Lubowan Resort, etc.
However, the traffic between resorts is blocked,
travelling by car is the only way, which hinders the
After a series of researches on tourists, two important

findings occurred to us. One is that long distance
between scenic spots but no public transportation.
The other one is according to the limited routes offered
by tourist bus, freedom is badly in need. We found
that a single vehicle cant meet the needs of tourists.
A concept of a service system connecting scenic spots
emerged. It provides EV renting and other ancillary
services based on EV digital platform (Xin& Wang,
2010), mobile phone and cloud platform. It solves the
problem of the traffic between different scenic spots,
209
Fig.1. System map of EVGO service
aiming at bringing more freedom to tourists. Tourists

can go to more scenic spots with less money. Finally the
whole tour experience of tourists increases.
3 Solution development
Application: Mobile application is an important part of
the service system, such as the function of payment,
unlock, as well as sharing experiences will give visitors
an unprecedented feeling. Before the start of the trip,
visitors can scan the two-dimensional code on the
vehicle to complete online reservation, payment and
other projects. Then a message will be sent to tourists
phone to inform the electric key number through NGC
technology unlock. The wonderful experience recorded
by phone on the journey can also be synchronized to
the vehicle platform to share.
Cloud platform: The vehicle platform as a cloud
platform features to help visitors get more information
about scenic spots in the form of dynamic information,
offer road navigation, as well as to share experience
with friends. Visitors choose their favorite photos to
share, and the photos can provide references for others
visitors to organize their own travel plans.
Fig. 2. Scenarios of EVGO
digital platform for scenic spots recommendation,

navigation, finding companions and sharing etc.
A PSS as a win-win solution: winning for all the stake
holders. Value is created by taking interaction among
all the stakeholders. In this project, tourists are able to
visit different scenic spots freely. The resort is benefit
from getting more visitors and increasing recognition.
While the service provider can earn money from it.
EVGO PSS opens up a new market for electric vehicles.
On one hand, Electric vehicles are environmentallyfriendly. On the other hand, Electric vehicles cant support
long distance with one-time charge, however, it can
perfectly meet the tourists need within one scenic spot.
5 Acknowledgement
We thank to all colleagues and teachers who give
supports during the projects and its partly supported
by Xin Dayang electric vehicle company. Its partly
supported by Ministry of Education Humanities and
Social Science Youth Fund (11YJC760018).
Reference:
1. Xin Zheng. Development of business models with Internet
of Things[D]. Beijing:Beijing University of Posts and
Telecommunications, 2011. 20-36
4 Discussion and conclusion

A single product cant meet the comprehensive
requirements of users in many cases. So a systematic
perspective for designers is needed. In this project, a
product service system (PSS) (Manzini & Vezzoli 2003,
Morelli 2003) is designed. It consists of a car renting
service for the connection of scenic spots in Wuxi
Taihu National Tourist Resort, an electric vehicles
210
2. Fengying Xin, Haibo Wang. Electric vehicle development

and commercialization prospects Situation Analysis[J].
International Petroleum Economics, 2010, (07): 20-24
3. N. Morelli(2003) Product-service systems, a perspective shift
for designers: A case study: the design of a telecentre. Design
Studies.24(1),74-99
4. Manzini, E. & Vezzoli, C. (2003). Product-Service Systems and
Sustainability. Opportunities for sustainable solutions. UNEP.SS
Joep Frens1, Mathias Funk1, Jun Hu1, Shengxiong Zhang2 , Kai Kang2 , Feng Wang2
1
Department of Industrial Design, Eindhoven University of Technology, The Netherlands
School of Digital Media, Jiangnan University, China
Exploring the Concept of

Interactive Patina of Culture
Abstract
Designing for different cultures is a mandatory discipline
of Industrial Design in our modern, globalized world.
We observe that this has a first world touch most of
the time: industrialized countries designing for less
developed countries, where the design is often only
executed and products are manufactured. This results in
a clear mismatch of artifacts and culture. In this paper,
we present the results of a workshop on designing for
cultures and introduce the concept of the Interactive
Patina of Culture as a different take on this. This more
balanced concept spans from designer and the process
of designing to the resulting designs and products.
The concept is backed by our field exploration during
the workshop where designers of two cultures meet
and jointly realize the Interactive Patina of Culture in
several products.
cultural identity of the user, also known as graceful

ageing, however, this cannot be limited to the designed
artifacts: the process of designing and the designer are
likewise to be included in this notion. That means that
design practice and the designer will over time work
in, with, and for a cultural context.
Recently we organized a student workshop around the
concept of Interactive Patina of Culture. The workshop
was setup in China (Suzhou area) with participants
both from the Netherlands and China. Our aim was to
investigate mechanisms for cooperation on the basis
of equality, by means of bringing students from two
cultures together. The assignment of the workshop was
to design a series of interactive public installations to be
located in the city of Taicang. The concept of Interactive
Patina of Culture acted as a mechanism to bring qualities
of the two cultures together.
Introduction
The concept of Interactive Patina of Culture is a concept
in the space between cultural studies, experience design
and human-computer interaction. It is one way to
understand recent globalization in design, namely not in
terms of cross-cultural design as a one-fits-all formula,
but instead as a set of different designs centered
around a common rationale, but rooted in different
cultural contexts. At the same time, patina conveys
the understanding that time and use of artifacts will
result in ageing, in becoming closer to the person and
In the following, we will give a short introduction on

the background of this concept, how it developed and
how we pursue its further development in a crosscultural setting. Rather than bringing the concept full
circle we see this paper as work-in-progress and
as an instrument to further our understanding the
concept of Interactive Patina of Culture, which will be
instrumental in preparing for the next workshop that
will elaborate on the topic.
211
The concept of Interactive Patina of Culture
The workshop
The concept of interactive patina of culture (IPC) is a

multi-layered concept, which will be explained in the
following by briefly unpacking the elements that make
up the concept. By doing so we move from a rather
metaphorical starting point, as most would anticipate the
now decontextualized word patina, towards a deeper
understanding of it, beyond the literal application:
The workshop took place in Taicang, a city near

Shanghai and had an intentionally multi-cultural
character: we explored new ways to cooperate between
different cultures by bringing students and staff from
different universities and continents together for two
weeks. Thirty-six Chinese students and nine students
from the Netherlands participated and formed nine
project-groups. The project-brief was the following:
A patina is often understood as a trace of repeated use,

as a slight and partial erosion of an objects surface caused
by an agent gradually and over time. The stem of a hammer
acquires a sheen, a polish of repeated use by a carpenter.
This polish tells a story about how an object was used.
Connecting the concept of patina to culture changes
the meaning of it. We posit that culture rubs off onto
the artifacts in use. That is to say the artifacts in a
culture are shaped by the values of that culture; they are
part of the patina of culture. This is best elucidated by
an example: during the workshop in China the authors
of this paper felt intrigued by a particular rolling pin in
the kitchen section of a local supermarket. A few days
later one of the Chinese hosts asked us which variant of
rolling pin we were so interested in, and explained that
different varieties of rolling pins exist that have different
thicknesses and lengths and these properties are
instrumental in the type of dough that is created when
using them. To us this told stories about how the way
that food is enjoyed has influence on how it is made.
The values of a culture are expressed in the artifacts
that it produces.
Connecting interactivity to the concept of patina of
culture extends the concept further, but not in ways
that we could immediately predict, i.e. in terms of
graceful ageing and degradation in interaction design
[4, 5]. In contrast, this was for us the challenge for
exploration. Still, we did not extend the concept
towards interactivity without reason. Bringing inter
activity to the concept of patina of culture gave the
participants of the workshop the opportunity to
explore how they could express the qualities of their
cultures in interactive ways. Thus it extends the concept
of patina of culture beyond the artifact into new
territories where the dynamism of culture is opened
up and explored and where cooperation is both a
mechanism to come to insights and a result.
212
Interactive Patina of Culture: Design an interactive

installation that engages the public in the act of transforming
a nondescript public space into a classy dwelling. It lets the
space meaningfully grow by the interactions with the public.
These interactions range from the intentional to the implicit
behavior. Thus the public is instrumental in growing a
valuable and thus socially meaningful public image of their city.
Below we present three case studies that were
produced in the workshop:
Figure 1. Interactive Fountain
Interactive Fountain. There is a Chinese tale of red

carps jumping over a dragon gate. In the tale, if a red
carp swims hard enough against the stream, it will reach
a dragon gate. If the carp tries hard enough to jump
high out of water, it will jump over the gate and become
a dragon. Over the years this tale have been used as a
metaphor for encouraging people to study to change
their own lives and for endorsing the support for ones
study from the family and the society. There is nothing
more noble than studying (Ancient Chinese saying).
The Interactive Fountain uses back projection to project
animated red carps to a (real, physical) water fountain, see
figure 1. The carps swim at a lower part of the fountain
and they seem to be trying hard to swim to higher
positions in the fountain. One can support a carp

by placing a hand next to it to bring it to a higher level
in the fountain. This goes on until it reaches a certain
height in the fountain where it then jumps over the gate.
The installation refers to the ancient tale of the carp
and engages the public in bringing the carp to higher
levels. It makes the audience aware of the importance
of education, but also of the cyclic nature of generations
following each other and supporting each other.
Figure 3. Moon Rising from Sea
Moon Rising from Sea. Taicang is a natural sea port

and became the number one port of the world in the
Ming dynasty, where Zheng He set off his voyages
to explore the world [1]. Taicangs cultural heritage
is deeply rooted in being a seaport to the world
ever since. Recently Taicang decided to exploit this
past importance and fame. A building is planned that
symbolizes the cultural significance of Taicangs seaport.
This building is roughly 10 by 10 meters on its base and
8 meters high and it features a large sail, and the moon
rising from the waves on its roof.
Figure 2. The Temple Puzzle
Temple Puzzle. Chinese temples are built of wood,

with stone elements. Often these temples burnt down
either by accident or during wars. It has always been
a tradition for people to contribute to the effort of
rebuilding these temples on the ruins. A ruin of an old
temple is located in one of the parks of Taicang, one
of the student groups took this as inspiration for their
interactive installation.
The Temple Puzzle installation uses the ruin of the
temple as a real-world tangible interface for children to
rebuild the temple that once stood there, see figure 2.
The act of jumping on the stones from the ruins of the
temple will reveal the original look of a certain part of
the temple that is related to the stones. The new temple
is projected on a canvas that is located close to the ruin.
The installation uses an existing ruin to reinterpret the
tradition of communally rebuilding the fragile wooden
temples of before in case they burnt. The installation
was meant as a means of playfully educating children
about the significance of the past. Teaching them
something about their culture but also relating their
environment and even their actions to their history.
The installation of Moon Rising from Sea utilize this

building to allow the public to contribute their photos
from social media, for an interactive photo show
projected on the sail, see figure 1. People from Taicang,
who live and work overseas, will be able to send back
photos and show to the people of Taicang what they
are doing and seeing elsewhere in the world. Local
viewers are able to browse through the photos by
turning a steering wheel.
The installation enables both people who have left
Taicang and people who live in Taicang to be in touch
through images and allows for a connection to be
crafted between people that is valuable now but also
significant in light of the cultural heritage of Taicang
as seaport.
In all these examples we found elements that fitted
well with the concept of Interactive Patina of Culture:
(1) the repeated engagement with the fountain and the
cultural connotation in the first installation, (2) the
cultural connotation and the relation between actions
and environment in the second, and the historical
reframing and the aspects of connecting people in a
way that is meaningful on multiple levels for the third.
213
From a more critical perspective we acknowledge that

none of the examples that we present in this short
paper extents the concept of interactive patina of
culture substantially beyond simple metaphor. The case
studies take an existing cultural element and make it
interactive. The richness that is implied in the concept
of IPC in terms of historical perspectives on culture
and meaning created through repeated interaction has
not yet panned out. Also, the concepts are still rather
generic rather than specific to the Chinese culture.
References
We see the value not in the results of the workshop

but primarily in the process towards the results. The
participants learned to value each others perspectives
and skills in the process of designing. What is more,
the workshop taught us how the concept of IPC can be
deepened; it set us up to create a better workshop in
the fall semester.
5. Florins, M., & Vanderdonckt, J. (2004). Graceful degradation
1. Levathes, L., When China ruled the seas. 1994: Oxford

University Press.
2. Trotto, A. Rights Through Making: Skills for Pervasive Ethics,
2011, Unpublished doctoral thesis, TU/e, retrievable from
http://dqi.id.tue.nl/docs/Trotto2011.pdf
3. Sennett, Richard. The craftsman. Yale University Press, 2008.
4. Saridakis, T. (2009). Design Patterns for Graceful
Degradation. In J. Noble & R. Johnson (Eds.), Transactions on
Pattern Languages of Programming I (Vol. 5770, pp. 67-93).
Springer Berlin / Heidelberg.
Conclusion
Looking back at the starting point of the workshop and
the results that it yielded we feel that we can further
elaborate the concept of Interactive Patina of Culture.
The execution of the workshop taught us very valuable
lessons both on our own motives, the group process
and the concept of IPC itself. In retrospect we recognize
in the challenge a mechanism to cooperate between
cultures. In line with Trotto we see the value of making
together [2], of designing together as a mechanism to
generate respect and thus create a fertile ground for
further cooperation. However, we also identify that we
build our workshop on the implicit recognition of the
unique identity of different cultures: the Chinese culture
knows different artifacts and customs than the Dutch
culture does. Although these differences have always been
meaningful to us the present consumption, society and
also design seems not to recognize these values anymore.
True cultural fit with its side-effect diversity stands in the
way of efficient progression and monetization it seems.
Our next step is to take the concept of IPC into the
direction of culturally inspired design for diversity and we
take further inspiration from the traditional artifacts that
are strongly embedded in culture. Here we take cues from
Sennetts concept of craftsmanship [3] and see the IPC as
a value-based approach for cultural design that takes the
cultural rooting of maker, the process of making and the
designed artifact as the major driver of the design process.
Culture is not the goal, but the starting point of design.
214
of user interfaces as a design method for multiplatform

systems. Proceedings of the 9th international conference
on Intelligent User Interface IUI 04, 140-147. ACM Press.
215
216
Acknowledgement
We would like to thank the following reviewers for their valuable

contribution to the DeSForM 2013 conference:
Christoph Bartneck, Tim Bell, Edgar Berdahl, Baptiste Caramiaux,
Lin-Lin Chen, Tom Djajadiningrat, Marco Donnarumma, Loe Feijs,
Simon Fraser, Joep Frens, Mathias Funk, Miaosen Gong, Michael Gurevich,
Abram Hindle, Michael Hohl, Jun Hu, Alexander Refsum Jensenius,
Andrew Johnston, Steven Kyffin, Michael Lyons, Edgar Rodriguez Ramirez,
PHOTO: FENG WANG
PHOTO: JUN HU
Dagmar Steffen, Bill Verplank, Stephan Wensveen
ISBN 978-90-386-3462-3
All rights reserved.

Des Form 2013

Uploaded by

Copyright:

Available Formats

Des Form 2013

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Des Form 2013

Uploaded by

Copyright:

Available Formats

Lin-Lin Chen, Tom Djajadiningrat, Loe Feijs, Simon Fraser, Jun Hu, Steven Kyffin, Dagmar Steffen

Design and semantics

DeSForM 2013 is supported by:

A catalogue record is available from the Eindhoven University of Technology Library.

Design and semantics

DeSForM 2013 Program Committee

Programme DeSForM 2013

Design and semantics of form and movement

Design and semantics of form and movement

Design and semantics of form and movement

DeSForM 2013 Program Committee

Feng Wang (Co-chair)

Loe Feijs (Co-chair)

Interactive Demos Chair

Design Competition Chair

DeSForM 2013 International

Design and semantics of form and movement

Welcome to Wuxi! Welcome to Jiangnan University!

Imaging Laboratory and The Digital Interactive

Wuxi is a beautiful city located in southeast China,

The DeSForM conference is intrigued by the notion that

Emerging from the Department of Animation in the

Design and semantics of form and movement

potentially pivotal effects of computational networks

we assign meaning to the movement of artifacts and our

Another change is continued globalization. The

Prof. David Frohlich, director of Digital World

And still some of the core-questions of DeSForM

Prof. Xiaoyu He, vice president of Nanjing University

Dr. Stephan Wensveen, associate professor

Design and semantics of form and movement

environment. It will emphasize not on the interactive

Design and semantics of form and movement

decision support in a design process, from merely

Special thanks to our local organizers and the

Wuxi, August 18, 2013

Design and semantics of form and movement

Programme DeSForM 2013

Design and semantics of form and movement

Paper session 3: Big data and social design

Design and semantics of form and movement

At a moments notice, according to the pleasure

And so: do Ming Dynasty circulating treasure notes raise important

Barry, K. (2007) The Aesthetics of Paper Money: national differences during

My title quotes the British political economist David Ricardo

Design and semantics of form and movement

Dorsett, C. (2011) Things and Theories: the unstable presence of exhibited

Design and semantics of form and movement

Framing the design of new media

David Frohlich is Director of Digital

Design and semantics of form and movement

Design Research through Practice.

Stephan Wensveen (1970) is Associate

Design and semantics of form and movement

Interaction & Fusion

Design and semantics of form and movement

Deputy Director of Committee of

Intention, not just in the minds

Xin holds a PhD in Design from

Design and semantics of form and movement

Programme DeSForM 2013

A report of this project

For a report of this

For a report of this

For a report of this