disability, design, art, and technology by Florian Grond
This essay is an account of Remote Feelings, a collaborative art project conceived of by the auth... more This essay is an account of Remote Feelings, a collaborative art project conceived of by the authors in 2016. Shaped by sensory difference, remote location and the use of digital technology, namely 3D scanning and printing technology, this project, like this essay, explores haptic experience, the pairing of analogue and digital methods, and the work and influence of Austrian artist and innovator Raoul Hausmann (1886-1971). The essay also shares information on our individual artistic practices and preoccupations, including Florian Grond's use of digital information as a starting point for the exploration of sensory experience and David Johnson's use of his blindness to inform his search for a new aesthetics.
Abstract—The process of designing affective technology as an assistive device needs to take ethic... more Abstract—The process of designing affective technology as an assistive device needs to take ethical questions into account. Although biomusic, an affective technology, has been used effectively as an assistive communication device in the context of healthcare, its use with persons with autism in a broader context poses complex challenges. In order to understand and respond to these challenges, a 3- day workshop was organized in Montreal to gather information from stakeholders, i.e. users on the spectrum, family members and persons who work with them in educational, work, and cultural settings, on the potential uses and ethical issues of biomusic. In this paper, we report some of the outcomes of this workshop from a design perspective, which we used to generate a framework for biomusic as affective technology. This framework proposes three distinct lenses: a technological one, an ecological one, and a human-centered one. We illustrate how this framework can make visible the ethical issues that can emerge during the design process and promote collaborative and concrete solutions that respond to user concerns.
In this article, we present the concept of sonic boundary objects by discussing
the case of binau... more In this article, we present the concept of sonic boundary objects by discussing
the case of binaural sound recordings as a tool, which helps to negotiate
between blind and sighted experiences of an urban soundscape. We begin
with developing the theoretical underpinning of the concept of a sonic
boundary object by briefly elaborating on the relevance of the concept of
the boundary object in the context of disability and technology; we then
explain the special properties of binaural recordings and revisit the
musicological concept of the sound object, whose relationship with
technology partly inspired the work in this article. After introducing these
concepts and related practices we proceed by proposing the term of a sonic
boundary object. We discuss the relationship of the sonic boundary object with
the concept of simulation in the context of disability studies and we then
elaborate on its potential impact as a complementary practice to blind
ethnography and to participatory design approaches. We will demonstrate its
ethnographic potential with an account of the authors’ experience using
binaural recordings to exchange on what blind navigation is like in Montreal.
The article concludes with a first-person reflection on this experience and
possible fields of application.
We present in this paper SonicFunction, a prototype for the interactive sonification of mathemati... more We present in this paper SonicFunction, a prototype for the interactive sonification of mathematical functions. Since many approaches to represent mathematical functions as auditory graphs exist already, we introduce in SonicFunction three new aspects related to sound design. Firstly, SonicFunction features a hybrid approach of discrete and continuous sonification of the function values f(x). Secondly, the sonification includes information about the derivative of the function. Thirdly, SonicFunction includes information about the sign of the function value f(x) within the timbre of the sonification and leaves the auditory graph context free for an acoustic representation of the bounding box. We discuss SonicFunction within the context of existing function sonifications, and report the results from an evaluation of the program with 14 partially sighted and blind students.
Journal on Multimodal User Interfaces, 2011
... Dalia El-Shimy · Florian Grond · Adriana Olmos · Jeremy R. Cooperstock ... Capitalizing on th... more ... Dalia El-Shimy · Florian Grond · Adriana Olmos · Jeremy R. Cooperstock ... Capitalizing on the innate human ability to localize sound [1] may of-fer an effective substitute to the visual modality as a repre-sentation of the user's surroundings. ...
We present in this paper SingingFunction, a vowel-based sonification strategy for mathematical fu... more We present in this paper SingingFunction, a vowel-based sonification strategy for mathematical functions. Within the research field of auditory graphs as representation of scalar functions, we focus in SingingFunction on important aspects of sound design, which allow to better distinguish function shapes as auditory gestalts. SingingFunction features the first vowel-based synthesis for function sonification, and allows for a seamless integration of higher derivatives of the function into a single sound stream. We present further the results of a psycho physical experiment, where we compare the effectiveness of function sonifications based on either mapping only $f'(x)$, or including hirarchically further information about the first derivatives $f'(x)$, or the second derivative $f''(x)$. Further we look at interactivity as an important factor and report interesting effects across all 3 sonification methods by comparing interactive explorations versus simple playback of sonified functions. Finally, we discuss SingingFunction within the context of existing function sonifications, and possible evaluation methods.
multimodal and auditory display by Florian Grond
We present in this paper SingingFunction, a vowel-based sonification strategy for mathematical fu... more We present in this paper SingingFunction, a vowel-based sonification strategy for mathematical functions. Within the research field of auditory graphs as representation of scalar functions, we focus in SingingFunction on important aspects of sound design, which allow to better distinguish function shapes as auditory gestalts. SingingFunction features the first vowel-based synthesis for function sonification, and allows for a seamless integration of higher derivatives of the function into a single sound stream. We present further the results of a psycho physical experiment, where we compare the effectiveness of function sonifications based on either mapping only f′(x), or including hierarchically further information about the first derivatives f′(x), or the second derivative f″(x). Further we look at interactivity as an important factor and report interesting effects across all 3 sonification methods by comparing interactive explorations versus simple playback of sonified functions. Finally, we discuss SingingFunction within the context of existing function sonifications, and possible evaluation methods.
F. Grond and J. Berger, “Parameter mapping sonification,” in The Sonification Handbook (T. Hermann, A. Hunt, and J. Neuhoff, eds.), ch. 15, pp. 363–397, Berlin, Germany: Logos Publishing House, 2011. This chapter introduces the benefits and challenges of parameter mapping sonification. The chapte... more This chapter introduces the benefits and challenges of parameter mapping sonification. The chapter explores features of data and their preparation for sonification; synthesis parameters, both in the signal and perceptual domains, and methods of creating mapping functions to bridge between data and sound synthesis.
Abstract: This case study introduces interactive sonification to evolutionary strategies (ES) for... more Abstract: This case study introduces interactive sonification to evolutionary strategies (ES) for global optimization. We briefly describe the specific strengths of sonification as a tool for monitoring, the emerging trend of interactive sonification, and what it can add to the field of ...
The desire to make data accessible through the sense of listening has led to ongoing research in ... more The desire to make data accessible through the sense of listening has led to ongoing research in the fields of sonification and auditory display since the early 1990s. Coming from the disciplines of computer sciences and human computer interface (HCI), the conceptualisation of sonification has been mostly driven by application areas and methods. On the other hand, the sonic arts, which have always participated in the auditory display community, have a genuine focus on sound. Despite these close interdisciplinary relationships between communities of sound practitioners, a rich and sound- or listening-centred concept of sonification is still missing for design guidelines. Complementary to the useful organisation by fields of application, a proper conceptual framework for sound needs to be abstracted from applications and also to some degree from tasks, as both are not directly related to sound. As an initial approach to recasting the thinking about sonification, we propose a conceptualisation of sonifications along two poles in which sound serves either a normative or a descriptive purpose. According to these two poles, design guidelines can be developed proper to display purposes and listening modes.
Abstract: In this paper, we present building blocks for the synthesis of vowel sounds in the prog... more Abstract: In this paper, we present building blocks for the synthesis of vowel sounds in the programming language SuperCollider. We discuss the advantages of using vowel based synthesis, and make a review where it has been used in sonifications already. Then we describe in detail the main class Vowel which handles all parameters related to the formants that are typically used for vowel synthesis.
Gesture in Embodied Communication and Human-Computer Interaction, 2010
Through advanced recording and simulation possibilities the amount of 3D movement data is constan... more Through advanced recording and simulation possibilities the amount of 3D movement data is constantly growing. This data type stems from different origins ranging from sports, HCI and music research (body movements and gestures). Although these data sources look at first glance very disparate, they are in fact structurally very similar since they represent dynamic processes in 3D space with many degrees of freedom. The standard technique to investigate alike data is the scientific 3D visualization of moving points or models. However, the rather young research field of sonification offers novel inspection techniques that complement visual analysis by transforming the data into audible sound so that particularly dynamic patterns can be understood by listening. This is beneficial for several reasons: Firstly, if properly used, sonification is ideal for representing multivariate data sets with temporally complex information such as fast transient motions. Secondly, in many applications the eyes are already occupied with a specific task and have therefore limited capacities to focus on additional information thereby limiting the cognitive load of the operator. Thirdly, sound requires neither a particular orientation of the user nor directed attention. In this paper we develop and test two sonification methods for 3D movement data. Specific sonifications for this purpose have already been developed to some extend by the authors [3] [7] and others [2] but there is still a lack of generality as far as methods are concerned. In contrast, we aim towards more generic methods for this data type and therefore we use in our sonification designs as little a priori knowledge as possible. Furthermore we conduct psychophysical experiments with these methods using concrete data samples together with meaningful visual stimuli in order to assess their efficiency. For that purpose we chose to sonify the ancillary gestures of clarinetists. Ancillary gestures are those body movements which are not directly involved in the sound production [1] . For clarinet players, lip and finger motions are effective gestures, whereas motions like weight transfer and body curvature for instance are ancillary gestures. Their importance is due to the fact that they tend to align with musical motives in the score [9] and are therefore an integral part of the player's performance as these movement patterns show consistency even for various levels of expressiveness . For instrumentalists, sonifications have potential benefits during instructions and practicing. Our primary goal in this paper
In this paper, we introduce space filling curves (SFC) as a useful possibility to organize data f... more In this paper, we introduce space filling curves (SFC) as a useful possibility to organize data for sonification. First, we give a brief overview about the history of SFCs and their graphical construction. Then we focus on the mapping properties of SFCs from 2D to one dimension. We present the acoustic results of an implementation of the described method, in which we took the Hilbert curve as one particular example of an SFC. The actual sonification program features different methods for real-time interaction. These methods take advantage of the particular properties of SFCs. We further discuss their restrictions, how they can be circumvented, and give an outlook to future applications, where we also make suggestions as to how the properties of SFCs can be combined with methods of data reduction.
In this paper we present SUMO, an open source software environment, which is designed to facilita... more In this paper we present SUMO, an open source software environment, which is designed to facilitate the open development of molecular sonifications for everyday research in chemistry and structural biology. Sonifications of chemical data are developed since more than 25 years but surprisingly auditory display is not yet a scientifically established mode to interact and explore molecular data. Before presenting SUMO we introduce the implications of presenting molecular data to the sonification community. For chemists and structural biologists, we briefly review different sonification approaches made so far and discus their potential. Within this broader scope we situate SUMO, the lab proof sonification framework. We describe the software environment in detail and present two implementations of methods for sonifying conformations of amino acids and B factors.
Abstract. Through advanced recording and simulation possibilities the amount of 3D movement data ... more Abstract. Through advanced recording and simulation possibilities the amount of 3D movement data is constantly growing. This data type stems from different origins ranging from sports, HCI and music research (body movements and gestures). Although these data sources look at ...
Uploads
disability, design, art, and technology by Florian Grond
the case of binaural sound recordings as a tool, which helps to negotiate
between blind and sighted experiences of an urban soundscape. We begin
with developing the theoretical underpinning of the concept of a sonic
boundary object by briefly elaborating on the relevance of the concept of
the boundary object in the context of disability and technology; we then
explain the special properties of binaural recordings and revisit the
musicological concept of the sound object, whose relationship with
technology partly inspired the work in this article. After introducing these
concepts and related practices we proceed by proposing the term of a sonic
boundary object. We discuss the relationship of the sonic boundary object with
the concept of simulation in the context of disability studies and we then
elaborate on its potential impact as a complementary practice to blind
ethnography and to participatory design approaches. We will demonstrate its
ethnographic potential with an account of the authors’ experience using
binaural recordings to exchange on what blind navigation is like in Montreal.
The article concludes with a first-person reflection on this experience and
possible fields of application.
multimodal and auditory display by Florian Grond
the case of binaural sound recordings as a tool, which helps to negotiate
between blind and sighted experiences of an urban soundscape. We begin
with developing the theoretical underpinning of the concept of a sonic
boundary object by briefly elaborating on the relevance of the concept of
the boundary object in the context of disability and technology; we then
explain the special properties of binaural recordings and revisit the
musicological concept of the sound object, whose relationship with
technology partly inspired the work in this article. After introducing these
concepts and related practices we proceed by proposing the term of a sonic
boundary object. We discuss the relationship of the sonic boundary object with
the concept of simulation in the context of disability studies and we then
elaborate on its potential impact as a complementary practice to blind
ethnography and to participatory design approaches. We will demonstrate its
ethnographic potential with an account of the authors’ experience using
binaural recordings to exchange on what blind navigation is like in Montreal.
The article concludes with a first-person reflection on this experience and
possible fields of application.