Research

In this paper, we describe MusicStory, a system that automatically creates videos to accompany music with lyrics. MusicStory uses common search engines, photo-sharing websites, and simple analysis of the dynamics and tempo of the music to create personalized photo-narratives. Video pacing and content is based on the content of the song and structure of the image repositories selected. The image associations MusicStory presents amplify the emotional experience by externalizing the imagery in song lyrics with the content found within a social network. The resulting work juxtaposes the meanings inherent in the social network with those in the song.

In this workshop, we describe and expose the mysterious creative process. We discuss models (both classical and contemporary) of creative practice and experience and their potential application to new media arts and technology. Models that connect the roles of creator and viewer/participant are of particular interest. The goal is to catalyze new ideas and foster creative collaboration across disciplines.

We investigate the practice of sharing short messages (microblogging) around live media events. Our focus is on Twitter and its usage during the 2008 Presidential Debates. We find that analysis of Twitter usage patterns around this media event can yield significant insights into the semantic structure and content of the media object. Specifically, we find that the level of Twitter activity serves as a predictor of changes in topics in the media event. Further we find that conversational cues can identify the key players in the media object and that the content of the Twitter posts can somewhat reflect the topics of discussion in the media object, but are mostly evaluative, in that they express the poster's reaction to the media.

Television broadcasters are beginning to combine social micro-blogging systems such as Twitter with television to create social video experiences around events. We looked at one such event, the first U.S. presidential debate in 2008, in conjunction with aggregated ratings of message sentiment from Twitter. We begin to develop an analytical methodology and visual representations that could help a journalist or public affairs person better understand the temporal dynamics of sentiment in reaction to the debate video. We demonstrate visuals and metrics that can be used to detect sentiment pulse, anomalies in that pulse, and indications of controversial topics that can be used to inform the design of visual analytic systems for social media events.

Going to movies in a group and inviting friends over to watch TV are common social activities. This social engagement both improves the viewing experience and helps us stay close with our friends and family. To bring this feeling of co-presence to the Internet, we developed a set of prototypes that enable people to feel more connected by watching web video together in sync. We present the preliminary results of a quantitative usage study and show initial evidence that simultaneous video sharing online can help people feel closer and more connected to their friends and family.

In today's fast-paced world, it is increasingly difficult to understand and act promptly upon the content of the many information streams available. Temporal pooling addresses this problem by producing a visual summary of recent stream content. This summary is often in motion to incorporate newly arrived content and to stay up-to-date. We therefore review the perception of motion and change blindness, offering several guidelines for the use of motion in visualization. We then describe TextPool, a tool for visualizing live text streams such as newswires, blogs and closed captioning. TextPool's temporal pooling summarization is a dynamic textual collage that clusters related terms. We tested TextPool with the content of several RSS newswire feeds, streaming 800 stories per day. TextPool was able to handle this bandwidth well, producing useful summarizations of stream content. We then examine TextPool in the light of our guidelines, and speculate on other possible implementations of temporal pooling.

In this article, we explore the structure of the web as an indicator of popular culture and its use in Multimedia Exhibits. In a series of art and technology installations, the software agency needs to keep 'grounded' to what people can readily understand. We administered a survey to understand how people perceived word and phrase obscurity related with frequency information gathered from a popular Web search engine. We found the frequency data gathered from the engine closely matched judgments gathered from people. The results of this study point to the new applications of the WWW in art and multimedia exhibits as an indicator of popular culture.

In today's fast-paced world, it is becoming increasingly difficult to understand and act promptly upon the content of the many information streams available to us. TextPool addresses this problem by quickly summarizing recent content in live text streams, such as newswires and closed captioning. The summarization is a dynamically changing textual collage that clusters related terms. We tested TextPool with the content of several RSS newswire feeds, which are updated roughly every five minutes. TextPool was able to handle this bandwidth well, and produced useful summarizations of feed content.

In this article, we explore the role of emotion in digital theater. Unlike heavily scripted productions, we create a theatrical agency which uses the world wide web to determine what content to deliver and how to deliver it with emotional competency. We describe the use of the web as a cultural artifact and how an autonomous agency can externalize these online connections during a performance. We address the problems that arise in bringing affect to an automated theatrical experience, and present our solution.

We explore applications for enriching experiences around live visual media events by leveraging conversational activity on short messaging services. We investigate the application of existing methods to discover the structure and content of media events and develop methods for exposing the discovered informational cues to users. We demonstrate these approaches using video footage and Twitter activity during two broadly watched media events: the first 2008 USA Presidential Debate and the Inauguration of Barack Obama. For the debates, we demonstrate a method for segmenting and annotating the media via conversational activity for the purpose of watching the video after the event has already passed. For the inauguration, we demonstrate approaches for exposing an awareness of the current topics of discussion on Twitter and the apparent levels of interest via a real-time feedback display. The primary contribution of this CSCW Horizons note is an initial exploration of approaches for applying cues mined from community conversation towards enhanced experiences around multimedia events and an invitation for a discussion of further approaches for applying these techniques to a growing number of domains and applications.

This paper describes a digital theater installation called Buzz. Buzz consists of virtual actors who express the collective voice generated by weblogs (blogs). These actors find compelling stories from blogs and perform them. In this paper, we explore what it means for a story to be compelling and describe a set of techniques for retrieving compelling stories. We also outline an architecture for high level direction of a performance using Adaptive Retrieval Charts (ARCs), allowing a director-level of interaction with the performance system. Our overall goal in this work is to build a model of human behavior on a new foundation of query formation, information retrieval and filtering.

We have developed a user-trainable query by humming (QBH) system that develops an error probability model of a user's singing. While the training is effective, it is also tedious and time consuming, requiring the user to sing dozens of melodies to the system before the system can be trained. To make training fun, we introduce a new interactive, distributed karaoke game, called Karaoke Callout, played over a cell phone. The user selects a song and sings it into the cell phone. The audio is sent to a server which rates the quality of the singing by measuring how closely it resembles a canonical example of the song stored in the server database, sending a score back to the user. The user may then challenge anyone in the phone's contact list. An SMS text challenge is sent to the challenged person's cell phone. The challenged person sings the song, attempting to better the performance of the challenger. This challenge may then be repeated, with either party selecting a new song with which to "call out" the other party. Over the course of an interaction, numerous examples of each party's singing are created and stored. These may then be used to train a QBH to the idiosyncrasies of each user's singing, as well as providing new query targets for the system.

Digital photomontage Image segmentation Common-sense reasoning a b s t r a c t Detecting manipulated images has become an important problem in many domains (including medical imaging, forensics, journalism and scientific publication) largely due to the recent success of image synthesis techniques and the accessibility of image editing software. Many previous signal-processing techniques are concerned about finding forgery through simple transformation (e.g. resizing, rotating, or scaling), yet little attention is given to examining the semantic content of an image, which is the main issue in recent image forgeries. Here, we present a complete workflow for finding the anomalies within images by combining the methods known in computer graphics and artificial intelligence.

In this article, we present the Association Engine, a multimedia installation that explores the space of language and exposes connections between words. It externalizes meaningful associations to remind the viewer of forgotten connections but also introduce her to new ones. It accomplishes this by performing a series of improvisational games. It stands out from previous efforts in the realm of computers and theater as the agency has been empowered with a cultural understanding, more specifically, it uses the Web as a source of cultural reality.