To appear in CAST01, Bonn, Germany, September 21-22, 2001.

Alices Adventures in New Media: An Exploration of Interactive Narratives in Augmented Reality

Emmanuel Moreno Motive Design motiv-8@mindspring.com Blair MacIntyre College of Computing blair@cc.gatech.edu Jay David Bolter School of Literature, Communication and Culture jay.bolter@lcc.gatech.edu GVU Center Georgia Institute of Technology Atlanta, GA 30332
Abstract: Alices Adventures in New Media is an Augmented Reality (AR) experience based on A Mad Tea Party, a chapter from Lewis Carrolls book Alices Adventures in Wonderland. The user assumes the role of Alice and sits at the tea party with three interactive characters: the Mad Hatter, the Dormouse, and the March Hare. The video-based characters are presented through a head-mounted display, and appear to be seated at the same physical table as the user. As a developing medium, AR has yet to establish itself as a narrative form. By comparing the unique characteristics of AR with established media, the project explores AR as a storytelling medium. Innovations include the refashioning of older media (such as film) for the development of an AR narrative, the use of simple procedural characters to create an immersive interactive experience, and the development of tools that enable the production of AR experiences. Keywords: Augmented Reality (AR), video actors, video texture mapping, digital video, interactive narrative, procedural interactive characters, Macromedia Director, remediation Project URL: http://www.cc.gatech.edu/projects/ael/armedia/alice.html Year the Work was created: 2001 Project Partners: Graphics, Visualization and Usability Center (GVU) The Wesley Center for New Media

Introduction
One important aspect of living with mixed reality is to show how the techniques of mixed reality can define new media forms for artistic expression and for entertainment. Such forms will help to convince our society of the cultural and even commercial value of the new technologies of mixed reality. For this reason, our work focuses on creating the formal conventions and the technology to support dramatic and narrative experiences in augmented reality (AR). AR combines the physical world with virtual elements. Typically, the user wears a head-mounted display (HMD) that mixes the view of the physical environment with computer-generated elements through the use of semi-transparent HMDs or opaque videomixed HMDs (real-time video of the physical environment is mixed with virtual elements and displayed on an opaque HMD) [7]. Often a new medium such as AR develops from the work of technical innovators. Initial research focuses on the mechanics of the technology, while issues of effective use of the technology as a medium are often secondary. The new medium may enjoy some initial success as an entertainment form based completely on the novelty of the technology. In contrast, established media rarely depend solely on technology to provide a gratifying experience. The development of an experience for an established medium is more often a synergy of technical mechanics and storytelling; narrative conventions are accepted and understood by the audience culture, while production tools and methods are in place to support the creation of experiences. A new medium faces the challenges of

technical innovation as well as the development of narrative conventions.

Aims of Research
Our research has three main goals. The first goal is to borrow and refashion a sense of authenticity from one or more earlier media, such as film and interactive CDROMS. We are drawing here on the theory of remediation by Bolter and Grusin [1]. Remediation is important because it promotes acceptance and understanding of AR by showing how it relates to earlier and established media. The second goal is to "surpass" the earlier media in some way in this case by exploring interaction techniques to which AR is particularly well suited, namely interaction between virtual and physical elements in the user's environment. Finally, we are developing tools that enable both artists and technologists to work, experiment and collaborate in AR as a new interactive narrative form.

Figure 1. The Mad Hatter (at the users left) has just been splashed with tea.

1. Project Description
The experience is based on A Mad Tea Party, a chapter from Lewis Carrolls Alices Adventures in Wonderland [8]. The user assumes the role of Alice and sits at the tea party with three interactive characters: the Mad Hatter, the Dormouse, and the March Hare. The characters are computer-controlled video actors displayed in the users HMD and appear to be sitting at the same physical table as the user (we describe video actors in [9]). The characters can interact with the user and with each other. The users objective is to get directions to the garden, located somewhere in Wonderland. The characters view the user as an interruption to the party already in progress and continue about their own business. They are frequently reluctant to acknowledge the user and often ignore the user altogether the user discovers that she cannot simply ask for directions and must participate in the tea party. Each character has a set of primitive actions that they can perform, including serving tea, receiving tea, sipping tea, asking riddles, and various reactions to events that may occur in the story environment. If properly provoked, a character may splash the user (or another character) with tea. Procedural behaviors govern how the character will react to events that occur in the environment (instigated by the user or by other characters). An example is shown in Figures 1-3. In Figure 1, the user splashes the Mad Hatter with tea. The March Hare reacts with laughter in Figure 2. Finally the sleepy Dormouse is awakened by all the noise (Figure 3). The user also has a range of gestures for virtually serving, receiving, sipping and throwing tea. The user can also address a character through simple audio level sensing. The characters have procedural behaviors that govern how each character acts or reacts to the user or other characters in the scene. Each action represents a primitive story element the progression of these elements builds the overall narrative experience.

Figure 2. The March Hare (at the users right) reacts with laughter.

Figure 3. The Dormouse (opposite the user) will soon awaken from the noise.

1.1 Relationships to older media

Alices Adventures in New Media refashions the conventional film screenplay [4, 6] as a formulaic basis for objective-based procedural characters, following proposals made by Janet Murray [10]. Murrays procedural concepts are combined with the interactive narrative techniques of conventional CD-ROMs [2, 11] to develop the idea of the procedural narrative node.

1.2 Unique characteristics of AR

The spatial characteristics of AR are unique compared to interactive CD-ROMs, web sites and Virtual Reality (VR). While other digital forms immerse the user by presenting an alternative virtual world, AR integrates the story world into the physical world in real time. A conventional CD-ROM might pause while waiting for user interaction. The pause reveals story time

disjointed from real time, frozen while the user contemplates a choice. In VR, the environment can change in an instant, transporting the participant from one place to another. Users have grown accustomed to this convention. Many VR environments strive to be fantasy-like rather than simulations of reality [3]. In contrast, most virtual objects and characters in AR are world-stabilized to the physical realm, a world that is not completely replaced by the machine and continues whether the machine is on or off. The medium depends on a delicate tension between the virtual and the physical to immerse the user in the story. If the video actors in Alice have to freeze and reload new video segments each time the user interacts, this discontinuity would disrupt the immersive experience for the user. Alice is an experience that requires story time to be insync with real time. Alice strives to create an environment responsive to the user, as well as the illusion that the story world exists whether or not the user takes action. While these goals may seem contradictory at first, they are complimentary they each help to synchronize story time with real time. The interactivity and spatial characteristics also distinguish AR from film. Film has the advantage of a well-defined form of linear writing embodied in the formulaic screenplay. A non-linear equivalent is needed to enable widespread production of AR experiences.

Procedural authorship has roots in the oral bard storytelling tradition of ancient Greece. The bardic tradition worked by using a formulaic system to substitute basic story elements or phrases to construct a coherent narrative [10]. In a procedurally authored story, the author creates basic building blocks, or primitives, that can be arranged differently to construct a coherent story. In Murrays model, the primitives are the basic actions or gestures of the user as structured by the author. The computer as story-presenter responds to users gestures first by capturing and analyzing the gestures, then by applying procedural rules to determine the appropriate story element to present [10]. Rather than producing several varied linear scenes as in a cul-de-sac, this project focuses on developing primitive story elements attached to the basic scripted actions of the user and the interactive characters. The actions and the corresponding story primitives fit within the framework of a linear narrative spine. The users actions vary the arrangements of story primitives and influence the actions of the other characters.

1.5 Scripting the Interactor

The use of a familiar story is our first strategy for scripting the interactor. It is reasonable to expect users (at least in Anglo-American culture) to be at least vaguely familiar with Carrolls A Mad Tea Party. Even if the user is not familiar with the story, a tea party is not difficult to understand. The teacup and teapot are seen physically on the table and virtually within the story-world. The coexistence is a cue to the user that the objects have significance. As the user witnesses other characters using teacups and teapots, the significance is further revealed. The characters wake up the Dormouse or address and answer each other, subtly encouraging the user to participate in the same manner. Finally, character actions can discourage inappropriate user behavior. If the Hatter is asking the user a riddle, the Hares attention is turned towards the Hatter, cueing the user to look as well. Additionally, if the user tries to address the Hare during the Hatters riddle, the user is ignored.

1.3 The interactive cul-de-sac

A common interactive technique described by Samsel and Wimberly is the interactive cul-de-sac [11], or userchoice node, located on a linear narrative spine. The node consists of user options and pre-written optional scenes. The user makes a choice, and the corresponding scene option is presented. The users choice simply leads to a pre-determined end condition, keeping the linear narrative intact. The users actions simply alter the presentation of the story. The use of a cul-de-sac poses technical problems for AR. This technique assumes the user is typically passive, only taking an active role when presented a choice at a cul-de-sac. After choosing, the user typically experiences the resulting scene through an avatar that represents the user-as-character in the story-world. The disjointed real-world/story-world (separated by the computer screen) of CD-ROMs and web-based interactive narratives tolerates and perhaps requires this convention. When users assume roles in AR, typically their own physical bodies become the avatars for users-as-characters in the story-world. The author does not have direct control of the users bodies. The users must actively perform in character to move the story forward. The author must have a strategy for scripting the user on a level more basic than choosing occasional optional pathways or varied scenes. The users actions must be scripted (predetermined and encouraged), evaluated and used in a way the cul-de-sac cannot support.

1.6 The illusion of independent characters

In order to create the illusion of a story world independent of the user, the experience must appear to continue whether or not the user takes action. If the characters are solely reactive to the user, the story will pause when the user is inactive, again disjointing story time from real time. Alice incorporates characters with their own objectives, and procedures to pursue their objectives. If the user is inactive, events still occur in the story world; however the user can still interact at any time without intruding on the story. Alice borrows from film conventions to define character objectives within a narrative structure. Dan Deckers model of the character structure and objective drive in the American film is used as a formulaic basis for describing the procedural behaviors used by the characters in pursuit of their objective or in reaction to other characters and story events [4]. Note that these behaviors are simple procedures to address varying property states and fall short of artificial intelligence.

1.4 The procedural node

The procedural node progresses from the cul-de-sac towards what Murray calls procedural authorship.

The intent is to create the illusion of independent character action, not to create truly intelligent agents.

2. Tools and technology

In order to promote the development of AR as a narrative form, we believe it is vital to engage the creative talents of filmmakers, stage producers, and new media designers. We are working to make our development system easy to use, at least for those with some background in digital video and multimedia design. It is unrealistic to expect such designers to familiarize themselves with OpenGL or C++ programming in order to create AR experiences. For that reason we are focusing on the use of Macromedia Director, perhaps the most commonly used tool for digital presentation and interaction. We are currently building native plug-ins (Xtras) to turn Director into a full-fledged AR prototyping environment for optical and video-mixed AR, including support for tracking, spatialized sound, gesture recognition, and so on. Director Lingo is used to describe procedural character behaviors, control the display of character actions and manage property states. For example, if the Mad Hatter needs to splash tea on the Dormouse, Lingo is used to load the appropriate video segments to show the Hatter turning towards the Dormouse, then splashing him with tea. In response, video clips of the Dormouse reacting and the March Hare laughing are loaded. When the action is completed, the properties for each character are updated, and new character states are set. Director 8.5 features built-in support for 3D sprites. 3D sprites combined with position tracking and orientation are used to place the characters in 3D space. Audio level sensing for Alice is achieved using the GetSoundInLevel (GSIL) Xtra [13].

gesture recognition and only a limited set of procedural behaviors has been completed. Work continues with full implementation as a goal. Additionally, our use of bitmap sequences extracted from video to create character actions allows us to integrate novel video playback techniques. In particular, we are attempting to address looping video artifacts using Video Textures [12], a technique that can be used to generate infinitely long, non-repetitious video sequences, potentially providing more natural-looking transitions between character actions and idle states.

References
[1] Bolter, Jay David, and Richard Grusin. Remediation: Understanding New Media. Cambridge: MIT Press, 1999. [2] Bonime, Andrew and Ken C. Pohlmann. Writing for New Media: The Essential Guide to Writing for Interactive Media, CD-ROMs, and the Web. New York: John Wiley & Sons, 1998. [3] Craven, Mike, Ian Taylor, Adam Drozd, Jim Purbrick, Chris Greenhalgh, Steve Benford, Mike Fraser, John Bowers, Kai-Mikael J-Aro, Bernd Lintermann, Michael Hoch. Exploiting Interactivity, Influence, Space and Time to Explore non-Linear Drama in Virtual Worlds. In Proc. CHI 2001, Seattle, WA, March 31 April 5, 2001. [4] Decker, Dan. Anatomy of a Screenplay: Writing the American Screenplay from Character Structure to Convergence. Chicago: The Screenwriters Group, 1998. [5] Feiner, Steven, Blair MacIntyre, and Tobias Hllerer. Wearing It Out: First Steps Toward Mobile Augmented Reality Systems. In Proc. International Symposium on Mixed Reality, pp.363377, Yokohama, Japan, March 9-11, 1999. [6] Field, Syd. Screenplay: The Foundations of Screenwriting. New York: Dell Pub, 1982. [7] Fuchs, Henry and Jeremy Ackerman. Displays for Augmented Reality: Historical Remarks and Future Prospects. In Proc. International Symposium on Mixed Reality, pp.31-41, Yokohama, Japan, March 9-11, 1999. [8] Gardner, Martin ed. The Annotated Alice: The Definitive Edition. New York: W. W. Norton and Company, 2000. [9] MacIntyre, B., Lohse, M., Bolter, J.D., and Moreno, E. (2001) Ghosts in the Machine: Integrating 2D Video Actors into a 3D AR System. In Proc. International Symposium on Mixed Reality, Yokohama, Japan, March 1415, 2001. [10] Murray, Janet. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. New York: Simon and Schuster, 1997. [11] Samsel, Jon and Darryl Wimberley. Writing for Interactive Media: The Complete Guide. New York: Allworth P, 1998. [12] Schdl, A., R. Szeliski, D. Salesin, and I. Essa. Video Textures, In Proc. ACM SIGGRAPH 2000, pp.489-498, New Orleans, LA, January 2000 [13] Smith, Geoff. http://PhysicalBits.com/Xtras. June 2, 2000.

3. Conclusions and future directions

. Alice's Adventures in New Media demonstrates that AR can be used to create short, relatively simple narrative experiences based on procedural characters. The techniques demonstrated here did have communicative power. Most users could understand that the use of the teacup represented a significant gesture (among others). Alices Adventures in New Media was demonstrated at Georgia Techs New Media Center in April 2001. One particular young girl persuaded her mother to participate. Before the mother could begin, the young girl explained, Be polite to everyone and say hi to them, before you splash them with tea because then they get really mad! The emotional loops of the characters easily recognizable even by a young child fall short of the independent story world goal. However, the fact that the child recognized scripted actions and story primitives (splashing tea modifies the story presentation), and that she so easily humanized the characters as people to be greeted, to be polite to, and to be angered is encouraging.

3.1 Future directions

At the printing time of this document, a Polhemus magnetic tracker has been partially implemented for