This demonstration presents "The Market Scene," a virtual-reality scene in which the us... more This demonstration presents "The Market Scene," a virtual-reality scene in which the user interacts with multiple virtual agents and initiates conversations with these agents via proxemics and gaze. The application is implemented in Unity with the VAIF authoring tool for rapid development of social agents. In the Market Scene, the user learns the background of the "Gods of the Neon City" adventure story through interaction in a crowded market rather than by listening to a narrator. The application demonstrates a form of mixed-initiative interaction, in that the user can initiate a conversation by coming near to and gazing at an agent, and an agent can initiate a conversation if a user approaches.
Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animat... more Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animation skills, advanced programming knowledge, and in some cases artificial intelligence or the integration of complex interaction models. Features like lip-syncing to an audio file, recognizing the users' speech, or having the character move at certain times in certain ways, are inaccessible to researchers that want to build and use these agents for education, research, or industrial uses. VAIF, the Virtual Agent Interaction Framework, is an extensively documented system that attempts to bridge that gap and provide inexperienced researchers the tools and means to develop their own agents in a centralized, lightweight platform that provides all these features through a simple interface within the Unity game engine.
The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality ... more The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality applications with embodied conversational agents. VAIF is intended for use by both expert and non-expert users, in contrast with more sophisticated and complex development tools such as the Virtual Human Toolkit. To determine if VAIF is actually usable by a range of users, we conducted a two-phase summative usability test, with a total of 43 participants. We also tried porting to VAIF a scene from an earlier VR application. The results of the usability study suggest that people with little or even no experience in creating embodied conversational agents can install VAIF and build interaction scenes from scratch, with relatively low rates of encountering problem episodes. However, the usability testing disclosed aspects of VAIF and its user’s guide that could be improved to reduce the number of problem episodes that users encounter.
The published research examining usability and playability of games is largely theoretical. A pri... more The published research examining usability and playability of games is largely theoretical. A prior empirical study of a game with an embodied conversational agent found that most frustration episodes could be understood in terms of both usability and playability, but this study was based on a game in which the interaction by both player and agent were limited to verbal communication. To explore whether these results would hold for a game in which the player and agent communicated with both speech and gesture, we conducted an empirical formative user-experience evaluation of a multimodal game. Our findings strongly confirmed that frustration episodes can be understood as issues of both usability and playability. However, the relative frequencies of the categories of usability and playability issues differed between the speech-only and the speech-and-gesture games. Much of this difference likely arose because higher levels of engagement and rapport between player and agent in the speech-and-gesture game led to the players having greater, and in many cases unfulfilled, expectations for the capabilities of the agent.
Overview This course introduces the theory and methods of natural language processing (NLP). NLP ... more Overview This course introduces the theory and methods of natural language processing (NLP). NLP systems understand and produce human language for applications such as information extraction, machine translation, automatic summarization, question-answering, and interactive dialog systems. The course covers knowledge-based and statistical approaches to language processing for syntax (language structures), semantics (language meaning), and pragmatics/discourse (the interpretation of language in context). For graduate students, the course will also cover aspects of current research in NLP.
How would humans would try to build rapport with an agent rather than with another human? To addr... more How would humans would try to build rapport with an agent rather than with another human? To address this issue, we created a virtual-reality application, Gods of the Neon City, in which an agent might (or might not) betray its human partner. The results from 45 participants suggest that it is possible to build a VR adventure with agents in which a central character can be perceived as untrustworthy, and that participants' perceptions of an agent's trustworthiness can be affected by systematic messages from other agents. However, the study's data are insufficient to demonstrate that participants try to address perceived untrustworthiness by increasing the length of their utterances.
International Conference on Design of Communication, Oct 12, 2003
Welcome to San Francisco and SIGDOC 2003. This year's theme, "Finding Real-World Solutio... more Welcome to San Francisco and SIGDOC 2003. This year's theme, "Finding Real-World Solutions for Documentation: How Theory Informs Practice and Practice Informs Theory" continues conversations from years past that evoke our deep concerns with understanding users and their needs, seeking and creating technical solutions to make our work easier, learning new ways to work better in teams. We have stories from the field and theory from the academy.The conference provides a venue for the stories and the theories to improve each other. In the broad field of design of communication, the circle of experience, abstraction, testing and use can be completed in both directions. So SIGDOC 2003 provides a forum in which documentation specialists, researchers, and educators can contribute to a field in which theory informs practice and practice informs theory. Together we work to answer some of the following questions: How can we bring new theoretical or methodological frameworks to bear on old problems? How can we take advantage of practitioners' experience to advance, improve, or combine the methods and methodologies used in documentation? What are the benefits and costs if practitioners just ignore user modeling and charge ahead? How can we productively model context, activity, documents, and users? How does documentation experience affect user modeling? What tools enable quick and effective production of documentation and allow writers to own the content longer? .To help us understand the complexity of these issues, our 2003 Rigo Award winner, JoAnn Hackos will deliver the Keynote Address on how might academe and industry work more closely together to understand how people need and use technical information. We will also hear from Susan Mills, IBM Director, Corporate User Technologies, who will accept the Diana Award for IBM.
The aim of this workshop is to identify and to synthesize current needs for language-technology e... more The aim of this workshop is to identify and to synthesize current needs for language-technology evaluation. The first part of the workshop will focus on one of the most challenging current issues in language engineering: the evaluation of dialogue systems and models. The second part will extend the discussion to address the problem of evaluation in language engineering more broadly and on more theoretical grounds.The space of possible dialogues is enormous, even for limited domains like travel information servers. The generalization of evaluation methodologies across different application domains and languages is an open problem. Review of published evaluations of dialogue models and systems suggests that usability techniques are the standard method. Dialogue-based system are often evaluated in terms of standard, objective usability metrics, such as task-completion time and number of user actions. In the past, researchers have proposed and debated theory-based methods for modifying and testing the underlying dialogue model, but the most widely used method of evaluation is usability testing, although more precise and empirical methods for evaluating the effectiveness of dialogue models have been proposed. For task-based interaction, typical measures of effectiveness are time-to-completion and task outcome, but the evaluation should focus on user satisfaction rather than on arbitrary effectiveness measurements. Indeed, the problems faced in current approaches to measurement of effectiveness dialogue models and systems include:1. Direct measures are unhelpful because efficient performance on the nominal task may not represent the most effective interaction2. Indirect measures usually rely on judgment and are vulnerable to weak relationships between the inputs and outputs3. Subjective measures are unreliable and domain-specificRepresentative questions to be addressed include but are not limited to:1. How do we deal with the combinatorial explosion of dialogue states?2. How can satisfaction be measured with respect to underlying dialogue models?3. Are there useful direct measures of dialogue properties that do not depend on task efficiency?4. What is the role of agent-based simulation in evaluation of dialogue models?Of course, the problems faced in evaluating dialogue and system models are found in other domains of language engineering, even for non-interactive processes such as part-of-speech tagging, parsing, semantic disambiguation, information extration, speech transcription, and audio document indexing. So the issue of evaluation can be viewed at a more generic level, raising fundamental, theoretical questions such as:1. What are the interest and benefits of evaluation for language engineering?2. Do we really need these specific methodologies, since a form of evaluation should always be present in any scientific investigation?3. If evaluation is needed in language engineering, is it the case for all domains?4. What form should it take? Technology evaluation (task-oriented in laboratory environment) or field/user Evaluation (complete systems in real-life conditions)?5. We have seen before that the the evaluation of dialogue models is still unsolved, but for domains where metrics already exists, are they satisfactory and sufficient? How can we take into account or abstract from the subjective factor introduced by human operators in the process?6. Do similarity measures and standards offer appropriate answers to this problem? Most of the efforts focus on evaluating process, but what about the issue of language resources evaluation?In the second part of the workshopwe wish to address the problem of evaluation both from a broader perspective, including novel applications domain for evaluation, new metrics for known tasks and resource evaluation, as well as look at the problem from a more theoretical point of view, including the isssue of formal theory of evaluation and infrastructural needs of language engineering.
This paper describes a system for embodied conversational agents developed by Inmerssion and one ... more This paper describes a system for embodied conversational agents developed by Inmerssion and one of the applications—Young Merlin: Trial by Fire —built with this system. In the Merlin application, the ECA and a human interact with speech in virtual reality. The goal of this application is to provide engaging VR experiences that build rapport through storytelling and verbal interactions. The agent is fully automated, and his attitude towards the user changes over time depending on the interaction. The conversational system was built through a declarative approach that supports animations, markup language, and gesture recognition. Future versions of Merlin will implement multi-character dialogs, additional actions, and extended interaction time.
This paper describes a system for embodied conversational agents (ECAs) developed at the Universi... more This paper describes a system for embodied conversational agents (ECAs) developed at the University of Texas at El Paso by the Advanced aGent ENgagement Team (AGENT) and one of the applications -- Survival on Jungle Island -- built with this system. In the Jungle application, the ECA and a human interact with speech and gesture for approximately 40 -- 60 minutes in a game composed of 23 scenes (to maintain the demonstration feasible, participants will interact only with select scenes that showcase the capabilities of our system). Each scene comprises a collection of speech input, speech output, gesture input, gesture output, scenery, triggers, and decision points.
This demonstration presents "The Market Scene," a virtual-reality scene in which the us... more This demonstration presents "The Market Scene," a virtual-reality scene in which the user interacts with multiple virtual agents and initiates conversations with these agents via proxemics and gaze. The application is implemented in Unity with the VAIF authoring tool for rapid development of social agents. In the Market Scene, the user learns the background of the "Gods of the Neon City" adventure story through interaction in a crowded market rather than by listening to a narrator. The application demonstrates a form of mixed-initiative interaction, in that the user can initiate a conversation by coming near to and gazing at an agent, and an agent can initiate a conversation if a user approaches.
Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animat... more Creating an embodied virtual agent is often a complex process. It involves 3D modeling and animation skills, advanced programming knowledge, and in some cases artificial intelligence or the integration of complex interaction models. Features like lip-syncing to an audio file, recognizing the users' speech, or having the character move at certain times in certain ways, are inaccessible to researchers that want to build and use these agents for education, research, or industrial uses. VAIF, the Virtual Agent Interaction Framework, is an extensively documented system that attempts to bridge that gap and provide inexperienced researchers the tools and means to develop their own agents in a centralized, lightweight platform that provides all these features through a simple interface within the Unity game engine.
The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality ... more The Virtual Agent Interaction Framework (VAIF) is an authoring tool for creating virtual-reality applications with embodied conversational agents. VAIF is intended for use by both expert and non-expert users, in contrast with more sophisticated and complex development tools such as the Virtual Human Toolkit. To determine if VAIF is actually usable by a range of users, we conducted a two-phase summative usability test, with a total of 43 participants. We also tried porting to VAIF a scene from an earlier VR application. The results of the usability study suggest that people with little or even no experience in creating embodied conversational agents can install VAIF and build interaction scenes from scratch, with relatively low rates of encountering problem episodes. However, the usability testing disclosed aspects of VAIF and its user’s guide that could be improved to reduce the number of problem episodes that users encounter.
The published research examining usability and playability of games is largely theoretical. A pri... more The published research examining usability and playability of games is largely theoretical. A prior empirical study of a game with an embodied conversational agent found that most frustration episodes could be understood in terms of both usability and playability, but this study was based on a game in which the interaction by both player and agent were limited to verbal communication. To explore whether these results would hold for a game in which the player and agent communicated with both speech and gesture, we conducted an empirical formative user-experience evaluation of a multimodal game. Our findings strongly confirmed that frustration episodes can be understood as issues of both usability and playability. However, the relative frequencies of the categories of usability and playability issues differed between the speech-only and the speech-and-gesture games. Much of this difference likely arose because higher levels of engagement and rapport between player and agent in the speech-and-gesture game led to the players having greater, and in many cases unfulfilled, expectations for the capabilities of the agent.
Overview This course introduces the theory and methods of natural language processing (NLP). NLP ... more Overview This course introduces the theory and methods of natural language processing (NLP). NLP systems understand and produce human language for applications such as information extraction, machine translation, automatic summarization, question-answering, and interactive dialog systems. The course covers knowledge-based and statistical approaches to language processing for syntax (language structures), semantics (language meaning), and pragmatics/discourse (the interpretation of language in context). For graduate students, the course will also cover aspects of current research in NLP.
How would humans would try to build rapport with an agent rather than with another human? To addr... more How would humans would try to build rapport with an agent rather than with another human? To address this issue, we created a virtual-reality application, Gods of the Neon City, in which an agent might (or might not) betray its human partner. The results from 45 participants suggest that it is possible to build a VR adventure with agents in which a central character can be perceived as untrustworthy, and that participants' perceptions of an agent's trustworthiness can be affected by systematic messages from other agents. However, the study's data are insufficient to demonstrate that participants try to address perceived untrustworthiness by increasing the length of their utterances.
International Conference on Design of Communication, Oct 12, 2003
Welcome to San Francisco and SIGDOC 2003. This year's theme, "Finding Real-World Solutio... more Welcome to San Francisco and SIGDOC 2003. This year's theme, "Finding Real-World Solutions for Documentation: How Theory Informs Practice and Practice Informs Theory" continues conversations from years past that evoke our deep concerns with understanding users and their needs, seeking and creating technical solutions to make our work easier, learning new ways to work better in teams. We have stories from the field and theory from the academy.The conference provides a venue for the stories and the theories to improve each other. In the broad field of design of communication, the circle of experience, abstraction, testing and use can be completed in both directions. So SIGDOC 2003 provides a forum in which documentation specialists, researchers, and educators can contribute to a field in which theory informs practice and practice informs theory. Together we work to answer some of the following questions: How can we bring new theoretical or methodological frameworks to bear on old problems? How can we take advantage of practitioners' experience to advance, improve, or combine the methods and methodologies used in documentation? What are the benefits and costs if practitioners just ignore user modeling and charge ahead? How can we productively model context, activity, documents, and users? How does documentation experience affect user modeling? What tools enable quick and effective production of documentation and allow writers to own the content longer? .To help us understand the complexity of these issues, our 2003 Rigo Award winner, JoAnn Hackos will deliver the Keynote Address on how might academe and industry work more closely together to understand how people need and use technical information. We will also hear from Susan Mills, IBM Director, Corporate User Technologies, who will accept the Diana Award for IBM.
The aim of this workshop is to identify and to synthesize current needs for language-technology e... more The aim of this workshop is to identify and to synthesize current needs for language-technology evaluation. The first part of the workshop will focus on one of the most challenging current issues in language engineering: the evaluation of dialogue systems and models. The second part will extend the discussion to address the problem of evaluation in language engineering more broadly and on more theoretical grounds.The space of possible dialogues is enormous, even for limited domains like travel information servers. The generalization of evaluation methodologies across different application domains and languages is an open problem. Review of published evaluations of dialogue models and systems suggests that usability techniques are the standard method. Dialogue-based system are often evaluated in terms of standard, objective usability metrics, such as task-completion time and number of user actions. In the past, researchers have proposed and debated theory-based methods for modifying and testing the underlying dialogue model, but the most widely used method of evaluation is usability testing, although more precise and empirical methods for evaluating the effectiveness of dialogue models have been proposed. For task-based interaction, typical measures of effectiveness are time-to-completion and task outcome, but the evaluation should focus on user satisfaction rather than on arbitrary effectiveness measurements. Indeed, the problems faced in current approaches to measurement of effectiveness dialogue models and systems include:1. Direct measures are unhelpful because efficient performance on the nominal task may not represent the most effective interaction2. Indirect measures usually rely on judgment and are vulnerable to weak relationships between the inputs and outputs3. Subjective measures are unreliable and domain-specificRepresentative questions to be addressed include but are not limited to:1. How do we deal with the combinatorial explosion of dialogue states?2. How can satisfaction be measured with respect to underlying dialogue models?3. Are there useful direct measures of dialogue properties that do not depend on task efficiency?4. What is the role of agent-based simulation in evaluation of dialogue models?Of course, the problems faced in evaluating dialogue and system models are found in other domains of language engineering, even for non-interactive processes such as part-of-speech tagging, parsing, semantic disambiguation, information extration, speech transcription, and audio document indexing. So the issue of evaluation can be viewed at a more generic level, raising fundamental, theoretical questions such as:1. What are the interest and benefits of evaluation for language engineering?2. Do we really need these specific methodologies, since a form of evaluation should always be present in any scientific investigation?3. If evaluation is needed in language engineering, is it the case for all domains?4. What form should it take? Technology evaluation (task-oriented in laboratory environment) or field/user Evaluation (complete systems in real-life conditions)?5. We have seen before that the the evaluation of dialogue models is still unsolved, but for domains where metrics already exists, are they satisfactory and sufficient? How can we take into account or abstract from the subjective factor introduced by human operators in the process?6. Do similarity measures and standards offer appropriate answers to this problem? Most of the efforts focus on evaluating process, but what about the issue of language resources evaluation?In the second part of the workshopwe wish to address the problem of evaluation both from a broader perspective, including novel applications domain for evaluation, new metrics for known tasks and resource evaluation, as well as look at the problem from a more theoretical point of view, including the isssue of formal theory of evaluation and infrastructural needs of language engineering.
This paper describes a system for embodied conversational agents developed by Inmerssion and one ... more This paper describes a system for embodied conversational agents developed by Inmerssion and one of the applications—Young Merlin: Trial by Fire —built with this system. In the Merlin application, the ECA and a human interact with speech in virtual reality. The goal of this application is to provide engaging VR experiences that build rapport through storytelling and verbal interactions. The agent is fully automated, and his attitude towards the user changes over time depending on the interaction. The conversational system was built through a declarative approach that supports animations, markup language, and gesture recognition. Future versions of Merlin will implement multi-character dialogs, additional actions, and extended interaction time.
This paper describes a system for embodied conversational agents (ECAs) developed at the Universi... more This paper describes a system for embodied conversational agents (ECAs) developed at the University of Texas at El Paso by the Advanced aGent ENgagement Team (AGENT) and one of the applications -- Survival on Jungle Island -- built with this system. In the Jungle application, the ECA and a human interact with speech and gesture for approximately 40 -- 60 minutes in a game composed of 23 scenes (to maintain the demonstration feasible, participants will interact only with select scenes that showcase the capabilities of our system). Each scene comprises a collection of speech input, speech output, gesture input, gesture output, scenery, triggers, and decision points.
Uploads
Papers by David Novick