Prerequisites for Affective Signal Processing (ASP) - Part II

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP) Egon L. van den Broek∗ Center for Telematics and Information Technology, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands vandenbroek@acm.org Joris H. Janssen∗ , Joyce H.D.M. Westerink User Experience Group, Philips Research Europe, High Tech Campus 34, 5656 AE Eindhoven, The Netherlands {joris.h.janssen,joyce.westerink}@philips.com Jennifer A. Healey Intel Corporation, Corporate Technology Group, 3600 Juliette Lane SC12-319 Santa Clara CA 95054, USA jennifer.healey@intel.com Keywords: Affective Signal Processing (ASP), Emotion, Validation, Physiology-driven, Triangulation Abstract: Although emotions are embraced by science, their recognition has not reached a satisfying level. Through a concise overview of affect, its signals, features, and classification methods, we provide understanding for the problems encountered. Next, we identify the prerequisites for successful Affective Signal Processing: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions of the signal processing community. Using these directives, a critical analysis of a real-world case is provided. This illustrates that the prerequisites can become a valuable guide for Affective Signal Processing. When dealing with people, let us remember that we are not dealing with creatures of logic; we are dealing with creatures of emotion ... (Dale Carnegie, 1936; p. 41) Dale Carnegie (1888-1955) 1 1 INTRODUCING EMOTIONS While a century ago emotions were considered as too spiritual and human’s health was solely explained in physical (e.g., injuries) and physiological terms (e.g., bacteria, viruses), it is now generally acknowledged that emotions have their impact on health and illness. It has been shown that emotions influence our cognitive and social functioning as well as our cardiovascular system (Schuler and O’Brien, 1997) and, as a consequence, can even either shorten or prolong life (Frederickson et al., 2000). ∗ Joris H. Janssen and Egon L. van den Broek did equally contribute to this article; hence, they are shared first authors. 1 For the interested reader, we refer to “Historical foundations of social effectiveness? Dale Carnegie’s principles” (Duke and Novicevic, 2008), which illustrates the timeless significance of Carnegie’s work. Medicine’s interest in emotions was followed by that of Artificial Intelligence (AI), which envisioned that emotions both lead the path to true AI and enhance the communication between man and machine (or even environment) (Picard, 1997; Minsky, 2006). This expresses the intrinsic need for automated sensing of emotions. Often this is done through speech or face analysis; see Cowie et al. (2001) and Zeng et al. (2009) for reviews. Alternatively, physiological signals are used to identify emotions; see also Table 1 and 2, Box 1, and (Westerink et al., 2008a). This paper discusses the last approach, which we will denote as Affective Signal Processing (ASP). Physiological signals have the advantage that they are free from social masking and have the potential of being measured by non-invasive sensors; e.g., (Samboa et al., 2009; Westerink et al., 2008a), making them suited for a wide range of applications. In contrast, recognizing facial expressions is notoriously problematic and speech is often either absent or suffers from severe distortions in many in real-world applications (Cowie et al., 2001; Zeng et al., 2009). In the next section, we provide both an overview and a review of ASP. After that, we introduce four prerequisites for successful ASP. Moreover, a real world case will be analyzed, using these prerequisites (Box 1). We finish the paper with a brief conclusion in which we denote some applications of ASP. Table 1: An overview of common physiological signals and features used in ASP. Physiology (source) Features Unit Remark Cardiovascular activity through ECG or BVP (Berntson et al., 1997) Heart rate (HR) SD IBIs RMSSD IBIs LF power (0.05Hz - 0.15Hz) HF power (0.15HZ - 0.40Hz) VLF power ( < 0.05Hz) LF/HF Pulse Transit Time (PTT) Mean, SD SCL Nr of SCRs SCR amplitude SCR 1/2 recovery time SCR rise time Mean, SD temp Respiration rate Amplitude Resps Mean, SD corrugator supercilii Mean, SD zygomaticus major Mean, SD upper trapezius Mean, SD inter-blink interval beats / min s s ms2 ms2 ms2 HRV index HRV index Sympathetic activity Parasympathetic activity Electrodermal Activity (EDA) (Boucsein, 1992) Skin temperature (ST) Respiration (Grossman and Taylor, 2007) Muscle activity through EMG (Reaz et al., 2006) (Westerink et al., 2008b) ms µS µS s s oC µV µV µV ms Tonic Sympathetic Activity Rate Phasic Activity Phasic Activity Frowning Smiling Notes. SD: Standard deviation; RMSSD: Root Mean Sum of Square Differences; IBI: Inter-beat interval; LF: Low frequency; HF: High frequency; VLF: Very Low Frequency; SCL: Skin Conductance Level; SCR: Skin Conductance Response; ECG: Electrocardiogram; EMG: Electromyogram; BVP: Blood volume pulse. 2 AFFECTIVE SIGNAL PROCESSING (ASP) A broad range of affective signals are used in affective sciences. When processing such signals some general issues have to be taken in consideration: 1) Affective signals are typically derived through noninvasive methods to determine changes in physiology and, as such, are indirect measures. Hence, a delay between the actual change in emotional state and the recorded change in signal has to be taken into account. 2) Physiological sensors are unreliable: they contain movement artifacts and are sensitive to differences in bodily position. 3) Most sensors are obtrusive, preventing their integration in real world applications. 4) Affective signals are influenced by (the interaction among) a variety of factors (Cacioppo and Tassinary, 1990). Some of these sources are located internally (e.g., a thought) and some are among the broad range of possible external factors (e.g., a signal outside). This makes affective signals inherently noisy, which is most prominent in real world research. 5) Physiological changes can evolve in a matter of milliseconds, seconds, minutes or even longer. Some changes hold for only a brief moment, while others can even be permanent. Although seldom reported, the expected time windows of change are of interest. In particular since changes can add to each other, even when having a different origin. 6) Humans are no linear time (translation or shift) invariant systems (Boucsein, 1992), they habituate. This increases the complexity of ASP substantially, since most signal processing techniques rely on this assumption. 7) Affective signals have large individual differences. This calls for methods and models tailored to the individual. It has been shown that personal approaches increase the performance of ASP (Bailenson et al., 2008). 2.1 Classification of Affective Signals To enable processing of the signals, in most cases comprehensive sets of features have to be identified for each affective signal. To extract these features, the affective signals are processed in the time (e.g., statistical moments (Westerink et al., 2008b)), frequency (e.g., Fourier), time-frequency (e.g., Wavelets), or power domain (e.g., Periodogram and Autoregression). In Table 1, we provide a brief overview of the signals most often applied, including their best known features, with reference to their physiological source. The features obtained from the affective signals (see Table 1) are fed to pattern recognition methods, which can be classified as: template matching, syn- tactic or structural matching, and statistical classification; e.g., artificial neural networks (ANN). The former two are not or seldom used in affective signal processing; most ASP schemes use the latter. Statistical pattern recognition distinguishes supervised and unsupervised (e.g., clustering) pattern recognition; i.e., respectively, with or without a set of (labeled) training data. With unsupervised pattern recognition, the distance/similarity measure used and the algorithm applied to generate the clusters are key elements. Supervised pattern recognition relies on learning from a set of examples (training set). Statistical pattern recognition uses input features, a discriminant function (or network function for ANN) to recognize the features, and an error criterion in its classification process. In the field of ASP, several studies have been conducted, using a broad range of signals, features, and classifiers; see Table 2 for an overview. Nonetheless, both the recognition performance and the number of emotions that the classifiers were able to discriminate are disappointing. Moreover, comparing the different studies is problematic because of the different settings the research was applied in, ranging from controlled lab studies to real world testing, the type of emotion triggers used, the number of target states to be discriminated, and the signals and features employed. This illustrates the need for a set of prerequisites for ASP. 3 PREREQUISITES 3.1 Validity In the pursuit to trigger emotions in a more or less controlled manner, a range of methods have been applied: actors, images (IAPS), sounds (e.g., music), (fragments of) movies (Westerink et al., 2008b), speech (Van den Broek, 2004), commercials (Hazlett and Hazlett, 1999; Poels and Dewitte, 2006), games, agents / serious gaming / virtual reality (Slater et al., 2006; Westerink et al., 2008a), reliving of emotions (Van den Broek, 2004), and real world experiences (Healey and Picard, 2005); see also Box 1. However, how to know which of these methods actually triggered participants’ true emotions? This is a typical concern of validity, which is a crucial issue for ASP. Validity can be best obtained through four approaches: content, criteria-related, construct, and ecological validation, which we will discuss in relation to ASP; in addition, see Box 1. Content validity refers to a) The agreement of experts on the domain of interest; e.g., limited to a specific application or group of patients; b) The degree to which a feature (or its parameters) of a given signal represents a construct; and c) The degree to which a set of features (or their parameters) of a given set of signals adequately represents all facets of the domain. For instance, employing only skin conductance level (SCL) for ASP will lead to a weak content validity when trying to measure emotion, as SCL is known to relate to the arousal component of an emotion, but not to the valence component. However, when trying to measure only emotional arousal, measuring only SCL may form strong content validity; see also Box 1. Criteria-related validity handles the quality of the translation from the preferred measurement to an alternative, rather than to what extend the measurement represents a construct. Emotions are preferably measured at the moment they occur; however, measurements before (predictive) or after (postdictive) the particular event are sometimes more feasible; e.g., through subjective questionnaires. The quality of these translations are referred to as predictive or postdictive validity. A third form of criteria-related validity is concurrent validity: a metric for the reliability of measurements applied in relation to the preferred standard. For instance, the more affective states are discriminated the higher the concurrent validity. A construct validation process aims to develop a nomological network (i.e., a ground truth), or possibly an ontology or semantic network, build around the construct of interest. Such a network requires theoretically grounded, observable, operational definitions of all constructs and the relations between them. Such a network aims to provide a verifiable theoretical framework. The lack of such a network is one of the most pregnant problems ASP is coping with; e.g., see Box 1. A frequently occurring mistake is that emotions are denoted, where moods (i.e., longer object-unrelated affective states with very different physiology) are meant. This is very relevant for ASP, as it is known that moods are accompanied by very different physiological patterns than emotions are (Gendolla and Brinkman, 2005). Ecological validity refers to the influence of the context on measurements. We identify two issues: 1) Natural affective events are sparse, which makes it hard to let participants cycle through a range of affective states in a limited time frame and 2) The affective signals that occur are easily contaminated by contextual factors; so, using a similar context as the intended ASP application for initial learning is of vital importance; see also Box 1. Although understandable from a measurement-feasibility perspective, emotion measurements are often done in controlled laboratory settings. This makes results poorly generalizable to realworld applications. Hence, the need for unobtrusive Table 2: A summary of 18 studies that have tried to infer affect from physiological signals. Information source Signals Part Fea Sinha & Parsons, 1996 Picard et al., 2001 M C ,E ,R ,M 27 1 18 40 Scheirer et al., 2002 Nasoz et al., 2003 Takahashi, 2003 Haag et al., 2004 C ,E C ,E ,S C ,E ,B C ,E ,S ,M ,R 24 31 12 1 5 3 18 13 Kim et al., 2004 Lisetti & Nasoz, 2004 C ,E ,S C ,E ,S 175 29 Wagner et al., 2005 C ,E ,R ,M 1 32 Yoo et al., 2005 Choi & Woo, 2005 Healey & Picard, 2005 Rani et al., 2006 C ,E 6 E C ,E ,R ,M 9 C ,E ,S ,M ,P 15 5 3 22 46 Zhai & Barreto, 2006 Leon et al., 2007 Liu et al., 2008 Katsis et al., 2008 Yannakakis & Hallam, 2008 Kim & André, 2009 C ,E ,S ,P C ,E C ,E ,S ,M C ,E ,M ,R C ,E C ,E ,M ,R 11 5 35 15 20 110 32 8 6 10 72 3 Sel / Red SFS, Fisher Viterbi SFS, Fisher PCA Fisher DBI ANOVA SBS Classifiers Target Result LDA LDA 2 emotions 8 emotions 86% 81% HMM kNN, LDA SVM MLP 2 frustrations 6 emotions 6 emotions val / aro SVM kNN, LDA, MLP kNN, LDA, MLP MLP MLP LDA kNN, SVM, RT, BN SVM AANN SVM SVM, ANFIS SVM, MLP LDA, EMDC 3 emotions 6 emotions 64% 69% 42% 90% 97% 78% 84% 4 emotions 92% 4 emotions 4 emotions 3 stress levels 3 emotions 80% 75% 97% 86% 2 stress levels 3 emotions 3 affect states 4 affect states 2 fun levels 4 emotions 90% 71% 83% 79% 70% 70/95% Notes. Part: the number of participants; Fea: the number of features; Sel / Red: Algorithms used for selection or reduction of features; C : Cardiovascular activity; E : Electrodermal activity; R : Respiration; M : Electromyogram; B : Electroencephalogram; S : Skin temperature; P : Pupil Diameter; MLP: MultiLayer Perceptron; HMM: Hidden Markov Model; RT: Regression Tree; BN: Bayesian Network; AANN: Auto-Associative Neural Network; SVM: Support Vector Machine; LDA: Linear Discriminant Analysis; kNN: k Nearest Neighbors; ANFIS: Adaptive Neuro-Fuzzy Inference System; DBI: Davies-Bouldin Index; PCA: Principal Component Analysis; SFS: Sequential Forward Selection; SBS: Sequential Backward Selection; EMDC: Emotion-specific Multilevel Dichotomous Classification. sensors and methods to make longitudinal real-world studies possible is pressing. 3.2 Triangulation We propose to adopt the principle of Triangulation on ASP, as applied in social sciences and humancomputer interaction. Heath (2001) defines triangulation as “the strategy of using multiple operationalizations of constructs to help separate the construct under consideration from other irrelevancies in the operationalization”. Using this strategy provides several advantages: 1. Distinct signals can be used to validate each other; 2. Extrapolations can be made based on multiple data sets, providing more certainty. In turn, corrections can be made to errors in a result set that clearly defy from other results; and 3. More solid ground, or even a ground truth (see also Section 3.1), is obtained for the interpretation of signals, as multiple perspectives are used. Triangulation was, for example, successfully employed by Bailenson et al. (2008) and Healey and Picard (2005) . As (one of) the first, Bailenson et al. (2008) has shown that using both physiological signals and facial expressions leads to better ASP than using one of them; see also Section 1. Hence, we advise to record three affective signals, or have at least three features derived from them, for each construct under investigation, in well controlled research. Moreover, qualitative and subjective measures should accompany the signals (e.g., questionnaires, video recordings, interviews, and Likert scales); e.g., see (Hazlett and Hazlett, 1999; Healey and Picard, 2005; Slater et al., 2006; Van den Broek, 2004; Westerink et al., 2008a). Please consult also Section 3.1 on this topic. Box 1: Application of the prerequisites in practice One of the first large-scale real-world cases in which ASP is applied is the driving application of Healey and Picard (2005). They applied ASP on ECG, EMG, EDA, and respiration to determine the stress of 24 participants, during at least 50 minutes real world driving, completed by questionnaires and video recordings. Healey and Picard (2005) were able to distinguish between three stress levels, using five minute time windows. In addition, through ASP, they developed a continuous stress metric. In this box, we apply our ASP prerequisites for a critical analysis of this case; each prerequisite is denoted separately in relation to the case. Validation Section 3.1 denotes content, criteria-related, construct, and ecological validation. The continuous stress metric developed by Healey and Picard predicts three situations: sitting still, driving on the highway, and driving in town. These situations are likely to be confounded by movement. Movement, however, is not necessarily correlated with stress level. And even when movement reflects a more excited state, it does not always come with a negative bias. As affective signals are also influenced by movement, this makes it uncertain whether the actually measured construct is movement, stress, or a combination of both. This, together with a weak definition, leads to a limited content validity. For the continuous stress metric employed, Healey and Picard show a very high inter-observer reliability. Moreover, this stress metric has a very high temporal resolution. Nonetheless, as the authors elaborate, this was in some cases not enough to deal with the short latencies of some physiological signals. In addition, subjective stress reports could not be done during driving, so they were conducted after the drive. As the drives were quite long, this might have lead to a decrease in postdictive validity. However, they were congruent with averages of the continuous stress metric. Taken together, the criteria-related validity of the measurements is high. Construct validation refers to a nomological framework built around the construct of interest. They nicely describe their stress hypothesis in terms of the activity of the sympathetic and parasympathetic nervous systems. However, a description of the relation between most of the physiological signals and these dimensions is only elaborated to a limited extent. Moreover, the HRV measurement is questionable as it was not corrected for respiration (Grossman and Taylor, 2007). All in all, the construct validity is poor. The ecological validity of Healey and Picard (2005) is excellent: the research results can be generalized to driving in general, most likely to all driving situations. One could argue that the results are hard to generalize to other domains of application. This is indeed a valid observation; however, the authors do (correctly) not claim to want so. Triangulation Healey and Picard (2005) use multiple signals for the construct under investigation, complemented by subjective self-reports and observer scores. Healey and Picard correlated different physiological signals to see to what extent they describe the same construct and used self-reports and observer scores to validate the physiological changes and, hence, created a higher validity; see also Section 3.2. Hence, they successfully adopted the principle of triangulation. Physiology-driven approach Instead of continuously inferring the stress level of the driver, it might be sufficient to express a high stress level in terms of one or a few physiological signals. For instance, (Healey and Picard, 2005) show that EDA is strongly correlated with their continuous stress metric. So, using EDA as a stress scale might be sufficient for actual applications. By setting some thresholds in the EDA, it can also be used for music selection or distraction management (e.g., cell phones), as the authors propose. This reduces the amount of sensors and computing power needed, making it more feasible for practical application; see also Section 3.3. Signal processing contributions Healey and Picard (2005) state that “Each signal was sampled at a rate appropriate for capturing the information contained in the signal . . . ”. Such an ill specified statement is in line with other research where similar statements are made or the subject is ignored completely. However, Healey and Picard (2005) continue with “ . . . constrained by the sampling rates available . . . ”, which explains at least partly the reported sample frequencies; i.e., ECG: 496 Hz; EDA and respiration: 31 Hz; and EMG: 15.5 Hz. These are not reported previously as standard frequencies, if reported at all. Tailored filters as proposed in the current paper are absent. The only filter defined is the 0.5 seconds averaging filter. Why this is applied is not reported. Possibly, more filters were applied; however, their specifications are omitted. On the one hand, this makes it hard to reproduce the research; on the other hand, it illustrates the lack of attention for filtering. On several occasions throughout the paper, comparable work on aircraft pilots was mentioned. It would be of interest to compare the rich set of data Healey and Picard (2005) gathered with other data sets. Then, the robustness of the variety of signals, their features and parameters could be fully accessed. To extend from a few single initiatives to a more general practice, a benchmark should be set up; see also Section 3.4. Conclusion The work of Healey and Picard (2005) is already a showcase for triangulation. Their results illustrate that their case is per excellence also suitable for a physiologically driven approach. An additional advantage of this direction is that it can be further explored even without the preferred thorough theoretical framework. This would ease the way of their work to possible application areas. Moreover, the applicability of the processing schemes proposed by Healey and Picard (2005) could be verified with future benchmarks and the robustness of their approach could be strengthened through the usage of tailored filters and ASP techniques. Hence, using the framework introduced in this paper, the case discussed could be substantially improved, even post-hoc, although its foundation is already good and unique in its kind. 3.3 A physiology-driven approach 4 CONCLUSION A final prerequisite stems from the idea that ASP can never be entirely based on psychological changes. As discussed in Section 2, there are many factors outside one’s affective state that contaminate affective signals. Beside validation and triangulation, another way to deal with this is to employ a more physiologydriven perspective (Tractinsky, 2004). Instead of expressing the goals of ASP directly in terms of affective states, they can often be stated in terms of the affective signals themselves (Slater et al., 2006). For instance, instead of inferring an air-traffic controller’s or driver’s stress level, thresholding SCL might be sufficient; see also Box 1. Note that there always remains an interpretation in affective states. Then, the use of syntactic or structural pattern recognition for ASP should be (further) explored. Its hierarchical approach to simplifying complex patterns in affective signals is expected to be valuable for ASP. This paper provided both an overview and a review of ASP and explained the lack of success of ASP; see Section 2 and Tables 1 and 2. Next, in Section 3, four prerequisites are introduced from which ASP is expected to benefit significantly: validation, the principle of triangulation, and a physiologicaldriven approach, and contributions of the signal processing community. In addition, a real world case is discussed in Box 1, which illustrated the use of the proposed prerequisites. With the guidelines provided and the future’s progress ahead, we envision embedding of ASP in various professional and consumer settings, as a key factor of our every day life. A broad range of probes have been developed over the years (Westerink et al., 2008a), which illustrate the feasibility of embedding ASP in various settings. Let us briefly denote three of them: 1) For more than a decade, ASP is already applied to determine the impact of advertisements on people (Hazlett and Hazlett, 1999); for a review on emotion measurement in advertising, see (Poels and Dewitte, 2006). 2) Almost half a century ago, with the development of Eliza (Weizenbaum, 1966) both the possible implications of AI (for medicine) and the limitations of (classic) AI became apparent. Among many others, Liu et al. (2008), Slater et al. (2006), and Van den Broek (2004) denote how ASP can help AI to mature and to be of real value in this field. 3) ASP has often been applied with pilots and in automotive industry (Healey and Picard, 2005; Westerink et al., 2008a), as is also denoted in Box 1. Healey and Picard (2005) also provide a brief overview of literature on this topic. With Ambient Intelligence evolving, (wireless) sensor networks becoming more mature, and with prerequisites that enable the exploitation of ASP’s full potential, empathic machines should come within reach. And would it not be an appealing idea to live in an empathic surrounding that adapts to your mood and emotions, which can calm you or help you to concentrate when required? 3.4 Contributions of Signal Processing The majority of research on ASP is conducted by psychology, physiology, medicine, human-computer interaction, or artificial intelligence. Hence, true signal processing expertise is often missing; see also Box 1. In particular, expertise from biomedical signal processing could significantly contribute to ASP’s progress. We will now address a triplet of issues: The development of filters tailored to the specifications of ASP sensors and to ASP’s applications could significantly boost the performance of ASP. Affective signals are mostly sampled in a discrete fashion. The required AD conversion, however, can distort the signal; i.e., aliasing. For all possible signals, with all possible amplifiers and sensors, it is recommended to determine the relation between sample frequency and signal loss / distortion. So far, this has not been done and guidelines are provided founded on weak assumptions; see also Box 1. Then, for all signals, also the Nyquist frequency could be defined. A benchmark should be founded with verified affective signals. This would enable objective performance measurements of signal processing and pattern recognition techniques. The principle of triangulation (see Section 3.2) could be applied using it and the generic applicability of techniques could be tested. Moreover, it could be used for concurrent validation (see Section 3.1); e.g., through comparing different signals or apparatus that can substitute each other. ACKNOWLEDGMENTS The authors would like to thank Marjolein van der Zwaag and Stijn de Waele for their comments on an earlier version of this paper. Furthermore, we would like to thank the anonymous reviewers, who provided us the opportunity to improve this paper. REFERENCES Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A., Nass, C., and John, O. (2008). Real-time classification of evoked emotions using facial feature tracking and physiological responses. International Journal of Human-Computer Studies, 66(5):303–317. Berntson, G. G., Bigger, J. T., Eckberg, D. L., Grossman, P., Kaufmann, P. G., Malik, M., Nagaraja, H. N., Porges, S. W., Saul, J. P., Stone, P. H., and van der Molen, M. W. (1997). Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology, 34(6):623–648. Boucsein, W. (1992). Electrodermal activity. New York, NY, USA: Plenum Press. Cacioppo, J. and Tassinary, L. (1990). Inferring psychological significance from physiological signals. American Psychologist, 45(1):16–28. Choi, A. and Woo, W. (2005). Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. Lecture Notes in Computer Science (Affective Computing and Intelligent Interaction), 3784:590–597. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., and Taylor, J. G. (2001). Emotion recognition in human– computer interaction. IEEE Signal Processing Magazine, 18(1):32–80. Duke, A. and Novicevic, M. M. (2008). Historical foundations of social effectiveness? Dale Carnegie’s principles. Social Influences, 3(2):132–142. Frederickson, B. L., Manusco, R. A., Branigan, C., and Tugade, M. M. (2000). The undoing effect of positive emotions. Motivation and Emotion, 24(4):237–257. Gendolla, G. H. E. and Brinkman, K. (2005). The role of mood states in self-regulation: Effects on action preferences and resource mobilization. European Psychologist, 10(3):187–198. Grossman, P. and Taylor, E. W. (2007). Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology, 74(2):263–285. Haag, A., Goronzy, S., Schaich, P., and Williams, J. (2004). Emotion recognition using bio-sensors: First steps towards an automatic system. Lecture notes in computer science (Affective Dialogue Systems), 3068:36–48. Hazlett, R. L. and Hazlett, S. Y. (1999). Emotional response to television commercials: Facial EMG vs. self-report. Journal of Advertising Research, 39(2):7–23. Healey, J. A. and Picard, R. W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2):156–166. Heath, L. (2001). Triangulation: Methodology, pages 15901–15906. Elsevier Science Ltd.: Oxford, UK, 1 edition. ISBN: 978-0-08-043076-8. Katsis, C. D., Katertsidis, N., Ganiatsas, G., and Fotiadis, D. I. (2008). Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics–Part A: Systems and Humans, 38(3):502–512. Kim, J. and André, E. (2008). Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis Machine Intelligence, 30(12):2067–2083. Kim, K. H., Bang, S. W., and Kim, S. R. (2004). Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing, 42(3):419– 427. Leon, E., Clarke, G., Callaghan, V., and Sepulveda, F. (2007). A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence, 20(3):337–345. Lisetti, C. L. and Nasoz, F. (2004). Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing, 2004(11):1672–1687. Liu, C., Conn, K., Sarkar, N., and Stone, W. (2008). Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. International Journal of Human-Computer Studies, 66(9):662– 677. Minsky, M. (2006). The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. New York, NY, USA: Simon & Schuster. Nasoz, F., Alvarez, K., Lisetti, C. L., and Finkelstein, N. (2003). Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work, 6:4–14. Picard, R. W. (1997). Affective Computing. Boston MA, USA: MIT Press. sue on affective computing. Computers, 16(4):751–757. Interacting with Picard, R. W., Vyzas, E., and Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10):1175–1191. Van den Broek, E. L. (2004). Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics), 103:118–125. Poels, K. and Dewitte, S. (2006). How to capture the heart? reviewing 20 years of emotion measurement in advertising. Journal of Advertising Research, 46(1):18–37. Wagner, J., Kim, J., and André, E. (2005). From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). Rani, P., Liu, C., Sarkar, N., and Vanman, E. (2006). An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis & Applications, 9(1):58–69. Reaz, M. B. I., Hussain, M. S., and Mohd-Yasin, F. (2006). Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online, 8(1):11– 35. Samboa, H., Silva, F., Silva, H., and ao, R. F. (2009). PLUX – Biosignals Aquisition and Processing. URL: http://www.plux.info [Last accessed on January 21, 2009]. Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W. (2002). Frustrating the user on purpose: a step toward building an affective computer. Interacting with Computers, 14(2):93–118. Schuler, J. L. H. and O’Brien, W. H. (1997). Cardiovascular recovery from stress and hypertension factors: A meta-analytic view. Psychophysiology, 34:649–659. Sinha, R. and Parsons, O. A. (1996). Multivariate response patterning of fear. Cognition and Emotion, 10(2):173–198. Slater, M., Guger, C., Edlinger, G., Leeb, R., Pfurtscheller, G., Antley, A., Garau, M., Brogni, A., and Friedman, D. (2006). Analysis of physiological responses to a social situation in an immersive virtual environment. Presence: Teleoperators and Virtual Environments, 15(5):553– 569. Takahashi, K. (2003). Remarks on emotion recognition from bio-potential signals. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, volume 2, pages 1655–1659, Palmerston North, New Zealand. Tractinsky, N. (2004). Tools over solutions? comments on interacting with computers special is- Weizenbaum, J. (1966). ELIZA – a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45. Westerink, J. H. D. M., Ouwerkerk, M., overbeek, T., Pasveer, W. F., and de Ruyter, B. (2008a). Probing Experiences: From Academic Research to Commercial Propositions, volume 8 of Philips Research Book Series. Springer: Dordrecht, The Netherlands. Westerink, J. H. D. M., van den Broek, E. L., Schut, M. H., van Herk, J., and Tuinenbreijer, K. (2008b). Computing emotion awareness through galvanic skin response and facial electromyography, volume 8 of Philips Research Book Series, chapter 14 (Part II: Probing in order to Feed Back), pages 137–150. Springer: Dordrecht, The Netherlands. Yannakakis, G. N. and Hallam, J. (2008). Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies, 66(10):741–755. Yoo, S. K., Lee, C. K., Park, J. Y., Kim, N. H., Lee, B. C., and Jeong, K. S. (2005). Neural network based emotion estimation using heart rate variability and skin resistance. Lecture Notes in Computer Science (Advances in Natural Computation), 3610:818–824. Zeng, Z., Pantic, M., Roisman, G. I., and Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis Machine Intelligence, 31(1):39–58. Zhai, J. and Barreto, A. (2006). Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Science Instrumentation, 42:495–500.