Finding Emotion in Image Descriptions: Crowdsourced Data
This dataset contains 660 images, each annotated with descriptions and mood labels. The images we... more This dataset contains 660 images, each annotated with descriptions and mood labels. The images were originally created by users of the WordsEye text-to-scene system (https://www.wordseye.com/) and were downloaded from the WordsEye gallery. For each image, we used Amazon Mechanical Turk to obtain: (a) a literal description that could function as a caption for the image, (b) the most relevant mood for the picture (happiness, sadness, anger, surprise, fear, or disgust), (c) a short explanation of why that mood was selected. We published three AMT HITs for each picture, for a total of 1980 captions, mood labels, and explanations. This data was used for the machine learning experiments presented in: Morgan Ulinski, Victor Soto, and Julia Hirschberg. Finding Emotion in Image Descriptions. In Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining, WISDOM '12, pages 8:1-8:7. Please cite this paper if you use this data.
One-sentence descriptions of each picture from the Picture Series for Positional Verbs (Ameka et ... more One-sentence descriptions of each picture from the Picture Series for Positional Verbs (Ameka et al., 1999) and each video clip from the Motion Verb Stimulus Kit (Levinson, 2001). 163 English sentences, 165 Spanish sentences, 157 German sentences, 158 Egyptian Arabic sentences. All sentences are tokenized and annotated with lemma, part of speech, morphological features, dependency label and head. We use the universal POS tags, universal features, and universal dependency relations. Treebank is in conll format.
Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages ... more Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages can shed light on the morphology-syntax interface: one aspect of their polysynthetic nature is that information often encoded in phrases and clauses in other languages is instead found in a single morphological word. In this paper, we look at another instance, the Australian Aboriginal language Arrernte, and in particular at complex predicates within the language, to examine the implications for the morphology-syntax interface. Following from this, we show how a glue semantics-based approach can be applied to Arrernte complex predicates, in a way that fits neatly with the use of glue semantics to model lexical functions in LFG in a multilingual natural language generation environment.
Proceedings of the 2014 Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2014
In this paper, we describe how field linguists can use the WordsEye Linguistics Tool (WELT) to st... more In this paper, we describe how field linguists can use the WordsEye Linguistics Tool (WELT) to study endangered languages. WELT is a tool under development for eliciting endangered language data and formally documenting a language, based on WordsEye (Coyne and Sproat, 2001), a text-to-scene generation tool that produces 3D scenes from text input. First, a linguist uses WELT to create elicitation materials and collect language data. Next, he or she uses WELT to formally document the language. Finally, the formal models are used to create a textto-scene system that takes input in the endangered language and generates a picture representing its meaning.
Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2014
We describe the WordsEye Linguistics tool (WELT), a novel tool for the documentation and preserva... more We describe the WordsEye Linguistics tool (WELT), a novel tool for the documentation and preservation of endangered languages. WELT is based on Words-Eye (Coyne and Sproat, 2001), a text-toscene tool that automatically generates 3D scenes from written input. WELT has two modes of operation. In the first mode, English input automatically generates a picture which can be used to elicit a description in the target language. In the second mode, the linguist formally documents the grammar of an endangered language, thereby creating a system that takes input in the endangered language and generates a picture according to the grammar; the picture can then be used to verify the grammar with native speakers. We will demonstrate WELT's use on scenarios involving Arrernte and Nahuatl.
Proceedings of the 13th Linguistic Annotation Workshop, 2019
We address the issue of acquiring quality annotations of hedging words and phrases, linguistic ph... more We address the issue of acquiring quality annotations of hedging words and phrases, linguistic phenomenona in which words, sounds, or other constructions are used to express ambiguity or uncertainty. Due to the limited availability of existing corpora annotated for hedging, linguists and other language scientists have been constrained as to the extent they can study this phenomenon. In this paper, we introduce a new method of acquiring hedging annotations via crowdsourcing, based on reformulating the task of labeling hedges as a simple word sense disambiguation task. We also introduce a new hedging corpus we have constructed by applying this method, a collection of forum posts annotated using Amazon Mechanical Turk. We found that the crowdsourced judgments we obtained had an inter-annotator agreement of 92.89% (Fleiss' Kappa=0.751) and, when comparing a subset of these annotations to an expert-annotated gold standard, an accuracy of 96.65%.
LREC ... International Conference on Language Resources & Evaluation : [proceedings]. International Conference on Language Resources and Evaluation, 2020
This paper proposes a representation framework for encoding spatial language in radiology based o... more This paper proposes a representation framework for encoding spatial language in radiology based on frame semantics. The framework is adopted from the existing SpatialNet representation in the general domain with the aim to generate more accurate representations of spatial language used by radiologists. We describe Rad-SpatialNet in detail along with illustrating the importance of incorporating domain knowledge in understanding the varied linguistic expressions involved in different radiological spatial relations. This work also constructs a corpus of 400 radiology reports of three examination types (chest X-rays, brain MRIs, and babygrams) annotated with fine-grained contextual information according to this schema. Spatial trigger expressions and elements corresponding to a spatial frame are annotated. We apply BERT-based models (BERT-Base and BERT- Large) to first extract the trigger terms (lexical units for a spatial frame) and then to identify the related frame elements. The resu...
When you listen to the evening news, or read a newspaper, book or web site, there is a good chanc... more When you listen to the evening news, or read a newspaper, book or web site, there is a good chance that you will hear or see a term — perhaps a name, perhaps a technical term — that you have never seen before. Such words are often novel or rare and are often names (of people, places, organizations. . . ). They are hard for humans to process, but they are even harder for automatic speech and language processing systems. For a single language, a speech recognition or text-to-speech system needs to know how to pronounce a word to recognize or say it. For two languages, in particular a pair with different writing systems, a search engine or document summarizer needs to know how to transcribe one word to another to retrieve or distill across languages. For example, the soccer player written in English text as Maciej Zurawski would appear as 마치에이주라브스키 (ma-chi-e-i ju-ra-beu-seu-ki) in Korean In this project we attack both problems – unusual term pronunciation and term transcription. For pr...
We describe our evaluation of the WordsEye text-to-scene generation system. We address the proble... more We describe our evaluation of the WordsEye text-to-scene generation system. We address the problem of evaluating the output of such a system vs. simple search methods to find a picture to illustrate a sentence. To do this, we constructed two sets of test sentences: a set of crowdsourced imaginative sentences and a set of realistic sentences extracted from the PASCAL image caption corpus (Rashtchian et al., 2010). For each sentence, we compared sample pictures found using Google Image Search to those produced by WordsEye. We then crowdsourced judgments as to which picture best illustrated each sentence. For imaginative sentences, pictures produced by WordsEye were preferred, but for realistic sentences, Google Image Search results were preferred. We also used crowdsourcing to obtain a rating for how well each picture illustrated the sentence, from 1 (completely correct) to 5 (completely incorrect). WordsEye pictures had an average rating of 2.58 on imaginative sentences and 2.54 on r...
We present experiments in incrementally learning a dependency parser. The parser will be used in ... more We present experiments in incrementally learning a dependency parser. The parser will be used in the WordsEye Linguistics Tools (WELT) (Ulinski et al., 2014) which supports field linguists documenting a language’s syntax and semantics. Our goal is to make syntactic annotation faster for field linguists. We have created a new parallel corpus of descriptions of spatial relations and motion events, based on pictures and video clips used by field linguists for elicitation of language from native speaker informants. We collected descriptions for each picture and video from native speakers in English, Spanish, German, and Egyptian Arabic. We compare the performance of MSTParser (McDonald et al., 2006) and MaltParser (Nivre et al., 2006) when trained on small amounts of this data. We find that MaltParser achieves the best performance. We also present the results of experiments using the parser to assist with annotation. We find that even when the parser is trained on a single sentence from...
This paper introduces SpatialNet, a novel resource which links linguistic expressions to actual s... more This paper introduces SpatialNet, a novel resource which links linguistic expressions to actual spatial configurations. SpatialNet is based on FrameNet (Ruppenhofer et al., 2016) and VigNet (Coyne et al., 2011), two resources which use frame semantics to encode lexical meaning. SpatialNet uses a deep semantic representation of spatial relations to provide a formal description of how a language expresses spatial information. This formal representation of the lexical semantics of spatial language also provides a consistent way to represent spatial meaning across multiple languages. In this paper, we describe the structure of SpatialNet, with examples from English and German. We also show how SpatialNet can be combined with other existing NLP tools to create a text-to-scene system for a language.
Proceedings of the Workshop on Computational Semantics beyond Events and Roles, 2018
We describe a novel method for identifying hedge terms using a set of manually constructed rules.... more We describe a novel method for identifying hedge terms using a set of manually constructed rules. We present experiments adding hedge features to a committed belief system to improve classification. We compare performance of this system (a) without hedging features, (b) with dictionary-based features, and (c) with rule-based features. We find that using hedge features improves performance of the committed belief system, particularly in identifying instances of non-committed belief and reported belief.
Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages ... more Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages can shed light on the morphology-syntax interface: one aspect of their polysynthetic nature is that information often encoded in phrases and clauses in other languages is instead found in a single morphological word. In this paper, we look at another instance, the Australian Aboriginal language Arrernte, and in particular at complex predicates within the language, to examine the implications for the morphology-syntax interface. Following from this, we show how a glue semantics-based approach can be applied to Arrernte complex predicates, in a way that fits neatly with the use of glue semantics to model lexical functions in LFG in a multilingual natural language generation environment. 1 Authors' note: Other abbreviations we use in examples are ASSOC MOTION = Associated Motion, ATTEN = Attenuative reduplicant, DO.COMING = type of Associated Motion marker, EMPH = emphasis particle, FOC = focus particle, INTENS = intensifier, ITER = iterative, IV = intransitive verbaliser, NOMLSR = nominaliser, NUM = number, PRES = present tense, PRIOR.MOTION = type of Associated Motion marker, PST = past tense, PV = preverb, QUICK:DO&GO.BACK = type of Associated Motion marker, RECIP = reciprocal, REFL = reflexive, SR = switch reference, SS = same subject, TV = transitive verbaliser, V = verb.
In this paper, we approach the problem of classifying emotion in image descriptions. A method is ... more In this paper, we approach the problem of classifying emotion in image descriptions. A method is proposed to perform 6-way emotion classification and is tested against two labeled datasets: a corpus of blog posts mined from LiveJournal and a corpus of descriptive texts of computer generated scenes. We perform feature selection using the mRMR technique and then use a multi-class linear predictor to classify posts among the Ekman Big Six emotions (happiness, sadness, anger, surprise, fear, and disgust) [9]. We find that TFIDF scores on lexical features and LIWC scores are much more helpful in emotion classification than using scores calculated from existing sentiment dictionaries, and that our proposed method performs significantly better than a baseline classifier that chooses the majority class. On the blog posts, we achieve 40% accuracy, and on the corpus of image descriptions, we achieve up to 63% accuracy.
Finding Emotion in Image Descriptions: Crowdsourced Data
This dataset contains 660 images, each annotated with descriptions and mood labels. The images we... more This dataset contains 660 images, each annotated with descriptions and mood labels. The images were originally created by users of the WordsEye text-to-scene system (https://www.wordseye.com/) and were downloaded from the WordsEye gallery. For each image, we used Amazon Mechanical Turk to obtain: (a) a literal description that could function as a caption for the image, (b) the most relevant mood for the picture (happiness, sadness, anger, surprise, fear, or disgust), (c) a short explanation of why that mood was selected. We published three AMT HITs for each picture, for a total of 1980 captions, mood labels, and explanations. This data was used for the machine learning experiments presented in: Morgan Ulinski, Victor Soto, and Julia Hirschberg. Finding Emotion in Image Descriptions. In Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining, WISDOM '12, pages 8:1-8:7. Please cite this paper if you use this data.
One-sentence descriptions of each picture from the Picture Series for Positional Verbs (Ameka et ... more One-sentence descriptions of each picture from the Picture Series for Positional Verbs (Ameka et al., 1999) and each video clip from the Motion Verb Stimulus Kit (Levinson, 2001). 163 English sentences, 165 Spanish sentences, 157 German sentences, 158 Egyptian Arabic sentences. All sentences are tokenized and annotated with lemma, part of speech, morphological features, dependency label and head. We use the universal POS tags, universal features, and universal dependency relations. Treebank is in conll format.
Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages ... more Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages can shed light on the morphology-syntax interface: one aspect of their polysynthetic nature is that information often encoded in phrases and clauses in other languages is instead found in a single morphological word. In this paper, we look at another instance, the Australian Aboriginal language Arrernte, and in particular at complex predicates within the language, to examine the implications for the morphology-syntax interface. Following from this, we show how a glue semantics-based approach can be applied to Arrernte complex predicates, in a way that fits neatly with the use of glue semantics to model lexical functions in LFG in a multilingual natural language generation environment.
Proceedings of the 2014 Workshop on the Use of Computational Methods in the Study of Endangered Languages, 2014
In this paper, we describe how field linguists can use the WordsEye Linguistics Tool (WELT) to st... more In this paper, we describe how field linguists can use the WordsEye Linguistics Tool (WELT) to study endangered languages. WELT is a tool under development for eliciting endangered language data and formally documenting a language, based on WordsEye (Coyne and Sproat, 2001), a text-to-scene generation tool that produces 3D scenes from text input. First, a linguist uses WELT to create elicitation materials and collect language data. Next, he or she uses WELT to formally document the language. Finally, the formal models are used to create a textto-scene system that takes input in the endangered language and generates a picture representing its meaning.
Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2014
We describe the WordsEye Linguistics tool (WELT), a novel tool for the documentation and preserva... more We describe the WordsEye Linguistics tool (WELT), a novel tool for the documentation and preservation of endangered languages. WELT is based on Words-Eye (Coyne and Sproat, 2001), a text-toscene tool that automatically generates 3D scenes from written input. WELT has two modes of operation. In the first mode, English input automatically generates a picture which can be used to elicit a description in the target language. In the second mode, the linguist formally documents the grammar of an endangered language, thereby creating a system that takes input in the endangered language and generates a picture according to the grammar; the picture can then be used to verify the grammar with native speakers. We will demonstrate WELT's use on scenarios involving Arrernte and Nahuatl.
Proceedings of the 13th Linguistic Annotation Workshop, 2019
We address the issue of acquiring quality annotations of hedging words and phrases, linguistic ph... more We address the issue of acquiring quality annotations of hedging words and phrases, linguistic phenomenona in which words, sounds, or other constructions are used to express ambiguity or uncertainty. Due to the limited availability of existing corpora annotated for hedging, linguists and other language scientists have been constrained as to the extent they can study this phenomenon. In this paper, we introduce a new method of acquiring hedging annotations via crowdsourcing, based on reformulating the task of labeling hedges as a simple word sense disambiguation task. We also introduce a new hedging corpus we have constructed by applying this method, a collection of forum posts annotated using Amazon Mechanical Turk. We found that the crowdsourced judgments we obtained had an inter-annotator agreement of 92.89% (Fleiss' Kappa=0.751) and, when comparing a subset of these annotations to an expert-annotated gold standard, an accuracy of 96.65%.
LREC ... International Conference on Language Resources & Evaluation : [proceedings]. International Conference on Language Resources and Evaluation, 2020
This paper proposes a representation framework for encoding spatial language in radiology based o... more This paper proposes a representation framework for encoding spatial language in radiology based on frame semantics. The framework is adopted from the existing SpatialNet representation in the general domain with the aim to generate more accurate representations of spatial language used by radiologists. We describe Rad-SpatialNet in detail along with illustrating the importance of incorporating domain knowledge in understanding the varied linguistic expressions involved in different radiological spatial relations. This work also constructs a corpus of 400 radiology reports of three examination types (chest X-rays, brain MRIs, and babygrams) annotated with fine-grained contextual information according to this schema. Spatial trigger expressions and elements corresponding to a spatial frame are annotated. We apply BERT-based models (BERT-Base and BERT- Large) to first extract the trigger terms (lexical units for a spatial frame) and then to identify the related frame elements. The resu...
When you listen to the evening news, or read a newspaper, book or web site, there is a good chanc... more When you listen to the evening news, or read a newspaper, book or web site, there is a good chance that you will hear or see a term — perhaps a name, perhaps a technical term — that you have never seen before. Such words are often novel or rare and are often names (of people, places, organizations. . . ). They are hard for humans to process, but they are even harder for automatic speech and language processing systems. For a single language, a speech recognition or text-to-speech system needs to know how to pronounce a word to recognize or say it. For two languages, in particular a pair with different writing systems, a search engine or document summarizer needs to know how to transcribe one word to another to retrieve or distill across languages. For example, the soccer player written in English text as Maciej Zurawski would appear as 마치에이주라브스키 (ma-chi-e-i ju-ra-beu-seu-ki) in Korean In this project we attack both problems – unusual term pronunciation and term transcription. For pr...
We describe our evaluation of the WordsEye text-to-scene generation system. We address the proble... more We describe our evaluation of the WordsEye text-to-scene generation system. We address the problem of evaluating the output of such a system vs. simple search methods to find a picture to illustrate a sentence. To do this, we constructed two sets of test sentences: a set of crowdsourced imaginative sentences and a set of realistic sentences extracted from the PASCAL image caption corpus (Rashtchian et al., 2010). For each sentence, we compared sample pictures found using Google Image Search to those produced by WordsEye. We then crowdsourced judgments as to which picture best illustrated each sentence. For imaginative sentences, pictures produced by WordsEye were preferred, but for realistic sentences, Google Image Search results were preferred. We also used crowdsourcing to obtain a rating for how well each picture illustrated the sentence, from 1 (completely correct) to 5 (completely incorrect). WordsEye pictures had an average rating of 2.58 on imaginative sentences and 2.54 on r...
We present experiments in incrementally learning a dependency parser. The parser will be used in ... more We present experiments in incrementally learning a dependency parser. The parser will be used in the WordsEye Linguistics Tools (WELT) (Ulinski et al., 2014) which supports field linguists documenting a language’s syntax and semantics. Our goal is to make syntactic annotation faster for field linguists. We have created a new parallel corpus of descriptions of spatial relations and motion events, based on pictures and video clips used by field linguists for elicitation of language from native speaker informants. We collected descriptions for each picture and video from native speakers in English, Spanish, German, and Egyptian Arabic. We compare the performance of MSTParser (McDonald et al., 2006) and MaltParser (Nivre et al., 2006) when trained on small amounts of this data. We find that MaltParser achieves the best performance. We also present the results of experiments using the parser to assist with annotation. We find that even when the parser is trained on a single sentence from...
This paper introduces SpatialNet, a novel resource which links linguistic expressions to actual s... more This paper introduces SpatialNet, a novel resource which links linguistic expressions to actual spatial configurations. SpatialNet is based on FrameNet (Ruppenhofer et al., 2016) and VigNet (Coyne et al., 2011), two resources which use frame semantics to encode lexical meaning. SpatialNet uses a deep semantic representation of spatial relations to provide a formal description of how a language expresses spatial information. This formal representation of the lexical semantics of spatial language also provides a consistent way to represent spatial meaning across multiple languages. In this paper, we describe the structure of SpatialNet, with examples from English and German. We also show how SpatialNet can be combined with other existing NLP tools to create a text-to-scene system for a language.
Proceedings of the Workshop on Computational Semantics beyond Events and Roles, 2018
We describe a novel method for identifying hedge terms using a set of manually constructed rules.... more We describe a novel method for identifying hedge terms using a set of manually constructed rules. We present experiments adding hedge features to a committed belief system to improve classification. We compare performance of this system (a) without hedging features, (b) with dictionary-based features, and (c) with rule-based features. We find that using hedge features improves performance of the committed belief system, particularly in identifying instances of non-committed belief and reported belief.
Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages ... more Using the example of Murrinh-Patha, Seiss (2011) illustrates how Australian Aboriginal languages can shed light on the morphology-syntax interface: one aspect of their polysynthetic nature is that information often encoded in phrases and clauses in other languages is instead found in a single morphological word. In this paper, we look at another instance, the Australian Aboriginal language Arrernte, and in particular at complex predicates within the language, to examine the implications for the morphology-syntax interface. Following from this, we show how a glue semantics-based approach can be applied to Arrernte complex predicates, in a way that fits neatly with the use of glue semantics to model lexical functions in LFG in a multilingual natural language generation environment. 1 Authors' note: Other abbreviations we use in examples are ASSOC MOTION = Associated Motion, ATTEN = Attenuative reduplicant, DO.COMING = type of Associated Motion marker, EMPH = emphasis particle, FOC = focus particle, INTENS = intensifier, ITER = iterative, IV = intransitive verbaliser, NOMLSR = nominaliser, NUM = number, PRES = present tense, PRIOR.MOTION = type of Associated Motion marker, PST = past tense, PV = preverb, QUICK:DO&GO.BACK = type of Associated Motion marker, RECIP = reciprocal, REFL = reflexive, SR = switch reference, SS = same subject, TV = transitive verbaliser, V = verb.
In this paper, we approach the problem of classifying emotion in image descriptions. A method is ... more In this paper, we approach the problem of classifying emotion in image descriptions. A method is proposed to perform 6-way emotion classification and is tested against two labeled datasets: a corpus of blog posts mined from LiveJournal and a corpus of descriptive texts of computer generated scenes. We perform feature selection using the mRMR technique and then use a multi-class linear predictor to classify posts among the Ekman Big Six emotions (happiness, sadness, anger, surprise, fear, and disgust) [9]. We find that TFIDF scores on lexical features and LIWC scores are much more helpful in emotion classification than using scores calculated from existing sentiment dictionaries, and that our proposed method performs significantly better than a baseline classifier that chooses the majority class. On the blog posts, we achieve 40% accuracy, and on the corpus of image descriptions, we achieve up to 63% accuracy.
Uploads
Papers by Morgan Ulinski