Video Annotation

In this paper  we introduce the current  results of an ongoing three-year research and development project on automatic annotation of
human nonverbal behavior.  The present  output of the project is a
tool  that  provides algorithms and  graphical user interface  for the
generation  of ground-truth data  about  the subset of  facial and
body activities.  These data are essential for  the  experts who are
committed to unraveling  the  complexity  of  the  linkage  between
the  psychophysiological  state  and  the  nonverbal behavior  of a
human.  Our work relied on a Kinect sensor, which  computes
depth maps  together  with the coordinates of the  body  joints and
facial points.  Local binary patterns are  then  extracted from the
regions  of interests of  a  facial  video, which are  either  spatio-temporally aligned  with  the  depth maps  or  calculated  using  the
Active Shape Model. Another key idea of the proposed tool is that
the extracted feature vector is  semantically  associated with
ontological  concepts  in perspective  providing  annotations  for
most of the nonverbal activities.

If the purpose of reflection is to improve teaching, it is essential to understand how video-aided reflection influences teacher change. Yet, there is limited research addressing how video analysis influences the change process. The purpose of this study was to gain an in depth understanding of how video influences the process of teacher change. Teachers in three different teaching environments engaged in semester-long video-reflection groups. Through a descriptive analysis of these meetings, participants' own video-analyses, and individual interviews, six over-arching themes emerged across the different environments that describe the change process. Teachers reported that video encouraged change because it helped them: (a) focus their analysis, (b) see their teaching from a new perspective, (c) trust the feedback they received, (d) feel accountable to change their practice, (e) remember to implement changes, and (f) see their progress. We discuss how these results may help researchers and educators understand how video can be used to encourage teacher improvement.

This paper focuses on methods applied for sign language video processing. In the first part, we present a robust traking method which detects and tracks the hands and face of a person performing Signs' language communication. The method uses a model of skin color and three particle filters to track the two hands and the face, with re-sampling and annealed

Video games today increasingly situate play in imaginative 3D worlds. As a result, the industry devotes much time and effort to level design. However, this subject has received very little research. Documentation on the process of level design or how designers push or pull players through a level within a video game is very sparse. In this paper, we propose a set of design patterns for level design. The patterns were developed based on a process involving interviews with game designers as well as gameplay analysis of different games. We established face validity of these patterns through expert review; we also established reliability using inter-rater agreement. In addition, we also developed a timeline video annotation method based on these patterns. This visualization method provides a very effective approach to view players' play style and preference as well as level design problems. The patterns as well as the visualization method will be discussed in the paper.

Having developed gIBIS and QOC as Argumentation-based Design Rationale (DR) approaches in the late 1980s/mid-1990s, we report on the subsequent evolution of this DR paradigm. Our primary claim is that with robust hypermedia software being used by professionals in diverse contexts, representations of this sort are demonstrably both practical and useful to capture. The Compendium approach has maintained a strong

Ogni sapere ha a che fare anche con il vedere e ogni nostro rapporto con il mondo passa attraverso le immagini.
La video-analisi, approccio rinnovato ed emergente nel panorama della ricerca educativa ‒ anche grazie all’ampia disponibilit̀à di nuovi strumenti tecnologici che consentono di recuperare con facilit̀a immagini e video in contesti reali e situati ‒, va oltre il semplificato metodo di raccolta dati. Aspetti controversi, relativi anche a problematiche di privacy e tutela dei dati personali o di resistenze all’uso di alcuni strumenti e diffidenze di matrice tecnologica, sono questioni connesse a prospettive visuali, che fanno dell’immagine i presupposti di una rappresentazione del śe, degli eventi, del mondo attorno.
La video-analisi è una finestra sulle interazioni sociali ed educative, come quelle che avvengono in una classe scolastica.
Partendo da alcune riflessioni teoriche, nella prima parte si illustrano gli aspetti epistemologici della Visual Research Method, dalla etno-metodologia visuale, alla video ricerca partecipativa, alla videography, alla visual framing theory, valutando percorsi inesplorati per affrontare le ricerche con l’uso di video e immagini.
Nella seconda parte si presentano alcune ricerche sulle tematiche educative esemplificando il ragionamento conoscitivo che si può applicare in un lavoro di ricerca tramite dati visuali.
L’uso di software nella e per la video-analisìe illustrato con alcuni passaggi teorico-metodologici, anche in funzione di guida operativa per studenti, laureandi, docenti e ricercatori che si accostano alla video-analisi per la ricerca scientifica.

Video annotation tools are often compared in the literature, however, most reviews mix unstructured, semi-structured, and the very few structured annotation software. This paper is a comprehensive review of video annotations tools generating structured data output for video clips, regions of interest, frames, and media fragments, with a focus on Linked Data support. The tools are compared in terms of supported input and output data formats, expressivity, annotation specificity, spatial and temporal fragmentation, the concept mapping sources used for Linked Open Data (LOD) interlinking, provenance data support, and standards alignment. Practicality and usability aspects of the user interface of these tools are highlighted. Moreover, this review distinguishes extensively researched yet discontinued semantic video annotation software from promising state-of-the-art tools that show new directions in this increasingly important field.

This paper outlines our experiences with applying collaborative tagging in e-learning systems to supplement more traditional metadata gathering approaches. Over the last 10 years, the learning object paradigm has emerged in e-learning and has caused standards bodies to focus on creating metadata repositories based upon strict domain-free taxonomies. We argue that the social collection phenomena and flexible metadata standards are key in collecting the kinds of metadata required for adaptable online learning. This paper takes a broad look at tagging within elearning. It first looks at the implications for tagging within the domain through an analysis of tags students provided when classifying learning objects. Next, it looks at two case studies based on novel interfaces for applying tagging. These two systems emphasize tags being applied within learning content through the use of a highlighting metaphor.

Preservation of intangible cultural heritage, such as music and dance, requires encoding of background knowledge together with digitized records of the performances. We present an ontology based approach for designing a cultural heritage repository for that purpose. Since dance and music are recorded in multimedia format, we use Multimedia Web Ontology Language (MOWL) to encode the domain knowledge. We propose an architectural framework that includes a method to construct the ontology with a labelled set of training data and use of the ontology to automatically annotate new instances of digital heritage artefacts. The annotations enable creation of a semantic navigation environment in a cultural heritage repository. We have demonstrated the efficacy of our approach by constructing an ontology for the cultural heritage domain of Indian Classical Dance, and have developed a browsing application for semantic access to the heritage collection of Indian dance videos.

During the past two decades, information visualisation (InfoVis) re- search has created new techniques and methods to support data- intensive analyses in science, industry and government. These have enabled a wide range of analyses tasks to be executed, with tasks varying in terms of the type and volume of data involved. However, the majority of this research has focused on static datasets, and the analysis and visualisation tasks tend to be carried out by trained expert users. In more recent years, social changes and technological advances have meant that data have become more and more dynamic, and are consumed by a wider audience. Examples of such dynamic data streams include e-mails, status updates, RSS 1 feeds, versioning systems, social networks and others. These new types of data are used by populations that are not specifically trained in information visualization. Some of these people might consist of casual users, while others might consist of people deeply involved with the data, but in both cases, they would not have received formal training in information visualization. For simplicity, throughout this dissertation, I refer to the people (casual users, novices, data experts) who have not been trained in information visualisation as non-experts.
These social and technological changes have given rise to multiple challenges because most existing visualisation models and techniques are intended for experts, and assume static datasets. Few studies have been conducted that explore these challenges. In this dissertation, with my collaborators, I address the question: Can we empower non- experts in their use of visualisation by enabling them to contribute to data stream analysis as well as to create their own visualizations?
The first step to answering this question is to determine whether people who are not trained in information visualisation and the data sciences can conduct useful dynamic analysis tasks using a visualisation system that is adapted to support their tasks. In the first part of this dissertation I focus on several scenarios and systems where different sized crowds of non-InfoVis experts users (20 to 300 and 2 000 to 700 000 people) use dynamic information visualisation to analyse dynamic data.
Another important issue is the lack of generic design principles for the visual encoding of dynamic visualization. In this dissertation I design, define and explore a design space to represent dynamic data for non-experts. This design space is structured by visual tokens representing data items that provide the constructive material for the assembly over time of different visualizations, from classic represen- tations to new ones. To date, research on visual encoding has been focused on static datasets for specific tasks, leaving generic dynamic approaches unexplored and unexploited.
In this thesis, I propose construction as a design paradigm for non- experts to author simple and dynamic visualizations. This paradigm is inspired by well-established developmental psychological theory as well as past and existing practices of visualisation authoring with tangible elements. I describe the simple conceptual components and processes underlying this paradigm, making it easier for the human computer interaction community to study and support this process for a wide range of visualizations. Finally, I use this paradigm and tangible tokens to study if and how non-experts are able to create, discuss and update their own visualizations. This study allows us to refine our previous model and provide a first exploration into how non-experts perform a visual mapping without software. In summary, this thesis contributes to the understanding of dynamic visualisation for non-expert users.

A one-page brief guide (in Italian first and English below) on how to start transcribing with the annotation tool ELAN (https://archive.mpi.nl/tla/elan)

Una guida essenziale (in italiano e inglese) per iniziare a trascrivere con lo strumento di annotazione ELAN (https://archive.mpi.nl/tla/elan)

We present a collaborative tool for choreographers and dancers, the Choreographer's Notebook, which allows multimodal annotation of rehearsal videos during the dance production process. The dance production process is a creative process in which exploration and expression are paramount and we describe issues we have observed over four different dance production cycles. The Choreographer's Notebook addresses the recurring problem of scarcity of rehearsal time and space, which is a limiting factor for exploration and expression. The tool is a web-based video application that enables choreographers to provide multimodal, contextsensitive instructions and feedback to the dancers outside of formal rehearsal time. The novelty of this tool is its ability to provide a multimodal collaborative video-based workflow. This article also presents a thorough ethnographic investigation of the dance production process and results from the use of the Choreographer's Notebook as an artifact within that process.

Vision-based tracking is a basic elementary task in many computer vision-based applications such as video surveillance and monitoring, sensing and navigation in robotics, video compression, video annotation, and many more. However, reliable recovery of targets and their trajectories in an uncontrolled environment is affected by a wide range of conditions exhibited by the environment such as sudden illumination changes and clutter. This work addresses the problem of (i) combining information from a set of cues in order to obtain reasonably accurate estimates of multiple targets in uncontrolled environments and (ii) a collection of data association methods for cues containing less information for robust tracking through persistent clutter. Specifically, we introduce a novel geometric template constrained data association method for robust tracking of point features, while using the Joint Probabilistic Data Association (JPDA) method for blob cue measurements. Extensive experimental validation of the tracking and the data association framework is presented in the work for several real-world outdoor traffic intersection image sequences.

Effective usage of multimedia digital libraries has to deal with the problem of building efficient content annotation and retrieval tools. MOM (Multimedia Ontology Manager) is a complete system that allows the creation of multimedia ontologies, supports automatic annotation and creation of extended text (and audio) commentaries of video sequences, and permits complex queries by reasoning on the ontology.

The COVID-19 pandemic has forced much of the academic world to transition into online operations and online learning. Interactions between the teachers and students are carried out via online video conferencing software where possible. All video conferencing software available today is designed for general usage and not for classroom teaching and learning. In this study, we analyzed the features and effectiveness of more than a dozen major video conferencing software that are being used to replace the physical face-to-face learning experiences. While some of the video conferencing software has pause feature but none allow annotation and segmentation of the recording. We propose tagging and annotation during the live streaming to improve direct access to any portion of the recorded video. We also propose automatic segmentation of the video based on the tagging so that the video is short, targeted, and can easily be identified.

The past decade has witnessed a rapid proliferation of video cameras in all walks of life and has resulted in a tremendous explosion of video content. Several applications such as content-based video annotation and retrieval, highlight extraction and video summarization require recognition of the activities occurring in the video. The analysis of human activities in videos is an area with increasingly important consequences from security and surveillance to entertainment and personal archiving. Several challenges at various levels of processing-robustness against errors in low-level processing, view and rate-invariant representations at midlevel processing and semantic representation of human activities at higher level processing-make this problem hard to solve. In this review paper, we present a comprehensive survey of efforts in the past couple of decades to address the problems of representation, recognition, and learning of human activities from video and related applications. We discuss the problem at two major levels of complexity: 1) "actions" and 2) "activities." "Actions" are characterized by simple motion patterns typically executed by a single human. "Activities" are more complex and involve coordinated actions among a small number of humans. We will discuss several approaches and classify them according to their ability to handle varying degrees of complexity as interpreted above. We begin with a discussion of approaches to model the simplest of action classes known as atomic or primitive actions that do not require sophisticated dynamical modeling. Then, methods to model actions with more complex dynamics are discussed. The discussion then leads naturally to methods for higher level representation of complex activities.

Video indexing and retrieval have a wide spectrum of promising applications, motivating the interest of researchers worldwide. This paper offers a tutorial and an overview of the landscape of general strategies in visual content-based video indexing and retrieval, focusing on methods for video structure analysis, including shot boundary detection, key frame extraction and scene segmentation, extraction of features including static key frame features, object features and motion features, video data mining, video annotation, video retrieval including query interfaces, similarity measure and relevance feedback, and video browsing. Finally, we analyze future research directions.

Increasingly computer vision discipline needs annotated video databases to realize assessment tasks. Manually providing ground truth data to multimedia resources is a very expensive work in terms of effort, time and economic resources. Automatic and semi-automatic video annotation and labeling is the faster and more economic way to get ground truth for quite large video collections. In this paper, we describe a new automatic and supervised video annotation tool. Annotation tool is a modified version of ViPER-GT tool. ViPER-GT standard version allows manually editing and reviewing video metadata to generate assessment data. Automatic annotation capability is possible thanks to an incorporated tracking system which can deal the visual data association problem in real time. The research aim is offer a system which enables spends less time doing valid assessment models.

Nowadays visual information is gaining much more importance than textual information as pictures convey more than words. As image has richer content than text, Video database is increasing exponentially over the Internet. Content-based Video retrieval (CBVR) from large video storage has become a challenging and important issue for researchers over worldwide. There is lack of tools so it is difficult to distinguish between low and high level feature. There has been lot of CBVR systems developed but still there is a lack of efficient retrieval system, as many retrieval techniques are not suitable for extensive search and so it deteriorates the overall performance of system. In this paper we have given an outline of the sequential steps of existing CBVR systems.

As video content on the web continues to expand, it is increasingly important to properly annotate videos for effective search and mining. While the idea of annotating static imagery with keywords is relatively well known, the idea of annotating videos with natural language keywords to enhance search is an important emerging problem with great potential to improve the quality of video search. However, leveraging web-scale video datasets for automated annotation also presents new challenges and requires methods specialized for scalability and efficiency. In this chapter we review specific, state of the art techniques for video analysis, feature extraction and classification suitable for extremely large scale automated video annotation. We also review key algorithms and data structures that make truly large scale video search possible. Drawing from these observations and insights, we present a complete method for automatically augmenting keyword annotations to videos using previous annotations for a large collection of videos. Our approach is designed explicitly to scale to YouTube sized datasets and we present some experiments and analysis for keyword augmentation quality using a corpus of over 1.2 million YouTube videos. We demonstrate how the automated annotation of webscale video collections is indeed feasible, and that an approach combining visual features with existing textual annotations yields better results than unimodal models.

Researchers throughout the years have been searching for methods to improve the available technologies to ease the users worldwide. In medical industries, researchers have made many improvements and development in medical equipment's to ease the medical professionals in serving patients. This work is the continuation of the previous research work in providing complete set of data for analysis on human behavior to set an accurate threshold for device detection. In this work, heart rate is monitored to all age groups for both genders wirelessly from remote locations. Results shown that these various age group is giving average reading that complies to the benchmark reading and is able to provide the necessary threshold value for future device improvement and enhancement.

AdA Filmontologya machine-readable Film Analysis Vocabulary for Video Annotation "A film is difficult to explain because it is easy to understand"this famous quote by Christian Metz (1991, 69) sums up a key challenge of film analysis. The complexity of audiovisual ,

In this paper, we present the dynamic pictorial ontology paradigm for video annotation. Ontologies are often used to describe a given domain for different goals, including description of multimedia data. In the case of video annotation, the visual knowledge cannot be described using only abstract concepts but is more effectively represented in a visual form. To this aim, we introduce visual concepts, elicited from the data set as the most representative prototypes that specialize abstract concepts. The ontology created is intrinsically dynamic since it must embrace the perceptual and visual experience during annotation. Thus visual concepts can change, adapting to the multimedia content analyzed. Motivation for this new ontology paradigm are discussed together with a proposal of a framework for ontology creation, maintenance, and automatic annotation of video. The creation and usage of dynamic pictorial ontologies have been tested for soccer domain exploiting low level perceptual features and higher level domain features.

In this paper, we discuss the need for a standardised system of  nnotation for sign language corpora. Although several tools exist for the annotation of video data (such as ELAN or iLex), and some existing projects have annotation guidelines (e.g., Crasborn et al., 2007; Johnston, 2010), a widely adopted standard is currently unavailable. First, we discuss the purpose of a set of unified annotation standards for sign languages: such standards would provide a shared set of conventions for the easy exchange of data across different sign language corpus projects and may increase consistency within corpora. Next, we discuss the properties that would define a good set of shared annotation conventions (Beckman et al., 2009). We examine some of the proposed annotation standards for spoken language description, such as the ToBI conventions for prosody and the Leipzig Glossing Rules for morpho-syntax. Lastly, we discuss the relationship between theory and description. Dryer (2006) pointed out that  linguists often contrast ‘theoretical linguistics’ with ‘descriptive’ work. But if one accepts the argument that there is indeed no ‘atheoretical description’, then sign language linguists need to agree on a shared theory for basic sign  language description, and how this translates into annotation practices.

Multimodal aspects of human communication are key in several applications of Natural Language Processing, such as Machine Translation and Natural Language Generation. Despite recent advances in integrating multimodality into Computational Linguistics, the merge between NLP and Computer Vision techniques is still timid, especially when it comes to providing fine-grained accounts for meaning construction. This paper reports on research aiming to determine appropriate methodology and develop a computational tool to annotate multimodal corpora according to a principled structured semantic representation of events, relations and entities: FrameNet. Taking a Brazilian television travel show as corpus, a pilot study was conducted to annotate the frames that are evoked by the audio and the ones that are evoked by visual elements. We also implemented a Multimodal Annotation tool which allows annotators to choose frames and locate frame elements both in the text and in the images, while keepi...

Content-Based Image Retrieval (CBIR) locates, retrieves and displays images alike to one given as a query, using a set of features. It demands accessible data in medical archives and from medical equipment, to infer meaning after some processing. A problem similar in some sense to the target image can aid clinicians. CBIR complements text-based retrieval and improves evidence-based diagnosis, administration, teaching, and research in healthcare. It facilitates visual/automatic diagnosis and decision-making in real-time remote consultation/screening, store-and-forward tests, home care assistance and overall patient surveillance. Metrics help comparing visual data and improve diagnostic. Specially designed architectures can benefit from the application scenario. CBIR use calls for file storage standardization, querying procedures, efficient image transmission, realistic databases, global availability, access simplicity, and Internet-based structures. This chapter recommends important and complex aspects required to handle visual content in healthcare.

Losing My Religion – Einsatz der Videoannotationsdatenbank Pan.do/ra in der kunstgeschichtlichen Analyse von Musikvideos 

Abstract
In dem vorliegenden Beitrag werden die Möglichkeiten, sowie der daraus resultierende Mehrwert der Anwendung eines Videoannotationsdatenbanksystems (Pan.do/ra) in der kulturwissenschaftlichen Untersuchung von Bewegtbildmaterial beleuchtet. Teil 1 (Thorsten Wübbena) beschäftigt sich konkret am Beispiel eines Musikvideos mit einer Auswahl der Nutzungsmöglichkeiten, während der anschließende Teil 2 (Matthias Arnold, Eric Decker) die Software in einem allgemeineren Kontext darstellt.

In this paper we discuss the potential and added value to art historical research in utilizing the Pan.do/ra video annotation system. In part 1 Thorsten Wübbena presents a variety of usage possibilities taking the annotation of a music video as example. In part 2 Matthias Arnold and Eric Decker introduce the software on a more general level.

Diese Präesntation ist Teil eines Vortrags, der gemeinsam mit Thorsten Wübbena auf der ersten Dhd-Tagung "Digital Humanities – methodischer Brückenschlag oder 'feindliche Übernahme'? Chancen und Risiken der Begegnung zwischen Geisteswissenschaften und Informatik", Dienstag, 25.03.2014 bis Freitag, 28.03.2014, Universität Passau, gehalten wurde.

In this technical demonstration we show an integrated web system for video search and annotation based on ontologies. The system is composed by three components: the Orione 1 ontology-based search engine, the Sirio 2 search interface, and the Pan 3 web-based video annotation tool. The system is currently being developed within the EU IM3I 4 project. The goal of the system is to provide an integrated environment for video annotation and retrieval of videos, for both technical and non-technical users. In fact, the search engine has different interfaces that permit different query modalities: free-text, natural language, graphical composition of concepts using Boolean and temporal relations and query by visual example. In addition, the ontology structure is exploited to encode semantic relations between concepts permitting, for example, to expand queries to synonyms and concept specializations. The annotation tool can be used to create ground-truth annotations to train automatic annotations systems, or to complement the results of automatic annotation, e.g. adding geolocalized information.

Labeling persons appearing in video frames with names detected in a corresponding video transcript helps improving video content annotation and search tasks. We develop a face naming method that learns from labeled and unlabeled examples using iterative label propagation in a graph of connected faces or name-face pairs. The advantage of this method is that it uses few labeled data points and incorporate the unlabeled data points during the learning process. Anchor detection boosts the face naming performance. On BBC News videos, the proposed method yields better results than a Support Vector Machine classifier and a nearest neighbor classifier trained on the same labeled data.

Due to the profile of strengths and weaknesses indicative of autism spectrum disorders (ASD), technology may play a key role in ameliorating communication difficulties with this population. This paper documents coding guidelines established through cross-disciplinary work focused on facilitating communication development in children with ASD using computerized feedback. The guidelines, referred to as A 3 (pronounced A-Cubed) or Annotation for

VIDANN is a computer program that allows participants to watch a video on a standard TV and to write their annotations (thought /feeling entries) on paper attached to a writing tablet. The system is designed as a Microsoft ActiveX module. It can be further adapted by the individual researcherthrough the use of a VBScript. All data, including the participant's handwriting, are stored in an XML database. An accompanying Wizard has been designed that enables researchers to generate VBScripts for standard configurations.

The paper focuses on an overview of the different existing methods in content-based video retrieval. During the last decade there was a~rapid growth of video posted on the Internet. This imposes urgent demands on video retrieval. Video has a complex structure and can express the same idea in different ways. This makes the task of searching for video more complicated. Video titles and text descriptions cannot give the hole information about objects and events in the video. This creates a need for content-based video retrieval. There is a semantic gap between low-level video features, that can be extracted, and the users' perception. Complex content-based video retrieval can be regarded as the bridge between traditional retrieval and semantic-based video retrieval.

Keywords: frames; near-duplicates video; scenes; shots; video annotation; video classification; video mining; video reranking; video retrieval.

В статье предлагается обзор различных существующих методов ассоциативного поиска по видео. В течение прошлого десятилетия наблюдался стремительный рост количества видео размещаемых в Интернете, что создало острую необходимость в появлении поиска по видео. Видео имеет сложную структуру. Одна и та же информация может быть выражена различными способами. Это серьёзно усложняет задачу видео-поиска. Заголовки и описания видео не могут дать полного представления о самом видео, что влечет за собой необходимость использования ассоциативного поиска по видео. Существует семантический разрыв между низкоуровневыми характеристиками видео и восприятием пользователей. Комплексный ассоциативный видео-поиск может рассматриваться как связующее звено между обычным поиском и смысловым поиском по видео.

Ключевые слова: анализ видео; аннотирование видео; видео-поиск; кадры; классификация видео; нечеткие дубликаты видео; ранжирование видео; сцены; съёмки.