Proceedings of the Thirteenth International Workshop on Multimedia Data Mining - MDMKDD '13, 2013
ABSTRACT Online social networks have attracted attention of people from both the academia and rea... more ABSTRACT Online social networks have attracted attention of people from both the academia and real world. In particular, the rich multimedia information accumulated in recent years provides an easy and convenient way for more active communication between people. This offers an opportunity to research people's behaviors and activities based on those multimedia content, which can be considered as social imagematics. One emerging area is driven by the fact that these massive multimedia data contain people's daily sentiments and opinions. However, existing sentiment analysis typically only pays attention to the textual information regardless of the visual content, which may be more informative in expressing people's sentiments and opinions. In this paper, we attempt to analyze the online sentiment changes of social media users using both the textual and visual content. In particular, we analyze the sentiment changes of Twitter users using both textual and visual features. An empirical study of real Twitter data sets indicates that the sentiments expressed in textual content and visual content are correlated. The preliminary results in this paper give insight into the important role of visual content in online social media.
While on the go, more and more people are using their phones to enjoy ubiquitous location-based s... more While on the go, more and more people are using their phones to enjoy ubiquitous location-based services (LBS). One of the fundamental problems of LBS is localization. Researchers are now investigating ways to use a phone-captured image for localization as it contains more scene context information than the embedded sensors. In this paper, we present a novel approach to mobile visual localization that accurately senses geographic scene context according to the current image (typically associated with a rough GPS position). Unlike most existing visual localization methods, the proposed approach is capable of providing a complete set of more accurate parameters about the scene geo-including the actual locations of both the mobile user and perhaps more importantly the captured scene along with the viewing direction. Our approach takes advantage of advanced techniques for large-scale image retrieval and 3D model reconstruction from photos. Specifically, we first perform joint geo-visual clustering in the cloud to generate scene clusters, with each scene represented by a 3D model. The 3D scene models are then indexed using a visual vocabulary tree structure. The phone-captured image is used to retrieve the relevant scene models, then aligned with the models, and further registered to the real-world map. Our approach achieves an estimation accuracy of user location within 14 meters, viewing direction within 9 degrees, and scene location within 21 meters. Such a complete set of accurate geo-parameters can lead to various LBS applications for routing that cannot be achieved with most existing methods. In particular, we showcase three novel applications: 1) accurate selflocalization, 2) collaborative localization for rendezvous routing, and 3) routing for photographing. The evaluations through user studies indicate these applications are effective for facilitating the perfect rendezvous for mobile users.
In this paper, we present a systematic framework for recognizing realistic actions from videos "i... more In this paper, we present a systematic framework for recognizing realistic actions from videos "in the wild." Such unconstrained videos are abundant in personal collections as well as on the web. Recognizing action from such videos has not been addressed extensively, primarily due to the tremendous variations that result from camera motion, background clutter, changes in object appearance, and scale, etc. The main challenge is how to extract reliable and informative features from the unconstrained videos. We extract both motion and static features from the videos. Since the raw features of both types are dense yet noisy, we propose strategies to prune these features. We use motion statistics to acquire stable motion features and clean static features. Furthermore, PageRank is used to mine the most informative static features. In order to further construct compact yet discriminative visual vocabularies, a divisive information-theoretic algorithm is employed to group semantically related features. Finally, AdaBoost is chosen to integrate all the heterogeneous yet complementary features for recognition. We have tested the framework on the KTH dataset and our own dataset consisting of 11 categories of actions collected from YouTube and personal videos, and have obtained impressive results for action recognition and action localization. Figure 1: Examples of our YouTube action dataset consist of 11 categories with about 1160 videos.
Social multimedia hosting and sharing websites, such as Flickr, Facebook, Youtube, Picasa, ImageS... more Social multimedia hosting and sharing websites, such as Flickr, Facebook, Youtube, Picasa, ImageShack and Photobucket, are increasingly popular around the globe. A major trend in the current studies on social multimedia is using the social media sites as a source of huge amount of labeled data for solving large scale computer science problems in computer vision, data mining and multimedia. In this paper, we take a new path to explore the global trends and sentiments that can be drawn by analyzing the sharing patterns of uploaded and downloaded social multimedia. In a sense, each time an image or video is uploaded or viewed, it constitutes an implicit vote for (or against) the subject of the image. This vote carries along with it a rich set of associated data including time and (often) location information. By aggregating such votes across millions of Internet users, we reveal the wisdom that is embedded in social multimedia sites for social science applications such as politics, economics, and marketing. We believe that our work opens a brand new arena for the multimedia research community with a potentially big impact on society and social sciences.
Geo-tagging is a fast-emerging trend in digital photography and community photo sharing. The pres... more Geo-tagging is a fast-emerging trend in digital photography and community photo sharing. The presence of geographically relevant metadata with images and videos has opened up interesting research avenues within the multimedia and computer vision domains. In this paper, we survey geo-tagging related research within the context of multimedia and along three dimensions: 1) Modalities in which geographical information can be extracted, 2) Applications that can benefit from the use of geographical information, and 3) The interplay between modalities and applications. Our survey will introduce research problems and discuss significant approaches. We will discuss the nature of different modalities and lay out factors that are expected to govern the choices with respect to multimedia and vision applications. Finally, we discuss future research directions in this field. 8
received a bachelor's degree (summa cum laude) from the University of Minnesota, an M.Sc. degree ... more received a bachelor's degree (summa cum laude) from the University of Minnesota, an M.Sc. degree in mathematics, an M.Sc. degree in computer science, and a Ph.D. degree in medical information sciences from Stanford University. Her research interests include statistical modeling and learning, data mining, computational biology, image processing, and image annotation and retrieval.
Sentiment analysis of online user generated content is important for many social media analytics ... more Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.
Semantic scene classification based only on low-level vision cues has had limited success on unco... more Semantic scene classification based only on low-level vision cues has had limited success on unconstrained image sets. On the other hand, camera metadata related to capture conditions provides cues independent of the captured scene content that can be used to improve classification performance. We consider three problems, indoor-outdoor classification, sunset detection, and manmade-natural classification. Analysis of camera metadata statistics for images of each class revealed that metadata fields, such as exposure time, flash fired, and subject distance, are most discriminative for each problem. A Bayesian network is employed to fuse content-based and metadata cues in the probability domain and degrades gracefully even when specific metadata inputs are missing (a practical concern). Finally, we provide extensive experimental results on the three problems using content-based and metadata cues to demonstrate the efficacy of the proposed integrated scene classification scheme.
C onsiderable research has been devoted to the problem of multimedia indexing and retrieval in th... more C onsiderable research has been devoted to the problem of multimedia indexing and retrieval in the past decade. However, limited by the state of the art in image understanding, the majority of the existing content-based image retrieval (CBIR) systems have taken a relatively low-level approach and fallen short of higher-level interpretation and knowledge. Recent research has begun to focus on bridging the semantic and conceptual gap that exists between man and computer by integrating knowledge-based techniques, human perception, scene content understanding, psychology, and linguistics. In this article, we provide an overview of exploiting context for semantic scene content understanding.
In classic pattern recognition problems, classes are mutually exclusive by deÿnition. Classiÿcati... more In classic pattern recognition problems, classes are mutually exclusive by deÿnition. Classiÿcation errors occur when the classes overlap in the feature space. We examine a di erent situation, occurring when the classes are, by deÿnition, not mutually exclusive. Such problems arise in semantic scene and document classiÿcation and in medical diagnosis. We present a framework to handle such problems and apply it to the problem of semantic scene classiÿcation, where a natural scene may contain multiple objects such that the scene can be described by multiple class labels (e.g., a ÿeld scene with a mountain in the background). Such a problem poses challenges to the classic pattern recognition paradigm and demands a di erent treatment. We discuss approaches for training and testing in this scenario and introduce new metrics for evaluating individual examples, class recall and precision, and overall accuracy. Experiments show that our methods are suitable for scene classiÿcation; furthermore, our work appears to generalize to other classiÿcation problems of the same nature.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005
Automatic image orientation detection for natural images is a useful, yet challenging research ar... more Automatic image orientation detection for natural images is a useful, yet challenging research area. Humans use scene context and semantic object recognition to identify the correct image orientation. However, it is difficult for a computer to perform the task in the same way because current object recognition algorithms are extremely limited in their scope and robustness. As a result, existing orientation detection methods were built upon lowlevel vision features such as spatial distributions of color and texture. In addition, discrepant detection rates have been reported. We have developed a probabilistic approach to image orientation detection via confidence-based integration of low-level and semantic cues within a Bayesian framework. Our current accuracy is approaching 90% for unconstrained consumer photos, impressive given the findings of a psychophysical study conducted recently. The proposed framework is an attempt to bridge the gap between computer and human vision systems, and is applicable to other problems involving semantic scene content understanding.
Semantic scene classification based only on low-level vision cues has had limited success on unco... more Semantic scene classification based only on low-level vision cues has had limited success on unconstrained image sets. On the other hand, camera metadata related to capture conditions provides cues independent of the captured scene content that can be used to improve classification performance. We consider two problems: indoor-outdoor classification and sunset detection. Analysis of camera metadata statistics for images of each class revealed that metadata fields, such as exposure time, flash fired, and subject distance, are most discriminative for both indooroutdoor and sunset classification. A Bayesian network is employed to fuse content-based and metadata cues in the probability domain and degrades gracefully, even when specific metadata inputs are missing (a practical concern). Finally, we provide extensive experimental results on the two problems, using content-based and metadata cues to demonstrate the efficacy of the proposed integrated scene classification scheme.
Scene content understanding facilitates a large number of applications, ranging from content-base... more Scene content understanding facilitates a large number of applications, ranging from content-based image retrieval to other multimedia applications. Material detection refers to the problem of identifying key semantic material types (such as sky, grass, foliage, water, and snow in images). In this paper, we present a holistic approach to determining scene content, based on a set of individual material detection algorithms, as well as probabilistic spatial context models. A major limitation of individual material detectors is the significant number of misclassifications that occur because of the similarities in color and texture characteristics of various material types. We have developed a spatial context-aware material detection system that reduces misclassification by constraining the beliefs to conform to the probabilistic spatial context models. Experimental results show that the accuracy of materials detection is improved by 13% using the spatial context models over the individual material detectors themselves.
Knowledge of the semantic classification of an image can be used to improve the accuracy of queri... more Knowledge of the semantic classification of an image can be used to improve the accuracy of queries in content-based image organization and retrieval and to provide customized image enhancement. We developed an exemplar-based system for classifying sunset scenes. However, the performance of such a system depends largely on the size and quality of the set of training exemplars, which can be limited in practice. In addition, variations in scene content, as well as distracting regions, may exist in many testing images to prohibit good matches with the exemplars. We propose using simulated spatial and temporal image recomposition to address such issues. The recomposition schemes boost the recall of sunset images from a reasonably large data set by 10%, while holding the false positive rate constant.
Scene categorization to indoor vs outdoor may be approached by using low-level features for infer... more Scene categorization to indoor vs outdoor may be approached by using low-level features for inferring high-level information about the image. Low-level features such as color and texture have been used extensively in image understanding research, however, they cannot solve the problem completely. In this paper, we propose the use of a Bayesian network for integrating knowledge from low-level and semantic features for indoor vs outdoor classification of images. Using ground truth data for sky and grass detection, we demonstrate that the classification performance can be significantly improved when semantic features are employed in the classification process.
Prior research in scene classiÿcation has focused on mapping a set of classic low-level vision fe... more Prior research in scene classiÿcation has focused on mapping a set of classic low-level vision features to semantically meaningful categories using a classiÿer engine. In this paper, we propose improving the established paradigm by using a simpliÿed low-level feature set to predict multiple semantic scene attributes that are integrated probabilistically to obtain a ÿnal indoor/outdoor scene classiÿcation. An initial indoor/outdoor prediction is obtained by classifying computationally e cient, low-dimensional color and wavelet texture features using support vector machines. Similar low-level features can also be used to explicitly predict the presence of semantic features including grass and sky. The semantic scene attributes are then integrated using a Bayesian network designed for improved indoor/outdoor scene classiÿcation.
Current research in content-based semantic image understanding is largely confined to exemplar-ba... more Current research in content-based semantic image understanding is largely confined to exemplar-based approaches built on low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features is expected to raise semantic image understanding to a new level. Belief networks, or Bayesian networks (BN), have proven to be an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditional independence relationships. In this paper, we present a general-purpose knowledge integration framework that employs BN in integrating both low-level and semantic features. The efficacy of this framework is demonstrated via three applications involving semantic understanding of pictorial images. The first application aims at detecting main photographic subjects in an image, the second aims at selecting the most appealing image in an event, and the third aims at classifying images into indoor or outdoor scenes. With these diverse examples, we demonstrate that effective inference engines can be built within this powerful and flexible framework according to specific domain knowledge and available training data to solve inherently uncertain vision problems.
Measuring perceptual saliency of regions in a scene is important for determining regions of inter... more Measuring perceptual saliency of regions in a scene is important for determining regions of interest. Color, texture and shape cues are good low-level features for detecting saliency. While self saliency refers to intrinsic attributes of a region, relative saliency as used to measure how salient a region is relative to its surrounding and, thus, needs to be defined within a spatial context. A few spatial context models are investigated in this study. In particular, we propose an auto-scaled, amorphous neighborhood as the context model to obtain reliable measurements of relative saliency features. Comparison of three context models has shown that the proposed model is capable of generating predicates more consistent with perceived saliency
Belief networks, such as Bayes nets, have emerged as an effective knowledge representation and in... more Belief networks, such as Bayes nets, have emerged as an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditionally independence relationships. Current research in contentbased image processing and analysis is largely limited to low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features would be very useful. In this paper, we present a general knowledge integration framework that incorporates Bayes networks and has been used in two applications involving semantic understanding of consumer photographs. The first application aims at detecting main photographic subjects in an image and the second aims at selecting the most appealing image in an event. With these diverse examples, we demonstrate that effective inference engines can be built according to specific domain knowledge and available training data to solve inherently uncertain vision problems.
Proceedings of the Thirteenth International Workshop on Multimedia Data Mining - MDMKDD '13, 2013
ABSTRACT Online social networks have attracted attention of people from both the academia and rea... more ABSTRACT Online social networks have attracted attention of people from both the academia and real world. In particular, the rich multimedia information accumulated in recent years provides an easy and convenient way for more active communication between people. This offers an opportunity to research people's behaviors and activities based on those multimedia content, which can be considered as social imagematics. One emerging area is driven by the fact that these massive multimedia data contain people's daily sentiments and opinions. However, existing sentiment analysis typically only pays attention to the textual information regardless of the visual content, which may be more informative in expressing people's sentiments and opinions. In this paper, we attempt to analyze the online sentiment changes of social media users using both the textual and visual content. In particular, we analyze the sentiment changes of Twitter users using both textual and visual features. An empirical study of real Twitter data sets indicates that the sentiments expressed in textual content and visual content are correlated. The preliminary results in this paper give insight into the important role of visual content in online social media.
While on the go, more and more people are using their phones to enjoy ubiquitous location-based s... more While on the go, more and more people are using their phones to enjoy ubiquitous location-based services (LBS). One of the fundamental problems of LBS is localization. Researchers are now investigating ways to use a phone-captured image for localization as it contains more scene context information than the embedded sensors. In this paper, we present a novel approach to mobile visual localization that accurately senses geographic scene context according to the current image (typically associated with a rough GPS position). Unlike most existing visual localization methods, the proposed approach is capable of providing a complete set of more accurate parameters about the scene geo-including the actual locations of both the mobile user and perhaps more importantly the captured scene along with the viewing direction. Our approach takes advantage of advanced techniques for large-scale image retrieval and 3D model reconstruction from photos. Specifically, we first perform joint geo-visual clustering in the cloud to generate scene clusters, with each scene represented by a 3D model. The 3D scene models are then indexed using a visual vocabulary tree structure. The phone-captured image is used to retrieve the relevant scene models, then aligned with the models, and further registered to the real-world map. Our approach achieves an estimation accuracy of user location within 14 meters, viewing direction within 9 degrees, and scene location within 21 meters. Such a complete set of accurate geo-parameters can lead to various LBS applications for routing that cannot be achieved with most existing methods. In particular, we showcase three novel applications: 1) accurate selflocalization, 2) collaborative localization for rendezvous routing, and 3) routing for photographing. The evaluations through user studies indicate these applications are effective for facilitating the perfect rendezvous for mobile users.
In this paper, we present a systematic framework for recognizing realistic actions from videos "i... more In this paper, we present a systematic framework for recognizing realistic actions from videos "in the wild." Such unconstrained videos are abundant in personal collections as well as on the web. Recognizing action from such videos has not been addressed extensively, primarily due to the tremendous variations that result from camera motion, background clutter, changes in object appearance, and scale, etc. The main challenge is how to extract reliable and informative features from the unconstrained videos. We extract both motion and static features from the videos. Since the raw features of both types are dense yet noisy, we propose strategies to prune these features. We use motion statistics to acquire stable motion features and clean static features. Furthermore, PageRank is used to mine the most informative static features. In order to further construct compact yet discriminative visual vocabularies, a divisive information-theoretic algorithm is employed to group semantically related features. Finally, AdaBoost is chosen to integrate all the heterogeneous yet complementary features for recognition. We have tested the framework on the KTH dataset and our own dataset consisting of 11 categories of actions collected from YouTube and personal videos, and have obtained impressive results for action recognition and action localization. Figure 1: Examples of our YouTube action dataset consist of 11 categories with about 1160 videos.
Social multimedia hosting and sharing websites, such as Flickr, Facebook, Youtube, Picasa, ImageS... more Social multimedia hosting and sharing websites, such as Flickr, Facebook, Youtube, Picasa, ImageShack and Photobucket, are increasingly popular around the globe. A major trend in the current studies on social multimedia is using the social media sites as a source of huge amount of labeled data for solving large scale computer science problems in computer vision, data mining and multimedia. In this paper, we take a new path to explore the global trends and sentiments that can be drawn by analyzing the sharing patterns of uploaded and downloaded social multimedia. In a sense, each time an image or video is uploaded or viewed, it constitutes an implicit vote for (or against) the subject of the image. This vote carries along with it a rich set of associated data including time and (often) location information. By aggregating such votes across millions of Internet users, we reveal the wisdom that is embedded in social multimedia sites for social science applications such as politics, economics, and marketing. We believe that our work opens a brand new arena for the multimedia research community with a potentially big impact on society and social sciences.
Geo-tagging is a fast-emerging trend in digital photography and community photo sharing. The pres... more Geo-tagging is a fast-emerging trend in digital photography and community photo sharing. The presence of geographically relevant metadata with images and videos has opened up interesting research avenues within the multimedia and computer vision domains. In this paper, we survey geo-tagging related research within the context of multimedia and along three dimensions: 1) Modalities in which geographical information can be extracted, 2) Applications that can benefit from the use of geographical information, and 3) The interplay between modalities and applications. Our survey will introduce research problems and discuss significant approaches. We will discuss the nature of different modalities and lay out factors that are expected to govern the choices with respect to multimedia and vision applications. Finally, we discuss future research directions in this field. 8
received a bachelor's degree (summa cum laude) from the University of Minnesota, an M.Sc. degree ... more received a bachelor's degree (summa cum laude) from the University of Minnesota, an M.Sc. degree in mathematics, an M.Sc. degree in computer science, and a Ph.D. degree in medical information sciences from Stanford University. Her research interests include statistical modeling and learning, data mining, computational biology, image processing, and image annotation and retrieval.
Sentiment analysis of online user generated content is important for many social media analytics ... more Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.
Semantic scene classification based only on low-level vision cues has had limited success on unco... more Semantic scene classification based only on low-level vision cues has had limited success on unconstrained image sets. On the other hand, camera metadata related to capture conditions provides cues independent of the captured scene content that can be used to improve classification performance. We consider three problems, indoor-outdoor classification, sunset detection, and manmade-natural classification. Analysis of camera metadata statistics for images of each class revealed that metadata fields, such as exposure time, flash fired, and subject distance, are most discriminative for each problem. A Bayesian network is employed to fuse content-based and metadata cues in the probability domain and degrades gracefully even when specific metadata inputs are missing (a practical concern). Finally, we provide extensive experimental results on the three problems using content-based and metadata cues to demonstrate the efficacy of the proposed integrated scene classification scheme.
C onsiderable research has been devoted to the problem of multimedia indexing and retrieval in th... more C onsiderable research has been devoted to the problem of multimedia indexing and retrieval in the past decade. However, limited by the state of the art in image understanding, the majority of the existing content-based image retrieval (CBIR) systems have taken a relatively low-level approach and fallen short of higher-level interpretation and knowledge. Recent research has begun to focus on bridging the semantic and conceptual gap that exists between man and computer by integrating knowledge-based techniques, human perception, scene content understanding, psychology, and linguistics. In this article, we provide an overview of exploiting context for semantic scene content understanding.
In classic pattern recognition problems, classes are mutually exclusive by deÿnition. Classiÿcati... more In classic pattern recognition problems, classes are mutually exclusive by deÿnition. Classiÿcation errors occur when the classes overlap in the feature space. We examine a di erent situation, occurring when the classes are, by deÿnition, not mutually exclusive. Such problems arise in semantic scene and document classiÿcation and in medical diagnosis. We present a framework to handle such problems and apply it to the problem of semantic scene classiÿcation, where a natural scene may contain multiple objects such that the scene can be described by multiple class labels (e.g., a ÿeld scene with a mountain in the background). Such a problem poses challenges to the classic pattern recognition paradigm and demands a di erent treatment. We discuss approaches for training and testing in this scenario and introduce new metrics for evaluating individual examples, class recall and precision, and overall accuracy. Experiments show that our methods are suitable for scene classiÿcation; furthermore, our work appears to generalize to other classiÿcation problems of the same nature.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005
Automatic image orientation detection for natural images is a useful, yet challenging research ar... more Automatic image orientation detection for natural images is a useful, yet challenging research area. Humans use scene context and semantic object recognition to identify the correct image orientation. However, it is difficult for a computer to perform the task in the same way because current object recognition algorithms are extremely limited in their scope and robustness. As a result, existing orientation detection methods were built upon lowlevel vision features such as spatial distributions of color and texture. In addition, discrepant detection rates have been reported. We have developed a probabilistic approach to image orientation detection via confidence-based integration of low-level and semantic cues within a Bayesian framework. Our current accuracy is approaching 90% for unconstrained consumer photos, impressive given the findings of a psychophysical study conducted recently. The proposed framework is an attempt to bridge the gap between computer and human vision systems, and is applicable to other problems involving semantic scene content understanding.
Semantic scene classification based only on low-level vision cues has had limited success on unco... more Semantic scene classification based only on low-level vision cues has had limited success on unconstrained image sets. On the other hand, camera metadata related to capture conditions provides cues independent of the captured scene content that can be used to improve classification performance. We consider two problems: indoor-outdoor classification and sunset detection. Analysis of camera metadata statistics for images of each class revealed that metadata fields, such as exposure time, flash fired, and subject distance, are most discriminative for both indooroutdoor and sunset classification. A Bayesian network is employed to fuse content-based and metadata cues in the probability domain and degrades gracefully, even when specific metadata inputs are missing (a practical concern). Finally, we provide extensive experimental results on the two problems, using content-based and metadata cues to demonstrate the efficacy of the proposed integrated scene classification scheme.
Scene content understanding facilitates a large number of applications, ranging from content-base... more Scene content understanding facilitates a large number of applications, ranging from content-based image retrieval to other multimedia applications. Material detection refers to the problem of identifying key semantic material types (such as sky, grass, foliage, water, and snow in images). In this paper, we present a holistic approach to determining scene content, based on a set of individual material detection algorithms, as well as probabilistic spatial context models. A major limitation of individual material detectors is the significant number of misclassifications that occur because of the similarities in color and texture characteristics of various material types. We have developed a spatial context-aware material detection system that reduces misclassification by constraining the beliefs to conform to the probabilistic spatial context models. Experimental results show that the accuracy of materials detection is improved by 13% using the spatial context models over the individual material detectors themselves.
Knowledge of the semantic classification of an image can be used to improve the accuracy of queri... more Knowledge of the semantic classification of an image can be used to improve the accuracy of queries in content-based image organization and retrieval and to provide customized image enhancement. We developed an exemplar-based system for classifying sunset scenes. However, the performance of such a system depends largely on the size and quality of the set of training exemplars, which can be limited in practice. In addition, variations in scene content, as well as distracting regions, may exist in many testing images to prohibit good matches with the exemplars. We propose using simulated spatial and temporal image recomposition to address such issues. The recomposition schemes boost the recall of sunset images from a reasonably large data set by 10%, while holding the false positive rate constant.
Scene categorization to indoor vs outdoor may be approached by using low-level features for infer... more Scene categorization to indoor vs outdoor may be approached by using low-level features for inferring high-level information about the image. Low-level features such as color and texture have been used extensively in image understanding research, however, they cannot solve the problem completely. In this paper, we propose the use of a Bayesian network for integrating knowledge from low-level and semantic features for indoor vs outdoor classification of images. Using ground truth data for sky and grass detection, we demonstrate that the classification performance can be significantly improved when semantic features are employed in the classification process.
Prior research in scene classiÿcation has focused on mapping a set of classic low-level vision fe... more Prior research in scene classiÿcation has focused on mapping a set of classic low-level vision features to semantically meaningful categories using a classiÿer engine. In this paper, we propose improving the established paradigm by using a simpliÿed low-level feature set to predict multiple semantic scene attributes that are integrated probabilistically to obtain a ÿnal indoor/outdoor scene classiÿcation. An initial indoor/outdoor prediction is obtained by classifying computationally e cient, low-dimensional color and wavelet texture features using support vector machines. Similar low-level features can also be used to explicitly predict the presence of semantic features including grass and sky. The semantic scene attributes are then integrated using a Bayesian network designed for improved indoor/outdoor scene classiÿcation.
Current research in content-based semantic image understanding is largely confined to exemplar-ba... more Current research in content-based semantic image understanding is largely confined to exemplar-based approaches built on low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features is expected to raise semantic image understanding to a new level. Belief networks, or Bayesian networks (BN), have proven to be an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditional independence relationships. In this paper, we present a general-purpose knowledge integration framework that employs BN in integrating both low-level and semantic features. The efficacy of this framework is demonstrated via three applications involving semantic understanding of pictorial images. The first application aims at detecting main photographic subjects in an image, the second aims at selecting the most appealing image in an event, and the third aims at classifying images into indoor or outdoor scenes. With these diverse examples, we demonstrate that effective inference engines can be built within this powerful and flexible framework according to specific domain knowledge and available training data to solve inherently uncertain vision problems.
Measuring perceptual saliency of regions in a scene is important for determining regions of inter... more Measuring perceptual saliency of regions in a scene is important for determining regions of interest. Color, texture and shape cues are good low-level features for detecting saliency. While self saliency refers to intrinsic attributes of a region, relative saliency as used to measure how salient a region is relative to its surrounding and, thus, needs to be defined within a spatial context. A few spatial context models are investigated in this study. In particular, we propose an auto-scaled, amorphous neighborhood as the context model to obtain reliable measurements of relative saliency features. Comparison of three context models has shown that the proposed model is capable of generating predicates more consistent with perceived saliency
Belief networks, such as Bayes nets, have emerged as an effective knowledge representation and in... more Belief networks, such as Bayes nets, have emerged as an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditionally independence relationships. Current research in contentbased image processing and analysis is largely limited to low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features would be very useful. In this paper, we present a general knowledge integration framework that incorporates Bayes networks and has been used in two applications involving semantic understanding of consumer photographs. The first application aims at detecting main photographic subjects in an image and the second aims at selecting the most appealing image in an event. With these diverse examples, we demonstrate that effective inference engines can be built according to specific domain knowledge and available training data to solve inherently uncertain vision problems.
Uploads
Papers by Jiebo Luo