POMDPs

In this work, we propose a multimodal interaction framework for robust human-multirobot communication in outdoor environments. In these scenarios, several human or environmental factors can cause errors, noise and wrong interpretations of the commands. The main goal of this work is to improve the robustness of human-robot interaction systems in similar situations. In particular, we propose a multimodal fusion method based on the following steps: for each communication channel, unimodal classifiers are firstly deployed in order to generate unimodal interpretations of the human inputs; the unimodal outcomes are then grouped into different multimodal recognition lines, each representing a possible interpretation of a sequence of multimodal inputs; these lines are finally assessed to recognize the human commands. We discuss the system at work in a real world case study in the SHERPA domain.

Partially Observable Markov Decision Processes (POMDP) is a widely used model to represent the interaction of an environment and an agent, under state uncertainty. Since the agent does not observe the environment state, its uncertainty is typically represented through a probabilistic belief. While the set of possible beliefs is infinite, making exact planning intractable, the belief space's complexity (and hence planning complexity) is characterised by its covering number. Many POMDP solvers uniformly discretize the belief space and give the planning error in terms of the (typically unknown) covering number. We instead propose an adaptive belief discretization scheme, and give its associated planning error. We furthermore characterise the covering number with respect to the POMDP parameters. This allows us to specify the exact memory requirements on the planner, needed to bound the value function error. We then propose a novel, computationally efficient solver using this scheme. We demonstrate that our algorithm is highly competitive with the state of the art in a variety of scenarios.

Partially Observable Markov Decision Processes (POMDPs) provide an efficient way to model real-world sequential decision making processes. Motivated by the problem of maintenance and inspection of a group of infrastructure components with independent dynamics, this letter presents an algorithm to find the optimal policy for a multi-component budget-constrained POMDP. We first introduce a budgeted-POMDP model (b-POMDP) which enables us to find the optimal policy for a POMDP while adhering to budget constraints. Next, we prove that the value function or maximal collected reward for a special class of b-POMDPs is a concave function of the budget for the finite horizon case. Our second contribution is an algorithm to calculate the optimal policy for a multi-component budget-constrained POMDP by finding the optimal budget split among the individual component POMDPs. The optimal budget split is posed as a welfare maximization problem and the solution is computed by exploiting the concavity of the value function. We illustrate the effectiveness of the proposed algorithm by proposing a maintenance and inspection policy for a group of real-world infrastructure components with different deterioration dynamics, inspection and maintenance costs. We show that the proposed algorithm vastly outperforms the policies currently used in practice. Index Terms-Optimization algorithms, Markov processes, distributed control. I. INTRODUCTION S EQUENTIAL decision-making is an integral component of many real world problems like machine maintenance, structural inspection and autonomous robotics [1]. Markov Decision Processes (MDPs) have provided an efficient framework to model and solve such problems while accounting for

La prise de décision est un problème omniprésent qui survient dés qu'on fait faceà plusieurs choix possibles. Ce problème est d'autant plus complexe lorsque les décisions, ou les actions, doiventêtre prise d'une manière séquentielle. En effet, l'exécution d'une actionà un moment donné entraîne un changementà l'environnement, ou au système qu'on veut contrôler, et un tel changement ne peut pasêtre prévu avec certitude. Le but d'un processus de prise de décision consiste alorsà choisir des actions en vue de se comporter d'une manière optimale dans un environnement incertain. Afin d'y parvenir, l'environnement est souvent modélisé comme un système dynamiqueà plusieursétats, et les actions sont choisies d'une telle manièreà ramener le système vers unétat désirable. Dans le cadre de cette thèse, nous avons proposé un ensemble de modèles stochastiques et d'algorithmes, afin d'améliorer la qualité du processus de prise de décision sous l'incertain. Les modèles développés sont une alternative aux Processus Décisionnels de Markov (MDPs), un cadre formel largement utilisé pour ce genre de problèmes. En particulier, nous avons montré que l'état d'un système dynamique peutêtre représenté d'une manière plus concise lorsqu'il est décrit en termes de prédictions de certainsévènements dans le futur. Nous avons aussi montré que le processus cognitif même du choix d'actions, appelé politique, peutêtre vu comme un système dynamique. Partant de cette observation, nous avons proposé une panoplie d'algorithmes, tous basés sur des représentations prédictives de politiques, pour résoudre différents problèmes de prise de décision, tels que la panification décentralisée, l'apprentissage par renforcement, ou bien encore l'apprentissage par imitation. Nous avons montré analytiquement et empiriquement que les approches proposées mènent a des réductions de la complexité de calcul età une amélioration de la qualité des solutions par rapport aux approches d'apprentissage et de planification standards.

Addressing current challenges in research on disruptive mood dysregulation disorder (DMDD), this study aims to compare executive function in children with DMDD, children with attention-deficit/hyperactivity disorder (ADHD), and children with oppositional defiant disorder (ODD). We also explore associations between irritability, a key DMDD characteristic, and executive function in a clinical sample regardless of diagnosis. Our sample include children (6-12 years) referred to child psychiatric clinics. Measures of daily-life (parent-reported questionnaire) and performance-based (neuropsychological tasks) executive function were applied. Identifying diagnoses, clinicians administered a standardized semi-structured diagnostic interview with parents. Irritability was assessed by parent-report. First, we compared executive function in DMDD (without ADHD/ODD), ADHD (without DMDD/ODD), ODD (without DMDD/ADHD) and DMDD + ADHD (without ODD). Second, we analyzed associations between executive function and irritability using the total sample. In daily life, children with DMDD showed clinically elevated and significantly worse emotion control scores compared to children with ADHD, and clinically elevated scores on cognitive flexibility compared to norm scores. Children with DMDD had significantly less working memory problems than those with ADHD. No differences were found between DMDD and ODD. Increased irritability was positively associated with emotional dyscontrol and cognitive inflexibility. For performance-based executive function, no diagnostic differences or associations with irritability were observed. We discuss how, in daily life, children with high irritability-levels get overwhelmed by feelings without accompanying regulatory capacities.

Recent work in the behavioural sciences has begun to overturn the long-held belief that human decision making is irrational, suboptimal and subject to biases. This turn to the rational suggests that human decision making may be a better source of ideas for constraining how machine learning problems are defined than would otherwise be the case. One promising idea concerns human decision making that is dependent on apparently irrelevant aspects of the choice context. Previous work has shown that by taking into account choice context and making relational observations, people can maximize expected value. Other work has shown that Partially observable Markov decision processes (POMDPs) are a useful way to formulate human-like decision problems. Here, we propose a novel POMDP model for contextual choice tasks and show that, despite the apparent irrationalities, a reinforcement learner can take advantage of the way that humans make decisions. We suggest that human irrationalities may offer a productive source of inspiration for improving the design of AI architectures and machine learning methods. Preprint. Under review.

As agents' technology becomes increasing more prevalent, coordination in mixed agent-human environments becomes a key issue. Agent-human coordination is becoming even more important in real life situations, where uncertainty and incomplete information exists and communication is costly. While abundant research has focused on aspects of computerized teamwork, little attention has been given to the issues raised in teams that consist of both computerized agents and people. In this paper we focus on teamwork between an agent and a human counterpart and present a novel agent designed to interact proficiently with people. In extensive simulations we matched our agent with people and compared it with another state-of-the-art agent. Our results demonstrate the significant improvement in coordination when our agent is involved.

We propose a novel approach to developing a tractable affective dialogue model for probabilistic frame-based dialogue systems. The affective dialogue model, based on Partially Observable Markov Decision Process (POMDP) and Dynamic Decision Network (DDN) techniques, is composed of two main parts: the slot-level dialogue manager and the global dialogue manager. It has two new features: (1) being able to deal with a large number of slots and (2) being able to take into account some aspects of the user's affective state in deriving the adaptive dialogue strategies. Our implemented prototype dialogue manager can handle hundreds of slots, where each individual slot might have hundreds of values. Our approach is illustrated through a route navigation example in the crisis management domain. We conducted various experiments to evaluate our approach and to compare it with approximate POMDP techniques and handcrafted policies. The experimental results showed that the DDN–POMDP policy outp...

Planning to see: A hierarchical approach to planning visual actions on a robot using POMDPs This journal article describes a novel approach that enables a mobile robot to autonomously tailor vision-based sensing and information processing to each of a wide range of tasks. Visual processing management is posed as an instance of probabilistic sequential decision-making, and more specifically as a partially observable Markov decision process. Automatic belief propagation is achieved by incorporating a hierarchical decomposition whose levels match the functional and cognitive requirements of visual processing on a mobile robot. As a result, the mobile robot is able to operate reliably and efficiently, and collaborate with humans, in dynamic indoor environments.

This paper proposes a novel hierarchical representation of POMDPs that for the first time is amenable to real-time solution. It will be referred to in this paper as the Robot Navigation-Hierarchical POMDP (RN-HPOMDP). The RN-HPOMDP is utilized as a unified framework for autonomous robot navigation in dynamic environments. As such, it is used for localization, planning and local obstacle avoidance. Hence, the RN-HPOMDP decides at each time step the actions the robot should execute, without the intervention of any other external module. Our approach employs state space and action space hierarchy, and can effectively model large environments at a fine resolution. Finally, the notion of the reference POMDP, that holds all the information regarding motion and sensor uncertainty is introduced, which makes our hierarchical structure memory efficient and enables fast learning. The RN-HPOMDP has been tested extensively in a real-world environment.

We analyze a single-item periodic-review inventory system with random yield and finite capacity operating in a random environment. The primary objective is to extend the model of Gallego and Hu (2004) to the more general case when the environment is only partially observable. Although our analysis is specific to inventory systems, it can also be applied to production systems by replacing the fixed capacity supplier with a fixed capacity producer. Using sufficient statistics, we consider single-period, multipleperiod and infinite-period problems to show that a state-dependent modified inflated base-stock policy is optimal. Moreover, we show that the multiple-period cost converges to the infinite-period cost as the length of the planning horizon increases.

Objective: To develop a scale for emotional regulation using item response theory. Method: Eighteen Swanson Nolan and Pelham (SNAP-IV) items that loaded on an emotional dysregulation factor were submitted to Rasch analysis. After eliminating the items that violated Rasch criteria, the remaining items were examined for reliability and validated against the Conners' emotional lability index. Results: A nine-item scale for emotional regulation was developed that satisfies the Rasch model and reliably distinguishes emotionally dysregulated/irritable children and adolescents. A score of 4 or higher in this scale has optimal accuracy for identifying children and adolescents with current significant dysfunction in emotional regulation. Among youth with ADHD inattentive, hyperactive-impulsive, and combined types, 42%, 56%, and 71% met the Clinical Evaluation of Emotional Regulation-9 (CEER-9) threshold for emotional lability, respectively. Conclusion: A nineitem scale whose sum total is a measure of emotional regulation is proposed as a tool for clinical and research purposes.

Methods of deep machine learning enable to to reuse low-level representations efficiently for generating more abstract high-level representations. Originally, deep learning has been applied passively (e.g., for classification purposes). Recently, it has been extended to estimate the value of actions for autonomous agents within the framework of reinforcement learning (RL). Explicit models of the environment can be learned to augment such a value function. Although "flat" connectionist methods have already been used for model-based RL, up to now, only modelfree variants of RL have been equipped with methods from deep learning. We propose a variant of deep model-based RL that enables an agent to learn arbitrarily abstract hierarchical representations of its environment. In this paper, we present research on how such hierarchical representations can be grounded in sensorimotor interaction between an agent and its environment.

We formalize decision-making problems in robotics and automated control using continuous MDPs and actions that take place over continuous time intervals. We then approximate the continuous MDP using finer and finer discretizations. Doing this results in a family of systems, each of which has an extremely large action space, although only a few actions are "interesting". We can view the decision maker as being unaware of which actions are "interesting". We can model this using MDPUs, MDPs with unawareness, where the action space is much smaller. As we show, MDPUs can be used as a general framework for learning tasks in robotic problems. We prove results on the difficulty of learning a near-optimal policy in an an MDPU for a continuous task. We apply these ideas to the problem of having a humanoid robot learn on its own how to walk.

High dimensionality of belief space in Partially Observable Markov Decision Processes (POMDPs) is one of the major causes that severely restricts the applicability of this model. Previous studies have demonstrated that the dimensionality of a POMDP can eventually be reduced by transforming it into an equivalent Predictive State Representation (PSR). In this paper, we address the problem of finding an approximate and compact PSR model corresponding to a given POMDP model. We formulate this problem in an optimization framework. Our algorithm tries to minimize the potential error that missing some core tests may cause. We also present an empirical evaluation on benchmark problems, illustrating the performance of this approach.

Recent research has shown that effective dialogue management can be achieved through the Partially Observable Markov Decision Process (POMDP) framework. However past research on POMDP-based dialogue systems usually assumed the parameters of the decision process were known a priori. The main contribution of this paper is to present a Bayesian reinforcement learning framework for learning the POMDP parameters online from data, in a decision-theoretic manner. We discuss various approximations and assumptions which can be leveraged to ensure computational tractability, and apply these techniques to learning observation models for several simulated spoken dialogue domains.

Identifying an object of interest, grasping it, and handing it over are key capabilities of collaborative robots. In this context we propose a fast, supervised learning framework for learning associations between human hand gestures and the intended robotic manipulation actions. This framework enables the robot to learn associations on the fly while performing a task with the user. We consider a domestic scenario of assembling a kid's table where the role of the robot is to assist the user. To facilitate the collaboration we incorporate the robot's gaze into the framework. The proposed approach is evaluated in simulation as well as in a real environment. We study the effect of accurate gesture detection on the number of interactions required to complete the task. Moreover, our quantitative analysis shows how purposeful gaze can significantly reduce the amount of time required to achieve the goal.

A common approach to the control problem in partially observable environments is to perform a direct search in policy space, as defined over some set of features of history. In this paper we consider predictive features, whose values are conditional probabilities of future events, given history. Since predictive features provide direct information about the agent's future, they have a number of advantages for control. However, unlike more typical features defined directly over past observations, it is not clear how to maintain the values ...

The problem of developing good policies for partially observable Markov decision problems (POMDPs) remains one of the most challenging areas of research in stochastic planning. One line of research in this area involves the use of reinforcement learning with belief states, probability distributions over the underlying model states. This is a promising method for small problems, but its application is limited by the intractability of computing or representing a full belief state for large problems. Recent work shows that, in many settings, we can maintain an approximate belief state, which is fairly close to the true belief state. In particular, great success has been shown with approximate belief states that marginalize out correlations between state variables. In this paper, we investigate two methods of full belief state reinforcement learning and one novel method for reinforcement learning using factored approximate belief states. We compare the performance of these algorithms on several well-known problem from the literature. Our results demonstrate the importance of approximate belief state representations for large problems.

One of the main challenges when it comes to designing a dialogue system is error handling. The Automatic Speech Recognition (ASR) technology is not perfect and the user may use words or expressions that are unknown to the system (off-domain). Traditional dialogue systems wait until the end of the user's utterance before processing and cannot be interrupted after taking the floor. On the contrary, incremental dialogue systems process the user's speech signal on the fly. Therefore they are able to react quickly whenever an error is suspected. In this paper, we devise a simulated environment to show that incremental dialogue systems make it possible to design new strategies for efficient error handling.

A common approach to the control problem in partially observable environments is to perform a direct search in policy space, as defined over some set of features of history. In this paper we consider predictive features, whose values are conditional probabilities of future events, given history. Since predictive features provide direct information about the agent's future, they have a number of advantages for control. However, unlike more typical features defined directly over past observations, it is not clear how to maintain the values ...

Active Object Recognition (AOR) has been approached as an unsupervised learning problem, in which optimal trajectories for object inspection are not known and are to be discovered by reducing label uncertainty measures or training with reinforcement learning. Such approaches have no guarantees of the quality of their solution. In this paper, we treat AOR as a Partially Observable Markov Decision Process (POMDP) and find near-optimal policies on training data using Belief Tree Search (BTS) on the corresponding belief Markov Decision Process (MDP). AOR then reduces to the problem of knowledge transfer from near-optimal policies on training set to the test set. We train a Long Short Term Memory (LSTM) network to predict the best next action on the training set rollouts. We sho that the proposed AOR method generalizes well to novel views of familiar objects and also to novel objects. We compare this supervised scheme against guided policy search, and find that the LSTM network reaches higher recognition accuracy compared to the guided policy method. We further look into optimizing the observation function to increase the total collected reward of optimal policy. In AOR, the observation function is known only approximately. We propose a gradient-based method update to this approximate observation function to increase the total reward of any policy. We show that by optimizing the observation function and retraining the supervised LSTM network, the AOR performance on the test set improves significantly.

Existing algorithms for discrete partially observable Markov decision processes can at best solve problems of a few thousand states due to two important sources of intractability: the curse of dimensionality and the policy space complexity. This paper describes a new algorithm (VDCBPI) that mitigates both sources of intractability by combining the Value Directed Compression (VDC) technique [13] with Bounded Policy Iteration (BPI)[14]. The scalability of VDCBPI is demonstrated on synthetic network management problems with ...

We describe a new approximation algorithm for solving partially observable MDPs. Our bounded policy iteration approach searches through the space of bounded-size, stochastic finite state controllers, combining several advantages of gradient ascent (efficiency, search through restricted controller space) and policy iteration (less vulnerability to local optima).

We consider the problem belief-state monitoring for the purposes of implementing a policy for a partially-observable Markov decision process (POMDP), specifically how one might approxi mate the belief state. Other schemes for belief state approximation (e.g., based on minimizing a measure such as KL-divergence between the true and estimated state) are not necessarily appropri ate for POMDPs. Instead we propose a frame work for analyzing value-directed approximation schemes, where approximation quality is deter mined by the expected error in utility rather than by the error in the belief state itself. We propose heuristic methods for finding good projection schemes for belief state estimation-exhibiting anytime characteristics-given a POMDP value function. We also describe several algorithms for constructing bounds on the error in decision qual ity (expected utility) associated with acting in ac cordance with a given belief state approximation.

We consider the problem of learning the behavior of a POMDP (Partially Observable Markov Decision Process) with deterministic actions and observations. This is a challenging problem due to the fact that the observations can only partially identify the states. Recent work by Holmes and Isbell offers an approach for inferring the hidden states from experience in deterministic POMDP environments. We propose an alternative algorithm that ensures more accurate predictions, and we show that in fact it produces the minimal ...

There is a tendency in decision-making research to treat uncertainty only as a problem to be overcome. But it is also a feature that can be leveraged, particularly in social interaction. Comparing the behavior of profitable and unprofitable poker players, we reveal a strategic use of information processing that keeps decision makers unpredictable. To win at poker, a player must exploit public signals from others. But using public inputs makes it easier for an observer to reconstruct that player's strategy and predict his or her behavior. How should players trade off between exploiting profitable opportunities and remaining unexploitable themselves? Using a recent multivariate approach to information theoretic data analysis and 1.75 million hands of online two-player No-Limit Texas Hold'em, we find that the important difference between winning and losing players is not in the amount of information they process, but how they process it. In particular, winning players are bette...

In human-robot interactive scenarios communication and collaboration during task execution are crucial issues. Since the human behavior is unpredictable and ambiguous, an interactive robotic system is to continuously interpret intentions and goals adapting its executive and communicative processes according to the users behaviors. In this work, we propose an integrated system that exploits attentional mechanisms to flexibly adapt planning and executive processes to the multimodal human-robot interaction.

My PhD was a period full of excitement, of intense learning on both the scientific and personal level. At the same time, there were many hard moments, where I had real doubts on its success. There are many people without whom I couldn't have completed this journey, and I would like to thank them in this section. First, I would like to thank my wonderful mother and sister, for always accepting and supporting my choice of moving abroad. A special thank goes to my best friend, my love, and my soulmate, Maria Carmela. You were always there for me, even when we were far away, or when I had to focus more on my research than on you, and I know you will always be. I would like to thank my supervisor, Rachid Alami, for giving me this wonderful opportunity, for his wise advise, and for all the time he spent with me during these years. I would also like to thank my two rapporteurs, Yiannis Demiris and Abdel-Illah Mouaddib, for taking the time to read my dissertation and for providing such insightful comments, which increased the quality of the thesis. My sincere thanks to Félix Ingrand and Mohamed Chetouani, for being part of my jury and for their thoughtful questions, which incented me to widen my research. I would also like to thank all the people involved in the SPENCER project, for supporting my research. Looking back, I really think we did something innovative, and I'm proud of that work and those moments. In particular, I would like to thank my fellow student Harmish Khambhaita, who accompanied me in most of this work. Finally, I cannot not thank all the wonderful people of LAAS, who shaped these three and half years to such an unforgettable experience. In particular, I would like to thank my fellow students Mamoun Gharbi, Grégoire Milliez, and Raphaël Lallement. Moving to a different country without speaking its language can be hard, but you were such good friends that it wasn't.

Attracted by their easy-to-use interfaces and captivating benefits, conversational systems have been widely embraced by many individuals and organizations as side-by-side digital co-workers. They enable the understanding of user needs, expressed in natural language, and on fulfilling such needs by invoking the appropriate backend services (e.g., APIs). Controlling the conversation flow, known as Dialogue Management, is one of the essential tasks in conversational systems and the key to its success and adoption as well. Nevertheless, designing scalable and robust dialogue management techniques to effectively support intelligent conversations remains a deeply challenging problem. This article studies dialogue management from an in-depth design perspective. We discuss the state of the art approaches, identify their recent advances and challenges, and provide an outlook on future research directions. Thus, we contribute to guiding researchers and practitioners in selecting the appropriate dialogue management approach aligned with their objectives, among the variety of approaches proposed so far.

POMDPs.jl is an open-source framework for solving Markov decision processes (MDPs) and partially observable MDPs (POMDPs). POMDPs.jl allows users to specify sequential decision making problems with minimal effort without sacrificing the expressive nature of POMDPs, making this framework viable for both educational and research purposes. It is written in the Julia language to allow flexible prototyping and large-scale computation that leverages the high-performance nature of the language. The associated JuliaPOMDP community also provides a number of state-of-the-art MDP and POMDP solvers and a rich library of support tools to help with implementing new solvers and evaluating the solution results. The most recent version of POMDPs.jl, the related packages, and documentation can be found at https://github.com/JuliaPOMDP/POMDPs.jl.

Motion planning under uncertainty is essential to autonomous robots. Over the past decade, the scalability of such planners have advanced substantially. Despite these advances, the problem remains difficult for systems with non-linear dynamics. Most successful methods for planning perform forward search that relies heavily on a large number of simulation runs. Each simulation run generally requires more costly integration for systems with non-linear dynamics. Therefore, for such problems, the entire planning process remains relatively slow. Not surprisingly, linearization-based methods for planning under uncertainty have been proposed. However, it is not clear how linearization affects the quality of the generated motion strategy, and more importantly where to and where not to use such a simplification. This paper presents our preliminary work towards answering such questions. In particular, we propose a measure, called Statistical-distance-based Non-linearity Measure (SNM), to identify where linearization can and where it should not be performed. The measure is based on the distance between the distributions that represent the original motion-sensing models and their linearized version. We show that when the planning problem is framed as the Partially Observable Markov Decision Process (POMDP), the difference between the value of the optimal strategy generated if we plan using the original model and if we plan using the linearized model, can be upper bounded by a function linear in SNM. We test the applicability of this measure in simulation via two venues. First, we compare SNM with a negentropy-based Measure of Non-Gaussianity (MoNG)-a measure that has recently been shown to be a suitable measure of non-linearity for stochastic systems [1]. We compare their performance in measuring the difference between a general POMDP solver [2] that computes motion strategies using the original model and a solver that uses the linearized model (adapted from [3]) on various scenarios. Our results indicate that SNM is more suitable in taking into account the effect that obstacles have on the effectiveness of linearization. In the second set of tests, we use a local estimate of SNM to develop a simple on-line planner that switches between using the original and the linearized model. Simulation results on a car-like robot with second order dynamics and a 4-DOFs and 6-DOFs manipulator with torque control indicate that our simple planner appropriately decides if and when linearization should be used.

Low-cost navigation solutions for indoor environments have a variety of real-world applications ranging from emergency evacuation to mobility aids for people with disabilities. Challenges for indoor navigation include robust localization in the absence of GPS, intuitive recognition of user navigation goals, and efficient route-planning and replanning techniques. In this paper, we present an architecture for indoor navigation that integrates observed behavior for recognizing user navigation goals and estimating future paths without direct input from the user. Our architecture comprises of three core components: effective localization, map representation and route planning, and plan recognition. The outlined architecture is unique in its integration of the core navigation and prediction components. To evaluate the feasibility of the proposed architecture, we develop a prototype application on a commercial smart-phone. The developed application was tested in an indoor environment and was found to accurately predict intended destination and to provide effective navigation guidance to the user.

We formalize decision-making problems in robotics and automated control using continuous MDPs and actions that take place over continuous time intervals. We then approximate the continuous MDP using finer and finer discretizations. Doing this results in a family of systems, each of which has an extremely large action space, although only a few actions are "interesting". We can view the decision maker as being unaware of which actions are "interesting". We an model this using MDPUs, MDPs with unawareness, where the action space is much smaller. As we show, MDPUs can be used as a general framework for learning tasks in robotic problems. We prove results on the difficulty of learning a near-optimal policy in an an MDPU for a continuous task. We apply these ideas to the problem of having a humanoid robot learn on its own how to walk.

Bayesian approaches provide a principled solution to the explorationexploitation trade-off in Reinforcement Learning. Typical approaches, however, either assume a fully observable environment or scale poorly. This work introduces the Factored Bayes-Adaptive POMDP model, a framework that is able to exploit the underlying structure while learning the dynamics in partially observable systems. We also present a belief tracking method to approximate the joint posterior over state and model variables, and an adaptation of the Monte-Carlo Tree Search solution method, which together are capable of solving the underlying problem near-optimally. Our method is able to learn efficiently given a known factorization or also learn the factorization and the model parameters at the same time. We demonstrate that this approach is able to outperform current methods and tackle problems that were previously infeasible.

This Dagstuhl Seminar also stood as the 11th European Workshop on Reinforcement Learning (EWRL11). Reinforcement learning gains more and more attention each year, as can be seen at the various conferences (ECML, ICML, IJCAI, . . . ). EWRL, and in particular this Dagstuhl Seminar, aimed at gathering people interested in reinforcement learning from all around the globe. This unusual format for EWRL helped viewing the field and discussing topics differently. Seminar 04.–09. August, 2013 – www.dagstuhl.de/13321 1998 ACM Subject Classification I.2.6 Learning, I.2.8 Problem Solving, Control Methods, and Search, I.2.9 Robotics, I.2.11 Distributed Artificial Intelligence, G.3 Probability and Statistics (Markov processes)

Abstract—A mobile robot must have the ability of building a representation of its environment and the objects in it. To build a three-dimensional (3D) model of a physical object, several scans must be taken at different locations. Selecting each location is the next-best-view problem. Search based methods, where candidate views are generated and evaluated by a utility function, are a solution. However, such methods are slow for high resolution models given that the evaluation requires visibility computation in 3D. We propose a scene representation by octrees with a hierarchical ray tracing that reduces the visibility computation time. Such method performs a coarse ray tracing, except for the interesting volumes where a finer resolution is applied. The method decreases the computation time at least one order of magnitude. Saving time with this method leads to evaluate more constraints and more candidate views in high resolution models. I.

In this paper the approach of using a partially observable Markov model for games with dynamical difficulty adjustment is introduced. This approach leads implicitly to a strategy which balances gathering information about the player through his or her behavior with adjusting the game to the estimated player's abilities and preferences. We will show how this approach can be used in a stroke rehabilitation system, where a person plays a game in which the controller is a rehabilitation device. We show that the parameters of the model have a clear influence on the behavior of the system and that aspects of the player's abilities and characteristics can be measured by observing the behavior.

Brain-imaging technology has boosted the quantification of neurobiological phenomena underlying human mental operations and their disturbances. Since its inception, drawing inference on neurophysiological effects hinged on classical statistical methods, especially, the general linear model. The tens of thousands of variables per brain scan were routinely tackled by independent statistical tests on each voxel. This circumvented the curse of dimensionality in exchange for neurobiologically imperfect observation units, a challenging multiple comparisons problem, and limited scaling to currently growing data repositories. Yet, the always bigger information granularity of neuroimaging data repositories has lunched a rapidly increasing adoption of statistical learning algorithms. These scale naturally to high-dimensional data, extract models from data rather than prespecifying them, and are empirically evaluated for extrapolation to unseen data. The present article portrays commonalities ...

Others can have a different perception of the world than ours. Understanding this divergence is an ability, known as perspective taking in developmental psychology, that humans exploit in daily social interactions. A recent trend in robotics aims at endowing robots with similar mental mechanisms. The goal then is to enable them to naturally and efficiently plan tasks and communicate about them. In this paper we address this challenge extending a state-of-the-art goal-oriented dialogue management framework, the Hidden Information State (HIS). The new version makes use of the robot's awareness of the users' belief in a reinforcement learning-based situated dialogue management optimisation procedure. Thus the proposed solution enables the system to cope with the communication ambiguities due to noisy channel but also with the possible misunderstandings due to some divergence among the beliefs of the robot and its interlocutor in a Human-Robot Interaction (HRI) context. We show the relevance of the approach by comparing different handcrafted and learnt dialogue policies with and without divergent belief reasoning in an in-house Pick-Place-Carry scenario by mean of user trials in a simulated 3D environment.

Many robotic projects use simulation as a faster and easier way to develop, evaluate and validate software components compared with on-board real world settings. In the human-robot interaction field, some recent works have attempted to integrate humans in the simulation loop. In this paper we investigate how such kind of robotic simulation software can be used to provide a dynamic and interactive environment to both collect a multimodal situated dialogue corpus and to perform an efficient reinforcement learning-based dialogue management optimisation procedure. Our proposition is illustrated by a preliminary experiment involving real users in a Pick-Place-Carry task for which encouraging results are obtained.

Automatically generating solutions to general multi-robot coordination problems with communication limitations is challenging, but crucial in many domains. As one way to address this problem, we describe a probabilistic framework for synthesizing control policies for general multi-robot systems based on decentralized partially observable Markov decision processes with macro-actions (MacDec-POMDPs). MacDec-POMDPs are a general model of decision-making where a team of robots cooperates to optimize a shared objective in the presence of uncertainty. MacDec-POMDPs also consider communication limitations, so execution is decentralized. We describe how, in contrast to most existing methods that are specialized to a particular problem class, we can synthesize control policies that exploit whatever opportunities for coordination are present in the problem, while balancing uncertainty, sensor information, and information about other robots.

Online, sample-based planning algorithms for POMDPs have shown great promise in scaling to problems with large state spaces, but they become intractable for large action and observation spaces. This is particularly problematic in multiagent POMDPs where the action and observation space grows exponentially with the number of agents. To combat this intractability, we propose a novel scalable approach based on sample-based planning and factored value functions that exploits structure present in many multiagent settings. This approach applies not only in the planning case, but also in the Bayesian reinforcement learning setting. Experimental results show that we are able to provide high quality solutions to large multiagent planning and learning problems.

With rapid profusion of video data, automated surveillance and intrusion detection is becoming closer to reality. In order to provide timely responses while limiting false alarms, an intrusion detection system must balance resources (e.g., time) and accuracy. In this paper, we show how such a system can be modeled with a partially observable Markov decision process (POMDP), representing possible computer vision filters and their costs in a way that is similar to human vision systems. The POMDP representation can be optimized to produce a dynamic sequence of operations and achieve a tradeoff between time and detection quality, taking into account uncertainty in the filter predictions. In a set of experiments on actual video data, we show that our method can both outperform static “expert” models and scale to large dynamic domains. These results suggest that our method could be used in real-world intrusion detection systems.

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for decentralized decision making under uncertainty. However, they typically model a problem at a low level of granularity, where each agent's actions are primitive operations lasting exactly one time step. We address the case where each agent has macroactions: temporally extended actions which may require different amounts of time to execute. We model macroactions as options in a factored Dec-POMDP model, focusing on options which depend only on information available to an individual agent while executing. This enables us to model systems where coordination decisions only occur at the level of deciding which macro-actions to execute, and the macro-actions themselves can then be executed to completion. The core technical difficulty when using options in a Dec-POMDP is that the options chosen by the agents no longer terminate at the same time. We present extensions of two leading Dec-POMDP algorithms for generating a policy with options and discuss the resulting form of optimality. Our results show that these algorithms retain agent coordination while allowing near-optimal solutions to be generated for significantly longer horizons and larger state-spaces than previous Dec-POMDP methods.

POMDPs and their decentralized multiagent counterparts, DEC-POMDPs, offer a rich framework for sequential decision making under uncertainty. Their high computational complexity, however, presents an important research challenge. One way to address the intractable memory requirements of current algorithms is based on representing agent policies as finite-state controllers. Using this representation, we propose a new approach that formulates the problem as a nonlinear program, which defines an optimal policy of a desired size for each agent. This new formulation allows a wide range of powerful nonlinear programming algorithms to be used to solve POMDPs and DEC-POMDPs. Although solving the NLP optimally is often intractable, the results we obtain using an off-the-shelf optimization method are competitive with state-of-the-art POMDP algorithms and outperform state-of-the-art DEC-POMDP algorithms. Our approach is easy to implement and it opens up promising research directions for solving POMDPs and DEC-POMDPs using nonlinear programming methods.

In recent years, there has been much debate regarding the most appropriate diagnostic classification of children exhibiting emotion dysregulation in the form of irritability and severe temper outbursts. Most recently, this has resulted in the addition of a new diagnosis, Disruptive Mood Dysregulation Disorder (DMDD) in the DSM 5. The impetus for including this new disorder was to reduce the number of diagnoses that these children would typically receive; however, there is concern that it has only complicated matters rather than simplifying them. For example, a recent epidemiologic study shows that DMDD cannot be differentiated from oppositional defiant disorder (ODD) based on symptoms alone. Thus, these children are an ideal population in which to apply RDoC constructs in order to obtain greater clarity in terms of underlying processes and ultimately, inform nosology and appropriate interventions. The aim of this article is to provide a foundation for future research by examining extant theoretical and empirical evidence for the role of four key RDoC constructs in DMDD.