POMDPs

In this paper the approach of using a partially observable Markov model for games with dynamical difficulty adjustment is introduced. This approach leads implicitly to a strategy which balances gathering information about the player through his or her behavior with adjusting the game to the estimated player's abilities and preferences. We will show how this approach can be used in a stroke rehabilitation system, where a person plays a game in which the controller is a rehabilitation device. We show that the parameters of the model have a clear influence on the behavior of the system and that aspects of the player's abilities and characteristics can be measured by observing the behavior.

This research attempts to propose an approach to make decisions under uncertainty for designing a dialogue manager for Amharic spoken dialogue system, Amharic as under-resourced language. A prototype Amharic Spoken Dialogue System was implemented on hotel and restaurant address information domain for experimentation. Data for this research was collected through a method called Wizard of Oz and domain knowledge is prepared using address search websites. How to design a dialogue manager which is robust for languages especially with low performing Automatic Speech Recognition unit is a fundamental question of this research. Previous studies on designing of spoken dialogue system for under resourced languages focused mainly on Automatic Speech Recognizer. We reviewed methods and frameworks on dialogue management. Design of Partially Observable Markov Decision Processes (POMDP) based dialogue manager, which provides a principled framework to plan under uncertainty, yields robustness. Low performing Spoken Dialogue System (SDS) components, especially the Automatic Speech Recognizer (ASR) were considered the causes of uncertainty. Maintaining multiple hypotheses (evidences) improves the correctness of the dialogue manager. We conducted experiments to test correctness score, error rate and robustness. With a maximum of 6 N-best list and 20 partitions the correctness score grow by 14.19%. Increasing the number of n-best list of 6 reduced the error rate by 5.78% and with 6 n-best list. The belief updated below 0.12 seconds with 20 partitions and 6-nbest list. And the dialogue manager was able to complete a task with an average 8.75 turns by 50% Word Error Rate. The finding from the research illustrates that POMDP-based design approach to dialogue management is robust and possible to develop an improving spoken dialogue system for under-resourced languages.

A primary goal in robotics research is to provide means for mobile platforms to perform autonomously within their environment. Depending on the task at hand, autonomous performance can be defined as the execution by the robot, without human intervention, of certain navigational tasks. In mobile robotics literature, commonly addressed navigation tasks include the localization, mapping, path planning and obstacle avoidance tasks. Solving any of these tasks is a hard problem by itself. The reason stems from the inherent complexity associated with both the robot and its environment, each of them being an extremely complex dynamical system with uncertainty involved.

In this thesis, we propose a probabilistic framework for mobile robot navigation in dynamic environments based on the Partially Observable Markov Decision Process (POMDP) model. The proposed model is able to perform the navigation tasks of localization, path planning and obstacle avoidance. POMDPs are models for sequential decision making where the world in which the robot operates is partially observable, i.e. the true underlying robot's state is not known, and the outcome of actions it executes is modelled probabilistically. As such POMDPs perform localization and path planning in a probabilistic manner.

POMDPs have the major shortcoming of their extreme computational complexity and hence they have been mainly used in robotics as high level path planners only. In this thesis, we propose a novel hierarchical representation of POMDPs, specifically designed for the autonomous robot navigation problem and termed as the Robot Navigation-Hierarchical POMDP (RN-HPOMDP). The proposed hierarchical POMDP can efficiently model large real-world environments and is amenable to real time solution. This is achieved mainly due to the design choice of modelling the state transition and observation functions dependent only on the robot motion model and not on the environment as it is commonly used in the POMDP literature. Furthermore, the notion of the reference POMDP (rPOMDP) is introduced that infers the robot motion model in a very small POMDP and it transfers this information to the hierarchical structure while being solved. The environment specific information is modelled within the reward function of the RN-HPOMDP. The employed model is utilized as a unified probabilistic navigation framework that accommodates for localization, path planning and obstacle avoidance. Hence, real-time solution of the RN-HPOMDP is essential since no other external modules are utilized and paths have to be replanned at each time step.

The RN-HPOMDP has been developed for the application of robot navigation in dynamic real-world environments that are highly populated. Thus, it is desirable for the robot to perform obstacle avoidance in a manner that resembles the human motion for obstacle avoidance. That is, the robot should be able to decide the most suitable obstacle avoidance behavior based on the state of the environment. Therefore, the robot can decide to either perform a detour or follow a completely new path to the goal and also modify its speed of movement (increase it or decrease it) to bypass an obstacle or let it move away respectively. Any of the above four distinct behaviors for obstacle avoidance should be decided well before the robot comes too close to the obstacle. For that reason, future motion prediction of obstacles is employed. Two kinds of prediction are utilized: short-term and long-term prediction. Short term prediction refers to the one-step ahead prediction whereas long-term prediction refers to the prediction of the final destination point of the obstacle's movement. Both kinds of prediction are integrated into the reward function of the RN-HPOMDP and the speed decision is performed through a modified solution of the RN-HPOMDP. As a result, the RN-HPOMDP can decide the optimal obstacle avoidance behavior based on the current and the predicted state of the environment without the intervention of any other external module.

Experimental results have shown the applicability and effectiveness of the proposed framework for the navigation task. The robustness and the probabilistic nature of the RN-HPOMDP as well as the future motion prediction are required to be able to perform efficiently and effectively in dynamic real-world environments that are highly populated.

This paper presents a simple method for exact online inference and approximate decision making, applicable to large or partially observable Markov decision processes. The approach is based on a closed form Bayesian inference procedure for a class of context models which contains variable order Markov decision processes. The models can be used for prediction, and thus for decision theoretic planning. The other novel step of this paper is to use the belief (context distribution) at any given time as a compact representation of system state, in a manner similar to predictive state representations. Since the belief update is linear time in the worst case, this allows for computationally efficient value iteration and reactive learning algorithms such as Q-learning for this class of models.

Designing and developing affective dialogue systems have been receiving much interest from the dialogue research community . Previous work was mainly focused on showing the system's emotion to the user in order to achieve designer's goals such as helping students practice nursing tasks or persuading users to change their dietary behavior . A more challenging issue is to develop a robust dialogue manager that can infer the user's affective state and adapt the system behavior accordingly. For example, in the automatic call routing domain, if the system detects that the user is irritated, the system should verify the information provided by the user or forward the call to a human operator .

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for decentralized decision making under uncertainty. However, they typically model a problem at a low level of granularity, where each agent's actions are primitive operations lasting exactly one time step.  We address the case where each agent has macro-actions:  temporally extended actions which may require different amounts of time to execute. We model macro-actions as \emph{options} in a factored Dec-POMDP model, focusing on options which depend only on information available to an individual agent while executing. This enables us to model systems where coordination decisions only occur at the level of deciding which macro-actions to execute, and the macro-actions themselves can then be executed to completion. The core technical difficulty when using options in a Dec-POMDP is that the options chosen by the agents no longer terminate at the same time. We present extensions of two leading Dec-POMDP algorithms for generating a policy with options and discuss the resulting form of optimality. Our results show that these algorithms retain agent coordination while allowing near-optimal solutions to be generated for significantly longer horizons and larger state-spaces than previous Dec-POMDP methods.

This paper introduces timeline trees, which are partial models of partially observable environments. Timeline trees are given some specific predictions to make and learn a decision tree over history. The main idea of timeline trees is to use temporally abstract features to identify and split on features of key events, spread arbitrarily far apart in the past (whereas previous decision-tree-based methods have been limited to a finite suffix of history). Experiments demonstrate that timeline trees can learn to make high quality predictions in complex, partially observable environments with high-dimensional observations (e.g. an arcade game).

An effective collision avoidance system for unmanned aircraft will enable them to fly in civil airspace and greatly expand their applications. One promising approach is to model aircraft collision avoidance as a partially observable Markov decision process (POMDP) and automatically generate the threat resolution logic for the collision avoidance system by solving the POMDP model. However, existing discrete-state POMDP algorithms cannot cope with the high-dimensional state space in collision avoidance POMDPs. Using a recently developed algorithm called Monte Carlo Value Iteration (MCVI), we constructed several continuous-state POMDP models and solved them directly, without discretizing the state space. Simulation results show that our 3-D continuous-state models reduce the collision risk by up to 70 times, compared with earlier 2-D discrete-state POMDP models. The success demonstrates both the benefits of continuous-state POMDP models for collision avoidance systems and the latest algorithmic progress in solving these complex models.

Automatic Target Recognition (ATR) algorithm performance is highly dependent on the sensing conditions under which the input data is collected. Open-loop fly-bys often produce poor results due to less than ideal measurement conditions. In addition, ATR algorithms must be extremely complicated to handle the diverse range of inputs with a resulting reduction in overall performance and increase in complexity. Our approach, closed-loop ATR (CL-ATR), focuses on improving the quality of information input to the ATR algorithms by optimizing motion, sensor settings and team (vehicle-vehicle-human) collaboration to dramatically improve classification accuracy. By managing the data collection guided by predicted ATR performance gain, we increase the information content of the data and thus dramatically improve ATR performance with existing ATR algorithms. CL-ATR has two major functions; first, an ATR utility function, which represents the performance sensitivity of ATR-produced classification labels as a function of parameters that correlate to vehicle/sensor states. This utility function is developed off-line and is often available from the original ATR study as a confusion matrix, or it can be derived through simulation without direct access to the inner working of the ATR algorithm. The utility function is inserted into our CL-ATR framework to autonomously control the vehicle/sensor. Second, an on-board planner maps the utility function into vehicle position and sensor collection plans. Because we only require the utility function on-board, we can activate any ATR algorithm onto a unmanned aerial vehicle (UAV) platform no matter how complex. This pairing of ATR performance profiles with vehicle/sensor controls creates a unique and powerful active perception behavior.

Simulation is becoming more and more important for robotics research, especially for multi-robot system research. Simatch, originating from the Robot World Cup (RoboCup) Middle Size League (MSL) match, is proposed to simulate the highly dynamic confrontation between multi-robot systems and validate the adversarial multi-robot strategies. It consists of three independent subsystems: the simulation subsystem setting up the simulated environment, the strategy subsystem realizing the multi-robot strategies and the scene subsystem describing the specific scene. Due to the independence of these subsystems, it is easy to employ Simatch for various applications. Simatch is validated by a series of tests, including a test of the omnidirectional locomotion, a simulated MSL match among ten robots and a simulated encirclement between sixteen robots. Since 2016, Simatch has been the official software for the MSL simulation match in the Chinese Robotics Competition. Besides, an MSL simulation project based on Simatch has been proposed to promote the development of MSL in the 2017 MSL international workshop. The proposed simulation system facilitates the development of multi-robot cooperation algorithms and related research.

A long-standing goal of AI is to enable robots to plan in the face of uncertain and incomplete information, and to handle task failure intelligently. This paper shows how to achieve this. There are two central ideas. The first idea is to organize the robot's knowledge into three layers: instance knowledge at the bottom, commonsense knowledge above that, and diagnostic knowledge on top. Knowledge in a layer above can be used to modify knowledge in the layer(s) below. The second idea is that the robot should represent not just how its actions change the world, but also what it knows or believes. There are two types of knowledge effects the robot's actions can have: epistemic effects (I believe X because I saw it) and assumptions (I'll assume X to be true). By combining the knowledge layers with the models of knowledge effects, we can simultaneously solve several problems in robotics: (i) task planning and execution under uncertainty; (ii) task planning and execution in open worlds; (iii) explaining task failure; (iv) verifying those explanations. The paper describes how the ideas are implemented in a three-layer architecture on a mobile robot platform. The robot implementation was evaluated in five different experiments on object search, mapping, and room categorization.

The Multi-Agent paradigm is becoming increasingly popular as a way of capturing complex control processes with stochastic properties. Many existing modelling tools are not flexible enough for these purposes, possibly because many of the modelling frameworks available inherit their structure from single agent frameworks. This paper proposes a new family of modelling frameworks called FASP, which is based on state encapsulation and powerful enough to capture multi-agent domains. It identifies how the FASP is more flexible, and describes systems more naturally than other approaches, demonstrating this with a number of robot football (soccer) formulations. This is important because more natural descriptions give more control when designing the tasks, against which a group of agents' collective behaviour is evaluated and regulated.

The default mode network (DMN) is believed to subserve the baseline mental activity in humans. Its higher energy consumption compared to other brain networks and its intimate coupling with conscious awareness are both pointing to an unknown over-arching function. Many research streams speak in favor of an evolutionarily adaptive role in envisioning experience to anticipate the future. In the present work, we propose a process model that tries to explain how the DMN may implement continuous evaluation and prediction of the environment to guide behavior. The main purpose of DMN activity, we argue, may be described by Markov decision processes that optimize action policies via value estimates through vicarious trial and error. Our formal perspective on DMN function naturally accommodates as special cases previous interpretations based on (a) predictive coding, (b) semantic associations, and (c) a sen-tinel role. Moreover, this process model for the neural optimization of complex behavior in the DMN offers parsimonious explanations for recent experimental findings in animals and humans. K E Y W O R D S artificial intelligence, human intelligence, systems neuroscience

Partially observable Markov decision processes (POMDPs) have been successfully applied to various robot motion planning tasks under uncertainty. However, most existing POMDP algorithms assume a discrete state space, while the natural state space of a robot is often continuous. This paper presents Monte Carlo Value Iteration (MCVI) for continuous-state POMDPs. MCVI samples both a robot's state space and the corresponding belief space, and avoids inefficient a priori discretization of the state space as a grid. Both theoretical results and preliminary experimental results indicate that MCVI is a promising new approach for robot motion planning under uncertainty.

The Multi-Agent paradigm is becoming increasingly pop- ular as a way of capturing complex control processes with stochastic properties. Many existing modelling tools are not flexible enough for these purposes, possibly because many of the modelling frame- works available inherit their structure from single agent frameworks. This paper proposes a new family of modelling frameworks called FASP, which is based

Aberrant decision-making characterizes various pediatric psychopathologies; however, deliberative choice strategies have not been investigated. A transdiagnostic sample of 95 youths completed a child-friendly sequential sampling paradigm. Participants searched for the best offer by sampling a finite list of offers. Participants’ willingness to explore was measured as the number of offers sampled, and ideal task performance was modeled using a Markov decision-process model. As in previous findings in adults, youths explored more offers when lists were long compared with short, yet participants generally sampled fewer offers relative to model-estimated ideal performance. Searching deeper into the list was associated with choosing better price options. Analyses examining the main and interactive effects of transdiagnostic anxiety and irritability symptoms indicated a negative correlation between anxiety and task performance (p = .01, ηp2 = .08). Findings suggest the need for more research on exploratory decision impairments in youths with anxiety symptoms.

An effective collision avoidance system for unmanned aircraft will enable them to fly in civil airspace and greatly expand their applications. One promising approach is to model aircraft collision avoidance as a partially observable Markov decision process (POMDP) and automatically generate the threat resolution logic for the collision avoidance system by solving the POMDP model. However, existing discrete-state POMDP algorithms cannot cope with the high-dimensional state space in collision avoidance POMDPs. Using a recently developed algorithm called Monte Carlo Value Iteration (MCVI), we constructed several continuous-state POMDP models and solved them directly, without discretizing the state space. Simulation results show that our 3-D continuous-state models reduce the collision risk by up to 70 times, compared with earlier 2-D discrete-state POMDP models. The success demonstrates both the benefits of continuous-state POMDP models for collision avoidance systems and the latest algorithmic progress in solving these complex models.

With rapid profusion of video data, automated surveillance and intrusion detection is becoming closer to reality. In order to provide timely responses while limiting false alarms, an intrusion detection system must balance resources (e.g., time) and accuracy. In this paper, we show how such a system can be modeled with a partially observable Markov decision process (POMDP), representing possible computer vision filters and their costs in a way that is similar to human vision systems. The POMDP representation can be optimized to produce a dynamic sequence of operations and achieve a tradeoff between time and detection quality, taking into account uncertainty in the filter predictions. In a set of experiments on actual video data, we show that our method can both outperform static "expert" models and scale to large dynamic domains. These results suggest that our method could be used in real-world intrusion detection systems.

Peshkin et al.'s example is illustrated in Fig. 3. There are two team-mates, V1 and V2, and an opponent O, each agent has partial observ-ability and can only see if the 4 horizontally and vertically adjacent squares are occupied, or not. Also, players V1 and V2 have an extra ...

We describe a simple method for exact online coinfernece and decision making for partially observable and large Markov decision processes. This is based on a closed form Bayesian update procedure for certain classes of models exhibiting a special conditional independence structure, which can be used for prediction, and consequently for planning.

Partially-observable Markov decision processes provide a very general model for decision-theoretic planning problems, allowing the trade-offs between various courses of actions to be determined under conditions of uncertainty, and incorporating partial observations made by an agent. Dynamic programming algorithms based on the information or belief state of an agent can be used to construct optimal policies without explicit consideration of past history, but at high computational cost. In this paper, we discuss how structured representations of the system dynamics can be incorporated in classic POMDP solution algorithms. We use Bayesian networks with structured conditional probability matrices to represent POMDPs, and use this representation to structure the belief space for POMDP algorithms. This allows irrelevant distinctions to be ignored. Apart from speeding up optimal policy construction, we suggest that such representations can be exploited to great extent in the development of useful approximation methods.