POMDP

This is an overview of partially observable Markov decision processes (POMDPs). We describe POMDP value and policy iteration as well as gradient ascent algorithms. The emphasis is on solution methods that work directly in the space of policies.

Today within the AAMAS community, we see at least four competing approaches to building multiagent systems: beliefdesire-intention (BDI), distributed constraint optimization (DCOP), distributed POMDPs, and auctions or game-theoretic approaches. While there is exciting progress within each approach, there is a lack of cross-cutting research. This paper highlights hybrid approaches for multiagent teamwork. In particular, for the past decade, the TEAMCORE research group has focused on building agent teams in complex, dynamic domains. While our early work was inspired by BDI, we will present an overview of recent research that uses DCOPs and distributed POMDPs in building agent teams. While DCOP and distributed POMDP algorithms provide promising results, hybrid approaches help us address problems of scalability and expressiveness. For example, in the BDI-POMDP hybrid approach, BDI team plans are exploited to improve POMDP tractability, and POMDPs improve BDI team plan performance. We present some recent results from applying this approach in a Disaster Rescue simulation domain being developed with help from the Los Angeles Fire Department.

Partially Observable Markov Decision Processes (POMDPs) have been demonstrated empirically to be good models for robust spoken dialogue design. This chapter shows that such models are also very appropriate for designing affective dialogue systems. We describe how to model affective dialogue systems using POMDPs and propose a novel approach to develop an affective dialogue model using factored POMDPs. We apply this model for a single-slot route navigation dialogue problem as a proof of concept. The experimental results demonstrate that integrating user's affect into a POMDP-based dialogue manager is not only a nice idea but is also helpful for improving the dialogue manager performance given that the user's affect influences their behavior. Further, our practical findings and experiments on the model tractability are expected to be helpful for designers and researchers who are interested in practical implementation of dialogue systems using the state-of-the-art POMDP techniques.

A long-standing goal of AI is to enable robots to plan in the face of uncertain and incomplete information, and to handle task failure intelligently. This paper shows how to achieve this. There are two central ideas. The first idea is to organize the robot's knowledge into three layers: instance knowledge at the bottom, commonsense knowledge above that, and diagnostic knowledge on top. Knowledge in a layer above can be used to modify knowledge in the layer(s) below. The second idea is that the robot should represent not just how its actions change the world, but also what it knows or believes. There are two types of knowledge effects the robot's actions can have: epistemic effects (I believe X because I saw it) and assumptions (I'll assume X to be true). By combining the knowledge layers with the models of knowledge effects, we can simultaneously solve several problems in robotics: (i) task planning and execution under uncertainty; (ii) task planning and execution in open worlds; (iii) explaining task failure; (iv) verifying those explanations. The paper describes how the ideas are implemented in a three-layer architecture on a mobile robot platform. The robot implementation was evaluated in five different experiments on object search, mapping, and room categorization.

In a spoken dialog system, determining which action a machine should take in a given situation is a difficult problem because automatic speech recognition is unreliable and hence the state of the conversation can never be known with certainty. Much of the research in spoken dialog systems centres on mitigating this uncertainty and recent work has focussed on three largely disparate techniques: parallel dialog state hypotheses, local use of confidence scores, and automated planning. While in isolation each of these approaches can improve action selection, taken together they currently lack a unified statistical framework that admits global optimization. In this paper we cast a spoken dialog system as a partially observable Markov decision process (POMDP). We show how this formulation unifies and extends existing techniques to form a single principled framework. A number of illustrations are used to show qualitatively the potential benefits of POMDPs compared to existing techniques, and empirical results from dialog simulations are presented which demonstrate significant quantitative gains. Finally, some of the key challenges to advancing this method -in particular scalability -are briefly outlined.

In most of the papers on inventory models operating in a random environment, the state of the environment in each period is assumed to be fully observed with perfect information. However, this assumption is not realistic in most real-life situations and we provide a remedy in this paper by assuming that the environment is only partially observed with imperfect information. We accomplish this by analyzing two formulations of single-item models with periodic-review and random supply in a random environment. In the first one, supply is random due to random capacity of production and random availability of transportation. We show that state-dependent base-stock policy is optimal if the capacity and all costs are observed, while demand and availability are unobserved. In the second model, we consider a model with random availability only with fixed-ordering cost. We show that state-dependent (s,S) policy is optimal if the availability process is observable.

A new approach to solve the sensor control problem is proposed, formulated based on multi-object Bayes filtering in the partially observable Markov decision process (POMDP) context, where the multi-object states are assumed to be random finite sets with multi-Bernoulli distributions. We introduce a novel cost function that is reliable in real-time environment. In each filtering iteration, after predicting the multi-Bernoulli parameters, estimates for the number and states of the targets are extracted. For each admissible control command, Monte-Carlo samples of measurements corresponding to the estimated target states are generated. Then, for each measurement sample, the CB-MeMBer update is performed and the average cost function is computed. The best command is the one incurring the minimum cost. The simulation results involve a challenging case of detecting and tracking up to 5 manoeuvring targets using a controllable sensor, and show that our method outperforms competing methods both in terms of tracking accuracy (measured in using OSPA metric) and in terms of computational cost.

We demonstrate the In Situ testbed, a system that aids in evaluating computational models of learning, including artificial neural networks. The testbed models contingencies of reinforcement using an extension of Mechner's (1959) notational system for the description of behavioral procedures. These contingencies are input to the model under test. The model's output is displayed as cumulative records. The cumulative record can then be compared to one produced by a pigeon exposed to the same contingencies. The testbed is tried with three published models of learning. Each model is exposed to up to three reinforcement schedules (testing ends when the model does not produce acceptable cumulative records): continuous reinforcement and extinction, fixed ratio, and fixed interval. The In Situ testbed appears to be a reliable and valid testing procedure for comparing models of learning.

This paper considers a scenario in which a secondary user (SU) opportunistically accesses a channel allocated to some primary network (PN) that switches between idle and active states in a time-slotted manner. At the beginning of each time slot, SU can choose to stay idle or to carry out spectrum sensing to detect the state of PN. If PN is detected to be idle, SU can carry out data transmission. Spectrum sensing consumes time and energy and introduces false alarms and mis-detections. The objective is to dynamically decide, for each time slot, whether SU should stay idle or carry out sensing, and if so, for how long, to maximize the expected reward. We formulate this as a partially observable Markov decision process and prove important properties of the optimal control policies. Heuristic control policies with low complexity and good performance are also proposed. Numerical results show the significant performance gain of our dynamic control approach for opportunistic spectrum access.

We consider the problem of joint network coding and packet scheduling for multimedia transmission from the Access Point (AP) to multiple receivers in 802.11 networks. The state of receivers is described by a hidden Markov model and the AP acts as a decision maker which employs ...

This extended abstract discusses various approaches to the constraining of Partially Observable Markov Decision Processes (POMDPs) using social norms and logical assertions in a dynamic logic framework. Whereas the exploitation of synergies among formal logic on the one hand and stochastic approaches and machine learning on the other is gaining significantly increasing interest since several years, most of the respective approaches fall into the category of relational learning in the widest sense, including inductive (stochastic) logic programming. In contrast, the use of formal knowledge (including knowledge about social norms) for the provision of hard constraints and prior knowledge for some stochastic learning or modeling task is much less frequently approached. Although we do not propose directly implementable technical solutions, it is hoped that this work is a useful contribution to a discussion about the usefulness and feasibility of approaches from norm research and formal logic in the context of stochastic behavioral models, and vice versa.

Today within the AAMAS community, we see at least four competing approaches to building multiagent systems: beliefdesire-intention (BDI), distributed constraint optimization (DCOP), distributed POMDPs, and auctions or game-theoretic approaches. While there is exciting progress within each approach, there is a lack of cross-cutting research. This paper highlights hybrid approaches for multiagent teamwork. In particular, for the past decade, the TEAMCORE research group has focused on building agent teams in complex, dynamic domains. While our early work was inspired by BDI, we will present an overview of recent research that uses DCOPs and distributed POMDPs in building agent teams. While DCOP and distributed POMDP algorithms provide promising results, hybrid approaches help us address problems of scalability and expressiveness. For example, in the BDI-POMDP hybrid approach, BDI team plans are exploited to improve POMDP tractability, and POMDPs improve BDI team plan performance. We present some recent results from applying this approach in a Disaster Rescue simulation domain being developed with help from the Los Angeles Fire Department.

Today within the AAMAS community, we see at least four competing approaches to building multiagent systems: beliefdesire-intention (BDI), distributed constraint optimization (DCOP), distributed POMDPs, and auctions or game-theoretic approaches. While there is exciting progress within each approach, there is a lack of cross-cutting research. This paper highlights hybrid approaches for multiagent teamwork. In particular, for the past decade, the TEAMCORE research group has focused on building agent teams in complex, dynamic domains. While our early work was inspired by BDI, we will present an overview of recent research that uses DCOPs and distributed POMDPs in building agent teams. While DCOP and distributed POMDP algorithms provide promising results, hybrid approaches help us address problems of scalability and expressiveness. For example, in the BDI-POMDP hybrid approach, BDI team plans are exploited to improve POMDP tractability, and POMDPs improve BDI team plan performance. We present some recent results from applying this approach in a Disaster Rescue simulation domain being developed with help from the Los Angeles Fire Department.

Interactions between humans and service robots are more natural when emotions can be synthesized in those robots. Cognitive appraisal theory of emotions provides a theoretical basis for designing artificial emotion generation systems for robots. Computational algorithms to implement the cognitive appraisal theory were proposed in this research. The algorithms were based on a probabilistic description about the world. The proposed model was applied to a sample of interactive tasks, and the robot's emotions during the task execution can lead to a more positive human-robot interaction experience.

We present a framework for sensor actuation and control in sentient spaces, in which sensors are used to observe a physical phenomena. We focus on sentient spaces that enable pervasive computing applications, such as smart video surveillance and situational awareness in instrumented office environments. Our framework utilizes the spatiotemporal statistical properties of an observed phenomena, with the goal of maximizing an application-specified reward. Specifically, we define an observation of a phenomena by assigning it a discrete value (state) and we model its semantics as the transition between these values (states). This semantic model is used to predict the future states in which the phenomena is likely to be at, based on partially-observed past states. To accomplish realtime agility, we designed an approximate, adaptive-grid solution for Partially Observable Markov Decision Processes (POMDPs) that yields practically good results, and in some cases, guarantees on the quality of the approximation. We use our framework to control and actuate a large-scale camera network so as to maximize the number and type of captured events. To enable real-time control, we implement an action schedule using a table lookup and make use of a factored probability model to capture state semantics. To the best of our knowledge, we are the first to address the problem of actuating a large-scale sensor network based on a real-time POMDP formulation.

... Mürsel Yildiz, Ahmet Cihat Toker, Fikret Sivrikaya, Seyit Ahmet Camtepe, Sahin Albayrak ... assume certain implementation conventions and can not be applied to all products [7]. Similarly, delay time between scheduled and actual transmission time of periodic beacon frames can ...

This extended abstract discusses various approaches to the constraining of Partially Observable Markov Decision Processes (POMDPs) using social norms and logical assertions in a dynamic logic framework. Whereas the exploitation of synergies among formal logic on the one hand and stochastic approaches and machine learning on the other is gaining significantly increasing interest since several years, most of the respective approaches fall into the category of relational learning in the widest sense, including inductive (stochastic) logic programming. In contrast, the use of formal knowledge (including knowledge about social norms) for the provision of hard constraints and prior knowledge for some stochastic learning or modeling task is much less frequently approached. Although we do not propose directly implementable technical solutions, it is hoped that this work is a useful contribution to a discussion about the usefulness and feasibility of approaches from norm research and formal logic in the context of stochastic behavioral models, and vice versa.

ABSTRACT Dynamic Spectrum Access is a key capability of Cognitive Radio (CR) networks to increase the efficiency in the use of the available spectrum resources. In this respect, this paper focuses on the spectrum selection problem when a number of radio links has to be established in a CR network to support applications with different bit rate requirements. A novel strategy based on a Partially Observable Markov Decision Process (POMDP) is proposed, whose target is to maximize a reward function that reflects the suitability of the available spectrum blocks to the application requirements. The proposed strategy combines partial observations of the interference state in the different spectrum blocks together with a statistical characterization of the interference dynamics. Thanks to this feature, the performance comparison of the algorithm against different reference strategies reveals that it achieves a very similar performance than a strategy operating under full knowledge of the real interference state of all the spectrum blocks, while at the same time it has much less requirements in terms of measurement needs and associated signaling.

We consider the problem of joint network coding and packet scheduling for multimedia transmission from the Access Point (AP) to multiple receivers in 802.11 networks. The state of receivers is described by a hidden Markov model and the AP acts as a decision maker which employs a partially observable Markov decision process (POMDP) to optimize the media transmission. Importantly, we introduce a simulation-based dynamic programming algorithm as a solution tool for our POMDP abstract. Our simulation-based algorithm simplifies the modeling process as well as reduces the computational complexity of the solution process. Our simulation results demonstrate that the proposed scheme provides higher performance than the network coding scheme without using optimization techniques and traditional retransmission scheme.

The context is sensor control for multi-object Bayes filtering in the framework of partially observed Markov decision processes. The current information state is represented by the multi-object probability density function (PDF), while the reward function, associated with each sensor control (action), is the information gain measured by the alpha or Rényi divergence. Assuming that both the predicted and updated state can be represented by independent identically distributed (IID) cluster random finite sets (RFS's) or, as a special case, the Poisson RFS's, the paper derives the analytic expressions of the corresponding Rényi divergence based information gains. The implementation of Rényi divergence via the sequential Monte Carlo method is presented. The performance of the proposed reward function is demonstrated by a numerical example, where a moving range-only sensor is controlled to estimate the number and the states of several moving objects using the PHD filter.