Systems Challenges for Trustworthy Embodied Systems

EI Systems Challenges fortiss View on Systems Challenges for Trustworthy Embodied Intelligence1 Harald Rueß (ruess@fortiss.org) fortiss Research Institute of the Free State of Bavaria Guerickestr. 25 D-80805 München 10th of January, 2022. “I am an optimist and I believe that we can create AI for the good of the world. That it can work in harmony with us. We simply need to be aware of the dangers, identify them, employ the best possible practice and management, and prepare for its consequences well in advance” Stephen Hawkins at Web Summit in Nov. 2017 1 Disclaimer: this research is supported by the BMWi-funded project Embodied Intelligence - The Next Big Thing, and the CASSAI project as funded by the Bavarian Ministry of Economics in the context of the fortiss AI Center. This report heavily draws on the results on the Agenda CPS and ongoing discussions and constructive feedback to earlier versions by Prof. Dr. Dr. h.c. Manfred Broy, is strongly influenced by ongoing work by SRI colleagues on the rigorous design of increasingly autonomous machines, in particular Dr. John Rushby and Dr. Natarajan Shankar, and also exchanges during a research stint in November 2019. It is also based on regular exchanges with fortiss colleagues, in particular, Dian Balta, Dr. Markus Duchon, Dr. Johannes Kroß, Prof. Dr. Daniel Mendez, Prof. Dr. Rute Sofia, and Dr. Henrik Putzer. It has also benefitted from a recent BIG workshop on challenges for embodied systems with participating colleagues from Siemens and fortiss. The views in the section on robust AI have been developed while preparing the fortiss-IBM joint Center for AI, and the section on human-centered engineering is heavily based on the corresponding fortiss whitepaper by Dr. Yuanting Liu and Dr. habil. Hao Shen on this topic, and the views on uncertainty quantification have mainly been formed through interactions within the CASSAI projects with the fortissians Carmen Carlan, Amit Sahu, Tewodros Beyene, and Julian Bernhard. It is a safe assumption, however, that most probably none of the contributors mentioned above could agree with all the described hypotheses on systems challenges for embodied systems as outlined here. 1 EI Systems Challenges Table of Contents TABLE OF CONTENTS ..............................................................................................................................2 1. INTRODUCTION ...........................................................................................................................3 2. EMBODIED SYSTEMS .................................................................................................................7 SOFTWARE ACTORS.................................................................................................................................8 ROBOTIC COMPANIONS...........................................................................................................................9 SERVICE FEDERATIONS .........................................................................................................................11 3. CHARACTERISTICS ..................................................................................................................14 COGNITIVE SYSTEMS ............................................................................................................................15 INTENT-DRIVEN SYSTEMS.....................................................................................................................17 FEDERATED SYSTEMS ...........................................................................................................................18 AUTONOMOUS SYSTEMS .......................................................................................................................19 SELF-LEARNING SYSTEMS....................................................................................................................20 4. TRUSTWORTHINESS ................................................................................................................22 5. CHALLENGES .............................................................................................................................27 ROBUST AI/ML .....................................................................................................................................28 HUMAN-CENTERED AI/ML ..................................................................................................................31 COGNITIVE ARCHITECTURES...............................................................................................................34 UNCERTAINTY QUANTIFICATION ........................................................................................................36 SELF-INTEGRATION ..............................................................................................................................40 ANALYSIS ...............................................................................................................................................43 TESTING ..................................................................................................................................................43 SYMBOLIC ANALYSIS .............................................................................................................................45 RUNTIME VERIFICATION AND RECOVERY..............................................................................................46 ASSURANCE............................................................................................................................................48 6. CONCLUSIONS ...........................................................................................................................56 2 EI Systems Challenges 1. I NTRODUCTION A new generation of increasingly autonomous machinery is about to be developed and embodied into all kinds of aspects of everyday life. This machinery is used beyond mere automation and assistance to humans, as manufacturing robots make way for autonomous machine workers, business and administrational services are performed by autonomous virtual organizations, and processes and value chains in the real and the virtualized worlds are executed by coalitions of autonomous machine actors. In this way, computational machines are acting as self-sufficient entities in our very economic and societal fabric. Next generation autonomous machines may also acquire and improve necessary skills and behavior in real world contexts by means of action, reaction, and interaction in and with physical and social environments. 2 But how exactly do the required cognitive skills and behavior that could possibly be termed intelligent3 unfold in time? Various answers to this fundamental quest on the emergence of intelligence, including Descartes’ brain-body duality, have been developed throughout the history of philosophy. These considerations have also been motivating the field of embodied intelligence (EI), which is the computational approach to the design and understanding of intelligent behavior in embodied and situated actors through the consideration of the strict coupling between the actor and its environment (situatedness), mediated by the constraints of the actor’s own body, perceptual and motor system, and brain (embodiment).4 Based on this paradigm it could be demonstrated that computational machines may develop and improve at least some physical dexterities and cognitive skills by means of interactions between 2 Striedinger, Kaspar, der „rätselhafte Findling“, Lebensläufe aus Franken, Herausgegeben im Auftrag der Gesellschaft für Fränkische Geschichte von Anton Chrous, III. Band, 1927. 3 Deary, Intelligence: A very short introduction, Oxford University Press, 2020. 4 Cangelosi, Bongard, Fischer, Nolfi. Embodied Intelligence. Handbook of Computational Intelligence, Springer, 2015. 3 EI Systems Challenges the physical self and the physical environment.5 6 7 8 9 10 11 12 13 14 In this way, EI robots have been built that can move, see, speak, and interact with other robots effectively.15 16 Current approaches to EI also rely on training actors in virtualized playgrounds for accelerated learning and for enabling non-destructive learning from failure.17 18 19 20 Despite all this progress on EI, however, it still remains open to speculation if and how “higher-level” thinking, symbol grounding, natural language understanding, consciousness, and emotions may emerge in machines through embodiment.21 22 23 A central but largely untouched challenge of EI is concerned with embodying computational actors into real-world environments with all their intricacies, including uncertainty, complexity, and unpredictability. This is not an easy task as actors may demonstrate unexpected and even emergent behavior when deployed in new operating environments; for example, when being moved from a model-based virtual playground into a real-world physical context.24 5 There is growing evidence that at least some sensorimotor behavior and cognitive skills are realized in the body and that social interactions can bootstrap learning (for example, Ballard et al, Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 1997; McGeer. Passive dynamic walking. International Journal of Robotics Research, 1990; O’Regan, Noe. A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 2001) 6 Pfeifer, Scheier. Understanding intelligence, MIT Press, 2001. 7 Breazeal. Designing sociable robots. MIT Press, 2002. 8 Beer: A dynamical systems perspective on agent-environment interaction. Artificial Intelligence 72, 1994. 9 Brooks: Elephants don’t play chess, Robotics and Autonomous Systems 6(1), 1990. 10 Cangelosi: Grounding language in action and perception: From cognitive agents to humanoid robots, Physics of Life Reviews, 7(2), 2010. 11 Chiel, Beer: The brain has a body: Adaptive behavior emerges from interactions of nervous system, body and environment. Trends in Neurosciences 20, 1997. 12 Keijzer: Representation and Behavior, MIT Press, 2001. 13 Nolfi, Floreano: Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines, MIT Press, 2000. 14 Pfeifer, Bongard: How the Body Shapes the Way We Think: A New View of Intelligence, MIT Press, 2006. 15 Workshop on Embodied AI at CVPR’21 (https://embodied-ai.org) 16 Pfeifer, Iida, Embodied Artificial Intelligence, 2003. 17 Such as visual exploration, visual navigation, and embodied question-answering. In particular, the question ”Is there still milk in the fridge?” may require the embodied actor to unlock new insights that are momentous in answering and to initiate corresponding tasks such as moving to the kitchen and opening the fridge. 18 For example, workshop on Embodied AI, http://https://embodied-ai.org/, CVPR, 2020. 19 For example, ongoing Embodied AI programmes, for example, at Intel Labs or Facebook AI Research. 20 These simulation environments currently include SUNCG, Matterport3D, iGibson, Replica, Habitat, and DART. 21 Smith, Gasser, The development of embodied cognition: Six lessons from babies, Artificial life, vol. 11, no. 1-2, 2005. 22 Rushby, Sanchez, Technology and Consciousness, SRI Technical Report, 2018. 23 For example, Integrated information theory (IIT) asserts that it is the intrinsic causal powers of the brain that really matter. And those powers cannot be simulated but must be part and parcel of the physics of the underlying mechanism. 24 Remark: the very notion of a “digital twin” suggests an unequivocalness of the model with respect to the real world no model can actually deliver; as Plato already realized such a model is nothing more than a “shadow on the wall”. 4 EI Systems Challenges Another key point of moving actors from innocent training playgrounds into a physical or social context is that actions have real consequences. Tay,25 a pretrained chatbot, who quickly turned nasty, when put into the context of a (so-called) social network, may serve as cautionary tale of what can go wrong all too easily and quickly. While this problem was solved by simply taking Tay offline, it is easy to imagine other cases of self-learning actors neglecting human moral expectations with long-lasting and perhaps even more severe impact. The sad tale of Tay also reminds us on the importance of a meaningful human control of EI. We clearly face serious social, economic, and legal challenges when deploying EI in the real world,26 as it is crucial to coordinate the behavior of EI actors in a beneficial manner, ensure their compatibility with our human-centered social norms and values, and design verifiably safe and reliable human-machine interaction. Notice that interpretations of the terms “beneficial” and “social norms and values” are heavily context- and application-specific. A simple robot companion, for example, might be called beneficial if it acts according to the intents of its human companion. This does not necessarily imply, however, that the robot companion acts according to the intents of others, which may even be contradictory. Let alone that it acts according to relevant human-centered social values. In our quest for developing and deploying EI in a meaningful way we therefore need to find adequate solutions to central engineering challenges: § How can we ensure that an EI actor behaves beneficially? That is, it functions as intended and it behaves in accordance with higher-level societal goals and standards. § How can we ensure that learning-enabled EI actors are robust across their whole lifecycle? That is, the behavior or EI-enabled actors is dependable, safe, and predictable (up to quantified tolerances) in uncertain, complex, and unpredictable environments. § How can we ensure that a meaningful (human) control over an EI actor is enabled during operation? Traditionally, the field of systems engineering tackles these kinds of questions for assuring quality-of-service and the dependable functioning of software-intensive systems. Systems engineering, however, has so far mainly been concerned with relatively small-scale, centralized, deterministic, non-evolvable, automated, and task-specific embedded and cyberphysical systems (CPS), which are operating in well-defined and largely predictable operating environments. The state-of-the-practice in safety engineering,27 for instance, is restricted to deterministic systems operating in well-defined operating contexts, and it usually relies on 25 https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist 26 Vazdanan et al., Responsibility Research for Trustworthy Autonomous Systems, 2021. 27 For instance, industrial safety engineering standards such as DO 178C in aerospace and ISO 26262 in the automotive industry. 5 EI Systems Challenges ultimate fallback mechanisms to a human operator (as is the case, for example, for current airplane autopilots). Current safety engineering practice therefore does not support certification and corresponding operating readiness of the envisioned new generation of EI acting in realworld environments. In Section 2 we introduce the notion of embodied systems. This class of systems is based on the central EI concepts of situatedness and embodiment. As such, the design of embodied systems may benefit from any progress on EI, but their success doesn’t hinge on reaching EI’s ultimate quest for human-like machine intelligence. We illustrate the defining features and the disruptive potential of embodied systems by means of three different scenarios, namely software assistants, robot companions, and federations of services. In Section 3 we analyze the main characteristics of embodied systems, and in Section 4 we discuss how to possibly increase trust into embodied systems. In Section 5 we deduce system challenges for designing trustworthy embodied systems from the main characteristics of embodied systems. We conclude with some final remarks in Section 6. 6 EI Systems Challenges 2. E MBODIED S YSTEMS In contrast to traditional EI, which is largely driven by its fundamental quest for emerging intelligent behavior in computational machines, we are embracing a seemingly utilitarian stance in that the embodiment of actors in a real-world context, physical or social, serves § the development of flexible, inventive, and optimized actions § for beneficial, goal-oriented, and robust behavior § in uncertain, complex, and unpredictable real-world environments. The central challenge is to enable the beneficial, dependable, safe, and predictable, at least up to acceptable tolerances, operation of embodied actors when woven into our very economic and social fabric. Actors28 are supposed to purposefully cooperate with other actors, both human and robot. As such, embodied systems are supporting humans in various real-world tasks, and in an increasingly autonomous, self-guided manner. Embodied systems, however, should not be restricted to an assistance role. Instead, their self-learning capabilities should well support the co-evolution of humans and machine as necessary for bootstrapping overall capabilities.29 Embodied systems 30 may be viewed as increasingly autonomous and self-learning cyberphysical systems. 31 More specifically, embodied systems are decentralized, dynamic, selflearning, and self-organizing federations of actors that collaborate to accomplish complex tasks and missions in partially unpredictable real-world operating contexts; these systems therefore are: § Situational in that they are aware of the operating context and itself. § Embodied in that real-world actions, reactions, and interactions are instrumental for the flexible development of inventive actions for goal-oriented and robust behavior in uncertain, complex, and unpredictable real-world environments. § Open to interact and collaborate with others in a mutually synergistic manner, while still operating as self-sufficient individually purposeful systems. 28 Notice that we use the terms embodied system and embodied actor almost interchangeably. 29 Bardini, Bootstrapping : Douglas Engelbart, Coevolution, and the Origins of Personal Computing, Stanford University Press, 2000. 30 Notice that we use the terms embodied system and embodied actors almost interchangeably; if the embodied system is regarded as a selfsufficient component, however, we tend to call it an actor. Moreover, populations of actors are usually refered to as embodied systems. 31 Broy, Geisberger (Eds.), Agenda CPS, acatech, 2013 (https://www.fortiss.org/en/results/scientific-publications/details/agenda-cpsintegrierte-forschungsagenda-cyber-physical-systems) 7 EI Systems Challenges § Adaptive in that they may adjust and improve behavior through experience and targeted exploration. Some scenarios might help to illustrate the main concepts, benefits, and challenges of embodied systems. S OFTWARE ACTORS Consider a mission- and communication-oriented software-based actor32 which autonomously gathers information, for example, from seminar organizers, composes announcements of next week's seminars, and mails them each week to a list that it keeps updated, all without the supervision of a human. Similarly, consider a software companion for automated assignment of tasks to employees in a large organization that automates the workload of hundreds of human counterparts. Automated software assistants typically are “embodied” in and interact with a largely virtual world of information sources, with a dedicated interface to a human operator. These kinds of autonomous software assistants do exist, with their main architectural principles rooted in cognitive architectures such as the global workspace theory 33 or multi-actor communication frameworks such as KQML 34 or OAA. 35 They have often been shown to outperform human counterparts on many real-life exploration and complex planning, scheduling, and dispatching tasks such as customer service, risk evaluation, product inspection, and data mining. Their actions are based on a situational awareness of the operating environment together with built-in intents and goals for managing the unpredictable. Traditional autonomous software assistants, however, have been lacking main attributes of embodied systems as they are purpose-built for solving specific tasks only, and their ability to adapt to changing environments through interactions with their environment, including human operators, is limited. We can easily imagine a new generation of integrated and more versatile software-based actor, which are embodied in the sense that they are learning our intents through collaborative interactions, and which are able to automatically detect and adapt to changing conditions (for example, a specific source of information is temporarily unavailable), thereby also optimizing future problem-solving strategies. Such a personal software assistant should be able to largely automate managerial tasks such as doing taxes, planning trips, ordering food, or controlling household appliances according to an overall situational assessment and intents. 32 The actor paradigm in AI is based upon the notion of reactive, internally motivated entities embedded in changing, uncertain worlds which they perceive, and in which they act. 33 Franklin, Patterson, The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous, Software Agent, IDPT, 2006. 34 Finin,Labrou,Mayfield, KQML as an agent communication language, Software Agents, MIT Press, 1997. 35 Martin, Cheyer, Moran, The Open Agent Architecture: A Framework for Building Distributed Software Systems, Applied AI, 1999. 8 EI Systems Challenges R OBOTIC COMPANIONS 36 Figure 1 depicts an embodied system with the sole purpose of landing an airplane. In a slightly more complex scenario, we envision such a robot to act as a co-pilot in a single-pilot cockpit. In this way, on long flights with two pilots, one can sleep while the other flies with the assistance from the robotic companion. Such a robot needs to be more like a human copilot than conventional flight management system or functionally automated autopilot. In particular, the robot companion needs to perform heterogeneous and complementary tasks including, say, radio communications, interpreting weather data and making route adjustments, pilot monitoring tasks, shared tasks (flaps, gear), ground taxi, and communication with the cabincrew (emergency evacuation). The robot companion also needs to integrate these tasks to accomplish a safe flight, it needs to base its decisions and actions on an overall situational assessment. In case things go wrong, the robot companion needs to find effective explanations based on fault diagnosis, and it needs to engage in an effective resolution process with the (human) pilot, based on a model of the pilot’s beliefs. Figure 1. Robotic copilot.37 In extreme situations the automated robot copilot might even take over control; for instance, if there is smoke in the cockpit. The robotic copilot might also be better suited to maintain pitch and thrust in extreme situation as the human pilot. 38 In these rare cases, the “robot” must now also cope with inconsistencies (for example, in sensor readings) based on flight laws, training procedures, models of the physical environment, and unforeseen situations without the possibility of a structured hand-over to the human pilot. In extreme cases where flight laws 36 This scenario is heavily based on the one presented by J. Rushby on increasingly autonomous systems. 37 Source: HTTPS :// WWW . AEROTIME . AERO /23374- ROBOT - CO - PILOT - FLIES - LANDS - BOEING -737 38 For instance, http://understandingaf447.com/extras/18-4_minutes__23_seconds_EN.pdf 9 EI Systems Challenges suddenly change, due, for example, to severe damage to the body of the airplane, the mechanized co-pilot can even relearn to fly the aircraft under these new circumstances on the spot and in real-time. Altogether, the robotic copilot in Figure 1 augments sensing, acting, and mental capabilities of the pilot (see Figure 2). For its embodied “self” it may be operated in different cockpit settings. However, if the robotic copilot is only supposed to augment decision and acting capabilities of a pilot then an embodied “self” is superfluous. In these cases, it suffices to deploy an autonomous and self-learning software companion into the cockpit control system. F IGURE 2. A REAS OF H UMAN A UGMENTATION . The robotic copilot is a particular instance of a larger class of embodied companions. Personal companions for supporting and taking over tedious household chores and for assisting with tax declarations, including the communication with tax authorities, and suggesting new possibilities based on our intents are an old dream. Embodied companions are also designated to assist, say, truck drivers, ship captains, caregivers, investors, managers, workers, lawyers, and, in fact, everybody. Potential benefits of these kinds of embodied companions include increased safety, reliability, efficiency, affordability, and previously unattainable capabilities. 10 EI Systems Challenges F IGURE 3. F EDERATED R ETAIL . S ERVICE FEDERATIONS When using current shopping websites, the consumer selects an order from a large, but ultimately fixed set of product options and configurations, and the ordered goods are, by-andlarge, already produced and stored in a system of warehouses as to guarantee speedy delivery. As such, the traditional shopping experience has just moved from a physical store onto a retail web site, but the whole process builds on established stakeholder roles of consumers, producers, logisticians, and chains of retailers. We might think, however, of more flexible setups which start with a software-defined custom design instead of a custom order. An often-cited example are custom-designed sneakers which might even be 3-D knitted, say, in a store.39 Other obvious examples are software-defined custom cars or mission-specific drones. The journey thus starts with a custom design, which is usually constructed by instantiating available design patterns and by composing them into the desired design and a corresponding production plan. This new design and the production plan can be made available to others to build upon; analogous to an “app” store. Let’s assume, however, that the design now needs to be realized, that is the production plan needs to be executed. Instead of identifying a suitable production site upfront, a bidding process is started among several production sites and logistics provider to dynamically set up a suitable distributed production chain for realizing the custom design. 39 See Speed Factory (https://www.digitaletechnologien.de/DT/Redaktion/DE/Standardartikel/AutonomikFuerIndustrieProjekte/autonomik_fuer_industrie_projekt-speedfactory.html ) 11 EI Systems Challenges In Figure 3 this process of connecting the digital production plan with real-world production entities such as 3D printers, robots, flexible and production lines together with storage and transport capabilities for moving partly finished products is referred to as deployment.40 Some of these entities, such as the shipping service, may operate autonomously and is possibly organized as its own virtual company. Such a deployment might be chosen according to different criteria, including the traditional triad of costs, quality, and time-to-delivery. For example, the ordering customer is provided with the option of a reduced price if she accepts a, say, two-day delay of shipping. In addition, flexible deployment might also take other all-important attributes such as resilience to faults or even partial collapse of selected production or logistics capabilities, sustainability (for example, climate neutrality) and other ethical considerations (for example, no child labor). These attributes might now even be part of the ordering process itself, thereby enabling everybody to act on behalf of her own personal beliefs and social values, and to directly influence the world through seemingly mundane every-day acts such as ordering goods. Altogether, the customer is not (only) a customer anymore as she can order the manufacturing of the goods she wants or needs. Also, integrated production and logistics chains are broken up into highly specialized services, which are orchestrated flexibly and in a highly distributed fashion. The production and logistics services might be realized through increasingly autonomously acting actors, who might do their own bidding for providing services with the purpose of forming dynamic coalitions. From this point of view, there does not seem to be a direct need for a traditional retailer 41 as an intermediary between customers and producers anymore. By-and-large we might be able to realize a proof-of-concept prototype of the embodied retail system as depicted in Figure 3 with currently available technology. In fact, the attentive reader might already have noticed that the embodied retailer is structured according to the main principles of model-driven design, namely, platform independent model (PIM), platform dependent model (PDM), and a deployment relation between those two models. However, the extra flexibility of the embodied retail system comes with a cost, and we do not yet have a foundational understanding of the necessary operators for composing the required designs and service nor do we have anything that comes close to the required large-scale deployment schemes. There are also serious questions regarding the resilience, that is tolerance, isolation, and recovery from faults, breakdowns, and cyberattacks.42 From a business and societal perspective the main open questions involve feasibility of corresponding value chains, incentivization and regulation, (product) liability issues, 40 In general, such a deployment is an n:m relation between services in the virtual world and their physical embodiments. 41 Maybe as an embodied system? 42 Fault Detection, Isolation, Recovery (FDIR) 12 EI Systems Challenges intellectual property, payment systems, and tax laws. Indeed, systems such as the outlined federation for retail do not only have the potential of disrupting established economic cycles but also shake up established social, economic, and governmental structures and infrastructures. The embodied retail scenario has been designed to also demonstrate the possible elimination of middlemen between consumer and service providers, thereby enabling completely new value chains in the upcoming era of post-platform ecosystems. Particularly, these kinds of dynamic federations of embodied service systems may well make currently well-established retail platforms and middlemen such as Amazon, Uber, or AirBnB superfluous. They also rely on flexible infrastructural services for, say, payment systems, insurance, and arbitration boards. We can easily abstract from the main building principles of embodied retail to other societalscale life worlds to create similar scenarios; in particular, critical supply infrastructures (for example, energy/water), mobility (for example, integrated transportation), medical (for example, personalized medicine), law (for example, judicial Q&A advisors, attorneys at law, and software judges in line with Leibniz’ rational actors), and government (e.g. seamless and proactive government services). Numerous other examples of embodied federated systems can be found across many application domains. For instance, collaborative IoT systems for supply-demand interaction (e.g. machinemachine, human-machine) in industrial applications, prosumer interaction and collaboration under permanently changing conditions in the energy domain, ad hoc collaboration between smart infrastructure and autonomous vehicles, self-engineering and re-configuration of automation components in building automation, or self-managed production logistics and warehouse logistics with a heterogeneous fleet of AGVs which are able to self-adapt to heterogeneous production floors and tasks. Moreover, next-generation communication networks are expected to sense, compute, learn, reason, act on business intent, and to manage the ongoing explosion of data from an everincreasing number of connected intelligent devices and a multiplicity of new use cases, along with ever-changing network topologies. 43 Zero-touch network operations for coping with the growing system’s complexity, size, and reduced decision-making lead time lead to an increased level of network autonomy and self-adapting automation, where unforeseen situations, intents, and requirements can usually be addressed without human intervention. 43 Ericsson, Artificial intelligence in next-generation connected systems, Whitepaper, 2021. 13 EI Systems Challenges 3. C HARACTERISTICS Embodied systems are comprised of federations of collaborating actors, they operate in largely unpredictable environments, physical or not, and they recognize this environment through sensors. Moreover, they are informed about the intentions of other actors in their respective and immediate operating environments; they take non-trivial decisions based on reasoning, they influence their environment, including other actors, via actuators; they interact and cooperate with the elements of their operating environment, they influence elements in their environment to better meet own goals44; and they show a certain behavior based on skills; and they learn new and improved behavior during operation and through interactions. 45 Altogether, an embodied system is characterized as being: § Cognitive, as actions are based on situational awareness, model-building, and planning. § Intent-driven, as actions are based on capturing actors’ intents, tasks, and goals. § Federated, as actions of decentralized actors are coordinated in a collaborative manner between stakeholders and on an intentional level to accomplish joint tasks or missions. § Autonomous, as actions are increasingly determined by an actor’s, or federations of actors’, own knowledge, beliefs, intents, preferences, and choices. § Self-learning, as actions are adapted and improved through experience, exploration, and reasoning, both inductive and deductive, of a situated actor. Notice that embodied systems may therefore be viewed as autonomous and self-learning cyberphysical systems with intent-driven, goal-oriented, and collaborative behavior. Embodied systems therefore go well beyond current functionally automated systems, such as autopilots for landing an airplane, parking a car, or manufacturing robots, which are all designed to handle limited tasks in precisely specified operational contexts. Task execution in functionally automated systems is planned offline or during design time, and they usually do not learn during operation. Moreover, collaboration is restricted to the exchange of information about the system context, and they rely on fallback mechanism to a human operator when encountering unforeseen and difficult situations. Most current systems, including production or household robots, are functional automated systems, and we are on the edge of deploying intentand mission-oriented systems. Simple collaborative systems start to be established. Selflearning during operation, however, is not possible now for mission-critical and safety-related applications. 44 For example, Mechanism Design (see also: Roughgarden, Algorithmic Game Theory, CACM, 2010). 45 Putzer, Wozniak, Trustworthy Autonomous/Cognitive Systems, fortiss, 2021. (see also: https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Informationsmaterialien/fortiss_whitepaper_trustworthy_ACS_web. pdf) 14 EI Systems Challenges Cognitive Systems Cognition may be defined as the mental action or process of acquiring knowledge and understanding through experience and the senses. 46 Clearly, it is only in conjunction with cognitive faculties that systems and machines develop their full potential and address a constantly growing number of new challenges in daily use. In a system with cognitive faculties, individual or interconnected objects therefore cannot only perceive the physical environment but are also able to learn from the wealth of experience gained, derive new insights, understand contexts, and make important decisions in a supportive or autonomous manner. Essential cognitive faculties are:47 48 § Perception. Fusion and interpretation of a multitude of sensors, stimuli, and observed behavior; removal of vagueness and ambiguity in input data, and synthesis of relevant information such as the detection, localization, and classification of relevant surrounding objects. 49 § Interpretation. Construction and update of faithful representations (“digital twins”) of the exogeneous operating environment and the endogenous “self”, based on perceived inputs and other knowledge sources. § Imagination. Model-based capability of situational awareness, inductive and deductive reasoning, planning, and for the projection into the future (and the past) based on both exogeneous and endogenous world models, derived knowledge, and perceived input. § Action. Selecting and prioritizing appropriate goals for a given configuration of the environmental models, and current beliefs and intents to achieve the selected goals by balancing optimized performance with the need for resilience. § Learning. Ability to adapt and optimize situational behavior, to adapt internal models, goal management, and planning processes dynamically, and to acquire new knowledge through inductive and deductive reasoning. Situational-aware reasoning, planning, and learning therefore are the main ingredients of the sense-plan-act cycle (see Figure 4), which is the central architectural concept of cognitive actors in the field of Artificial Intelligence (AI).50 In other words, AI is concerned principally 46 Lexico.com 47 The selection of these cognitive capabilities is inspired by “axioms” which are considered to be necessary for machine intelligence (see also: Aleksander, Machine Consciousness, Progress in Brain Research, Elsevier, 2005). The big questions, however, centers around cognitive capabililies sufficient for machine consciousness. 48 See also: Metzler, Shea, Taxonomy of Cognitive Functions, Engineering Design, 2011. 49 Guidance, Navigation, and Control (GNC) 50 Russell, Norvig, Artificial Intelligence: a modern approach, Pearson, 2021. 15 EI Systems Challenges with designing the internals of a stream-transforming cognitive actors for mapping from a stream of raw perceptual data to a stream of actions, whereas EI primarily focuses on perception of and actions on a physical world. )XOOF\FOHRIDXWRQRPRXVV\VWHPVWKDW FDSWXUHVWKHSHUFHSWLRQWKHGHOLEHUD WLRQDQGWKHH[HFXWLRQ World / Environment Context Model In Perception • Multisensor Data Fusion • Object Detection • Object Classification • Intent Recognition Action Plan Deliberation • Desires & Goals • Strategy • Plan • Trajectory Out Execution • Action Selection • Priority • Reflexes • Precise Control Dependable Cognitive System (based on dependable AI) F IGURE 4. SENSE - PLAN - ACT CYCLE . Designs for cognitive actors vary enormously depending on the nature of the operating environment, the nature of the perceptual and motor connections between actors and environment, and the requirements of the task. For instance, cognitive faculties have been categorized into System 1 (fast) capabilities for performing intuitive, automated tasks that we can do instinctively, and System 2 (slow) capabilities for performing tasks that require conscious decision making and may be described verbally. 51 There has always been a tendency in AI to use a heterogeneous set of techniques for realizing these two categories of cognitive capabilities, as machine implementations of System 1 functionality are often based on logic and probability theory (“symbolic”), and the implementation of System 2 functionality primarily uses connectionist (“sub-symbolic”) approaches, which are inspired by the networked neural structure of the brain. Building on these two pillars of AI, cognitive architectures such as Soar 52 integrate logic with connectionist techniques for realizing autonomous robotic (both virtual and physical) systems that have some basic cognitive abilities of humans, but also in search of a unified theory of cognition53. With a similar motivation in mind, neuro-symbolic programming proposes integrated frameworks which have the pure neural, logical, and probabilistic methods as special cases.54 Symbolic reasoning capabilities, however, may also be encoded with connectionist networks. 51 According to the global workspace theory of consciousness. 52 Laird, The SOAR Cognitive Architecture, MIT Press, 2012. 53 Newell, Unified theories of cognition, Cambridge 1990. 54 Raedt, Manhaeve, Dumancic, Neuro-symbolic = neural + logic + probabilistic, IJCAI, 2019. 16 EI Systems Challenges Intent-driven Systems Everything an actor needs to know about its goals and expected actions must be defined by means of intents of both human and machine actors. The aim of intent-driven systems is to capture relevant intents and to act in accordance with these intents in an optimized and resilient manner. 55 Intent-driven networks, for example, can predict faults to proactively optimize performance and to carry out repairs.56 Intents of actors are ideally expressed explicitly and declaratively – that is, as utility-level goals57 that describes the properties (what?) of a satisfactory outcome rather than prescribing a specific solution (how?). This gives the system the flexibility to explore various solution options and find an optimized one. It also allows the system to optimize by choosing its own goals that maximize utility. One of the benefits of expressing intents as utility-level goals is that it supports the system to cope with conflicting objectives of multiple intents. This is vital, because an embodied actor often must take multiple intents into account before taking a decision. Unlike traditional software-intensive systems, where requirements are analyzed offline to detect and resolve conflicts prior to implementation, intents are added to an embodied system and modified during operation. Adaptation to changed intent as well as conflict detection and resolution are therefore essential capabilities of embodied systems. One of the main challenges for specifying goals and intents of embodied systems operating in the real world, however, is that there is very little chance that we can specify our objectives completely and correctly in such a way that the pursuit of those objectives by more capable machines has beneficial outcomes.58 In particular, we can expect a sufficiently capable machine pursuing a fixed objective to take preemptive steps to ensure that the stated objective is achieved, including acquiring physical and computational resource and defending against any possible attempt to interfere with goal achievement. 59 This ability to recognize plans, goals, and intents of other actors enables basic cognitive capabilities to reason about what other actors are doing, why they are doing it, and what they will do next. Consider, for example, a robotic butler for doing all the boring, repetitive tasks that we wish we didn’t have to do. Ultimately, we want household assistants that can anticipate our intentions and plans, and provably act in accordance with them. Building these kinds of 55 Silvander, Wnuk, Svahnberg, Systematic literature review on intent-driven systems, 2020. 56 Huawei. Huawei Launches the Intent-Driven Network Solution to Maximum Business Value, 2018. (see: http://www.huawei.com/en/press-events/news/2018/2/Huawei-Launches-the-Intent-Driven-Network-Solution ) 57 Including those that may have been considered “common sense” in human-operated systems. 58 “So tell me what you want, what you really, really want…“ (Spice Girls, Wannabe) 59 Russell, Artificial Intelligence and the Problem of Control, 2021. 17 EI Systems Challenges proactive assistant systems requires plan, action, and intent recognition for accurately capturing and tracking the operator’s requirements.60 Plan, goal, and intent recognition also enables different actors to negotiate with each other on behalf of their peers’ intent, as the basis for a potential mutually beneficial collaboration. For example, one actor may have the intent to deliver high-quality service, while another may want to minimize resource spending. Current AI technology resolves such conflicts either explicitly from weights that introduce relative importance or implicitly from properties of preferential outcomes as defined in utility-level goals. Federated Systems A federated system consists of an ensemble of interacting actors, both machines and humans, which are collaborating to jointly achieve a common task or goal through the mutual provision of actions that individual systems alone cannot achieve. 61 62 In doing so, these ensembles exchange relevant information, negotiate their goals, plans and intentions, and they adapt their own actions to the negotiated plan. This aspect of negotiation, that is the ability to confer with another as to arrive at the settlement of some matter63 is a central activity, and the realization of “confer” and “arrive at settlement” are among the challenges for designing collaborative, federated systems.64 Especially the community of distributed AI has developed a range of models of collaborative plans in the face of resource constraints and uncertainty. Collaborative federated systems are often safety-critical, operate in dynamic contexts, and must be capable of reacting to unforeseen situations without human intervention. On the one hand, they must be able to handle uncertainties due to the imprecision of sensors and the behavior of data-driven components for perceiving and interpreting the context to enable decisions to be made during operation. On the other hand, uncertainties can emerge from the collaboration in a collaborative group, related to the exchange of information (e.g., context knowledge) between collaborative systems. Uncertainty that can occur during operation should be considered systematically during engineering to enable collaborating systems to cope with uncertainties autonomously. Effective collaboration strongly relies on the assumption that most, if not all, actors operate as expected. Therefore, a level of trust and distrust between actors needs to be established as part 60 Sukthankar et al (Eds.)., Plan, action, and intent recognition, 2014. 61 Böhm et al., Model-based Engineering of Collaborative Embedded Systems, Springer Nature, 2021. 62 Grosz, Collaborative Systems, AAAI-94 Presidential Address, AI Magazine, 1996. 63 Mish (Ed), Ninth New Collegiate Dictionary, 1988. 18 EI Systems Challenges of the collaboration. 65 Measures need to be taken to tolerate a certain number of noncollaborating and ineffective actors, but also actors with malicious intentions. In particular, the security design of current decentralized systems with their heterogeneous components, including components-off-the-shelf (COTS) and software of unknown pedigree (SOUP) start with a zero-trust model, which is then mitigated with the right mix of security measures to create challenging barriers for attackers, including pervasive authentication and corresponding checks of all interactions. Autonomous Systems Current industrial practice is mainly concerned with developing remotely operated vehicles (“teleoperated driving”), self-operated systems for restricted time periods and for restricted objectives such as remotely piloted air systems in case of a lost data link, pilotless underwater vehicles, and driverless metros in controlled urban environments. Yet we are only at the very beginning of a new generation of autonomous systems, which are characterized by increasingly autonomous behavior in increasingly complex environments, fulfilling missions of increasing complexity, the ability to collaborate with other machines and humans, and the capability to learn from experience and to adopt their behavior appropriately. As it is designed to perform the equivalent operational tasks of understanding through experiencing and sensing, an autonomous system therefore may be viewed as a technical implementation of cognition. These systems perform and integrate heterogeneous sets of tasks based on an overall situational assessment. In contrast to mere automation, increasingly autonomous systems employ a never-give-up strategy even in the face of real difficulties, say inconsistencies, unforeseen situations, and authority limits (see Robotic companions). Potential benefits include increased safety, reliability, efficiency, affordability, and previously unattainable mission capability. Clearly, with more autonomy comes more and different forms of responsibility. In the absence of an adequately high level of autonomy that can be relied upon, substantial involvement by human supervisors and operators is required. Increasingly autonomous systems support humans in daily routine tasks, have humans in the loop for continuous control of evolution of subsystems, and ask humans for high-level decision-making. This kind of mixed human-machine systems creates significant new challenges in the areas of human-machine collaboration and mixed initiative control. The overarching goal is in achieving a sufficient mutual understanding of state and intent of both humans and technical systems as to optimally blend their competences in jointly acting towards overarching objectives, while respecting privacy. The challenge here is to model 19 EI Systems Challenges human behavior interactions and to provide the appropriate uncertainty characteristics related to the largely unpredictable behavior of humans under unforeseen circumstances. Moreover, as individual spheres of control may overlap arbitrarily, there is a pronounced need for orchestrating these processes such that they jointly serve, say, not only a single human, but can best-possibly multi-task in serving arbitrarily large groups at the same time despite uncorrelated requests and uncoordinated missions. Self-Learning Systems One of the distinctive features of embodied systems is their ability to continuously improve their knowledge and capabilities through experience, both positive and negative, and targeted exploration in the real world. This ability of self-learning through exploration is, of course, a far cry from supervised machine learning schemes for synthesizing, say, neural network representations for approximating functions from given input-output samples. The “learned” behavior may often not be transferred to other operating contexts. An “end-to-end” autonomous controller, for example, might work well for, say, driving around in the outskirts of Phoenix for which it was trained. But we most probably expect it to fail miserably when put into the streets of Algiers or to navigate around the Gate of India. This is because the neural network controller has not actually learned to drive, as we as humans are supposed to when visiting driving school. It was just trained to mimic a context-specific set of driving scenarios, and there is no reason to expect it to generalize its behavior to radically new contexts. In contrast, autopoietic systems extend autonomously their perception and attention, their situational representation and interpretation of the perceived world, their actions and their collaboration patterns, and they are able to communicate such learned capabilities with other systems:66 “An autopoietic machine is a machine organized (…) as a network of processes of production (transformation and destruction) of components which: (i) through their interactions and transformations continuously regenerate and realize the network of processes (relations) that produced them; and (ii) constitute it (the machine) as a concrete unity in space in which they (the components) exist by specifying the topological domain of its realization as such a network”. The ability of unsupervised learning by means of interacting with its operating context (including the “self”) is the major characteristic of autopoietic systems. This is close to human behavior and possibly also the ultimate dream of AI. Achieving higher levels of autonomy in open, that is uncertain, unstructured, and dynamic, environments and terrain increasingly involves data-driven machine learning techniques with many open systems science and engineering challenges. The prevalent approach in autonomous driving, for example, aims at reducing the uncertainty of the operating context by compiling and continuously extending huge sets of driving scenarios which sufficiently (up to tolerable quantities?) cover all possible situations; as the basis for continuous self-learning ecosystems of a global scale. 66 Maturana, Varela, Autopoiesis and cognition: the realization of the living, Kluwer, 1980. 20 EI Systems Challenges New and more efficient control regimes for reliable and safe exploration of unknown terrain are clearly needed, as embodied systems must necessarily act with complete, uncertain, and even inconsistent models of the world. Possible approaches that are being currently pursued include the learning and use of causal models, employment of an ensemble of models, and multifaced understanding.67 67 Minsky: “You don’t really understand something if you only understand it one way”. 21 EI Systems Challenges 4. T RUSTWORTHINESS Embodied systems are a new generation of increasingly autonomous systems operating in realworld societal contexts. Thus, actions of embodied systems do matter. The autonomous behavior of embodied systems also implies a real danger of losing control as self-learning systems may exhibit emergent behavior, they evolve much faster than we as humans may even comprehend, and they are able to self-organize in increasingly powerful dynamic federations. These essential features of embodied systems make it even harder to ensure that a meaningful (human) control over an embodied system is enabled in the field. Without embodied systems – and the human beings behind them – being demonstrably worthy of trust, unwanted consequences may ensue, and their uptake might be hindered, preventing the realization of the potentially vast social and economic benefits that they can bring. Trustworthiness therefore is a prerequisite for people and societies to develop, deploy and use embodied systems in a meaningful manner. Trust may be viewed either as a belief, attitude, intention, or behavior, and as such it is a complex notion in itself. It is most generally understood as a subjective evaluation of a truster on a trustee about something in particular; for example, the completion of a task.68 A classical definition from organization theory defines trust as the willingness of a party to be vulnerable to the actions of another party based on the expectation that the other will perform a particular action important to the trustor, irrespective of the ability to monitor or control that party.69 An expert group commissioned by the EC has recently identified three main ingredients for trustworthy AI-based systems;70 specifically, they recommend trustworthy systems to be at least: § Lawful; that is, complying with applicable laws and regulations, § Ethical; that is, ensuring adherence to applicable ethical principles and values, § Robust; that is, predictable and sustained functionality in the face of uncertainty, faults, and malicious attacks. In addition, respecting the vast amount of, largely, unwritten norms for accepted social behavior is also instrumental for building up trust. 68 Hardin, Trust and trustworthiness. Russell Sage Foundation, 2002. 69 Mayer, Davis, Schoorman, An integrative model of organizational trust. Academy of management review, Academy of Management Review, 1995. 70 https://digital-strategy.ec.europa.eu/en/policies/expert-group-ai 22 EI Systems Challenges Explicit lawful and ethical actors have been proposed based on implementing legal theories, humanlike competence, and ethical theories predicated on virtue ethics, deontology, and consequentialism.71 For example, when ethical principles are in conflict, they attempt to work out reasonable resolutions. For contexts where informing others of one‘s intention and reasoning is crucial, these actors communicate and even defend their reasoning. Explicit lawful and ethical actors, however, are inevitably tied to a specific societal context. Societal-scale CPS therefore have been proposed that are parameterized by social contexts. This approach is based on (1) understanding the nature, scope, and evolution of policies in the operation of societal-scale CPS in different societies, (2) investigating methods for the explicit formal representation of societal context, and (3) developing architectures that guarantee the enforcement of policy requirements. 72 Human agency and oversight Technical robustness and safety Privacy and data governance Transparency Diversity, nondiscrimination, and fairness Societal and environmental wellbeing Accountability Including fundamental rights, human agency, and human oversight. Including resilience to attack and security, fall back plan and general safety, accuracy, reliability, and reproducibility. Including respect for privacy, quality and integrity of data, and access to data. Including traceability, explainability, and communication. Including the avoidance of unfair bias, accessibility and universal design, and stakeholder participation. Including sustainability and environmental friendliness, social impact, society, and democracy. Including auditability, minimization and reporting of negative impact, trade-offs, and redress. T ABLE 1. P RINCIPLES BY THE EC FOR RESPONSIBLE AND TRUSTWORTHY AI. 73 We shortly describe different accounts in the EU and the US for trustworthy AI-based systems. Table 1 summarizes the lawful, ethical, and robustness attributes for responsible and trustworthy AI as developed on behalf of the European Commission. Figure 5displays these practices, which are recommended to be implemented and continuously evaluated throughout the system’s lifecycle. These are strong recommendations indeed, as current, largely manual, auditing frameworks are way to inefficient, slow, and error-prone for self-learning and everevolving systems, and we currently do not have adequate technical means for automated compliance audits. Moreover, trade-offs usually need to be made in real-life engineering for addressing seemingly contradictory attributes such as privacy and transparency. 71 Scheutz, The case for explicit ethical agents, AI Magazine, 2017. 72 NSF PIRE 16-571: Science of Design for Societal-Scale Cyber-Physical Systems 73 https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf 23 EI Systems Challenges F IGURE 5. A TTRIBUTES OF T RUSTWORTHY AI. 74 In a similar vein, the principles in Table 2 by the ACM for transparency and accountability 75 are designed to increase trust in all kinds of algorithmic systems. Awareness Access and redress Accountability Explanation Owners, designers, builders, users, and other stakeholders of analytic systems should be aware of the possible biases involved in their design, implementation, and use and the potential harm that biases can cause to individuals and society. Regulators should encourage the adoption of mechanisms that enable questioning and redress for individuals and groups that are adversely affected by algorithmically informed decisions. Institutions should be held responsible for decisions made by the algorithms that they use, even if it is not feasible to explain in detail how the algorithms produce their results. Systems and institutions that use algorithmic decision-making are encouraged to produce explanations regarding both the procedures followed by the algorithm and the specific decisions that are made. This is particularly important in public policy contexts. 74 Source: EU High-Level Expert Group on AI (https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf) 75 ACM Code of Ethics (http://ethics.acm.org) 24 EI Systems Challenges Data Provenance Auditability Validation and Testing A description of the way in which the training data was collected should be maintained by the builders of the algorithms, accompanied by an exploration of the potential biases induced by the human or algorithmic data-gathering process. Public scrutiny of the data provides maximum opportunity for corrections. However, concerns over privacy, protecting trade secrets, or revelation of analytics that might allow malicious actors to game the system can justify restricting access to qualified and authorized individuals. Models, algorithms, data, and decisions should be recorded so that they can be audited in cases where harm is suspected. Institutions should use rigorous methods to validate their models and document those methods and results. In particular, they should routinely perform tests to assess and determine whether the model generates discriminatory harm. Institutions are encouraged to make the results of such tests public. T ABLE 2. PRINCIPLES BY THE ACM FOR TRANSPARENCY AND ACCOUNTABILITY . 76 Due to the relative similarity of the underlying systems of social values we may observe a strong overlap of the trustworthiness requirements in the EU and the US. However, the ACM principles on transparency and accountability tend to be phrased more in technical terms, and consequently they seem to be more amenable to automated compliance. On the other hand, the ACM principles on transparency and accountability have clearly been formulated with current AI/ML techniques in mind. As such they do not adequately address the characteristics of the upcoming generation of increasingly autonomous and self-learning systems. Particularly, there might not even be clearly identifiable “institutions” anymore for operating an embodied system, and embodied systems might eventually need to be held responsible for their very own actions. Also, the demand on data provenance as formulated above may not be applicable to continuous, unsupervised learning. Depending on the intended role of embodied systems in societal contexts – ranging from mere assistants to fully autonomous actors - the ACM principles on transparency and accountability consequently therefore need to be revised, developed further, and agreed upon to fit the characteristics of a new generation of increasingly autonomous and self-learning systems. We also need to develop techniques for ensuring such principles of algorithmic transparency and accountability for dynamic federations of increasingly autonomous, learning-enabled, and embodied systems. 76 ACM Code of Ethics (http://ethics.acm.org) 25 EI Systems Challenges F IGURE 6. M APPING FROM MAIN CHARACTERISTICS OF E MBODIED S YSTEM CHALLENGES . 26 S YSTEMS TO CORRESPONDING EI Systems Challenges 5. C HALLENGES Traditional system engineering comes to a juncture from assuring quality-of-service, dependability, and safety attributes for relatively small-scale, centralized, deterministic and predictable, non-evolvable, automated embedded systems operating in well-defined and predictable environments to assuring the trustworthiness of larger-scale, federated, nonpredictable, self-learning, and increasingly autonomous embodied systems operating in uncertain and largely unknown environments. These differences between embedded and embodied systems are also summarized in Figure 7. Architecture Behavior Context Maintenance Requirement Human control Embedded Systems centralized deterministic well-defined updates dependability yes Embodied Systems federated largely unpredictable uncertain self-learning trustworthiness increasingly no F IGURE 7. F ROM E MBEDDED TO E MBODIED S YSTEMS . Thereby, we can build up trust by engineering embodied systems which are lawful, ethical, and robust. Compared with traditional embedded systems engineering we face additional challenges; in particular, embodied systems: 77 § Learn continuously and they adapt and optimize their behavior based on experience and targeted exploration. § Need to safely operate in partially unknown or uncertain environments, and they need to be robustness in the presence of inaccuracies, uncertainty, and errors in their world models (“known unknown”) and also in the presence of non-modeled phenomena (“unknown unknown”). § Increasingly lack the fallback to a responsible human being. § Offer a variety of new attack surfaces due to data-driven programming.77 § Exhibit largely unpredictable and emergent behavior due to data-driven programming. § Cannot be certified as current certification regimes require the system’s behavior and its intended operating context to be fully specified and verified prior to commissioning. vulgo: machine learning. 27 EI Systems Challenges For the continuous evolvability and self-organizing capabilities of embodied systems, the traditional design-build-commission-decommission life cycle of embedded systems is inadequate and clearly needs to be replaced with a design-operation continuum based on the combined functionality for design, simulation/verification, deployment, operation, and maintenance. Given the expected lifetime of embodied systems, they must be designed to cope with changing underlying technologies and hardware, changing regulatory settings, changing societal contexts, changing system requirements, and yet provide the same high level of dependability and quality of service throughout their evolution. It is even perceivable that certain design and engineering steps, including situational risk management, are eventually performed by the embodied systems themselves. Based on the characteristics of embodied systems as outlined in the Section 3 we are now deriving all-important systems challenges for developing, deploying, and operating trustworthy embodied systems. A high-level summary of this derivation is provided in Figure 6. Notice, however, that Figure 6 only depicts the most obvious and possibly the most important relations between characteristics and derived challenges for engineering trustworthy embodied systems. R OBUST AI/ML Machine learning (ML) techniques are ubiquitous with lots of successful applications. The main attraction is that functional requirements are stated in terms of data only, and a corresponding program for approximating such a function is synthesized in an automated fashion. Moreover, the engineering steps from data wrangling to architectural selection and optimization of the function approximation is increasingly being automated. 78 However, these advantages also come with some downsides. 78 § Current learning techniques based on, say, artificial neural networks are only as good as the available data (and their resource-intensive pre-processing and labeling requirements). § There is uncertainty on the input-output behavior of the learned function approximators are usually they are not robust in the sense that small changes to the input might result in unexpected behavior; for example, one-pixel attacks of trained classifiers are successful for many artificial neural networks. § The learned function is usually restricted to the context as encoded in the learning data with limited transferability.79 Xin, Zhao, Chu. AutoML: A Survey of the State-of-the-Art. Knowledge-Based Systems, 2021. 79 “The soccer bot lines up to take a shot at the goal. But instead of getting ready to block it, the goalkeeper drops to ground and wiggles its legs. Confused, the striker does a weird little sideways dance, stamping its feet and waving one arm, and then falls over. 1-0 to the goalie… 28 EI Systems Challenges § The input-output behavior of the learned function is implicit and not transparent in that the reasons for proposed decisions are opaque; for example, a neural network for classifying tumors as benign or not might show human-like performance, but there usually is no further explanation to the human diagnostician. § Current supervised learning algorithm based on, say stochastic gradient descent, are rather data-intensive and inefficient; small children, in particular, have the ability to form concepts such as “cow” based on only a few encounters. As we gain more experience with developing and deploying machine learning-based systems for real world challenges we realize some short-comings. 80 § Adequate data often is not or not readily available, and crucial data for specifying the intended behavior might only become available during development and operation. Traditional process models with clearly defined requirements engineering phases, however, do not adequately support the extra flexibility needed by data-driven AI development approach. § Building data-driven ML systems requires a comprehensive wealth of experience, even though this kind of expert knowledge (“which learning algorithm?”, “which architecture?”, “which hyperparameters?”) is increasingly being captured in automated meta-learning processes. 81 § An ML system is often more than just machine learning and building and running it is a serious software and systems engineering undertaking. However, generally accepted or even standardized processes, methods, and tools for the development and operation of predictable and transparent ML are largely missing. § Current ML applications usually do not adequately address trustworthiness attributes as outlined in Section 4. Moreover, most machine learning applications nowadays are based on supervised learning, which does not support the required self-learning capabilities of embodied systems. Reinforcement learning, which is based on optimizing objective functions, however, is a good starting point, where inverse reinforcement learning might be used for intent recognition. However, current approaches to reinforcement learning require a huge number of trials. Failed attempts in the real world which might result into undesired behavior, accidents, or other catastrophic events. AI trained using reinforcement learning can be tricked by … an adversarial policy.“ (quoted from: Heaven, Reinforcement learning AI are vulnerable to a new kind of attack, MIT Technology Review, Artificial Intelligence, 2020) 80 Standards on engineering trustworthy autonomous systems are currently emerging; in particular VDE-AR-E 2842-61 of the standardization organization DKE (cmp. https://www.dke.de/de/normen-standards/dokument?id=7141809&type=dke|dokument ) 81 For example, AutoML. 29 EI Systems Challenges Techniques for safe and predictable self-learning and exploration of uncertain and unpredictable real worlds are still in their infancy. Promising approaches for tackling the added complexities of autonomous actors in real world settings include hybrid combinations 82 of classical and learning-based algorithms and the incorporation of prior knowledge.83 How can one learner who does not know what there is to learn manage to learn anymore? Current machine learning approaches usually start with what needs to be learned for learning. We as humans, however, can discover both the tasks to be learned and the solution to those tasks through exploration, or non-goal-directed action. Machine learning has mainly concentrated on non-incremental learning tasks, tasks in which the entire training set is fixed at the start of learning and then is either presented in its entirety or randomly sampled. Embodied actors, however, need to learn incrementally and continuously through exploration. Machine learning also is increasingly being augmented with domain-specific knowledge and rules for increasing the efficiency and effectiveness of machine learning, rules and decision trees might also be compiled from learned behavior, which themselves can be used for improving further learning but also for making decisions transparent, say, to a human operator. In this way, domain knowledge, such as physical laws, is currently integrated into machine learning, by using techniques such as regularization, data augmentation, or post-processing. A recent survey on knowledge-augmented machine learning84 reviews the role of knowledge in machine learning, and it discusses its relation to the concept of invariance. Among others, neuro-symbolic integration (with logic, probability theory, and neural structures as projections) has been proposed as the basis for a new generation of dependable, predictable, transparent, and efficient data-driven programming techniques for realizing increasingly autonomous human assistants and/or for mission- and safety-relevant applications. Altogether, despite technological advances that have led to the proliferation of data-driven machine learning systems, there still is the question of the level of trust that we can put on these systems. A new generation of robust machine learning algorithms therefore is needed that,85 82 83 84 85 § in uncertain and largely unpredictable environments, § can make timely and confident decisions, § whose results are understandable and explainable to a human operator, § that are resilient to erroneous inputs and targeted attacks, Chaplol et al, Learning to explore using active neural slam, ICLR, 2020. Xin Ye and Yezhou Yang, From seeing to moving: As survey on learning for visual indoor navigation, arXiv:2002.11310, 2020. https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Whitepaper/fortiss_whitepaper_knowledge_as_invariance_web.pdf Stoica et al., A Berkeley View of Systems Challenges for AI, 2017 (https://arxiv.org/pdf/1712.05855.pdf). 30 EI Systems Challenges § that can process ever-increasing amounts of data, § from decentralized and heterogeneous data sources, § but can also extract useful insights from small amounts of data and sparse rewards, § without significant compromises in confidentiality and privacy in federated multi-actor settings. Traditionally, ML modeling techniques have relied on unsiloing data from multiple sources into a single data lake. Centralized data sources, however, pose serious privacy, data misuse, and security challenges for federated systems. Also, aggregating diverse data from multiple sources needs to meet regulatory concerns such as GDPR, HIPAA, or CCPA (we will be reconsidering these issues in the subsection below on Assurance). To overcome these challenges, several pillars of privacy preserving machine learning have been developed for unsiloing ML models with specific techniques that reduce privacy risk and ensure that data remains reasonably secure, namely federated machine learning, secure multi-party computation, differential privacy, and homomorphic encryption. Altogether, the prevailing methods for machine learning do not map to the ways that humans learn, as humans learn by seeing, moving, interacting, and speaking with others. Humans learn from sequential experiences, not from shuffled and randomized experiences. We need to come up with a new generation of machine learning techniques, possibly mimicking the ways humans learn, as to enable efficient self-learning for trustworthy embodied systems through targeted exploration and experience. H UMAN -C ENTERED AI/ML Embodied systems are machines to support humans in daily routine tasks, have humans in the loop for the continuous supervision of the evolution of subsystems, and ask humans for highlevel decision-making. The central challenge, as addressed in the field of human-centered engineering, is to enable symbiotic relationships, in which embodied systems and humans augment each other reciprocatively,86 and as the basis for co-evolutionary improvement of both machines and humans.87 Human and machine need to avoid “mode confusions” based on a mutual understanding of state and intent of both humans and machine as to optimally blend their competences in jointly acting towards overarching objectives, while respecting privacy. Moreover, in the absence of an adequately high level of autonomy that can be relied upon, substantial involvement by human 86 Cmp. IBM high-level AI framework. 87 Engelbart, The Bootstrap Paradigm (https://dougengelbart.org/content/view/248/) 31 EI Systems Challenges supervisors and operators is required. This creates significant new challenges in the areas of human-machine interaction and mixed initiative control. Embodied systems are learning-enabled. But, as discussed above, current ML techniques offer new attack surfaces, are largely non-transparent (implicit models), tend to be energy and data hungry, and they lack basic transferability capabilities as required for navigating unknown or uncertain terrain.88 It is therefore unclear if and how these technologies can be used beneficially in real-world applications requiring human-machine interaction or in mission- and safetycritical applications. 89 Moreover, the apparent success of ML in producing seemingly intelligent decisions brings along dangerous causes for misunderstandings in the communication between humans and machines. If we compare the behavior of ML systems and humans in decision making, significant differences are obvious. ML essentially provides efficient algorithmic solutions for optimizing a well-defined target function, enabling the learning of task- and data-specific patterns from a huge number of samples or observations. In contrast, a human would rather make decisions based on ground truth rules like causality and can transfer known solutions to new situations and domains. Although both types of decision making can be called forms of generalization, the human way of decision making is a harder form of generalization, sometimes termed horizontal, strong, or out-of-distribution generalization. Human decision making takes advantage of heterogeneous information sources such as interventions, domain shifts and temporal structures, which ML typically discards or even fails to model in learning processes. These shortcomings of ML lead to serious challenges in designing trustworthy systems based on machine learning for human users; in particular: § Low explainability: the decision-making mechanism of ML algorithms such as ANNs cannot be made fully transparent to humans and is difficult to interpret for humans. § Miscalibration of trust: ML seems to be both highly effective to humans and also largely predictable, thereby luring humans into accepting these technical systems as humanlike partners (anthropomorphization) which are trusted more than actually justified.90 § Low level of human control and involvement: most ML algorithms rely on either a hypothetical model of the distribution of data or concrete interpretation (labeling) of data. Such constructions have become one major hurdle to enabling ML systems with high levels of human control, such as human-like reasoning and generalization. 88 E.g. Keynote of AAAI President, http://web.engr.oregonstate.edu/~tgd/talks/dietterich-aaai-presidents-address-final.pdf 89 See: https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Whitepaper/fortiss_whitepaper_HCML_web.pdf 90 This phenomenon has previously also been demonstrated by Feigenbaum’s Eliza program. 32 EI Systems Challenges Specifically, it is rather difficult to find the right level of human control on which the system can effectively communicate with humans to obtain such input. The challenge here is to model human behavioral interactions with the technical system and to provide the appropriate uncertainty characteristics related to the largely unpredictable behavior of humans under unforeseen circumstances. Moreover, as individual spheres of control may overlap arbitrarily, there is a pronounced need for orchestrating these processes such that they jointly serve, say, not only a single human, but can best-possibly multi-task in serving arbitrarily large groups at the same time despite uncorrelated requests and uncoordinated missions. This is particularly challenging for complex mission tasks calling for collaboration and teaming among humans and machines. In these cases, AI-enabled systems may need to identify the “real” intent of human operators and their goals, interact with them in a goal-oriented manner based on models of human behavior, and, in extreme cases, also tolerate and adequately mitigate seemingly irrational behavior. Building trust based on explainability (“how?”, “why?”, “what-if?”) is essential for human operators to accept ML-based solutions and those systems incorporating decisions made by them. Explainability therefore is particularly useful § for increasing the confidence of human operators, § for building trust by supporting an increased understanding of the transferability of results to other problems of interest, § for avoiding misconception and ensuring that humans understand outcomes of learningenabled components, as a solid basis for intervening human actions, and § for increasing human confidence in the decisions and predictions made by a learningenabled component. There are, however, significant challenges in developing adequate methods for explainability. One of them is the trade-off between attaining the simplicity of algorithmic transparency and impacting the high-performing nature of complex but opaque ML models. Yet another challenge is to identify the right information for the user, where different levels of knowledge will come into play. Beyond selecting the level of knowledge retained by the user, generating a concise explanation also becomes a challenge. Most existing methods for explainability, however, focus on explaining the processes behind an ML-based decision, which is often useless in a particular application domain. In addressing these issues, current research is integrating ML algorithms with domain-specific (for example, laws of physics) and, possibly learned, knowledge. 33 EI Systems Challenges C OGNITIVE A RCHITECTURES The field of cognitive architectures creates programs than can reason about problems across different domains, develop insights, adapt to new situations, and even reflect on themselves. These programs realize cognitive functions including perception, memory, attention, social interaction, planning, motivation, actuation, reasoning, communication, learning, emotion, modeling self/other, building/creation and arithmetic capabilities. Prominent cognitive architectures include Soar, ACT-R, LIDA, CLARION, and EPIC. 91 However, with no clear definition and general theory of cognition, there are several hundreds more based on different sets of premises and assumptions, also coming from various backgrounds (computer science, psychology, philosophy, and neuroscience).92 It is not even clear at all what constitutes a cognitive architecture. Newell’s criteria for a cognitive architecture, for instance, include flexible behavior, real-time behavior, rationality, large knowledge base, learning, development, linguistic abilities, self-awareness, and brain realization. 93 Sun’s desiderata are broader and include, among others, cognitive realism, adaptation, modularity, routineness, and synergistic interaction.94 Many of these criteria for cognitive system are clearly of general interest also for the class of embodied systems as defined above. The obvious question therefore is, if and how principles of cognitive architectures are aiding in the design of embodied systems.95 For example, it has been recognized that machinery which is expected to behave “correctly” in a complex world may be akin to a model-based reflective predictive controller of a machine with a mission.96 97 Some cognitive architectures use one uniform representation and corresponding learning method yielding “grand unification and functional elegance”, 98 but loosing expressiveness. Others utilize quite general knowledge representations and many inference strategies 99 that result in higher expressiveness, but they cause difficulties with integrations of different components of the cognitive architecture. A substantial number of cognitive architectures are hybrid (for example, Soar, ACT-R, LIDA, CLARION, EPIC) in that they are combining both 91 Nancy, Balamurugan, Vijakkumar, A Comparative Analysis of Cognitive Architecture, JARTET, 2016. 92 Kotseruba et al. A Review of 40 Years of Cognitive Architecture Research, arXiv:1610.08602, 2016 93 Anderson, Lebiere, The Newell Test for a Theory of Cognition, Beh. Brain Sci., 2003. 94 Sun, Desiderata for cognitive architectures, Philos. Psychol., 2004. 95 Irrespective of their relative in explaining the development of higher-level intelligent behavior and consciousness. 96 Sanz, Lopez, Rodriguez, Hernandez, Principles for consciousness in integrated cognitive control, Neural Networks Society, 2007. 97 Sanz, Thinking with the body: towards hierarchical scalable cognition, Handbook of Cognitive Science, An Embodied Approach, 2008. 98 Rosenbloom, Extending Mental Imagery in Sigma, LNAI 7716, 2012. 99 Goertzel, Pennachin, Geisweiller, Engineering General Intelligence, Atlantis Press, 2014. 34 EI Systems Challenges symbolic and sub-symbolic reasoning, thereby providing architectural concepts for integrating connectionist100 with logic-based AI technologies. Probabilistic programming provides yet another framework in which basic components of cognitive architectures are represented in a unified and elegant fashion.101 This probabilistic model cognition is destined to support aleatoric uncertainty, that is the “known unknown”. Notice also that probabilistic programming suggests a programming model for embodied systems based on well-known concepts program construction in computer science for specifying, developing, analyzing, synthesizing, and composing programs. There are numerous demonstrations of cognitive architectures for performing real-world including navigation, obstacle avoidance, object manipulation, and fetch-and-carry tasks for trash collecting102 or soda collecting 103 mobile robots. Applications from industrial domains include robotic crane operation,104 bridge construction,105 autonomous cleaning and deburring workstation,106 an automated stamp distribution center,107 and an analytics engine as inspired by the HTM cognitive architecture.108 Cognitive architectures have also proven to be useful for human performance modeling, human-robot interaction, natural language processing, categorization and clustering, and computer vision. Cognitive architectures might be able to support active perception109 for coupling perception with action of an embodied actor. For example, an actor may be spawned anywhere in the environment and may not immediately “see” the pixels containing the answer to its visual goal (for example, the car/goal may not be visible). Thus, the actor must move to succeed — controlling the pixels that it will perceive. The actor must learn to map its visual input to the correct action based on its perception of the world, the underlying physical constraints, and its understanding of the question. The observations that the actor collects are a consequence of the actions that the actor takes in the environment, and the actor is controlling the data distribution that is coming in. The actor controls the pixels it gets to see. One of the challenges of active perception is to be generally robust to variation. 100 Connectionist architectures are supposed to exhibit intelligent behavior without storing, retrieving, or otherwise operating on structured symbolic expressions. 101 Potapov, A Step from Probabilistic Programming to Cognitive Architectures, arXiv, 2016 102 Firby et al., An Architecture for Vision and Action, IJCAI, 1995. 103 Brooks, A robot that walks: emergent behaviors from a carefully evolved neural network, Neural Comp., 1989. 104 Lytle, Saidi, NIST research in autonomous construction, Auton. Robots, 2007. 105 Bostelman, Bunch, Delivery of an Advanced Double-Hull Ship Welding, ICSC, Symposia on Intelligent Industrial Automation and Soft Computing, 1999. 106 Murphy, Norcross, Proctor, CAD directed robotic deburring, Robotics and manufacturing research, education, and applications, 1988. 107 Albus, The NIST Real-time Control System (RCS): an approach to intelligent systems research, J. Exp. Theor. Artif. Intell. 1997 108 https://numenta.com/grok 109 Aloimonos (Ed.), Active Perception, Psychology Press, 1993. 35 EI Systems Challenges Finally, cognitive architectures and theories from psychology, such as cue theory, might serve as the basis and inspiration for designing novel control regimes for embodied actors capable of safely exploring and navigating the “unknown unknown”. In this way, careful terrain exploration has been approached by minimizing surprises, for example, based on active inference 110 and the free energy principle, 111 112 or, alternatively, by maximizing predictive information.113 U NCERTAINTY Q UANTIFICATION There is indeed a multitude of sources for uncertainty in the design and operation of embodied systems, as there is uncertainty about their operational context (for example, how many and which objects and actors are in the environment), there is uncertainty about corresponding hazards and risks, there is uncertainty about the behavior of learning-enabled components, there is uncertainty about safety envelopes, there is controller uncertainty due to nondeterminism and also probabilistic control algorithms, there is uncertainty on the internal models, and, last but not least, there is also uncertainty about the intentions, behaviors, and strategies actions of other embodied actors, both human114 and machines. Learning in the sense of replacing specific observations by general models is a process for inductive inference. Such models are never provably correct but only hypothetical and therefore uncertain, and the same holds true for the predictions produced by a model. For example, the input-output behavior of ANNs heavily relies on the selection of “complete” and “correct” sets of training and support data for faithfully specifying relevant operating contexts (input) and their intended internal representation (output). Another source of uncertainty for ANNs is due to the use of stochastic search heuristics, which may lead to incorrect recall even for inputs from the training data, and the largely unpredictable capability of generalizing from given data points. Uncertainty on the faithfulness of the training data representing operating contexts and uncertainty on the correctness and generalizability of training also combine in a, well, uncertain manner. One usually distinguishes between aleatoric and epistemic sources of uncertainty, whereas aleatoric115 uncertainty refers to the variability in the outcome of an experiment which is due to 110 Active inference: maintaining a model and its predictions through action to change the sensory inputs to minimize prediction error indirectly (if the sound is not getting louder, moving closer towards the train in order to hear the train getting louder). 111 Friston’s Free Energy Principle (FEP) is a leading formal theory of self-organizing system dynamics. It basically asserts that living systems must minimize the entropy of its sensory exchanges with the world; for example: Friston, The free-energy principle: a unified brain theory?, Nature reviews neuroscience, 2010). 112 Smith, A unified Framework for Intelligence based on the Free Energy Principle, 2019. 113 Ay, Bertschinger, Der, Güttler, Olbrich, Predictive information and explorative behavior of autonomous robots, European Physical Journal B, 2008. 114 Compare with the Human-Centered AI/ML challenge above. 115 Aka statistical, experimental, or “known unknown” 36 EI Systems Challenges inherently random effects, and epistemic116 uncertainty refers to uncertainty caused by a lack of knowledge.117 For example, incomplete knowledge of an embodied actor’s operating context is an epistemic source of uncertainty. As epistemic uncertainty refers to the ignorance of an actor, and hence to its epistemic state, it can in principle be reduced with additional information. The central challenge is uncertainty quantification, 118 that is, to systematically reduce uncertainty to acceptable level, and as the basis for trustworthy and (up to tolerable quantities) predictable embodied systems. Uncertainty quantification involves: § § § § § Identifying all relevant sources of uncertainty. Adequately quantifying and estimating uncertainty. Understanding how uncertainty accumulates, forward and inverse, along chains of computations. Reducing overall uncertainty below acceptable levels.119 Managing incremental change of uncertainty. Table 3 lists corresponding challenges for the rigorous design of embodied systems and for managing uncertainties throughout its lifecycle. The analysis and assurance challenges are also addressed below in the corresponding subsections on Analysis and Assurance. There are different uncertainty-reducing techniques for robust AI systems depending on the aleatoric or epistemic nature.120 The basic principle of uncertainty reduction also plays a key role in active learning121 and in learning algorithms.122 For example, indirect cues123 may cause the system to hypothesize the existence and certain objects of a relevant object, which needs to be confirmed by additional actions. In addition, uncertainty quantification approaches in engineering have been designed to demonstrate that, with high probability, a real-valued response function of a given physical system does not exceed a given safety threshold.124 116 Aka systematic, structural, or “unknown unknown” 117 Hüllermeier, & Waegeman, Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods,. Machine Learning, 110(3), 2021. 118 Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known. 119 For example, less than one hazardous behavior for 10^9 operational time 120 Dietterich, Steps Toward Robust Artificial Intelligence. AI Magazine, 38(3), 2017. 121 Aggarwal, Kong, Gu, Han, & Philip. (2014). Active learning: A survey. Data Classification: Algorithms and Applications. CRC Press. 122 Mitchell. (1980). The need for biases in learning generalizations. Tech. Rep. TR CBM–TR–117, Rutgers University. 123 Björkman. (1994). Internal cue theory: Calibration and resolution of confidence in general knowledge. Organizational Behavior and Human Decision Processes. 124 Owhadi, Scovel, Sullivan, McKerns, & Ortiz. (2013). Optimal uncertainty quantification. Siam Review, 55(2), 271-345. 37 EI Systems Challenges Specification • Challenge • • • Prediction Challenge • • • • • Assurance Challenge • • • Design Challenge • • Analysis Challenge • • Maintenance • Challenge • Provide means for constructing (and maintaining) safety envelopes, either deductively from safety analysis or inductively from safe nominal behavior. Provide means for minimizing uncertainties related to safety envelopes with a given level of effort. Provide means for deriving safety requirements for learning-enabled components, which are sufficient for establishing AI system safety. Provide means for reducing specification uncertainty by means of deriving data requirements for learning-enabled components. Identify all relevant sources of uncertainty for an AI system. Provide adequate means for measuring uncertainty. Calculate forward propagation of uncertainty, where the various sources of uncertainty are propagated through the model to predict overall uncertainty in the system response. Identify and solve relevant inverse 125 uncertainty quantification problems for safe AI (using, for example, a Bayesian approach). Predict (up to tolerable quantities) the unsafe behavior of AI systems operating in uncertain environments. Provide adequate measures of uncertainty for assuring AI system safety. Construct and maintain evidence-based arguments for supporting the certainty and for rebuting the uncertainty of safety claims. Identify useful safety case patterns for safe AI systems together with compositional operators on safety cases for managing uncertainty. Develop safety case patterns for different architectural designs of AI systems.126 Compositionally construct safe and quasi-predictable AI systems together with their safety cases. Provide adequate means for measuring and for reducing uncertainty on the input-output behavior of learning-enabled components. Define and measure the respective contribution of static and dynamic analysis techniques for learning-enabled systems, towards reducing safety-related uncertainty to tolerable levels. Identify incremental change operators for maintaining uncertainty and safety assurance of self-learning AI systems. Safely adapt and optimize the situational behavior of an AI system (together with its safety cases based on the principle of minimizing uncertainty. T ABLE 3. E NGINEERING C HALLENGES FOR U NCERTAINTY Q UANTIFICATION . 125 That is, calculating from a set of observations the causal factors that produced them. 126 In analogy to, say, Mils separation kernel protection profile. 38 EI Systems Challenges Uncertainty quantification also plays a pivotal role in reducing uncertainties for learningenabled components such as ANNs.127 128 Establishing resilience129 and other invariance130 properties, for example, are an important means for reducing behavioral uncertainty of ANNs. Moreover, measuring and estimating the uncertainty of the input-output behavior of learningenabled components is essential for, say, switching between performant and safe channels in a Simplex architecture, and uncertainty information is useful input for planning safe actions. Proposals for measuring behavioral uncertainty of learning-enabled components include: § The distance between neuron activations observed during training and the activation pattern for the current input are used for estimating the input-output uncertainty.131 § Ensemble learning techniques are used for estimating input-output uncertainty by training a certain number of ML components from different initializations and sometimes on differing versions of the dataset; the variance of the ensemble’s predictions is then interpreted as its epistemic uncertainty. § Certain instances of ensemble learning techniques such as Bayesian neural networks measure both epistemic uncertainty on model parameters, and the aleatoric uncertainty of the input-output behavior with respect to model parameters.132 What we should focus on, however, is not so much about reducing behavioral uncertainty of individual components but of the embodied system itself. Such an uncertainty on the systemlevel behavior is obtained, for example, by forward propagation133 of component uncertainties along chains of computation. Uncertainties can also be explicitly managed through assurance cases. 134 These structured arguments are comprehensive, defensible, and valid justification that the system fulfills crucial properties, at least up to a tolerable level of uncertainty, with the goal of increasing confidence and building up trust in the behavior of an embodied system. The purpose is, broadly, to demonstrate that the crucial risks associated with specific system concerns 135 have been 127 Czarnecki, Salay, Towards a framework for managing perception uncertainty for safe automated driving. Computer Safety, Reliability and Security. 2018, Springer. 128 Abdar. A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges. 2020 . 129 Cheng, Nührenberg, & Rueß, Maximum resilience of artificial neural networks, ATVA, 2017, Springer. 130 With respect to certain classes of input transformations such as stretching. 131 Cheng, Nührenberg, Yasuoka, Runtime monitoring neuron activation patterns, DATE, 2019, IEEE. 132 Jospin, Buntine, Boussaid, Laga, Bennamoun, Hands-on bayesian neural networks-a tutorial for deep learning users, arXiv:2007.06823, 2020. 133 For example, based on Bayesian inference. 134 We will be revisiting assurance cases in the subsection describing the Assurance challenge. 135 Including safety and security, but also applies to all the other attributes of trustworthiness. 39 EI Systems Challenges identified, are well-understood, have been appropriately mitigated, and that there are mechanisms in place to monitor the effectiveness of defined mitigations. Of particular interest is to capture how the influence of a learning-enabled component is captured and reasoned within the control structure of an embodied system. Recent extensions of assurance cases for reasoning about confidence and uncertainty seem to be a good starting point for a more thorough investigation into uncertainty quantification for embodied system. 136 137 Altogether, there is an increasing interest on various aspects of uncertainty quantification for embodied systems. What is still missing, however, is a comprehensive set of methods and tools for the rigorous design of embodied systems based on the principle of uncertainty quantification. S ELF -I NTEGRATION Figure 3. How and why do all the embodied actors, ranging from design, production, and logistics, form a collaborative federation in a productive manner, thereby supporting the intent of the buyer? Moreover, how does this federation tolerate real-world mishaps, such as a ship getting stuck in a channel? Intent-driven formation of purposeful federations of embodied systems requires the individual systems to be open to collaborate with others, while still operating as self-sufficient individually purposeful systems. Formation of these federations is based on self-integration, which seeks out other systems to support to meet their local and global intents and goals, which cannot be accomplished on their own. The Semantic Interoperability Logical Framework (SILF), 138 for example, facilitates dependable machine-to-machine information exchange, based on an extensive ontology to describe the content of messages, and an intent-aware mediation mechanism to translate messages as needed. These adapters may be synthesized automatically from ontological descriptions, whereas the purpose of the integration is represented in a task ontology.139 Notice that SILF focuses on the composition140 of systems but not compositionality,141 for enabling novel capabilities. More recently self-integration based on contract theory and negotiation has been used to purposefully self-integrate, for example, drones and wearable (IoT) devices. More precisely, a 136 Duan et al., Reasoning about confidence and uncertaintyin assurance cases: A survey, Software Engineering in Health Care, 2014. 137 Bloomfield, Littlewood, Wright, Confidence: its role independability cases for risk assessment, Dependable Systems and Networks, 2007. 138 NATO Science and Technology Organization, Neuilly-sur-Seine, Framework for Semantic Interoperatility, TR-IST-094.5, 2014. 139 Ford et al., Purpose-aware interoperability: the ONISTT ontologies and analyzer, Simulation Interoperability Workshop, 2007. 140 Requiring the preservation of local properties. 141 Compositionality requires the analysis of emergent properties of compositions, some of which are vital, as in safety and security. 40 EI Systems Challenges trust negotiation protocol for IoT devices has been developed to create an assume-guarantee contract that also includes a set of assessment procedures. 142 The contract yields additional assurance for dynamic integration from a shared, historical record of adaptation assessment. This additional assurance might also be managed using the concept of assurance cases. Other examples for self-integrating systems, including mobility scenarios in which cars and, say, traffic lights are purposefully interacting and adjusting their behavior for improving the flow of traffic, and intensive care unit scenarios in which, say, heart-lung machines and X-ray cameras recognize each other and negotiate their safe interaction.143 Some of these integrated systems could, of course, be readily constructed as bespoke one pony trick systems by suitably skilled teams. Automated self-integration, however, promises to be more flexible, more efficient, and less error prone. The scenarios above also demonstrate that, beyond automation of the integration, the challenge is to provide assurance for safety of the integrated systems. Forming intent-driven federations of increasingly autonomous embodied systems is a challenging endeavor. Indeed, composition of more traditional systems can often introduce new vulnerabilities,144 as in, say, exposed crypto keys and privacy violations. We therefore need to come up with suitable architectural principles and composition operators for constructing resilient and safe embodied systems from a (possibly dynamically changing) set of heterogeneous, and even untrusted, constituent systems. In this way, embodied systems may tolerate certain failures, unexpected events, and even malicious attacks. Modeling attacks and other hardware and software defects is an issue, since, almost by definition, cyber-attacks are very hard to predict.145 Yet providing some degree of resilience and to continuously improve resilience is a must for societal acceptance of embodied systems. And for the most advanced kinds of systems, it may be that what is needed is agreement on a shared system of ethics. Since embodied systems are acting in the real world with their wickerwork of societal norms, rules and laws, smart contracts are a central concept towards intent-driven dynamic federations of embodied systems. In this way, self-integration and self-orchestration might be approached as follows. § Software-based (“smart”) contracts define the service interfaces and service-level agreements for embodied actors. § Federations of embodied actors are formed through conclusion of contracts; for instance, through bidding in auctions and/or using a mediator. Smart contracts are executed until the purpose have the contract has been met. § 142 Riley et al, Toward a Negotiation Framework for Self-Integration, Autonomic Computing & Self-Organizing Systems Companion, 2020. 143 Rushby, Trustworthy Self-Integrating Systems, Distributed Computing and Internet Technology, 2016. 144 Neumann, How might we increase trustworthiness?, CACM, 2019. 145 Dutertre et al., Intrusion-Tolerant Enclaves, Security and Privacy, 2002. 41 EI Systems Challenges For example, a “ship” embodied actor offers smart contracts for “shipping A from B to C in exchange for D”. If there is a “customer” who needs to ship “a in A” from “b in B” to “c in C”, and is willing to provide “d in D” in exchange, then the “ship” and the “customer” might want to conclude a corresponding contract. If “c noting C”, for example, that is the “ship” did not intend to call at harbor “c”, then the “ship” might be willing to change her route for a small extra fee. Alternatively, mitigations of common mishaps might already be defined in the initial set of contracts. This is, of course, just how the current contract-based economy is designed to work. The offering and conclusion of contracts may be realized, for instance, by means of distributed execution of logic programs. Global invariants need to be maintained on a set of contracts. For example, the federation in Figure 3 needs to ensure that their mutual service level agreements enable timely and orderly delivery of the customer’s order. Other invariants need to be ensured such as regulatory rules, desiderata such as climate neutrality, or resiliency to some breach of contract. Not all contracts are being served to completion. What happens, for example, if the “ship” not able to fulfill the contract, since it got stuck, say, in some shipping channel. Now, the federation of actors in Figure 3 may need to reorganize to still be able to satisfy the customer order on time. Such a reorganization of the federation is based on successful renegotiation and cancellation of contracts. This process of resilient execution of smart contracts generalizes the fault-detection, isolation, and recovery (FDIR) cycle of fault-tolerant systems. Again, if contracts are negotiated by means of distributed logic programs, then resilient execution and renegotiation of contracts might, for example, be realized through backtracking and mechanisms for distributed incremental maintenance. A contract-based reconfiguration of a federation might also involve a change of the embodiment of a certain service. For example, if the ship refuses or is unable to call port “c”, the delivery federation might decide to replace this ship with other means of transportation. Using the slang of model-driven design, such a replacement involves changing the deployment of a PDM federation of virtualized services) to a PIM (embodiment of virtualized services in the physical world). Most importantly, actors and/or federations of actors need to be incentivized to honor contracts. also need to be held responsible146 for breach of contract, possible by some empowered higher instance, who identifies, collects evidence, and penalizes breach of contract. Clearly, the suggested contract-based composition and execution operator for embodied systems mimics a contract-based societal organization. The obvious question is, if and how these “smart contracts” may be integrated into existing judicial systems or variants thereof. These considerations point to a multitude of serious systems programming challenges. For instance, how do we specify smart contracts? What is the right framework for negotiating 146 Vazdanan et al., Responsibility Research for Trustworthy Autonomous Systems, Autonomous Agents and Multiagent Systems, 2021. 42 EI Systems Challenges contracts? How can we verify smart contracts? How can we provide evidence of the conclusion or breach of contract? How to incentivize/penalize embodied actors as to ensure beneficial behavior? It is also open to discussion if such federations should be deployed in social contexts without an orchestrating higher instance. A NALYSIS Analysis is the process of assuring that a system meets a set of given requirements. The verification challenge therefore involves identifying what kind of properties are expected of embodied systems and how to establish them. Analysis of embodied systems is challenging, among others, for their openness, adaptivity, situatedness, and for their largely non-predictable behavior in uncertain operating contexts. In addition to functional correctness, performance, dependability, and safety requirements as in classical embedded system, the analysis of embodied systems also focuses on establishing properties on their lawful, ethical, and robust, that is trustworthy, behavior. The verification of learning-enabled components of embodied systems poses yet another challenge. The verification of learning-enabled components of embodied systems poses yet another challenge. Consider, for example, the embodied retail system in Section 2, and assume that, in reaction to the customer’s request a dynamic federation of production and logistics services have been set up for cooperatively serving this request, then analysis might establish that this federation actually is able to fulfill the request within the agreed time frame, that the delivered product is according to functional, quality, and safety agreements. The federation might also be shown to be robust to common faults, such as breakdowns of logistics chains, and even malicious attacks. Moreover, analysis may also be used to demonstrate that the federation complies with applicable laws (for example, tax laws) and that certain social values such as climate neutrality are adhered to. State-of-the-practice techniques for safety analysis require deterministic behavior in welldefined operating contexts, and they usually rely on fallback mechanisms to a human operator. Clearly, these prerequisites are not fulfilled for embodied systems, and consequently current safety analysis methodology, as encoded in industrial standards such as DO 178C in aerospace or ISO 26262 in the automotive domain are not applicable – at least not directly so. There are three main analysis techniques, namely testing, symbolic verification, and runtime verification. We briefly describe some of associated challenges when applied to analyzing embodied systems with learning-enabled components. T ESTI NG . This is the most widely used and, arguably, also the most successful technique for analyzing software-intensive systems. Non-deterministic systems, however, are usually considered to be untestable because of the overwhelming number of cases to be considered. System tests are also performed with the assumption of fixed and well-described operating contexts. Embodied systems, however, need to be analyzed with respect to uncertain operating 43 EI Systems Challenges contexts, which may not even be known at design time. Finally, the analysis for learningenabled components requires establishing properties for all possible evolutions of such a component. For all these reasons, testing methodologies as developed for embedded systems are not directly applicable to embodied systems. Novel approaches to testing embodied systems are urgently needed. For example, scenario-based testing dynamically classifies relevant scenarios by means of automated clustering, and it generates a sufficient set of test cases from the classes thus obtained.147 More generally, probabilistic programs might be synthesized for capturing relevant scenarios,148 since probabilistic programs assign distributions to features of scenarios, and they impose hard and soft constraints over scenarios. Testing is usually decomposed into testing individual components of a system followed by testing the integrated system. But then: how can we test systems with learning-enabled components? For artificial neural networks (ANNs), traditional structural coverage criteria from software testing can usually not be applied directly to ANN. For example, neuron coverage is trivially fulfilled for an ANN by a single test case. Moreover, branch coverage, when applied to ANNs, may lead to an exponential (in the number of neurons) number of branches to be investigated, and are therefore not practical as typical ANNs are comprised of millions of neurons. As usual in testing, the balance between the ability to find bugs and the computational cost of test case generation is essential for the effectiveness of a test method.149 Therefore, ANN-specific non-structural test coverage criterion for the robustness, interpretability, completeness, and correctness of an ANN have been developed.150 A scenario coverage metric, for example, partitions the possible input space according to N attributes (e.g. snow, rainy, …), and proposes, based on existing work on combinatorial testing, efficient kprojection (for k = 0,…,N-1) coverage metrics as approximations of the exponential number of input partitions. The generation of falsifying/adversarial test cases is generally using search heuristics based on gradient descent or evolutionary algorithms.151 152 153 154 These approaches may be able to find falsifying examples efficiently, but they usually do not provide an explicit level of confidence about the nonexistence of adversarial examples in case the algorithm fails to find one. Various traditional techniques for test case generation such as fuzzing, symbolic execution, concolic testing, mutation testing, and metamorphic testing have been extended to ANNs. 147 Hauer, On Scenario-Based Testing of Automated and Autonomous Driving Systems, TUM, 2021. 148 Fremont et al. Scenic: Language-Based Scene Generation, UCB/EECS-2018-8, 2019. 149 Sun, Huang et al, Testing Deep Neural Networks, arXiv:1803.04792v4, 2019. 150 Cheng, Nührenberg, Rueß, Yasuaoka, Towards dependability metrics for neural networks, MEMOCODE, 2018, IEEE. 151 Goodfellow, Shlens, & Szegedy, Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014. 152 Papernot, McDaniel et al., The limitations of deep learning in adversarial settings, Security & Privacy, 2016, IEEE. 153 Carlini, Wagner, Towards evaluating the robustness of neural networks, Security&Privacy, 2017, IEEE. 44 EI Systems Challenges Despite their effectiveness in discovering various defects of ANNs together with their datacentric requirement specifications, however, it is not exactly clear how testing-based approaches can be efficiently integrated into the construction of convincing safety argumentations for learning-enabled components, let alone embodied systems. Altogether, testing methods seem to be effective at discovering defects of learning-enabled components such as ANNs. It is unclear, however, how to measure the effectiveness of test coverage metrics in building up sufficient confidence, or, dually, raising doubts. Also, most testing-based approaches assume a fixed ANN. However, ANNs are learning-enabled and trained continuously on new data/scenarios. The challenge is to come up with methodologies for efficiently - depending on the application context also in real-time - retesting safety requirements for continuously evolving ANNs. S YMBOLI C A NALYSI S . These analysis techniques generalize testing in that sets of test cases are evaluated on a system at once. These test sets are usually encoded as logical constraints for describing, possibly infinite, test sets. Symbolic analysis neither requires a complete system implementation nor a fully specified operational context, since unknown behavior may be represented logically by means of uninterpreted functions. Logical constraints on these uninterpreted function are used for expressing known (or learned) facts about these behaviors. In contrast to testing, symbolic analysis therefore may be applied to demonstrating that certain requirements hold for embodied systems operating in uncertain and only partially known operating contexts. Another use of symbolic analysis is to support the generation of safe trajectories during runtime.155 Recently, many different symbolic analysis techniques have been adapted to learning-enabled components such as ANNs.156 In particular, verification problems for ANNs have been reduced to constraint solving problems such as satisfiability in propositional logic,157 158 satisfiability modulo theories,159 160 161 162163 and mixed-integer linear programming.164 These approaches, however, typically do not scale up to the size of real-world ANNs with millions of neurons. Approximation techniques are applied to improve efficiency, but usually at the expense of 155 Althoff, Dolan, Online verification of automated road vehicles using reachability analysis, Transactions on Robotics, 2014, IEEE. 156 Huang et al., A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability, Computer Science Review, 2020. 157 Cheng, Nührenberg, Rueß, Verification of binarized neural networks, VSTTE, 2018. 158 Narodytska, Verifying properties of binarized deep neural networks, AAAI, 2018. 159 Huang, Kwiatkowska, Safety verification of deep neural networks, CAV, 2017 160 Pulina, An Abstraction-Refinement approach to verification of artificial neural networks, CAV, 2010. 161 Katz, Barrett e al., Reluplex: An efficient SMT solver for verifying deep neural networks, CAV, 2017. 162 Tuncali, Ito, Kapinski, Deshmukh, Reasoning about safety of learning-enabled components in autonomous cyber-physical systems, IEEE DAC, 2018. 164 Cheng, Nührenberg, Rueß, Maximal Resilience of Artificial Neural Networks, ATVA, 2017. 45 EI Systems Challenges precision. Recent approaches based on global optimization have the potential of dealing with larger networks.165 Compositional verification techniques for scaling up symbolic analysis are largely missing. Since symbolic analysis technologies work on abstract models, they might miss certain defects due to implementation issues (for example, rational numbers vs. IEEE floating points). It is also unclear how to efficiently apply these techniques to continuously evolving learning-enabled components. R UNTI ME V ERI FI CATI ON AND R ECOVERY . In runtime verification a monitor observes the concrete execution of the system in question and checks for violations of stipulated properties. When the monitor detects a violation of a property, it notifies a command module which then isolates the cause of the violation, followed by an attempt to recover from the violation. In this way, runtime verification is a central element of FDIR-based166 fault-tolerant systems. For the multitude of sources for uncertainty in AI systems, stringent real-time requirements, and everchanging learning-enabled components, runtime verification is an essential element for analyzing embodied systems. Architectural design principles for monitoring distributed systems are needed to ensure that monitoring does not perturb the system (at least, not too much).167 In particular, the tutorial discusses challenges on instrumenting real-time systems so that timing constraints of the system are respected. 168 A recent tutorial describes state-of-the-practice technology for generating runtime monitors that capture the safe operational environment of systems with AI/ML components.169 Altogether, runtime verification is an essential and attractive technique of any verification strategy for embodied systems. Unlike static verification techniques such as testing or symbolic analysis, there is no need for adaptation to learning-based components. In this way, runtime monitoring is an enabling verification technology for continuous assurance, based on the MAPE-K 170 loop from autonomic computing. The main challenge in deploying runtime monitoring, as is the case for any other cyber-physical system, is to embed monitors in an efficient (for example, energy-efficient) way, without perturbing the behavior of the embodied system too much. Runtime monitoring may also be used for measuring uncertainties in input-output behavior of learning-enabled components. For example, if an input is out-of-distribution of the training set, then one may conclude that the output may not actually be a “correct” one. Such information about the uncertainty of a perception result is useful input for deliberatively planning 165 Ruan, Wu, Sun, Huang, Reachability analysis of deep neural networks with provable guarantess, IJCAI, 2018. 166 Fault Detection, Isolation, and Recovery 167 Goodloe, Pike, Monitoring distributed real-time systems: a survey and future directions, NASA, 2010. 168 Bonakdarpour, Runtime-Monitoring of Time-Sensitive Systems, Runtime Verification, 2011. 169 https://uva-mcps-lab.github.io/RV21/paper10.1.html 170 Measure, Analyze, Plan, Execute; the K stands for Knowledge 46 EI Systems Challenges meaningful and safe actions. Uncertainty information about the perception unit is also used in Simplex architectures for switching to a safe(r) perception channel whenever the ANN output is doubtful. Clearly, the distance (in some given metric) of the input to the set of training input may serve as a measure of uncertainty of the input-output behavior of the learning-enabled component. Notice, however, that such a measure returns uncertainty zero even for “incorrect” behavior on training inputs. Alternatively, it has been proposed to monitor the neuronal activation pattern in ANN-based components, and to compare it with neuronal activation patterns as learned during the ANN training phase. 171 In addition, applicable background knowledge and physical laws may also be used in monitoring the plausibility of the input-output behavior of an ANN. In summary, due to the multitude of sources of uncertainty of embodied systems with learningenabled components and the partially unknown environments in which they operate, even if all the challenges for specification and verification are solved, it is likely that one will not be able to prove unconditional safe and correct operation. There will always be situations in which we do not have a provable guarantee of correctness. Therefore, techniques for achieving fault tolerance and error resilience at run time must play a crucial role. There is however not yet a systematic understanding of what kind of analysis cand be achieved at design time, how the design process can contribute to safe and correct operation of the embodied system at run time, and how the design-time and run-time analysis techniques can interoperate effectively. The distributed and dynamic nature of federations of embodied actors and their goals is particularly challenging for runtime verification. A runtime monitoring framework for embedded systems must support reasoning under uncertainty, 172 173 and also partially observable systems with nondeterministic and probabilistic dynamics.174 VerifAI is a runtime monitoring framework for autonomous systems with learning-enabled components. 175 It includes formal modeling of the autonomous system and its environment (in terms of probabilistic programs), automatic falsification of system-level specifications as well as other simulation-based verification and testing methods, automated diagnosis of errors, and automatic specification-driven parameter and component synthesis. Safety of systems with learning-enabled components in Simplex architectures176 often relies on a runtime monitorbased switch between a performant and a safe channel. Runtime monitoring of typical security 171 Cheng, Nührenberg, Yasuoka, Runtime monitoring neuron activation patterns, IEEE DATE, 2019. 172 Zheng, Julien, Verification and validation in cyber physical systems: Research challenges and a way forward. IEEE Software Engineering for Smart Cyber-Physical Systems, 2015. 173 Ma, Meiyi, et al., Predictive monitoring with logic-calibrated uncertainty for cyber-physical systems, TECS, 2021. 174 Viswanadha, Kim et al., Parallel and Multi-Objective Falsification with Scenic and VerifAI. Runtime Verification, 2021. 175 Torfah, Junges, Fremont, Seshia, Formal Analysis of AI-Based Autonomy: From Modeling to Runtime Assurance, Runtime Verification, 2021. 176 Desai, Ghosh, Seshia, Shankar, Tiwari, SOTER: a runtime assurance framework for programming safe robotics systems, IEEE DSN, 2019. 47 EI Systems Challenges hyperproperties, 177 178 privacy policies, 179 and contextual integrity 180 have also been considered. Given a set of properties and an embodied system the challenge is to generate and maintain sound and possibly complete runtime monitors, which are woven into the embodied system, as to keep interferences with the core behavior at a minimum. A SSURANCE A trustworthy embodied system operating in our very economic and social fabric is required to obey applicable regulation and laws, to act according to human-centered social values, and to be robust and safe (see Section 4). But how can we be assured that an embodied system indeed is worth of the trust we may put in it? Clearly analysis (see Subsection Analysis) of embodied systems is a central component of any mechanism for inspiring confidence into the lawful, ethical, and robust behavior of any technical system, and for building up trust. In addition, as an actor in a real-life context, an embodied system is also required to provide explicit evidence that indeed it is acting as required. Requirements. Embodied systems have the possibility of autonomously acting in regulated sectors such as healthcare, finance, insurance, accounting, or retail. As such they need to comply with applicable regulations and national laws; for example, the General Data Protection Rights (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), the GrammLeach-Bliley Act (GLBA), the Sarbanes-Oxley Act (SOA), or the Children’s Online Privacy Protection Act (COPPA). There is indeed no lack of these kinds of regulations and laws. For example, German law now obliges organizations to demonstrably ensure that basic labor and human rights standards are respected throughout global supply chains.181 The length of these regulations and laws, the opacity of the legal language, the complexity of these acts, and their contradictions182 however make it very difficult to determine and demonstrate compliance.183 Regulations have been formalized using a variety of policy languages.184 185 186 Formalisms and systems for expressing and checking privacy policy include P3P, 187 EPAL, 188 and 177 Finkbeiner, Hahn, Stenger, Tentrup, Monitoring hyperproperties, Formal methods in system design, 2019. 178 Bonakdarpour, Sanchez, Schneider, Monitoring hyperproperties by combining static analysis and runtime verification, Leveraging Applications of Formal Methods, 2018. 179 Chowdhury, Jia, Garg, Datta, Temporal mode-checking for runtime monitoring of privacy policies. CAV 2014. 180 Barth, Datta, Mitchell, Nissenbaum, Privacy and contextual integrity: Framework and applications, IEEE Security&Privacy, 2006. 181 https://www.bmz.de/de/entwicklungspolitik/lieferkettengesetz 182 Lam, Mitchell, Sundaram, A formalization of HIPAA for a medical messaging system, Trust, Privacy and Security in Digital Business, 2009, Springer. 183 Ness, A year is a terrible thing to waste: early experience with HIPAA, Annals of Epidemiology, 2005. 184 Anton, Earp, Reese. Analyzing website privacy requirements using a privacy goal taxonomy. Requirements Engineering 2002. 185 Anton, Eart, Vail, Jain, Gheen, Frink, HIPAA’s effect on web site privacy policies. Security & Privacy, 2007, IEEE. 186 Anton, He, Baumer. Inside JetBlue’s privacy policy violations, Security & Privacy, 2004, IEEE. 187 Cranor, P3P: Making privacy policies more useful, Security & Privacy, 2003. IEEE. 188 Ashley, Hada, Karjoth, Powers, Schunter, Enterprise privacy authorization language (EPAL), IBM Research, 2003. 48 EI Systems Challenges XACML,189 DKAL,190 and the Logic of Privacy and Utility, which is a formalization of the concept of contextual integrity. 191 In addition to compliance with applicable laws and norms, embodied systems are required to be demonstrably robust, safe, and secure. More precisely, embodied systems are expected § to be resilient to common and possibly also new kinds of breakdowns and malicious attacks, § the risk of unintended harm to humans, machinery and the environment is demonstrably below acceptable levels, and § identified confidentiality, integrity, and availability requirements are satisfied. Other requirements include transparency (for example, traceable use of data and information sources, published decision policies), demonstrable fairness of decisions (for example, all applicants are demonstrably being served equal), inverse privacy,192 or contextual integrity.193 194 The latter requirement is based on the hypothesis that privacy is a right to appropriate flow of information. In case of non-compliance to any of these requirements, culprits need to be identified and called to account. The challenge is to come up with adequate formalizations of adequate subsets of applicable laws and norms as a prerequisite for demonstrable compliance to these requirements. Previous attempts in this direction have often been based on some variant of temporal logic and/or logic programming. We now describe possible approaches and corresponding challenges for assuring compliance to these kinds of requirements. The challenge for a federation of embodied actors is to assure, and possibly demonstrably so, that it is indeed worth of our trust (Section Trustworthiness). For example, the largely autonomously acting federation of embodied retail actors (Section Service federations) is required to answer questions such as: is it able to deliver the requested product in time and according to order specs? Is it resilient to common planning breakdowns, failures, and malicious attacks? Are there contingency plans and never-give-up strategies in case of, say, operational, judicial, and financial quagmires? Are the actors of the federation acting safely? 189 Ramli, Nielson, Nielson, The logic of XACML, Science of Computer Programming, 2014. 190 Gurevich, Neeman, DKAL 2—A simplified and improved authorization language, Technical Report MSR-TR-2009–11, 2009. 191 Barth, Datta, Mitchell, Nissenbaum, Privacy and contextual integrity: Framework and applications, Security and privacy, 2006, IEEE. 192 Gurevich, Hudis, Wing, Viewpoint: Inverse privacy, CACM Volume 59, Number 7, 2016. 193 Nissenbaum, Privacy in Context: Technology, Policy, and the Integrity of Social Life, Stanford University Press, 2010. 194 Datta, Blocki, Christin, DeYoung, Garg, Jia, Sinha, Understanding and protecting privacy: Formal semantics and principled audit mechanisms, Information Systems Security, 2011. Springer. 49 EI Systems Challenges Does the federation obey applicable tax laws? Can the federation demonstrate that the delivered product is not based on child labor? Does the federation demonstrably act in a climate neutral or even climate positive manner? This list of questions to be answered is necessarily incomplete as there is indeed no lack of requirements when acting in real-world social contexts. Analysis-based assurance. Static analysis techniques such as testing ensure the system’s compliance to requirements, but only at a selected points of time in their evolution. Even if one succeeds in testing the compliance of an embodied system to its requirements, these results usually are outdated in further evolutions of the self-learning embodied system operating in a dynamic context, thereby requiring retesting. In a similar vein, traditional formal verification techniques may be applied to demonstrate, a priori during design time, conformance requirements of an embodied system operating in largely unknown environments and all its possible evolutions. Hereby, proof objects serve as evidence,195 and reflection principles are used to generate specific evidence for each possible evolution.196 But it seems unrealistic that, still largely manual, formal verification is applied to the analysis of non-trivial evolving embodied systems. By-and-large we therefore need to go beyond current static analysis techniques such as testing and formal verification to realistically assure compliance of everevolving embodied systems with their complex real-world requirements. Process-based assurance. Assurance has traditionally been approached in software and systems engineering through design processes following rigorous development standards, and by demonstrating compliance through analysis. Well-proven and successful assurance standards such as DO 178C in aerospace or ISO 26262 in the automotive domain are predicated on the assumptions that system behavior is deterministic, there is a clearly defined operating environment, there is always a fallback to a human-in-the-loop operator, and, once the system is deployed, it does not learn and evolve. These basic assumptions of traditional safety engineering do not hold anymore for embodied systems. Safety engineering therefore comes to a grand climacteric moving from deterministic, non-evolving embedded and cyber-physical systems operating in well-defined contexts to embodied systems. Process-based design for many embodied systems seems to be difficult, if not impossible, as these systems continuously integrate into dynamic coalitions, they are increasingly autonomous, and they are evolving based on previous experience. But changes in process-based assurance is notoriously slow and expensive. 197 Therefore, assurance for embodied systems increasingly needs to rely on assurance artifacts collected at runtime instead on process-based design-time assurance. Continuous assurance. It has been recommended to continuously audit trustworthiness of an embodied system throughout its lifecycle ( Table 1) to assure compliance to applicable laws, social norms, and technical robustness and safety requirements. This is easier said than done, as embodied systems 195 Necula, Proof-carrying code, POPL, 1997. 196 Rueß, Meta-Programming in the Calculus of Construction, U Ulm, 1995. 197 Changes of only one line of code in safety-cricital code in the aerospace industry have reportedly cost seven-figure sums (in US Dollars) in some cases. 50 EI Systems Challenges § are dynamic federations based on changing priorities and intent, § are composed of ever-evolving embodied systems with self-learning capabilities, § operate in largely unknown contexts based on experience and targeted exploration, and § applicable laws and regulations may change over time and are also location dependent. The challenge therefore is continuous assurance, an assurance system provided provisionally at design time, and continually monitored, updated, and evaluated at run-time as the system and its environment evolves. But continuous assurance for regulations such as GDPR has become a costly burden.198 199 The culprit is the heavy reliance on manual, checklist-based auditing in today’s compliance process, which is expensive, slow, and error-prone. It is also restricted to the compliance of static snapshots in the evolution of a system. Therefore, runtime analysis and assurance are all-important for continuously assuring compliance of ever-evolving embodied systems. Runtime assurance is a dynamic analysis technique based on a log for recording system events, which are relevant for establishing conformance with given requirements. These requirements are usually expressed in a variant of temporal logic, which is expressive enough for encoding regulation rules. 200 Now runtime verification techniques are used for establishing that the logged event trace indeed conforms with the given requirement. For example, an embodied actor logs events of its data handling operation, and runtime verification is used to establish compliance with a given privacy policy or detecting violations thereof. In addition, the root cause of such a violation may be computed from a log as the basis for calling the responsible embodied actors to account for the conformance violations they caused. It is open to discussion if the identified embodied culprits can and should be treated as legal entities, which can be held responsible for their actions in a traditional brick-and-mortar judicial system. Nevertheless, a workable judicial governance of one sort or the other is needed for promoting, by-and-large, compliant behavior - as the basic mechanism for a self-improving system behavior. Depending on the regulatory framework, compliance to these requirements by every actor might lead to functionally inferior behavior or may even be a barrier to innovation. For example, regulatory frameworks may well be contradictory, there are ongoing considerations of the relative importance of compliance with a wide range of different requirements, possibly from 198 According to Forbes ,GDPR cost US Fortune 500 companies $7.8 Billion as of 2018. (Forbes, The GDPR racket: Who’s making money from this $9bn business shakedown, 2020. https://www.forbes.com/sites/oliversmith/2018/05/02/the-gdpr-racket-whos-making-money-fromthis-9bn-business-shakedown/#54c0702034a2) 199 A recent report shows that 74% of small- or mid-sized organizations spent more than 100 k$to prepare for continuous compliance with GDPR and CCPA. (DataGrail, The age of privacy: The cost of continuous compliance, 2020, https://datagrail.io/downloads/GDPR-CCPAcost-report.pdf) 200 DeYoung, Garg, Dia, Kaynar, Datta, Experiences in the logical specification of the HIPAA and GLBA privacy laws. WPES 2010, ACM. 51 EI Systems Challenges different sources, and, finally, without minor transgressions of, say, traffic rules (such as crossing solid lines) traffic flow would often be less fluid. At a high-level, runtime assurance can be approached by compliance checks on recorded logs when demanded, say, by an audit authority, 201 by online checking of relevant events against the prevailing policy, 202 203 204 205 206 207 208 209 or by combinations thereof.210 In federations of embodied actors, system logs do not only need to contain all the relevant event in the right order, but they also need to be tamper-proof and they need to comply with applicable privacy policies. Distributed ledger technology such as various incarnations of blockchains have lately been tried, for instance, for tamper-proof logging. It is also challenging to monitor information flow policies such as non-interference or observational determinism, which relate multiple computation traces with each other.211 212 Assurance Cases. A fundamentally different approach to assurance is based on constructing and maintaining safety and assurance cases, 213 which are compelling, comprehensive, defensible, and valid justification of the compliance of a system. It is based on a structured argument of assurance considerations, across the system lifecycle, that can assist in convincing the various stakeholders that the system is acceptably safe. The purpose is, broadly, to demonstrate that the safety-related risks associated with specific system concerns214 have been identified, are well-understood, and have been appropriately mitigated, and that there are mechanisms in place to monitor the effectiveness of safety-related mitigations. In this sense, an assurance case is a structured argument for linking safety-related claims through a chain of arguments to a body of the appropriate evidence. 201 Garg, Jia, Datta, Policy auditing over incomplete logs: Theory, implementation and applications. CCS, 2011. 202 Basin, Klaedtke, Müller. Monitoring security policies with metric first-order temporal logic. SACMAT, 2010, ACM. 203 Basin, Klaedtke, Marinovic, Zalinescu, Monitoring compliance policies over incomplete and disagreeing logs, RV, 2012, Springer. 204 Basin, Klaedtke, Marinovic, Zalinescu, Monitoring of temporal first-order properties with aggregations, RV, 2013, Springer. 205 Chomicki, Efficient checking of temporal integrity constraints using bounded history encoding ACM Trans. Database Syst., 1995. 206 Chomicki, Niwinski, On the feasibility of checking temporal integrity constraints, PODS 1993, ACM. 207 Krukow, Nielsen, Sassone, A logical framework for history-based access control and reputation systems, J. Comput. Secur., 2008. 208 Bauer, Gore, Tiu, A first-order policy language for history-based transaction monitoring, ICTAC 2009. 209 Ozeer, φ comp: An Architecture for Monitoring and Enforcing Security Compliance in Sensitive Health Data Environment, ICSA-C, 2021, ACM. 210 Chowdhury, Jia, Garg, Datta, Temporal mode-checking for runtime monitoring of privacy policies, CAV, 2014, Springer. 211 Finkbeiner, Hahn, Stenger, Tentrup, Monitoring hyperproperties, RV, 2017, Springer. 212 Bonakdarpour, Finkbeiner, The complexity of monitoring hyperproperties, CSF, 2018, IEEE. 213 UK Ministry of Defence, 2007. 214 Including safety and security, but also applies to all the other attributes of trustworthiness. 52 EI Systems Challenges One of the main benefits for structured arguments in assurance cases is to explicitly capture the causal dependencies between assurance claims and the substantiating evidence, as obtained, for instance, by analysis. Assurance cases also determine the level of scrutiny needed for developing and operating systems which are acceptably safe. This kind of information from safety cases might in the future also be used by an embodied actor to safely explore and navigate in largely unknown operating contexts. On the other hand, assurance cases currently are at most semi-formal and their construction and maintenance requires significant manual efforts. Adequate notions of assurance case patterns and modular composition operators for assurance cases, however, are opening up new possibilities for aligning compliance arguments with system evolvements during runtime (such as the dynamic formation of federations). A major challenge is to measure confidence in assurance cases.215 216 217 218 219 Eliminative induction, for instance, increases confidence in assurance cases by removing sources of doubt and using Baconian 220 probability to represent confidence. 221 One systematic approach is through construction and dialectical consideration of counterclaims and countercases, where counterclaims are natural in confirmation measures as studied in Bayesian confirmation theory, and countercases are assurance cases for negated claims. Assurance cases are successfully applied to traditional safety-critical systems with clearly defined operating contexts, safety requirements, and fallback strategies to human operators. Given the increasing complexities and sources of uncertainty, however, the current assurance approach with prescribed and fixed verification and validation process activities, criteria, and metrics does not work well for assuring AI-based or even embodied systems (Alves, et al., 2018). Due to the multitude of sources of uncertainty, assurance arguments for increasingly autonomous embodied systems, need to (1) stress rigor in the assessment of the evidence and reasoning employed, and (2) automate the search for defeaters, the construction of cases and countercases, and the management and representation of dialectical examination.222 It is of particular interest to capture how the influence of a learning-enabled component is captured and reasoned within the control structure of an embodied actor. Rigorous assurance cases can be adapted, faster and more efficiently, to ever-evolving assurance needs of embodied systems. 215 Grigorova, Maibaum, Taking a page from the law books: Considering evidence weight in evaluating assurance case confidence, Software Reliability Engineering Workshops. 2013, IEEE. 216 Duan, Rayadurgam, Heimdahl, Sokolsky, Lee, Representing confidence in assurance case evidence, Computer Safety, Reliability, and Security, 2014, Springer. 217 Bloomfield, Littlewood, Wright, Confidence: its role in dependability cases for risk assessment, Dependable Systems and Networks (DSN), 2007, IEEE. 218 Rushby, Formalism in safety cases, Making Systems Safer, 2010, Springer. 219 Rushby, Logic and epistemology in safety cases, Computer Safety, Reliability, and Security, 2013, Springer. 220 https://ntrs.nasa.gov/api/citations/20160013333/downloads/20160013333.pdf 221 Goodenough, Weinstock, Toward a theory of assurance case confidence, CMU Technical Report, 2004. 222 Bloomfield, Rushby, Assurance 2.0: A Manifesto, arXiv:2004.10474, 2020. 53 EI Systems Challenges Rigorous assurance cases open new possibilities for dependable and safe exploration in largely unknown operating contexts, which have been obtained from relevant information of a safety case and its certainty assessment. For example, if there is only weak evidence on the fact that the traffic light in front of the ego cart is green, then the ego car might want to increase her assurance by strengthen this case, for example, by means of additional sensor activity. Therefore, rigorous assurance cases can be instrumental in online behavioral self-adaptation and for determining safe behavior when operating in uncertain contexts. Evidence-based Assurance is based on explicitly generating verifiable evidence for any transaction of and between embodied actors. The underlying mindset is “verify, then trust”. Consider, for example, a responsible visa-granting actor that returns evidence (think of it as a number) corresponding to the permission for the requester to enter the foreign country for a specific time frame. The border control, upon entry, verifies the identity of the presenter and verifies that the presented visa is valid for this person to enter the country now. This basic mechanism is applicable to all kinds of transactions which involve the exchange of physical evidence in the form of identity cards, driver’s licenses, money, checks, visas, airline tickets, traffic tickets, birth certificates, vaccination certificates, and stock certificates, as well as electronic evidence including PIN numbers, passwords, keys, certificates, digital coins, and nonces. It is easy to extrapolate from the above scenario to other uses of digital evidence in electronic commerce, business and administrative processes, and digital government. where actors execute smart contracts (as outlined in Subsection Self-Integration) to carry out specific tasks that require the exchange of authorization and authentication information. In this way, evidence is produced, say, for asserting the conclusion of contracts. The generated evidence then can be independently and automatically checked, for example, as the prerequisite for directing corresponding payments. Previously we have been defining such a framework for evidential transactions, called Cyberlogic, which is based on a public key infrastructure. 223 224 The basic ingredients are extremely simple. First, evidence is encoded by means of numbers using digital certificates and nonces. Second, predicates are signed by private keys so that a decryption of such a certificate with the corresponding public key is a proof or evidence for the assertion contained in the certificate. Third, protocols are distributed logic programs that gather evidence by using both ordinary predicates, digital certificates, and proof construction based on the Curry-Howard isomorphism. These simple building blocks are sufficient to encode all-important capabilities for delegation, retraction of permission, and time-stamped evidence. For example, some vaccination certificate might be handed out at a certain point of time. Now, new medical 223 Rueß, Shankar, Introducing Cyberlogic, High Confidence Software and Systems Conference, Baltimore, 2003. 224 Nigam, Reis, Rahmouni, S, Rueß, Proof Search and Certificates for Evidential Transactions, Automated Deduction (CADE), 2021. 54 EI Systems Challenges evidence on the ephemeral efficacy of vaccination results in the need for retracting corresponding capabilities. Evidential transaction in a Cyberlogic-like setting have recently been used to demonstrate that accountability for federated machine learning becomes paramount to fully overcoming legislative and jurisdictional constraints.225 In particular, it ensures that all entities' data are adequately included in the model and that evidence on fairness and reproducibility is curated towards trustworthiness. or realizing accountability for federated machine learning. Cyberlogic also forms the foundations for secure and trusted-by-design smart contracts.226 Even though Cyberlogic has proven to be instrumental in many cases, it is still unclear if proof-carrying capabilities can be scaled up for complex real-world systems. Possible directions for addressing these scalability issues are interactive proof systems and the use of advanced cryptographic mechanisms. Altogether, we have described challenges and possible approaches for the assurance of embodied systems acting in social contexts. It should have become clear that embodied systems, as increasingly autonomous actors in the real world, need to deal, like the rest of us, with the complexity and the rigmaroles associated with real-world norms and laws. As technical systems, however, they rely on rigorous and faithful digital representations thereof. Moreover, auditing, as a means for assuring compliance, needs to be aligned with the evolvement of actors. Auditing also needs to be largely automated, and correspondingly we have outlined promising base technologies based on assurance cases and evidential transactions. We therefore suggest that rigorous and automated assurance is key to a meaningful embodiment of ever evolving and increasingly autonomous systems into social contexts. It is also hoped for that these advancements eventually may also prove to be instrumental for a more rational and evidence-based development and maintenance of judicial and regulatory frameworks – as the basis for deforesting the convoluted jungle of applicable regulations and for creating space for more innovative behavior. 225 Balta, Sellami, Kuhn, Schöpp, Buchinger, Baracaldo, Altakrouri, Accountable Federated Machine Learning in Government: Engineering and Management Insights, Electronic Participation, 2021. Springer. 226 Dargaye, Kirchner, Tucci-Piergovanni, Towards secure and trusted-by-design smart contracts, Francophone Days of Application Languages, 2018. 55 EI Systems Challenges 6. C ONCLUSIONS We have been arguing that a new generation of increasingly autonomous and self-learning systems is about to be developed and embodied into all kinds of aspects of everyday life. A main driver for their deployment lies in their ubiquitous disruptive potential, as autonomy and unsupervised learning capabilities are widely believed to be the key technological base for initiating and driving the next economic and societal phase shift.227 Embodied systems are not a distant science fiction, as purpose-built embodied systems might be “hand crafted” with currently available technology. But only at very high cost and sometimes with unknown risks, as we do not yet have a mature science and technology to support the engineering of embodied systems in which we may put our trust. There is indeed a real risk both of becoming dependent and of losing a meaningful (human) control over increasingly autonomous and self-learning systems. When deploying these systems into a real-life context we therefore face various engineering challenges, as it is crucial to coordinate the behavior of embodied actors in a beneficial manner, ensure their compatibility with our human-centered social values, and design verifiably safe and reliable human-machine interaction.228 Traditional systems engineering, however, is coming to a climacteric from embedded to embodied systems, and with assuring the trustworthiness of a new generation of dynamic federations of situationally aware, intent-driven, explorative, ever-evolving, largely nonpredictable, and increasingly autonomous embodied systems in uncertain, complex, and unpredictable real-world contexts. Model-driven systems engineering has been the pillar for building dependable and safe embedded controls, say, for safely flying airplanes. Currently, we are working on data-driven engineering techniques for functionally automated systems with learning-enabled components. Next, we need to develop engineering capabilities for trustworthy embodied systems, which are based on a suitable mixture, adaptations, and further developments of model- and data-driven engineering methodologies. Such a framework for systems engineering needs to be as rigorous as possible, as embodied systems eventually are equipped with substantial self-engineering capabilities, including dataand experience-driven functional updates, zero-touch repair and maintenance capabilities, and the possibility of by-need-augmentation of sensing, cognitive, and acting capabilities. We also envision embodied systems which are performing their own risk analysis and are defining their own mitigation strategies based on their own understanding of socially acceptable behavior. 227 Davidow, Malone, The Autonomous Revolution: Reclaiming the Future we’ve Sold to Machines, 2020. 228 Vazdanan et al., Responsibility Research for Trustworthy Autonomous Systems, 2021. 56 EI Systems Challenges We have described the main characteristics of embodied systems, and we have derived from these characteristics a shortlist of central systems challenges: including robust and humancentric AI, architectures for autonomous systems, safe exploration of the unknown (unknown), trustworthy self-integration, and continual verification and assurance for federated self-learning systems. Notice that we have been restricting ourselves to technical systems challenges, thereby omitting all-important, but largely unsolved, economic, jurisdictional, and societal questions on embodying autonomous, technical systems into our everyday life. We consider the identified list of technical systems challenges as just comprising the most pressing ones, and we expect to identify additional ones as we gain a deeper understanding of the real consequences of embodying autonomous actors into economic and societal contexts. Moreover, the relative importance of each of the identified systems challenges is highly dependent on the envisioned application context and its associated requirements. Solving these challenges will require synergistic innovations in software systems engineering, architectures for autonomous systems, and core AI/ML algorithms. A crucial next step is to gain more experience and increase our theoretical understanding of autonomous and self-learning embodied systems in which we can put our trust. A series of increasingly challenging embodied systems might support us in leveraging and bootstrapping engineering knowledge for embodied systems in an accelerated fashion. At the same time, sound sociopolitical and legal conditions and frameworks must be created for embodying autonomously acting machines in essential real-world processes and structures, as failing to deploy embodied systems in a meaningful manner into our very economic and social fabric can all too easily and quickly become dystopian. 57