13/10/2024

BLG 435E: Artificial Intelligence

Intelligent Agents and Environment Types
Summary of the Last Lecture
Instructor: Professor Mehmet Keskinöz
ITU Artificial Intelligence Research and Development Center (ITUAI)
Faculty of Computer and Informatics
Computer Engineering Department
Istanbul Technical University, Istanbul, Turkey

Email: keskinoz@itu.edu.tr
ODS 2001 ODS 2001

What is AI? Acting Humanly

• Turing (1950) "Computing machinery and intelligence":

Views of AI fall into four categories: • "Can machines think?"  "Can machines behave
intelligently?”

Thinking humanly Thinking rationally

• Skills required:
– Natural language processing
Acting humanly Acting rationally – Knowledge representation
– Automated reasoning
– Machine learning
• Modern view (ie. Since 1990s): Acting rationally. • Predicted that by 2000, a machine might have a 30%
chance of fooling a lay person for 5 minutes
– Cleverbot.
– Alice, Kirk
– Eliza
– Loebner Prize

ODS 2001 ODS 2001

1
 13/10/2024

Thinking humanly: cognitive modeling Thinking rationally

• Aristotle: what are correct arguments/thought processes?

• Validate thinking in humans
• Several Greek schools developed various forms of logic:
– notation and rules of derivation for thoughts;
• Cognitive science brings together computer models from AI
and experimental techniques from psychology to construct • Direct line through mathematics and philosophy to modern
AI.
the working of the human mind.

ODS 2001 ODS 2001

Rational Action Thinking rationally

• Aristotle: what are correct arguments/thought processes?

• Rational behavior: doing the right thing
• Several Greek schools developed various forms of logic:
• The right thing: that which is expected to maximize – notation and rules of derivation for thoughts;
goal achievement, given the available information
• Does it require thinking? • Direct line through mathematics and philosophy to modern
– Not always. AI.
• Iroboclean?
• blinking reflex.
• Insects. Do dung beetles think?
– Thinking seems to lead to flexibility and robustness.

ODS 2001 ODS 2001

2
 13/10/2024

Rational Action Inspirations for AI

• Rational behavior: doing the right thing

• Major question:
• The right thing: that which is expected to maximize – “How are we going to get a machine to
goal achievement, given the available information act intelligently to perform complex tasks?”
• Does it require thinking?
– Not always.
• Iroboclean?
• blinking reflex.
• Insects. Do dung beetles think?
– Thinking seems to lead to flexibility and robustness.

ODS 2001 ODS 2001

Inspirations for AI Inspirations for AI

1. Logic 2. Introspection
– Studied intensively within mathematics
– Gives a handle on how to reason intelligently
– Humans are intelligent, aren’t they?
• Example: automated reasoning • Expert systems
– Proving theorems using deduction – Implement the ways (rules) of the experts
– http://www.youtube.com/watch?v=3NOS63-4hTQ • Example: MYCIN (blood disease diagnosis)
• Advantage of logic: – Performed better than junior doctors
– We can be very precise (formal) about our programs
• Disadvantage of logic:
– Not designed for uncertainty.

ODS 2001 ODS 2001

3
 13/10/2024

Inspirations for AI Inspirations for AI

3. Brains
4. Evolution
– Our brains and senses are what give us intelligence
– Our brains evolved through natural selection
• Neurologist tell us about:
– Networks of billions of neurons • So, simulate the evolutionary process
• Build artificial neural networks – Simulate genes, mutation, inheritance, fitness, etc.
– In hardware and software (mostly software now) • Genetic algorithms and genetic programming
• Build neural structures – Used in machine learning (induction)
– Interactions of layers of neural networks
– Used in Artificial Life simulation
• http://www.youtube.com/watch?v=r7180npAU9Y&NR=1

ODS 2001 ODS 2001

1.2 Inspirations for AI Rational Agents

• An agent is an entity that perceives and acts

5. Society • This course is about designing rational agents
– Humans interact to achieve tasks requiring intelligence • Abstractly, an agent is a function from percept histories to actions:
– Can draw on group/crowd psychology
• Software should therefore [ f: P*  A ]
– Cooperate and compete to achieve tasks
• For any given class of environments and tasks, we seek the agent
• Multi-agent systems (or class of agents) with the best performance.
– Split tasks into sub-tasks • The primary goal is performance, not thinking, consciousness or
– Autonomous agents interact to achieve their subtask intelligence. These may be means to achieve performance.
• http://www.youtube.com/watch?v=1Fn3Mz6f5xA&feature=related • Performance measure is usually given by the user or engineer.
• http://www.youtube.com/watch?v=Vbt-vHaIbYw&feature=related •
• Used in movies too. • computational limitations make perfect rationality unachievable
 design best program for given machine resources

ODS 2001 ODS 2001

4
 13/10/2024

AI prehistory

• Philosophy
– Can formal rules be used to draw valid conclusions?
– Where does knowledge come from?
– How does knowledge lead into action? Intelligent Agents
• Mathematics/Statistics
– What are the formal rules to draw valid conclusion?
– How do we reason with uncertain information?
– How do intelligent agents learn?

• Economics
– How should we make decisions to maximize payoff?
– How should we do this when others are making decisions too?

• Psychology
– How do humans and animals think?

• Computer
– How can we build efficient computers?

• Linguistics
– How does language relate to thoughts?
– knowledge representation, grammar

ODS 2001 ODS 2001

Agents
• Sensors and actuators are both key components in control systems and automation, but
they serve opposite functions:
• An agent is anything that can be viewed as perceiving • Sensors
its environment through sensors and acting upon that • Function: Sensors detect and measure physical properties or changes in the
environment through actuators environment and convert them into signals that can be read by a system or a device.
• Purpose: They gather information from the environment, acting as the "input" side of
• Human agent: the system.
– eyes, ears, and other organs for sensors; • Examples:
– hands, legs, mouth, and other body parts for actuators – Temperature sensors (thermometers)
– Light sensors (photocells)
• Robotic agent: – Motion detectors
– cameras and infrared range finders for sensors – Pressure sensors
– various motors for actuators – Cameras
• Use Case: In a smart home, a temperature sensor might detect the room temperature
and send that data to a thermostat.
ODS 2001 ODS 2001

5
 13/10/2024

• An actuator is a device that converts energy (usually electrical, hydraulic, or

• Actuators pneumatic) into mechanical motion. It is a critical component in systems that
• Function: Actuators convert signals (often electrical, hydraulic, or pneumatic) require movement or control, such as machinery, robotics, and smart home
into physical motion or action. devices.
• Purpose: They perform actions or outputs in response to control signals, making • There are different types of actuators depending on the form of energy they use:
them the "output" side of the system.
• Electric actuators: Use electrical energy to create motion, commonly used in
• Examples:
precise control applications.
– Electric motors
• Hydraulic actuators: Use pressurized liquid to generate motion, often used in
– Hydraulic pistons
heavy machinery.
– Pneumatic cylinders
• Pneumatic actuators: Use compressed air to produce mechanical movement,
– Heating elements
suitable for fast and high-force operations.
– Speakers
• In smart systems, like smart homes or cities, actuators are used to control things
• Use Case: In a smart home, an actuator would open or close an air vent based
like doors, windows, HVAC systems, and lighting by responding to commands
on commands from the thermostat.
from sensors or control systems. For example, a smart thermostat might use an
actuator to open or close air vents based on temperature settings.
ODS 2001 ODS 2001

• A percept sequence refers to the complete history of everything an agent (like a

robot, software, or AI system) has perceived through its sensors over time. It is
essentially a record of all the inputs the agent has received from its
• Key Difference
environment. These inputs (or "percepts") help the agent make decisions about
• Sensors collect and transmit data about the environment to a system (input),
how to act in its environment.
while actuators act on the environment by executing physical actions based on
commands (output). • For example, in the context of a robot navigating a room, the percept sequence
• In short, sensors are for detecting while actuators are for acting. might include sensor data like visual images, distance measurements, or sound
information from different moments in time. Based on this percept sequence,
the agent decides what actions to take, such as moving forward, turning, or
stopping.
• In artificial intelligence, particularly in intelligent agents, the percept sequence is
a fundamental concept, as the agent’s actions are typically based on this
accumulated information rather than just the current percept.

ODS 2001 ODS 2001

6
 13/10/2024

Agents and environments Vacuum-cleaner world

Demo:
http://www.ai.sri.com/~oreilly/aima3ejava/aima3ejavademos.h
tml

• Percepts: location and contents, e.g., [A,Dirty]

• Actions: Left, Right, Suck, NoOp
• The agent function maps from percept histories to actions: • Agent’s function  look-up table
– For many agents this is a very large table
[f: P*  A]

• The agent program runs on the physical architecture to

produce f
• agent = architecture + program

ODS 2001 ODS 2001

Rational agents Omniscient Agent

• Rationality
– Performance measuring success • An omniscient agent in artificial intelligence (AI) refers to a hypothetical or
– Agents prior knowledge of environment theoretical agent that has complete knowledge of the environment and all
– Actions that agent can perform possible outcomes of its actions. It knows the actual state of the world and
– Agent’s percept sequence to date how its actions will affect the future with absolute certainty.
• Key Features of an Omniscient Agent:
• Rational Agent: For each possible percept sequence, a • Complete Knowledge: It knows everything about the environment, including
rational agent should select an action that is expected to hidden or uncertain aspects.
maximize its performance measure, given the evidence • Perfect Prediction: It can predict the exact consequences of all possible
provided by the percept sequence and whatever built-in actions, knowing their outcomes ahead of time.
knowledge the agent has. • Optimal Decision-Making: Since it knows the best action to take in every
• possible situation, it can make optimal decisions to achieve its goal.

ODS 2001 ODS 2001

7
 13/10/2024

Rationality

• Important Distinction:
• Omniscience vs. Rationality: Real-world AI agents are not omniscient because
• Rational is different from omniscience they do not have perfect knowledge of the environment or outcomes. However,
– Percepts may not supply all relevant information they can be rational, meaning they make the best possible decisions based on
the information they have. A rational agent may not always make the best
– E.g., in card game, don’t know cards of others. decision but works to maximize its performance given its limited knowledge and
computational resources.
• Why Omniscience is Impractical:
• Rational is different from being perfect • In real-world applications, omniscience is not achievable due to:
– Rationality maximizes expected outcome while • Uncertainty in environments: The agent cannot know all factors influencing
perfection maximizes actual outcome. the environment.
• Incomplete data: Sensors and inputs can be noisy or incomplete.
• Computation limits: It is impossible to compute all possible actions and their
consequences due to time and resource constraints.
• Thus, most AI systems aim for rationality rather than omniscience.
ODS 2001 ODS 2001

Autonomy in Agents PEAS

• PEAS: Performance measure, Environment, Actuators,

The autonomy of an agent is the extent to which its Sensors
behaviour is determined by its own experience,
rather than knowledge of designer.
• Must first specify the setting for intelligent agent design
• Extremes • Consider, e.g., the task of designing an automated taxi
– No autonomy – ignores environment/data driver:
– Performance measure: Safe, fast, legal,
– Complete autonomy – must act randomly/no comfortable trip, maximize profits
program – Environment: Roads, other traffic, pedestrians,
• Example: baby learning to crawl customers

• Ideal: design agents to have some autonomy – Actuators: Steering wheel, accelerator, brake,
signal, horn
– Possibly become more autonomous with
experience – Sensors: Cameras, sonar, speedometer, GPS,
odometer, engine sensors, keyboard
ODS 2001 ODS 2001

8
 13/10/2024

PEAS PEAS

• Agent: Part-picking robot • Agent: Interactive English tutor

• Performance measure: Percentage of parts in • Performance measure: Maximize student's
correct bins score on test
• Environment: Conveyor belt with parts, bins • Environment: Set of students
• Actuators: Jointed arm and hand • Actuators: Screen display (exercises,
• Sensors: Camera, joint angle sensors suggestions, corrections)
• Sensors: Keyboard

ODS 2001 ODS 2001

Environment types
Fully observable (vs. partially observable)

• Fully observable (vs. partially observable)  Is everything an agent requires to choose its actions
• Deterministic (vs. stochastic) available to it via its sensors? Perfect or Full
• Episodic (vs. sequential) information.
• Static (vs. dynamic)  If so, the environment is fully accessible
• Discrete (vs. continuous)
 If not, parts of the environment are inaccessible
• Single agent (vs. multiagent):
 Agent must make informed guesses about world.

 In decision theory: perfect information vs. imperfect

information.
Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis
Fully Partially Partially Partially Fully Fully

Artificial Intelligence a modern approach

ODS 2001

9
 13/10/2024

Deterministic (vs. stochastic)

• An episode in the context of artificial intelligence (AI) and machine learning
refers to a single, self-contained interaction or sequence of events between an
• Does the change in world state agent and its environment, which starts in some initial state and ends when a
terminal condition is reached. During an episode, the agent takes actions,
– Depend only on current state and agent’s action? receives feedback (rewards or penalties), and transitions through various states.
• Non-deterministic environments • Key Features of an Episode:
• Start and End: An episode begins in a specific state and ends when a goal is
– Have aspects beyond the control of the agent
achieved, a failure occurs, or a time limit is reached.
– Utility functions have to guess at changes in world • Sequence of Actions: Within an episode, the agent takes a series of actions
Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis based on the environment's current state and receives rewards or penalties.
Deterministic Stochastic Stochastic Stochastic Stochastic Deterministic • Learning Context: Episodes are often used in reinforcement learning, where
the agent learns by interacting with the environment over multiple episodes,
improving its strategy over time.

ODS 2001 ODS 2001

Examples of Episodes: • Role of Episodes in AI:

• Game Playing: In a game like chess, one episode would be a complete game, • Training: During training, especially in reinforcement learning, the agent
starting with the first move and ending with a win, loss, or draw. undergoes multiple episodes to learn optimal strategies through trial and error.
• Robot Navigation: If a robot is tasked with moving from point A to point B, one • Evaluation: Episodes allow for clear evaluations of an agent’s performance, as
episode would cover the entire journey, from the starting point to the destination each episode has a clear outcome (success, failure, etc.).
or failure point.
• In summary, an episode is a complete cycle of interaction between an agent
• Image Classification: In an episodic environment, an episode could be a single and its environment, from the starting point to a defined end, allowing the agent
image classification task where the agent classifies an image and receives to learn and adapt based on the feedback it receives during each episode.
feedback on whether it was correct or not.

ODS 2001 ODS 2001

10
 13/10/2024

Episodic Environment Sequential Environment

• Independence of Actions: Each action or task in an episodic environment is • Sequential Environment:

independent of previous and future actions. What the agent does in one episode • Dependence of Actions: In a sequential environment, the agent's current
does not affect future episodes or states. actions affect future states and decisions. What the agent does now can impact
• No Long-Term Consequences: Actions have immediate effects, but they don’t the subsequent situation.
influence what happens next. The agent focuses only on the current situation. • Long-Term Consequences: Actions have lasting effects. The agent must plan
• Simplified Decision-Making: The agent does not need memory or planning ahead and consider how each action will influence not only the immediate next
over multiple steps. It only needs to consider the present percept and action. state but potentially many future states.
• Example: Classifying images, where each image is independent, and how the • Complex Decision-Making: The agent must keep track of previous actions and
agent classifies one image has no impact on the next. states, possibly maintain a memory of past interactions, and plan its actions
considering future outcomes.
• Task: Classify an image.
• Example: Playing chess, where each move influences the future configuration of
• Action: Based solely on the current image. the board and the overall outcome of the game.
• Outcome: The result of the classification does not affect the next image – Task: Play a move in chess.
classification task. – Action: The move affects future board states and the rest of the game.
– Outcome: Each move has consequences that influence future moves and
ODS 2001
the overall outcome. ODS 2001

Episodic versus Sequential Environment Episodic (vs. sequential):

• Key Differences:
• Independence vs. Dependence: In an episodic environment, each episode is
• Is the choice of current action
independent, whereas in a sequential environment, actions are dependent on
– Dependent on previous actions?
previous actions and affect future states.
– If not, then the environment is episodic
• Short-Term vs. Long-Term Impact: Episodic environments focus on short-term
effects, while sequential environments involve long-term planning and • In non-episodic environments:
consequences. – Agent has to plan ahead:
• Memory and Planning: Sequential environments often require the agent to • Current choice will affect future actions
maintain memory or anticipate future states, which is not needed in episodic
environments. Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis
• Summary: Sequential Sequential Sequential Sequential Episodic Episodic

• Episodic: Actions are isolated, and decision-making is simpler, focusing on one-

off tasks without future consequences.
• Sequential: Actions are interconnected, requiring the agent to plan and consider
the long-term effects of its actions.
ODS 2001 ODS 2001

11
 13/10/2024

Static (vs. dynamic): Discrete (vs. continuous)

• Static environments don’t change • A limited number of distinct, clearly defined percepts and
– While the agent is deliberating over what to do actions vs. a range of values (continuous)

• Dynamic environments do change

– So agent should/could consult the world when choosing actions
– Alternatively: anticipate the change during deliberation OR make
decision very fast Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis

• Semidynamic: If the environment itself does not change with Discrete Discrete Discrete Conti Conti Conti

the passage of time but the agent's performance score does.

Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis
Static Static Static Dynamic Dynamic Semi

Another example: off-line route planning vs. on-board navigation system

ODS 2001 ODS 2001

Single agent (vs. multiagent): Summary.

Observable Deterministic Episodic Static Discrete Agents

• An agent operating by itself in an environment or there are
many agents working together Cross Word Fully Deterministic Sequential Static Discrete Single

Poker Fully Stochastic Sequential Static Discrete Multi

Backgammon Partially Stochastic Sequential Static Discrete Multi

Cross Word Poker Backgammon Taxi driver Part picking robot Image analysis
Single Multi Multi Multi Single Single
Taxi driver Partially Multi
Stochastic Sequential Dynamic Conti

Part picking robot Partially Stochastic Episodic Dynamic Conti Single

Image analysis Fully Deterministic Episodic Semi Conti Single

ODS 2001 ODS 2001

12
 13/10/2024

Choice under (Un)certainty Agent types

• Four basic types in order of increasing

Fully
Observable generality:
yes – Simple reflex agents
no
Deterministic no – Reflex agents with state/model
yes – Goal-based agents
Certainty: Uncertainty – Utility-based agents
Search
– All these can be turned into learning agents
– http://www.ai.sri.com/~oreilly/aima3ejava/aima3ejavademo
s.html

ODS 2001 ODS 2001

Simple reflex agents Simple reflex agents

• Simple but very limited intelligence.

• Action does not depend on percept history, only on current percept.
• Therefore no memory requirements.
• Infinite loops
– Suppose vacuum cleaner does not observe location. What do you do
given location = clean? Left of A or right on B -> infinite loop.
– Fly buzzing around window or light.
– Possible Solution: Randomize action.
– Thermostat.
• Chess – openings, endings
– Lookup table (not a good idea in general)
• 35100 entries required for the entire game

ODS 2001 ODS 2001

13
 13/10/2024

States: Beyond Reflexes States and Memory: Game Theory

• Recall the agent function that maps from percept histories to • If each state includes the information about the
actions:
[f: P*  A] percepts and actions that led to it, the state
• An agent program can implement an agent function by space has perfect recall.
maintaining an internal state.
• The internal state can contain information about the state of the
• Perfect Information = Perfect Recall + Full
external environment. Observability + Deterministic Actions.
• The state depends on the history of percepts and on the history
of actions taken:
[f: P*, A* S A] where S is the set of states.
• If each internal state includes all information relevant to
information making, the state space is Markovian.

ODS 2001 ODS 2001

Model-based reflex agents Goal-based agents

 Know how world evolves

 Overtaking car gets closer from • knowing state and environment? Enough?
behind
 How agents actions affect the
– Taxi can go left, right, straight
world
 Wheel turned clockwise takes you • Have a goal
right
– A destination to get to
 Model base agents update their
state
• Uses knowledge about a goal to guide its actions
– E.g., Search, planning

ODS 2001 ODS 2001

14
 13/10/2024

Goal-based agents Utility-based agents

• Goals are not always enough

– Many action sequences get taxi to destination
– Consider other things. How fast, how safe…..
• A utility function maps a state onto a real
number which describes the associated degree
of “happiness”, “goodness”, “success”.
• Where does the utility measure come from?
• Reflex agent breaks when it sees brake lights. Goal based agent
reasons – Economics: money.
– Brake light -> car in front is stopping -> I should stop -> I should use brake
– Biology: number of offspring.
ODS 2001
– Your life? ODS 2001

Utility-based agents Learning agents

 Performance element is
what was previously the
whole agent
 Input sensor
 Output action
 Learning element
 Modifies performance
element.

ODS 2001 ODS 2001

15
 13/10/2024

Learning agents Learning agents(Taxi driver)

– Performance element
 Critic: how the agent is
doing • How it currently drives
 Input: checkmate? – Taxi driver Makes quick left turn across 3 lanes
 Fixed • Critics observe shocking language by passenger and other
drivers and informs bad action
 Problem generator • Learning element tries to modify performance elements
 Tries to solve the problem for future
differently instead of • Problem generator suggests experiment out something
optimizing. called Brakes on different Road conditions
 Suggests exploring new – Exploration vs. Exploitation
actions -> new problems.
• Learning experience can be costly in the short run
ODS 2001
• shocking language from ODS
other drivers
2001
• Less tip

The Big Picture: AI for Model-Based Agents The Picture for Reflex-Based Agents

Planning
Decision Theory Action Reinforcement Action
Learning
Game Theory Reinforcement
Learning
Knowledge Learning Learning
Logic Machine Learning
Statistics • Studied in AI, Cybernetics, Control Theory, Biology,
Probability
Psychology.
Heuristics
Inference

ODS 2001 ODS 2001

16
 13/10/2024

Summary

• Agents can be described by their PEAS.

• Environments can be described by several key properties: 64
Environment Types.
• A rational agent maximizes the performance measure for their
PEAS.
• The performance measure depends on the agent function.
• The agent program implements the agent function.
• 3 main architectures for agent programs.
• In this course we will look at some of the common and useful
combinations of environment/agent architecture.

ODS 2001

17