System z

Introduction to Virtualization
SHARE Boston, Session 13591

Romney White, IBM

romneyw@us.ibm.com
System z Architecture and Technology

© 2013 IBM Corporation

 Agenda

Introduction to Virtualization
– Concept
– Server Virtualization Approaches
– Hypervisor Implementation Methods
– Why Virtualization Matters

Virtualization on System z
– Logical Partitions
– Virtual Machines

2 System z Virtualization Technology © 2013 IBM Corporation

 Virtualization Concept

Virtual Resources
 Proxies for real resources: same interfaces/functions, different attributes
 May be part of a physical resource or multiple physical resources

Virtualization
 Creates virtual resources and "maps" them to real resources
 Primarily accomplished with software or firmware

Resources
 Components with architecturally-defined interfaces/functions
 May be centralized or distributed - usually physical
 Examples: memory, disk drives, networks, servers

 Separates presentation of resources to users from actual resources

 Aggregates pools of resources for allocation to users as virtual resources

3 System z Virtualization Technology © 2013 IBM Corporation

 Server Virtualization Approaches

Hardware Partitioning Bare-metal Hypervisor Hosted Hypervisor

Apps Apps Apps Apps Apps Apps

... ... ...
OS OS OS OS OS OS

Adjustable
partitions Hypervisor
Hypervisor
Partition
Controller Host OS

SMP Server SMP Server SMP Server

Server is subdivided into fractions Hypervisor provides fine-grained Hypervisor uses OS services to
each of which can run an OS timesharing of all resources do timesharing of all resources

Physical partitioning Hypervisor software/firmware Hypervisor software runs on

S/370™ SI-to-PP and PP-to-SI runs directly on server a host operating system
Sun Domains, HP nPartitions System z LPAR and z/VM® VMware GSX
Logical partitioning POWER™ Hypervisor Microsoft® Virtual Server
IBM eServer™ pSeries® LPAR VMware ESX Server HP Integrity VM
HP vPartitions Xen Hypervisor KVM

Characteristics:
• Bare-metal hypervisors offer high efficiency and availability
• Hosted hypervisors are useful for clients where host OS integration is important
• Hardware partitioning is less flexible than hypervisor-based solutions

4 System z Virtualization Technology © 2013 IBM Corporation

 Hypervisor Implementation Methods
Trap and Emulate Translate, Trap, and Emulate
Virt Mach Virt Mach • VM runs in user mode
• VM runs in user mode
L • All privileged instructions L • Some IA-32 instructions must
A cause traps A be replaced with trap ops
ST ST
PrivOp TrapOp
L Trap Hypervisor PrivOp L Trap Hypervisor PrivOp
... emulation code ... emulation code

Examples: CP-67, VM/370 Examples: VMware, Microsoft VS

Benefits: Runs unmodified OS Benefits: Runs unmodified, translated OS
Issues: Substantial overhead Issues: Substantial overhead

Hypervisor Calls (“Paravirtualization”) Direct Hardware Virtualization

Virt Mach Virt Mach • VM runs in normal modes
• VM runs in normal modes
L • OS in VM calls hypervisor L • Hardware does most of the
A to access real resources A virtualization (SIE architecture)
ST ST • Hypervisor provides control
Hcall PrivOp
L Call Hypervisor L Exit Hypervisor
service service
... ...

Examples: POWER Hypervisor, Xen Examples: System z LPAR, z/VM, Xen

Benefits: High efficiency Benefits: High efficiency, runs unmodified OS
Issues: OS must be modified to issue Hcalls Issues: Requires underlying hardware support

5 System z Virtualization Technology © 2013 IBM Corporation

 Server Virtualization Business Value

Roles: Possible Benefits:

Virtual Servers
Consolidations High resource utilization
Dynamic provisioning / hosting Great usage flexibility
W orkload management Enhanced workload QoS
W orkload isolation Virtualization High availability / security
Software release migration Low cost of availability
Mixed production and test Low management costs
Physical
Mixed OS types/releases Server Enhanced interoperability
Reconfigurable clusters Legacy compatibility
Low-cost backup servers Investment protection

Virtualization can fill many roles and provide many benefits

In the final analysis, its potential benefits take three forms:
• Help reduce hardware costs
– Help increase physical resource utilization
– Small footprints
• Can improve flexibility and responsiveness
– Virtual resources can be adjusted dynamically to meet new or changing needs
and to optimize service level achievement
– Virtualization is a key enabler of on demand operating environments such as cloud
• Can reduce management costs
– Fewer physical servers to manage
– Many common management tasks become much easier

6 System z Virtualization Technology © 2013 IBM Corporation

 System z Virtualization

HiperSockets & Virtual Networking and Switching

Multi-dimensional
virtualization
Web technology
Sphere WebSphere Traditional
SysAdmin Apache OLTP and – System z provides
Test FTP Test
Tools Batch logical (LPAR) and
software (z/VM)
Linux Linux CMS Linux WLM WLM Linux z/OS partitioning
z/OS z/OS – PR/SM enables
highly scalable
virtual server
z/VM Control Program z/VM hosting for
LPAR and z/VM
I/O & Network
virtual machine
environments
Memory Memory Memory Memory – IRD coordinates
allocation of CPU
IFL Processors Processors and I/O resources
among z/OS and
LPAR LPAR LPAR LPAR non-z/OS® LPARs*
Intelligent Resource Director (IRD)
Processor Resource / System Manager (PR/SM) * Excluding non-shared resources
like Integrated Facility for Linux
processors

7 System z Virtualization Technology © 2013 IBM Corporation

 LPAR Logical CPU Dispatching and Execution Control

Logical CPU SIE: Start Interpretive

Program
Instruction stream
“instruction” Execution
Instructions

Hardware Load
Load, Store, Add, ...
Store
Problem state
Instructions Instruction Add
Execution
Controls Hardware
or
Firmware
High-Frequency Start Subchannel, SSCH
Control
Instructions that Test Subchannel, ... TSCH
require
virtualization Instruction
SIE Interception Interpretation
to hypervisor
Low-Frequency LPAR Handling
Control
Instructions that hypervisor LPAR Virtualization
E.g., Modify SIE CPU
require hypervisor Assists
virtualization Subchannel STATE
Descriptor

Physical CPU
Instruction Execution Unit
8 System z Virtualization Technology © 2013 IBM Corporation
 LPAR Logical Processor Dispatching

z/VM Logical
z/OS Logical Partition z/VM Logical Partition Partition
General General zAAP zIIP General General General zAAP zIIP IFL IFL
Purpose Purpose Logical Purpose Purpose Purpose Logical
Logical Logical Logical Logical Logical Logical Logical Logical Logical
Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

JAVA General General General IFL IFL

z/OS & z/OS & DB2 Purpose Purpose Purpose zAAP zIIP
& Virtual Virtual Virtual
Virtual Virtual Virtual Virtual Processor Processor
DB2 DB2 XML Processor Processor
Processor Processor Processor

LPAR hypervisor dynamically dispatches:

1. General-purpose logical processors on general-purpose physical processors
2. zAAP logical processors on zAAP physical processors
3. zIIP logical processors on zIIP physical processors
4. IFL logical processors to IFL physical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared
General General General General zAAP zAAP zIIP zIIP IFL IFL
Purpose Purpose Purpose Purpose Physical Physical Physical Physical Physical Physical
Physical Physical Physical Physical
Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

Shared zAAP Shared zIIP Shared IFL

Shared General-Purpose
Physical Processor Physical Processor Physical Processor
Physical Processor Pool
Pool Pool Pool

9 System z Virtualization Technology © 2013 IBM Corporation

 LPAR Memory Partitioning

A
collection
of up to LPAR 1 LPAR 2 LPAR N
231- or A
64
2 -bytes collection
virtual of up to
address
spaces 264-bytes
z/VM Virtual Machine z/OS or Linux High- virtual
Each Performance Logical address
virtual Physical Memory Space spaces
machine Partition Physical Memory
may have Spaces z/OS and
its own Linux
virtual exploit
address multiple
spaces such
mapped virtual
to this address
common z/VM Hypervisor spaces
logical
partition Logical Partition Hypervisor
address
space

= the real partition memory pages associated with a virtual address space; that is, the sets of
dynamically-allocated physical memory pages necessary to run a z/OS task or a Linux
process

= the real partition memory pages associated with a virtual machine; that is, the sets of
dynamically-allocated physical memory pages necessary to run a guest operating system in
a virtual machine

10 System z Virtualization Technology © 2013 IBM Corporation

 LPAR Multiple Logical Channel Subsystems

P P P P P P P P P P P
A A A A A A A A A A A
R R R R R R R R R R R
T T T T T T T T T T T
I I I I I I I I I I I
T T T T T T T T T T T
I I I I I I I I I I I
O O O O O O O O O O O
N N N N N N N N N N N

CPU

Logical Channel Logical Channel Logical Channel Logical Channel

-
Subsystem Subsystem Subsystem Subsystem
Image
Image1 Image 2 Image 3 Image 4

System z Physical Channel Subsystem

FICON Channel Path OSA Ethernet Adapter
Transparently Shared by Transparently Shared By
All Logical Partitions Logical Partitions Configured
FICON Switch, to Channel Subsystem Switches,
CU - Devices, etc Image N Network Links, etc.

11 System z Virtualization Technology © 2013 IBM Corporation

 LPAR High-Performance I/O Sharing

Shared I/O Adapter

Storage Controller
Subchannel Physical
Logical Image For Channel Adapter Logical
Partition 1 Path Interface (e.g., Device
Partition 1 Image 1 a FICON
Subchannel channel path)
Image For Logical Physical
Partition 1 Device Device –
Subchannel Channel Volume
Path Logical
Logical Image for
Image 2 e.g., a
Partition 2 Device
Partition 2 Parallel
Subchannel Channel Access
Image for Path Logical
Logical Partition 3 Image 3
Volume
The I/O Device (PAV)
Partition 3
operations for
each logical Logical
Subchannel Channel partition are Device
Image for Path multiplexed
Logical Partition N Image N within the
Partition N adapter/channel
path and on the
associated I/O
interface

The I/O infrastructure (adapters/channels, their transmission links, and attached I/O resources are shared by
LPARs at native speeds (without hypervisor involvement)

I/O requests, their associated data transfers , and I/O interruptions flow between each OS instance and the
shared I/O components, just as if the I/O components were physically dedicated to a single OS instance
12 System z Virtualization Technology © 2013 IBM Corporation
 z/VM Virtual CPU Dispatching and Execution Control

Virtual CPU SIE: Start Interpretive

Program
Instruction stream
“instruction” Execution
Instructions

Hardware Load
Load, Store, Add, ...
Store
Problem state
Instructions Instruction Add
Execution
Controls Hardware
or
Firmware
High-Frequency Set Storage Key, SSKE
Control
Instructions that Signal Processor, ... SIGP
require
virtualization Instruction
SIE Interception Interpretation
to hypervisor
Control z/VM Handling
Instructions that
require hypervisor hypervisor z/VM Virtualization
E.g., Start SIE
virtualization CPU Assists
Subchannel STATE
Descriptor

Logical CPU => Physical CPU

Instruction Execution Unit
13 System z Virtualization Technology © 2013 IBM Corporation
 z/VM Virtual Processor Dispatching

z/OS Guest z/VM Guest Linux Guest

General General zAAP zIIP General General General zAAP zIIP
Purpose Purpose Virtual Purpose Purpose Purpose Virtual IFL Virtual IFL Virtual
Virtual Virtual Virtual Virtual Virtual Virtual Virtual
Processor Processor Processor Processor
Processor Processor Processor Processor Processor Processor Processor

JAVA General General General Linux Linux

z/OS & z/OS & DB2 Purpose Purpose Purpose zAAP zIIP
& Virtual Process Process
Virtual Virtual Virtual Virtual
DB2 DB2 XML Processor Processor
Processor Processor Processor

z/VM hypervisor dynamically dispatches:

1. General-purpose Virtual processors on general-purpose Logical processors
2. zAAP Virtual processors on zAAP or general-purpose Logical processors
3. zIIP Virtual processors on zIIP or general-purpose Logical processors
4. IFL Virtual processors to IFL or general-purpose Logical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared
General General General General zAAP zAAP zIIP zIIP IFL IFL
Purpose Purpose Purpose Purpose Physical Logical Logical Logical Logical Logical
Physical Logical Logical Logical
Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

General-Purpose Logical Processor zAAP Logical zIIP Logical IFL Logical

Pool Processor Pool Processor Pool Processor Pool

14 System z Virtualization Technology © 2013 IBM Corporation

 z/VM Memory Virtualization

LPAR 1
Virtual Machine 1 Virtual Virtual Machine N
Machine 2

z/OS or Linux Virtual

z/VM Virtual Machine
Machine Virtual Memory
Virtual Memory Space
Spaces

z/VM Hypervisor
z/VM Hypervisor
Logical Partition Hypervisor
= the guest real memory pages associated with a virtual address space; that is, the sets of
dynamically-allocated host virtual memory pages necessary to run a z/OS task or a Linux
process

= the guest real memory pages associated with a virtual machine; that is, the sets of
dynamically-allocated host virtual memory pages necessary to run a guest operating system
in a virtual machine

15 System z Virtualization Technology © 2013 IBM Corporation

 z/VM Disk Virtualization

Minidisk Cache
(High-speed, Virtual Disk Virtual Disk
in-memory disk in Memory in Memory
R/W cache)
T1 R/W R/W

R/W R/O R/O

N

Linux B Linux R/W Linux R/W

Shared
1 Application 2 A 3 C
Code

z/VM

TDISK: dynamic Minidisk: real disk

temporary disk partitioning technology
TDISK 1 Minidisk A
allocation pool
Fullpack N Minidisk B
TDISK space
Minidisk C Legend:
R/W = Read/Write
ESS 750, ESS 800, DS8000, DS6000 R/O = Read-Only

16 System z Virtualization Technology © 2013 IBM Corporation

 z/VM LAN and Switch Virtualization

Linux Linux Linux Linux Linux Linux VM

Guest Guest Guest Guest Guest Guest Switch
NIC NIC NIC NIC NIC NIC Controller

Port 65 Port 66 Port 67 Port 68 Port 69 Port 70

Load Balancer Aggregator / Multiplexer

LACP z/VM Virtual

Port 1 Port 2 Port 3 Port 4 Switch z/VM
System z LPAR

OSA OSA OSA OSA

Port 1 Port 2 Port 3 Port 4

LACP
(Link Aggregation Control Protocol)
Real Switch

17 System z Virtualization Technology © 2013 IBM Corporation

 z/VM Operations Automation

1. Send all Linux console

output to a single CMS
virtual machine
Virtual
server farm
Realtime
2. Use PROP and
REXX to interrogate
PROP Performance Graphs
Toolkit
console messages REXX Linux
Linux
Linux for VM
Linux
Console
CMS Console
Console
Virtual
Console
CMS Reports,
Historical Data

3. Issue hypervisor
commands on behalf
of Linux servers

Hypervisor CP
CP Operations Monitor

1. Use the CP Monitor to automatically 2. Use Performance

capture performance and resource Toolkit for VM to
Monitor
consumption data for each Linux server process Monitor data
Data

18 System z Virtualization Technology © 2013 IBM Corporation

 System z Virtualization Genetics

Over 45 years of continuous innovation in virtualization zEC12

– Refined to support modern business requirements z196
– Exploiting hardware technology for economical growth , ...
y
ibilit System z10
l e x
,F
e ss System z9 z/VM
s t n 64-Bit
obu zSeries Live Guest
Relocation
R VM/ESA
y,
ilit
Virtual Switch SSI
b 9672 ESA
li a Guest LANs Set Observer
, Re 9x21 VM/XA SMAPI
i t y Virtual Machine Resource Manager
il 3090
la b 31-Bit Virtual Disks in Storage Performance Toolkit
ca
:S 308x VM/HPO CMS Pipelines QDIO Enhanced Buffer State Mgmt
alue Accounting Facility Minidisk Cache HiperSockets xCAT
sV 303x 64 MB Real
i nes 4381 VM/SP Absolute | Relative SHARE SIE on SIE Automated Shutdown
s
Bu SMP Discontiguous Saved Segments Named Saved Systems I/O Priority Queuing
Instruction TRACE Start Interpretive Execution (SIE) Host Page-Management Assist
VM/370
Programmable Operator (PROP) LPAR Hypervisor Integrated Facility for Linux HyperSwap
S/370
Inter-User Communication Vehicle (IUCV) VM Assist Microcode Adapter Interruption Pass-Through
CP-67 Conversational Monitor System (CMS) Dedicated I/O Processors Multiple Logical Channel Subsystems (LCSS)
S/360
Diagnose Hypervisor Interface Program Event Recording (PER) Open Systems Adapter (OSA) Network Switching OpenStack
Control Program Hypervisor Translation Look-Aside Buffer (TLB) REXX Interpreter Large SMP Dynamic Virtual Machine Timeout
Dynamic Address Translation (DAT) Zone Relocation Expanded Storage Multiple Image Facility (MIF) N_Port ID Virtualization (NPIV)

1960s 1972 1980 1981 1988 1995 2004 2013

IBM System z – a comprehensive and sophisticated suite of virtualization function

19 System z Virtualization Technology © 2013 IBM Corporation