IBM Spectrum Scale Overview and updates

Mack Kigada
Executive Advisory Consultant
mkigada@ke.ibm.com

© Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
 Agenda

IBM Spectrum Scale – Introduction

IBM Spectrum Scale – Blueprints
IBM Spectrum Scale – Details
Outlook: Spectrum Scale 4.2
IBM Elastic Storage Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 2

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
What is IBM Spectrum Scale?

 Cluster file system,

concurrent/parallel
shared disk access to a Application Nodes
single global namespace with
IBM Spectrum Scale

 Standard POSIX file

system interface Infiniband / 10 GbE / 1GbE

 Seamless capacity and

performance scaling
IBM Spectrum
 All features* are included Scale
*refer to later slide – GPFS licensing

FC / SAS / FPO / LOCAL attached drives

© Copyright IBM Corporation 2016. Technical University/Symposia materials 3

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Flexible Topologies for GPFS Cluster Configuration

Storage Area Network (SAN) Model Network Shared Disk (NSD) Server Model

Application
TCP/IP Network
Nodes
Application
Nodes
TCP/IP or Infinband Network
Storage Network
I/O
Storage Servers

Storage Storage

Shared Nothing Cluster Model

TCP/IP or Infinband RDMA Network

© Copyright IBM Corporation 2016. Technical University/Symposia materials 4

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 IBM Spectrum Scale – unique solution feature richness

File Placement
Disk, tape Optimization
Non-disruptive and flash Cloud
subsystem exchange tiers Ready Commodity Hardware

Rolling Archive
Upgrades Integration

Advanced Mirroring
and Global Namespace Global Active
Caching Services File
Storage Resource Pools Management
(AFM)
Online system
expansion

Flash Native Encryption Quota, ACLs, Unified data access

and with File and
Acceleration Snapshots
Secure Erase Object Based Storage

© Copyright IBM Corporation 2016. Technical University/Symposia materials 5

IBM
5
Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
PCS / IBM Confidential IBM.
 Proven performance up to large scale
HPC
BlueGene
Roadrunner
Elastic Storage Elastic Storage Elastic Storage
driver driver driver

Watson
…

GPFS in RAM cache

10.000+ nodes | 200.000 disks | 405GB/s | 120 PB in one 'server'

Many Top-500 supercomputers run it.

Watson demonstrated an in-memory use case.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 6

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 © Copyright IBM Corporation 2016. Technical University/Symposia materials 7
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Options for Migration to Tape

LTFS EE TSM HSM

GPFS Cluster GPFS Cluster

TSM Server

LTFS EE
SAN TSM HSM LAN

Tape

Tape
 LTFS EE is installed on • TSM HSM client is installed on
Spectrum Scale cluster GPFS cluster
 LTFS EE migrates data directly • HSM migrates data to TSM server
to LTFS tape • TSM Server stores data on tape
 Data stays visible and is auto • Data stays visible and is auto
recalled on access recalled on access

© Copyright IBM Corporation 2016. Technical University/Symposia materials 8

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM. 8
 Active File Management - Evolution of the global
namespace

GPFS

GPFS

GPFS GPFS
GPFS
GPFS

GPFS introduced
concurrent file system Multi-cluster expands the global AFM takes global namespace truly
access from multiple namespace by connecting multiple global by automatically managing
nodes. sites asynchronous replication of data

1993 2005 2011

© Copyright IBM Corporation 2016. Technical University/Symposia materials 9

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Agenda

IBM Spectrum Scale – Introduction

IBM Spectrum Scale – Blueprints
IBM Spectrum Scale – Details
Outlook: Spectrum Scale 4.2
IBM Elastic Storage Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 10

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 My Opportunity – well fitted for Spectrum Scale?

Yes Maybe / Not

• Demanding performance  Replace a single appliance
requirements
 Complex NAS requirements,
• High capacity requirements 10thds of end user
• Regular growth expected
 Very small capacities, not really
• Archival of data allows tape demanding performance
integration requirements
 “Just another box in the corner”

© Copyright IBM Corporation 2016. Technical University/Symposia materials 11

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Most common: Compute Clusters powered by Elastic Storage

© Copyright IBM Corporation 2016. Technical University/Symposia materials 12

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Spectrum Scale with global name space for
many platforms & applications

Server/Applications
using ESS-Storage

Global name space via NAS and GPFS

Network (FDR-IB/40GigE/10GigE

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
ESS Storage
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

building blocks

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

ESS #1 ESS #2 ESS #3 ESS #...

Use one storage for all data

Leverage advanced functions (encryption, ILM, backup)
© Copyright IBM Corporation 2016. Technical University/Symposia materials 13
IBM
13
Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 IBM Spectrum Scale as disk pool underneath
Spectrum Protect (TSM)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 14

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Spectrum Scale for backup with Spectrum Protect (TSM)

• Spectrum Protect servers use ESS as storage (for DB and STG)

• Spectrum Protect servers connect to ESS via Infiniband
• Can also be 10 Gbit or 40 Gbit Ethernet
• Performance optimized client workload was used
Spectrum Protect v7.1 servers
• Client values as GPFS client
• Scalable performance – up to 9 GB/s in this setup
• Low TCO in peta-scale environments
• Easy to use and to manage
• Outstanding data protection
56 Gbit Infiniband

9 GB/sec

TS3500 Tape
Library

More information GNR Server

(348x 2TB NL-SAS)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 15

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Spectrum Scale Metro Cluster for NFS

© Copyright IBM Corporation 2016. Technical University/Symposia materials 16

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Spectrum Scale Metro Cluster for NFS

• Elastic Storage servers provide global file systems across

sites Site 1 Site 2
• Clustered NFS servers export the GPFS file systems via NFS
• NFS server replicates data using GPFS replication
• Users access the NFS exports on all NFS servers NFS

• Use DNS round-robin for load balancing

Clustered
• Automated failover of NFS servers in case of failures NFS Server
• Leverage GPFS quorum for cluster stability running GPFS

Ethernet / IB
• Client value
• High availability: Metro cluster with transparent NFS failover
• Disaster protection: Data is identical in two sites GPFS synchronous
replication
• Advanced cluster protection: based on quorum
• Leading performance and resiliency
• Advanced functions (encryption, ILM, snapshots, backup) Quorum

Spectrum Scale
synchronous replication

© Copyright IBM Corporation 2016. Technical University/Symposia materials 17

IBM
17
Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Massive scale analytics: Hadoop on IBM Elastic Storage

© Copyright IBM Corporation 2016. Technical University/Symposia materials 18

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Spectrum Scale for Analytics (Hadoop)

• Analytic servers run Hadoop and are connected to

Elastic Storage as GPFS clients
Hadoop server configured
• Elastic Storage integrates with Hadoop applications as GPFS clients
using a Hadoop connector (FPO)
• Provides Hadoop a single namespace
• Does not create three copies
• Uses fast and scalable GPFS file system in the
backend
• FPO provides value adding functions (also used
by IBM BigInsight)

• Client values:
• Lower space consumption, 1 instead of 3 copies
• Leading performance and resiliency
• Advanced functions in ESS (ILM, Snapshot,
backup)
Spectrum Scale

© Copyright IBM Corporation 2016. Technical University/Symposia materials 19

IBM
19
Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Platform for application server:
High speed, high capacity shared data access

© Copyright IBM Corporation 2016. Technical University/Symposia materials 20

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Genetech Omnicast Video surveillance software

• Spectrum driver on
Windows Application Server

• Shared file system across

all Servers

• Storage hosted by ESS pods

• Each ESS system will bring

1PB capacity to the cluster
(configuration specific)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 21

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Agenda

IBM Spectrum Scale – Introduction

IBM Spectrum Scale – Blueprints
IBM Spectrum Scale – Details
Outlook: Spectrum Scale 4.2
IBM Elastic Storage Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 22

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Fiber Channel based stretched cluster (Scenario A)
• active – active across sites
GPFS • Each file ingested into GPFS
quorum node written into two Failure
Groups (sync)
• All disks visible to all NSD
Servers, FC across DC
• Not supported by ESS (no
Data ingest Data ingest
FC)
• Third site with quorum node
required (VM?)
IP
• IP network used for heartbeat,
GPFS locking
FC

DataCenter 1 DataCenter 2
(GPFS Failure Group 1) (GPFS Failure Group 2)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 23

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 IP based stretched cluster (Scenario B)
 Each file written into GPFS
GPFS written into two Failure
quorum node Groups (sync)
 Active – Active cluster, high
GPFS
throughput
NSD Client  NSD client will write twice –
= Application node via IP network, no FC across
Data ingest sides required
IP
 ESS supported
 Third site with quorum node
GPFS required (VM?)
NSD Server

DataCenter 1 DataCenter 2
(GPFS Failure Group 1) (GPFS Failure Group 2)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 24

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Independent Cluster with AFM replication (Scenario C)

 Total separation –
independent Spectrum Scale
cluster
GPFS  Asynchronous mirroring of
NSD Client data
= Application node  AFM: All changes queued,
Data ingest
no file scans, permanent
replication
AFM Mirror
 No quorum node required
GPFS
 Combination with HSM to
NSD Server
Tape (LTFS, Tape) on DC2
supported

AFM – Active File Management

DataCenter 1 DataCenter 2
(GPFS Failure Group 1) (GPFS Failure Group 2)

© Copyright IBM Corporation 2016. Technical University/Symposia materials 25

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Cluster Export Services (CES) – Protocol Support

© Copyright IBM Corporation 2016. Technical University/Symposia materials 26

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Scope of delivery

• Software Offering - protocol support is added to GPFS

• Can be configured on existing GPFS clusters or new cluster
• Support for Intel and Power Systems
• RHEL 7 (more operating systems will be added later)

• Add support for the following protocols

• SMB
• NFS
• Object (HTTP REST)

• Cluster nodes are designated as “Protocol Gateway Nodes”

• Integrated management of the protocol services
• Active-Active clustering
• High availability through IP fail-over

© Copyright IBM Corporation 2016. Technical University/Symposia materials 27

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 IBM Spectrum Scale Licensing

Socket Based Licensing

Features Express Edition Standard Edition Advanced Edition
•Simpler, no more PVUs

Basic GPFS
Express Edition
functionality •gpfs.base (no ilm, afm, cnfs)
•gpfs.docs
ILM; Storage •gpfs.gpl
pools, Policy,
•gpfs.msg
mmbackup
•gpfs.gskit
Active File
Management Standard Edition
(AFM) •Add gpfs.ext
Clustered NFS Advanced Edition
CES / NFS / SMB •Add – gpfs.crypto

Encryption

© Copyright IBM Corporation 2016. Technical University/Symposia materials 28

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 IBM Spectrum Scale Licensing (2/2)
License Types
GPFS GPFS GPFS
Server FPO Client

l l l
Access data in GPFS file system
Node Tasks

l l
NSD Server for an FPO pool

l l
Access data in an FPO pool

l
NSD Server for shared pool

l
Quorum Node, Gateway Node

l
CES or Clustered NFS Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 29

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Agenda

IBM Spectrum Scale – Introduction

IBM Spectrum Scale – Blueprints
IBM Spectrum Scale – Details
Outlook: Spectrum Scale 4.2
IBM Elastic Storage Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 30

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 New in Spectrum Scale 4.2
New Feature Benefit
Client Experience Focus Easy to learn UI and integration across Spectrum Storage
• Common interface across Spectrum Portfolio portfolio

• GUI Phase 1 Simplify common management functions, including

• Enabling protocols
• Policy driven placement and ILM
• Monitoring
• Troubleshooting

Object Storage • Unified File and Object Single view of data with wither file or object read and write

• Extended S3 API support Enable applications originally written for AWS

Big Data & Analytics Higher performance and broader integration with HDFS
• Native Hadoop Support applications to go beyond Hadoop and embrace Map/Reduce
• Ambari Integration ecosystem

General Expanding functionality in Spectrum Scale data aware policy

• Quality of Service for File engine:
• Compression for File and Object • Performance reservations to meet SLAs – even by time of
• z Linux Asynch DR and Backup day
• Efficiently reduce data size by up to 50% based upon
business or technical policy
• Extending multi-site resiliency features to z-Linux

Spectrum Scale RAID • Expanding erasure code support on Elastic Higher resiliency and faster rebuild times for Elastic Storage
Storage Server Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 31

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Client Experience

• Reduce administration overhead

• Graphical User Interface for
common tasks
• Easy to adopt
• Base interface on common IBM
Storage Framework

• Integrated into Spectrum Control

• Storage portfolio visibility
• Consolidated management
• Multiple clusters

© Copyright IBM Corporation 2016. Technical University/Symposia materials 32

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Compression

• Improved storage efficiency

• Typically 2x improvement in storage efficiency

• Improved i/o bandwidth

• Read/write compressed data reduces load on storage backend

• Improved client side caching

• Caching compressed data increases apparent cache size

• Compression is controlled per file

• by administrator defined policy rules

© Copyright IBM Corporation 2016. Technical University/Symposia materials 33

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Performance Optimization

• Quality of Service
• Set IOPs thresholds to reserve performance
• Throttle background functions such as rebuild or async replication
• Set by flexible policy, such as day-of-week and time-of-day

• Highly Available Write Cache (HAWC)

• Improves performance of mall synchronous writes
• Small synch writes are written to the log. As log fills, rewrite to home location
• Relatively small amount NVDIMM required to maximize b/w to disk

• Local Read Only Cache (LROC)

• Extend the page pool memory to include local SSD for read caching
• Writes invalidate cache and are consistent across nodes

• Enable Metadata Acceleration

• Metadata includes directories, inodes, indirect blocks

• Lift data to the highest tiers based on the file’s “heat”

© Copyright IBM Corporation 2016. Technical University/Symposia materials 34

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Agenda

Spectrum Scale – Details

Outlook: Spectrum Scale 4.2
IBM Elastic Storage Server

© Copyright IBM Corporation 2016. Technical University/Symposia materials 35

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 http://www.snia.org/sites/default/education/tutorials/2012/fall/big_data/JohnWebster_Can_You_Manage_at_Petabyte_Scale.pdf

© Copyright IBM Corporation 2016. Technical University/Symposia materials 36

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 What is ESS?

 Elastic Storage Server (ESS) is an integrated file based storage solution:

 IBM Power System Servers
 Storage Enclosure and drives
 Spectrum Scale Software
including Spectrum Scale RAID software
 Networking components

 ESS is a building block solution for Spectrum Scale

 ESS’s can be scaled horizontally to provide massive scaling of:
 Capacity
 Throughput
 Number of files
 Number of Objects

 All of the intelligence in ESS resides in Spectrum Scale

and Spectrum Scale RAID Software

232 SAS Disks @ 6TB

© Copyright IBM Corporation 2016. Technical University/Symposia materials 37

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Elastic Storage Server (ESS) GL Models LARGE Capacity

“5146-GLx” 1010 TB Usable n+3

Power S822L Servers

1818-80e Expansion Chassis
“Twin Tailed” 612 TB Usable n+3 Management Server and HMC
JBOD 10 GbE, 40 GbE, FDR Inifiniband
SAS Disk Enclosure
Red Hat 7
Graphical User Interface
325 TB Usable n+3 Elastic Storage Software
Elastic Storage Native RAID

From 116 to 348 Spinning Disk

Building Block approach to Growth

Model GL2 Model GL4 Model GL6

Analytics Focused Analytics and Cloud PetaScale Storage
2 Enclosures, 12U 4 Enclosures, 20U 6 Enclosures, 28U
116 NL-SAS, 2 SSD 232 NL-SAS, 2 SSD 348 NL-SAS, 2 SSD
5+ GB/Sec 10+ GB/Sec 12+ GB/sec 38

© Copyright IBM Corporation 2016. Technical University/Symposia materials 38

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
Elastic Storage Server (ESS) GS Models SPEED
“5146-GSx”
 Smaller Configurations for High Velocity Ingest (or Lower Cost Entry Point)
 Uses 2U, 24 Drive Storage Enclosures
 400, 800 GB SSD Drives or 1.2 TB SAS Drives
 Highest “Performance per U” Delivered to Clients
 Deployable alone or as part of an ESS Configuration as “Platinum” tier
117 TB Usable n+3

FC 5887
78 TB Usable n+3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887

FC 5887
25 TB Usable n+3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
12 TB Usable n+3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
FC 5887

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
FC 5887

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
FC 5887

FC 5887

FC 5887

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Model GS1 Model GS2 Model GS4 Model GS6

24 SSD 46 SAS + 2 SSD or 94 SAS + 2 SSD or 142 SAS + 2 SSD
6 GB/Sec 48 SSD Drives 96 SSD Drives
2 GB/Sec SAS 5 GB/Sec SAS 7 GB/Sec
12 GB/Sec SSD 16 GB/Sec SSD
© Copyright IBM Corporation 2016. Technical University/Symposia materials 39
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 ESS advantages compared to custom build Spectrum Scale

• Integrity: end to end checksumming

• Declustered Raid
• Performance: Single IO up to 2MB!
• No controller saturation

• Flexibility: vdisk concept allows small fs striped across all disks

• Monitoring: Direct disk access allows monitoring of smart data

• 256 MB persistent NVRAM cache for caching of small IO‘s

© Copyright IBM Corporation 2016. Technical University/Symposia materials 40

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 © Copyright IBM Corporation 2016. Technical University/Symposia materials 41
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Session Evaluations

YOUR OPINION MATTERS!

1 2 3 4
Submit four or more session
evaluations by 5:30pm Wednesday
to be eligible for drawings!
*Winners will be notified Thursday morning. Prizes must be picked up at
registration desk, during operating hours, by the conclusion of the event.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 42

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Continue growing your IBM skills

ibm.com/training
provides a comprehensive
portfolio of skills and career
accelerators that are designed
to meet all your training needs.

If you can’t find the training that is right for you with our
Global Training Providers, we can help.

Contact IBM Training at dpmc@us.ibm.com

Global Skills Initiative

© Copyright IBM Corporation 2016. Technical University/Symposia materials 43

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.