VM Recovery Manager HA Deployment

The VM Recovery Manager HA package consists of filesets for installing

the KSYS, VM agent, and GUI.
• KSYS
1. Install the KSYS software.
2. Optional: Install the GUI server.
• VM
1. Install VM agents in the virtual Machines

The VM Recovery Manager HA solution uses other subsystems such as

Hardware Management Console (HMC) and Virtual I/O Server (VIOS)
that must exist in your production environment.
VIOS
– VIOS Version 3.1.0.1, or later, must be installed on all VIOS
partitions that are part of the host group. Host monitor,
which is a key component of the VM Recovery Manager HA
solution, is installed on the VIOS by default.
HMC
– HMC Version 9 Release 9.1.0, or later
Installing the KSYS software

1. Use installp command in the AIX LPAR to install KSYS filesets that
are included in the package.
# installp -ac -g -Y -d FILESETS_DIR ksys.hautils.rte ksys.ha.license
ksys.main.cmds ksys.main.msg.en_US.cmds ksys.main.rte
ksys.ui.agent ksys.ui.common

2. Verify whether the installation of filesets is successful by running

the following command:
# lslpp -l <fileset_name>

3. Run the following command to check the KSYS subsystem has

been created:
# /opt/IBM/ksys/ksysmgr

4. You may add the /opt/IBM/ksys directory to your PATH

environment variable so that you can access the ksysmgr
command easily.
Installing VM agents

Installing a VM agent in an AIX VM

1. Run the following command in the AIX virtual machine:
# installp -ac -g -XY -d FILESET_DIR ksys.vmmon.rte
2. Perform one of the following steps to verify whether the
Installation of VM agent is successful:
a. Run the lslpp command:
# lslpp -l ksys.vmmon.rte
b. Ensure that the ksysvmmgr command and the binary file
For the VM agent daemon exist in the following directories:
# ls -l /usr/sbin/ksysvmmgr
# ls -l /usr/sbin/ksys_vmmd
c. Run the lssrc -s ksys_vmm command to verify whether the
VM agent daemon is enabled. The status of the ksys_vmm
subsystem must be Active in the output of this command.
Installing a VM agent in a Linux VM
1. Ensure that the following Reliable Scalable Cluster Technology
(RSCT) packages are installed in the
– Linux VM:
– rsct.core
– rsct.opt.storagerm
– rsct.core.utils
– rsct.basic
– DynamicRM

2. Install the VM agent RPM packages based on the following

Linux distributions in the virtual machine.
- In Red Hat Enterprise Linux (RHEL) (little endian) virtual machines, run
the following command:
# rpm -ivh vmagent-1.3.0-1.0.el7.ppc64le
- In SUSE Linux Enterprise Server (SLES) (little endian) virtual machines,
run the following command:
# rpm -ivh vmagent-1.3.0-1.0.suse123.ppc64le
Installing GUI server filesets
• To use the VM Recovery Manager HA by using the graphical
interface, you must install the GUI server filesets on a system to
manage KSYS nodes by using the GUI.
• The logical partition, in which you want to install the GUI server
filesets, must be running IBM AIX 7.2 with Technology Level 2
Service Pack 1 (7200-02-01), or later.
You can choose to install the GUI server fileset on one of the KSYS
nodes.
• You must install the following GUI server filesets before you start
the graphical interface. The GUI agent filesets are automatically
Installed along with the KSYS filesets.
• To install the GUI server filesets, complete the following steps:
1. Install the GUI server filesets based on the following scenarios:
- If you are installing the GUI server filesets on one of
the KSYS nodes, run the following command:
# installp -ac -FYd FILESETS_DIR ksys.ui.server
- If you are installing the GUI server filesets on a
separate system that manages all the KSYS nodes, run
the following command:
# installp -ac -FYd FILESETS_DIR ksys.ha.license ksys.ui.server
ksys.ui.common
 2. Install the open source software packages, which are not included
in the installed filesets, based on the following scenarios:
2.1 If the GUI server LPAR is connected to the
internet, run the following command in the GUI
server LPAR:
# /opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh
This command downloads and installs the remaining files that are not
included in the filesets because these files are licensed under the
General Public License.
2.2 If the GUI server LPAR is configured to use an
HTTP proxy to access the internet, run the
following command in the GUI server LPAR to
specify the proxy information:

# /opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh -p
You can also specify the proxy information by using the http_proxy
environment variable.
 2.3 If the GUI server LPAR is not connected to the
internet, complete the following steps:
a. Copy the vmruiinst.ksh file
(/opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh) from the
GUI server LPAR to a system that is running the AIX operating
system and that has internet access.

b. Run # vmruiinst.ksh -d /DIR command where /DIR is the location

where you want to download the remaining files. For example,
# vmruiinst.ksh -d /tmp/vmrui_rpms

c. Download the following packages that are prerequisite packages

for GUI server:
info-4.13-3.aix5.3.ppc.rpm
cpio-2.11-2.aix6.1.ppc.rpm
readline-6.2-2.aix5.3.ppc.rpm
libiconv-1.13.1-2.aix5.3.ppc.rpm
bash-4.2-5.aix5.3.ppc.rpm
gettext-0.17-6.aix5.3.ppc.rpm
libgcc-4.9.2-1.aix6.1.ppc.rpm
libgcc-4.9.2-1.aix7.1.ppc.rpm
libstdc++-4.9.2-1.aix6.1.ppc.rpm
libstdc++-4.9.2-1.aix7.1.ppc.rpm
 d. Copy the downloaded files to a directory in the GUI server LPAR.
e. In the GUI server LPAR, run # vmruiinst.ksh -i /DIR command
where /DIR is the location where you copied the downloaded
files.
After the vmruiinst.ksh command is complete, a message displays a URL
for the VM Recovery Manager HA GUI server. Enter the specified URL in
a web browser in the GUI server LPAR and start using the VM Recovery
Manager HA GUI.
Setting up the KSYS subsystem
To create and initialize a KSYS cluster, complete the following steps in
each of the KSYS LPARs:
1. Configure a cluster and add the KSYS node to the cluster by running
the following command:
# ksysmgr add ksyscluster CLUSTER_NAME ksysnodes=ksysnode1
type=HA
2. Verify the KSYS cluster configuration by running the following
command:
# ksysmgr verify ksyscluster CLUSTER_NAME
3. Deploy the KSYS cluster by running the following command:
# ksysmgr sync ksyscluster CLUSTER_NAME
You can perform steps 1 - 3 by running the following command:
# ksysmgr add ksyscluster cluster_name ksysnodes=ksysnode1 sync=yes
type=HA
4. Verify the creation of KSYS cluster by running the following
command:
# ksysmgr query ksyscluster
# lsrpdomain
# lssrc -s IBM.VMRd
The output must display the state of the KSYS cluster as online.
Adding HMCs
1. Add the HMC with user name: hscroot, password: abc1234 by
running the following command:
# ksysmgr add hmc HMC_NAME login=hscroot password=abc123
ip=HMC_IP
2. Repeat the step 1 to add multiple HMCs if needed.
3. Verify the HMCs that you have added by running the following
command:
# ksysmgr query hmc
Example:
# ksysmgr query hmc
Name: my_hmc
Ip: 10.10.10.11
Login: hscroot

Managed Host List:

Host Name Uuid
========= ====
EGY-Server1_HQ 4715697c-a486-3cbb-8b5a-2637fa4db054
EGY-Server2_HQ 4715697c-a486-3cbb-8b5a-2637fa4db055
EGY-Server3_HQ d2038c84-8484-3f08-a608-6436e0f0f928
EGY-Server4_HQ 2b67b2b2-3eef-3047-8e9e-0c6d5f043201
Adding hosts
1. Add the managed host to the KSYS by running the following
command:
# ksysmgr add host HOST_NAME [uuid=uuid] [ip=FSP_IP]
If the host is connected to more than one HMC, you must specify the
UUID of the host, you can use the command ksysmgr query hmc
command to identify the host name and the host UUID as shown in the
previous page.
The IP address here is optional and meant by the FSP IP address, which
you can extract from lsrsrc IBM.MCP command.
2. Repeat Step 1 for all hosts that you want to add to the KSYS
subsystem.
3. Verify the hosts that you added by running the following command:
# ksysmgr query host
Name: EGY-Server1_HQ
UUID: 4715697c-a486-3cbb-8b5a-2637fa4db054
FspIp: Must run discovery first to populate
Host_group: No host_group defined
VIOS: S1-VIOS1
S1-VIOS2
HMCs: MY_HMC
HA_monitor: enable
VM_failure_detection_speed: normal
Name: EGY-Server2_HQ
UUID: 4715697c-a486-3cbb-8b5a-2637fa4db055
FspIp: Must run discovery first to populate
Host_group: No host_group defined
VIOS: S2-VIOS1
S2-VIOS2
HMCs: MY_HMC
HA_monitor: enable
VM_failure_detection_speed: normal

Creating host groups

You can group a set of hosts depending on your business requirements.

Each host in the KSYS subsystem must be a part of a host group.

The KSYS subsystem creates a health monitoring Shared Storage Pool

(SSP) cluster across the virtual I/O servers that are part of the host
group. The health cluster monitors health of all virtual I/O servers
across the cluster and retains the health data that is available to the
KSYS subsystem by using a VIOS in the host group. The SSP cluster is
used only by the KSYS. You must not use this SSP cluster for any other
purpose. You can continue to use virtual Small Computer System
Interface (vSCSI) or N_Port ID Virtualization (NPIV) modes of the cluster.
However, if an SSP cluster exists in your environment, the KSYS
subsystem does not deploy any new SSP clusters and instead, uses the
existing SSP cluster for health management. However, if an existing SSP
cluster is used, the KSYS subsystem might not support VIOS
management.
The KSYS subsystem requires two disks to create the health monitoring
SSP cluster across the Virtual I/O Servers in the host group. A disk of at
least 10 GB is required to monitor the health of all hosts, called as a
repository disk, and another disk of at least 10 GB is required to track
the health data, called as a HA disk, for each host group. These disks
must be accessible to all the managed Virtual I/O Servers on each of the
hosts in the host group. You must specify the disk details when you
create the host group or before you run the first discovery operation.
You cannot modify the HA disk after the discovery operation is run
successfully. If you want to modify the HA disk, you must delete the
host group and re-create the host group with the HA disk details.

• To create host group in the KSYS subsystem, complete the

following steps in the KSYS LPAR:

1. Identify all VIOSs that are managed by KSYS:

# ksysmgr query vios

Example
# ksysmgr query vios
Name: S1-VIOS1
UUID: 29798A5A-F115-4017-973F-0CF05FE26A97
Host: EGY-Server1_HQ
Version: VIOS 3.1.0.00
State: MANAGED
HM_versions: Unknown

Name: S1-VIOS2
UUID: 24F4082-8307-4BF4-8DA2-5CC1E3F0DA7A
Host: EGY-Server1_HQ
Version: VIOS 3.1.0.00
State: MANAGED
HM_versions: Unknown

Name: S2-VIOS1
UUID: 29798A5A-F115-4017-973F-0CF05FE26A98
Host: EGY-Server2_HQ
Version: VIOS 3.1.0.00
State: MANAGED
HM_versions: Unknown

Name: S2-VIOS2
UUID: 24F4082-8307-4BF4-8DA2-5CC1E3F0DA7B
Host: EGY-Server2_HQ
Version: VIOS 3.1.0.00
State: MANAGED
HM_versions: Unknown
 2. Identify all available shared disks by host UUIDs:
# ksysmgr query viodisk hosts=EGY-Server1_HQ, EGY-Server2_HQ

Example
# ksysmgr query viodisk hosts= EGY-Server1_HQ, EGY-Server2_HQ
Looking for free shared disks in these VIOSs:
S1-VIOS1
S1-VIOS2
S2-VIOS1
S2-VIOS2
This can take a few minutes
[..]

These are the shared free disks which appear on all VIOS in the list
provided:
DiskNames are as they appear on VIOS S1-VIOS1
DiskName Size ViodiskID
----------------------------------------------------------------------------
Hdisk2 20480 gwMDAwMDAwMDAwMDExMA==
Hdisk3 20480 gwMDAwMDAwMDAwMDExMQ==
3. Create the host group and add the host(s) and disks:
# ksysmgr add host_group MY_HG hosts= EGY-Server1_HQ, EGY-
Server2_HQ \
repo_disk=gwMDAwMDAwMDAwMDExMA== \
ha_disk=gwMDAwMDAwMDAwMDExMQ==
Repeat step 1 for all host groups that you want to create in the KSYS
subsystem.
4. Verify the host group(s) that you created by running:
# ksysmgr query host_group

# ksysmgr query host_group

Name: MY_HG
Hosts: EGY-Server1_HQ
EGY-Server2_HQ
Memory_capacity: Priority Based Settings
high: 100
medium: 100
low: 100
CPU_capacity: Priority Based Settings
high: 100
medium: 100
low: 100
Skip_power_on: No
HA_monitor: enable
Restart_policy: auto
Enabling the HA monitoring
It is very recommended to enable HA monitoring for the KSYS
subsystem to start monitoring the environment.
- To check HA monitoring at system level, enter the
following command:
# ksysmgr query system
System-Wide Persistent Attributes
auto_discovery_time: 00:00 hours
notification_level: low
dup_event_processing: yes
ha_monitor: disable
host_failure_detection_time: 90 seconds
vm_failure_detection_speed normal
User Scripts for Host Group:
- To enable HA monitoring at system level, enter the
following command:
# ksysmgr modify system ha_monitor=enable
KSYS ha_monitor has been updated
# ksysmgr query system
System-Wide Persistent Attributes
auto_discovery_time: 00:00 hours
notification_level: low
dup_event_processing: yes
ha_monitor: enable
host_failure_detection_time: 90 seconds
vm_failure_detection_speed normal
User Scripts for Host Group:
 - You can also enable HA monitoring at VM-level for
each VM by running the command:
# ksysmgr modify vm vm1[,vm2,...] ha_monitor=enable

Configuring VIOS

When you add hosts to the KSYS subsystem, all the Virtual I/O Servers
in the hosts are also added to the KSYS subsystem. The VM Recovery
Manager HA solution monitors the hosts and virtual machines by using
Virtual I/O Servers in the host.

The VM Recovery Manager HA solution requires at least two Virtual I/O

Servers per host. You can have a maximum of 24 Virtual I/O Servers,
spread across different hosts, in a single host group. If a host has more
than 2 Virtual I/O Servers, you can exclude specific VIOS partitions from
the HA management.

1. To exclude specific VIOS partitions from the HA management,

complete the following steps:
# ksysmgr unmanage vios VIOS_NAME

2. You can include the VIOS partition for the HA management at any
time by using the command:
# ksysmgr manage vios VIOS_NAME

3. Verify the existing Virtual I/O Servers by running the following

command:
# ksysmgr query VIOS_NAME
Configuring virtual machines
When a host is added to the KSYS subsystem, all the virtual machines in
the host are included by default in the HA management.
1. If you do not want high availability for any of the virtual machines,
you can exclude specific virtual machines from the HA
management by running one of the following commands:
# ksysmgr unmanage vm VM_NAME|LPAR_UUID
# ksysmgr unmanage vm name=VM_NAME host=hostname |
uuid=LPAR_UUID | ALL host=HOST_NAME | ALL host_group=HG_NAME

2. You can include the VM back in the HA management at any time

by using the following command:

# ksysmgr manage vm
Discovering and verifying the KSYS configuration
After adding various resources (HMCs, hosts, and host groups) to the
KSYS subsystem, you must run the discovery operation. During the
initial discovery operation, the KSYS subsystem creates the required
high availability setup to monitor the VMs and hosts.
The KSYS subsystem creates an SSP cluster based on the information
that is specified in the configuration steps. During any subsequent
discovery operations, the KSYS subsystem scans the environment for
any changes to the environment and adapts to the modified
environment. For example, when you add a host or when you run the
Live Partition Mobility (LPM) operation from one host to another host
that is outside of the current KSYS subsystem, the KSYS configuration
settings are updated in the next discovery operation.
By default, the KSYS subsystem automatically rediscovers sites once in
every 24 hours during midnight. You can change this period by
modifying the auto_discover_time system attribute.

After the KSYS subsystem discovers the resources, a verification is

required to ensure that the virtual machines can be restarted on
another host without any errors during a failover operation.
The first discovery operation can take a few minutes because the SSP
health cluster is deployed during the first discovery operation.
To discover and verify the configuration for a specific host group,
complete the following steps:

1. Discover the resources by running the following command:

# ksysmgr discover host_group HG_NAME

2. Verify the resources by running the following command:

# ksysmgr verify host_group HG_NAME

You must run the discovery and verification commands each time you
modify the resources in the KSYS subsystem.

To perform both the discovery and verification operations, run the

following command:
# ksysmgr discover host_group HG_NAME verify=yes

Example
# ksysmgr discover host_group MY_HG verify=yes
Running discovery on Host_group MY_HG, this may take few minutes...
Existing HA trunk adapter found for VIOS S2-VIOS2
Creating HA trunk adapter for VIOS S2-VIOS1
Finished creating HA trunk adapter for VIOS S2-VIOS1
Creating HA trunk adapter for VIOS S1-VIOS2
Finished creating HA trunk adapter for VIOS S1-VIOS2
Creating HA trunk adapter for VIOS S1-VIOS1
Finished creating HA trunk adapter for VIOS S1-VIOS1
SSP creation started for Host_group MY_HG
SSP creation completed for Host_group MY_HG
Preparing VIOS in EGY-Server1_HQ for HA management
VIOS in EGY-Server1_HQ prepared for HA management
Preparing VIOS in EGY-Server2_HQ for HA management
[..]
VIOS in EGY-Server2_HQ prepared for HA management
Discovery has started for VM lpar2
Configuration information retrieval started for VM lpar2
Discovery has started for VM lpar5
Configuration information retrieval started for VM lpar5
Discovery has started for VM lpar9
Configuration information retrieval started for VM lpar9
Configuration information retrieval completed for VM lpar2
Discovery for VM lpar2 is complete
Configuration information retrieval completed for VM lpar5
Discovery for VM lpar5 is complete
Configuration information retrieval completed for VM lpar9
Discovery for VM lpar9 is complete

Ahmed Mashhour
ahdmashr@eg.ibm.com