Steps to add a node in Oracle RAC 19c

infraxpertzz.com/steps-to-add-a-node-in-oracle-rac-19c

infra xpertzz August 11, 2021

In this post, we will add a node to our existing Oracle Clusterware.

Please click on the INDEX and browse for more interesting posts.

Existing node : Jack

New Node : Jill

NOTE : Due to resource crunch, we could add a node to a single node RAC system. The steps
are however same for any nodes except the ssh setup, which will be discussed as we go through
the document.

Steps for adding node in Oracle RAC (19c) environment :

For adding node to existing RAC environment, initially we need a setup Oracle RAC environment
to add nodes.

Existing /etc/hosts file for One-Node RAC Setup :

[oracle@jack bin]$ cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public
172.168.190.1 jack.infraxpertzz.com jack
# Private
192.168.190.1 jack-priv.infraxpertzz.com jack-priv
# Virtual
172.168.190.3 jack-vip.infraxpertzz.com jack-vip
# SCAN
172.168.190.10 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.20 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.30 crsprod-scan.infraxpertzz.com crsprod-scan

Configure new host entry in all nodes of /etc/hosts file :

1/35
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public
172.168.190.1 jack.infraxpertzz.com jack
172.168.190.2 jill.infraxpertzz.com jill
# Private
192.168.190.1 jack-priv.infraxpertzz.com jack-priv
192.168.190.2 jill-priv.infraxpertzz.com jill-priv
# Virtual
172.168.190.3 jack-vip.infraxpertzz.com jack-vip
172.168.190.4 jill-vip.infraxpertzz.com jill-vip
# SCAN
172.168.190.10 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.20 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.30 crsprod-scan.infraxpertzz.com crsprod-scan

Creating groups and user on new node with same group and user id of existing nodes :

Add the below groups and user in the new node.

Make the groups similar to the existing setup

Groups : oinstall (primary group)

dba, asmoper,asmadmin,asmdba,oper,dba (secondary group)

#groupadd -g 1005 oinstall

#groupadd -g 1006 dba
#groupadd –g 1001 asmoper
#groupadd –g 1002 asmadmin
#groupadd –g 1003 asmdba
#groupadd –g 1004 oper
#useradd -u 54323 -g oinstall -G dba,asmoper,asmadmin,asmdba,oper,dba oracle

ASM library Installation and Configuration :

Install the below RPM packages in the new node

rpm –Uvh oracleasmlib-2.0.12-1.el7.x86_64

rpm –Uvh oracleasm-support-2.1.8-3.1.el7.x86_64

For other RPM packages, please check the LINK

Configuration and Check ASM disks :

Configure,Start and check ASM disks on the new node

2/35
[root@jill Packages]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

[root@jill Packages]# oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

[root@jill Packages]# oracleasm scandisks

Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DATA1"
Instantiating disk "DATA2"
Instantiating disk "DATA4"
Instantiating disk "FRA8"
Instantiating disk "FRA7"
Instantiating disk "DATA5"
Instantiating disk "DATA3"
Instantiating disk "FRA6"

[root@jill Packages]# oracleasm listdisks

DATA1
DATA2
DATA3
DATA4
DATA5
FRA6
FRA7
FRA8

Create Directories

Create the necessary directories in the new node

mkdir -p /u01/app/19c/grid
mkdir -p /u01/app/oracle/product/19c/db_1
mkdir –p /u01/app/19c/gridbase
mkdir –p /u01/app/oracle/orabase
chown -R oracle:oinstall /u01
chmod -R 775 /u01/

3/35
Configure SSH for oracle user on all nodes :

Copy sshUserSetup.sh script to new node(jill) and execute it.

[oracle@jack gridbase]# cd /u01/app/19c/grid/deinstall

[oracle@jack gridbase]# scp sshUserSetup.sh oracle@jill:/home/oracle

oracle@jill's password:
sshUserSetup.sh 100% 32KB 31.6KB/s 00:00

Run sshUserSetup.sh

4/35
[oracle@jill ~]$ ./sshUserSetup.sh -hosts "jill" -user oracle -noPromptPassphrase

The output of this script is also logged into /tmp/sshUserSetup_2021-07-26-12-25-53.log

Hosts are jill
user is oracle
Platform:- Linux
Checking if the remote hosts are reachable
PING jill.infraxpertzz.com (172.168.190.2) 56(84) bytes of data.
64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=1 ttl=64 time=0.038 ms
64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=2 ttl=64 time=0.042 ms
64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=3 ttl=64 time=0.067 ms
64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=4 ttl=64 time=0.051 ms
64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=5 ttl=64 time=0.044 ms

--- jill.infraxpertzz.com ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 4108ms
rtt min/avg/max/mdev = 0.038/0.048/0.067/0.011 ms
Remote host reachability check succeeded.
The following hosts are reachable: jill.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost jill
numhosts 1
The script will setup SSH connectivity from the host jill.infraxpertzz.com to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host jill.infraxpertzz.com
and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.

NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEGES TO THESE
directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?
yes

The user chose yes

Please specify if you want to specify a passphrase for the private key this script will
create for the local host. Passphrase is used to encrypt the private key and makes SSH
much more secure. Type 'yes' or 'no' and then press enter. In case you press 'yes', you
would need to enter the passphrase whenever the script executes ssh or scp. no
The estimated number of times the user would be prompted for a passphrase is 2. In
addition, if the private-public files are also newly created, the user would have to
specify the passphrase on one additional occasion.
Enter 'yes' or 'no'.
yes

The user chose yes

Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host

5/35
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /home/oracle/.ssh/config, it would be backed up to
/home/oracle/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Generating public/private rsa key pair.
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
be:ba:8e:44:93:75:1b:0f:c7:b7:ce:4e:57:52:fa:87 oracle@jill.infraxpertzz.com
The key's randomart image is:
+--[ RSA 1024]----+
| |
| . |
| . + o . .|
| o . * . . o |
| + S . . o .|
| . . . o = |
| . . + E o|
| . . . o . .|
| ..+o. . |
+-----------------+
Creating .ssh directory and setting permissions on remote host jill
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME
DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host jill. If a config file
exists already at ~oracle/.ssh/config, it would be backed up to
~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host
jill.
Warning: Permanently added 'jill,172.168.190.2' (ECDSA) to the list of known hosts.
oracle@jill's password:
Done with creating .ssh directory and setting permissions on remote host jill.
Copying local host public key to the remote host jill
The user may be prompted for a password or passphrase here since the script would be using
SCP for host jill.
oracle@jill's password:
Done copying local host public key to the remote host jill
The script will run SSH on the remote machine jill. The user may be prompted for a
passphrase here in case the private key has been encrypted with a passphrase.
cat: /home/oracle/.ssh/known_hosts.tmp: No such file or directory
cat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directory
SSH setup is complete.

------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.

6/35
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user oracle.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the /sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--jill:--
Running /usr/bin/ssh -x -l oracle jill date to verify SSH connectivity has been setup from
local host to jill.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED
FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being
prompted for a passphrase may be OK but being prompted for a password is ERROR.
The script will run SSH on the remote machine jill. The user may be prompted for a
passphrase here in case the private key has been encrypted with a passphrase.
Mon Jul 26 12:26:42 IST 2021
------------------------------------------------------------------------
SSH verification complete.

Copy authorized keys to all nodes running under cluster environment

[oracle@jill .ssh]$ pwd

/home/oracle/.ssh
[oracle@jill .ssh]$ scp authorized_keys oracle@jack:/home/oracle/
oracle@jack's password:
authorized_keys 100% 484 0.5KB/s 00:00

[oracle@jack ~]$ cat authorized_keys >> .ssh/authorized_keys

[oracle@jack ~]$ cat .ssh/authorized_keys
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAAAgQCtedmNSm8amWPTQO7OM6UAdI+/fIQ8Uu4muNqb/vzg3b696AGiNVbHYJ2KcXf
oracle@jack.infraxpertzz.com
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAAAgQCtedmNSm8amWPTQO7OM6UAdI+/fIQ8Uu4muNqb/vzg3b696AGiNVbHYJ2KcXf
oracle@jack.infraxpertzz.com
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAAAgQCfgHCdG3yak6ozk9m4eXPK5G5CtR59l7f+a/vYiPP8w+Y/H+igL5H4bPQQEhB
oracle@jill.infraxpertzz.com
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAAAgQCfgHCdG3yak6ozk9m4eXPK5G5CtR59l7f+a/vYiPP8w+Y/H+igL5H4bPQQEhB
oracle@jill.infraxpertzz.com
[oracle@jack .ssh]$ pwd
/home/oracle/.ssh
[oracle@jack .ssh]$ scp authorized_keys jill:/home/oracle/.ssh/
oracle@jill's password:
authorized_keys 100% 484 0.5KB/s 00:00

Copy these authorized keys from Jack to any other existing Nodes. This is the only place where
we have extra steps. Rest everywhere you need to follow the same steps as in the document

7/35
Check Time Synchronization :

Run the below commands from all nodes.

Example for 1st node :

[oracle@jack bin]$ ssh jack date

Tue Jul 20 21:25:09 IST 2021
[oracle@jack bin]$ ssh jill date
Tue Jul 20 21:25:13 IST 2021

Run the same from your new node and check

Verify Cluster utility :

8/35
[oracle@jack bin]$ pwd
/u01/app/19c/grid/bin
[oracle@jack bin]$ ./cluvfy comp peer -n jill -refnode jack
Verifying Peer Compatibility ...
Verifying Physical memory ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 5.6877GB (5964032.0KB) 5.6877GB (5964032.0KB) matched

Verifying Available memory ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 4.711GB (4939860.0KB) 2.571GB (2695904.0KB) matched

Verifying Swap space ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 16GB (1.6777212E7KB) 16GB (1.6777212E7KB) matched

Verifying Free Space ...PASSED

Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /usr / 4.7529GB (4983808.0KB) 4.6016GB
(4825088.0KB) matched

Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /var / 4.7529GB (4983808.0KB) 4.6016GB
(4825088.0KB) matched

Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /etc / 4.7529GB (4983808.0KB) 4.6016GB
(4825088.0KB) matched

Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /sbin / 4.7529GB (4983808.0KB) 4.6016GB
(4825088.0KB) matched

Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /tmp /tmp 5.1816GB (5433344.0KB) 5.1777GB
(5429248.0KB) matched

Verifying User existence for "oracle" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------

9/35
jill oracle(1000) oracle(1000) matched

Verifying Group existence for "dba" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill dba(1006) dba(1006) matched

Verifying Group existence for "oinstall" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill oinstall(1005) oinstall(1005) matched

Verifying Group membership for "oracle" in "oinstall (Primary)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill yes yes matched

Verifying Group membership for "oracle" in "dba" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill yes yes matched

Verifying Run level ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 5 5 matched

Verifying System architecture ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill x86_64 x86_64 matched

Verifying Kernel version ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 4.1.12-61.1.18.el7uek.x86_64 4.1.12-61.1.18.el7uek.x86_64 matched

Verifying Kernel param "semmsl" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 250 250 matched

10/35
Verifying Kernel param "semmns" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 32000 32000 matched

Verifying Kernel param "semopm" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 100 100 matched

Verifying Kernel param "semmni" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 128 128 matched

Verifying Kernel param "shmmax" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 4398046511104 4398046511104 matched

Verifying Kernel param "shmmni" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 4096 4096 matched

Verifying Kernel param "shmall" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 0 0 matched

Verifying Kernel param "file-max" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 6815744 6815744 matched

Verifying Kernel param "ip_local_port_range" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 9000 65500 9000 65500 matched

11/35
Verifying Kernel param "rmem_default" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 262144 262144 matched

Verifying Kernel param "rmem_max" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 4194304 4194304 matched

Verifying Kernel param "wmem_default" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 262144 262144 matched

Verifying Kernel param "wmem_max" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 1048576 1048576 matched

Verifying Kernel param "aio-max-nr" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 1048576 1048576 matched

Verifying Kernel param "panic_on_oops" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill 1 1 matched

Verifying Package existence for "kmod (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill kmod-20-21.el7 (x86_64) kmod-20-21.el7 (x86_64) matched

Verifying Package existence for "kmod-libs (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill kmod-libs-20-21.el7 (x86_64) kmod-libs-20-21.el7 (x86_64) matched

12/35
 Verifying Package existence for "binutils" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill binutils-2.25.1-22.base.el7 binutils-2.25.1-22.base.el7 matched

Verifying Package existence for "compat-libcap1" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill compat-libcap1-1.10-7.el7 compat-libcap1-1.10-7.el7 matched

Verifying Package existence for "libgcc (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill libgcc-4.8.5-11.el7 (x86_64) libgcc-4.8.5-11.el7 (x86_64) matched

Verifying Package existence for "libstdc++ (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill libstdc++-4.8.5-11.el7 (x86_64) libstdc++-4.8.5-11.el7 (x86_64) matched

Verifying Package existence for "libstdc++-devel (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill libstdc++-devel-4.8.5-11.el7 (x86_64) libstdc++-devel-4.8.5-11.el7
(x86_64) matched

Verifying Package existence for "sysstat" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill sysstat-10.1.5-11.el7 sysstat-10.1.5-11.el7 matched

Verifying Package existence for "ksh" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill ksh-20120801-26.el7 ksh-20120801-26.el7 matched

Verifying Package existence for "make" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill make-3.82-23.el7 make-3.82-23.el7 matched

13/35
 Verifying Package existence for "glibc (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill glibc-2.17-157.el7 (x86_64) glibc-2.17-157.el7 (x86_64) matched

Verifying Package existence for "glibc-devel (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill glibc-devel-2.17-157.el7 (x86_64) glibc-devel-2.17-157.el7 (x86_64)
matched

Verifying Package existence for "libaio (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill libaio-0.3.109-13.el7 (x86_64) libaio-0.3.109-13.el7 (x86_64) matched

Verifying Package existence for "libaio-devel (x86_64)" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill libaio-devel-0.3.109-13.el7 (x86_64) libaio-devel-0.3.109-13.el7 (x86_64)
matched

Verifying Package existence for "nfs-utils" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.3.0-0.33.0.1.el7 matched

Verifying Package existence for "smartmontools" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill smartmontools-6.2-7.el7 smartmontools-6.2-7.el7 matched

Verifying Package existence for "net-tools" ...PASSED

Node Name Status Ref. node status Comment

------------ ------------------------ ------------------------ ----------
jill net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17.20131004git.el7
matched

Verifying Peer Compatibility ...PASSED

Verification of peer compatibility was successful.

CVU operation performed: peer compatibility

14/35
Date: Jul 20, 2021 9:29:58 PM
CVU home: /u01/app/19c/grid/
User: oracle

Verify new node pre-check :

[oracle@jack bin]$ $GRID_HOME/bin/cluvfy stage -pre nodeadd -n jill -fixup -verbose >
/home/oracle/cluvfy_pre_nodeadd.txt

Above node addition pre-check has to get passed to add nodes to the existing One-Node RAC
environment

Below is the output :

15/35
[oracle@jack ~]$ cat cluvfy_pre_nodeadd.txt

Verifying Physical Memory ...

Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 5.6877GB (5964032.0KB) 8GB (8388608.0KB) failed
jill 5.6877GB (5964032.0KB) 8GB (8388608.0KB) failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 2.5013GB (2622796.0KB) 50MB (51200.0KB) passed
jill 4.7025GB (4930904.0KB) 50MB (51200.0KB) passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 16GB (1.6777212E7KB) 5.6877GB (5964032.0KB) passed
jill 16GB (1.6777212E7KB) 5.6877GB (5964032.0KB) passed
Verifying Swap Size ...PASSED
Verifying Free Space: jack:/usr,jack:/var,jack:/etc,jack:/sbin ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr jack / 4.6016GB 25MB passed
/var jack / 4.6016GB 5MB passed
/etc jack / 4.6016GB 25MB passed
/sbin jack / 4.6016GB 10MB passed
Verifying Free Space: jack:/usr,jack:/var,jack:/etc,jack:/sbin ...PASSED
Verifying Free Space: jack:/u01/app/19c/grid ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/19c/grid jack /u01 31.3945GB 6.9GB passed
Verifying Free Space: jack:/u01/app/19c/grid ...PASSED
Verifying Free Space: jack:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp jack /tmp 5.1777GB 1GB passed
Verifying Free Space: jack:/tmp ...PASSED
Verifying Free Space: jill:/usr,jill:/var,jill:/etc,jill:/sbin ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr jill / 4.7471GB 25MB passed
/var jill / 4.7471GB 5MB passed
/etc jill / 4.7471GB 25MB passed
/sbin jill / 4.7471GB 10MB passed
Verifying Free Space: jill:/usr,jill:/var,jill:/etc,jill:/sbin ...PASSED
Verifying Free Space: jill:/u01/app/19c/grid ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/19c/grid jill /u01 46.6494GB 6.9GB passed
Verifying Free Space: jill:/u01/app/19c/grid ...PASSED
Verifying Free Space: jill:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp jill /tmp 5.1816GB 1GB passed
Verifying Free Space: jill:/tmp ...PASSED
Verifying User Existence: oracle ...

16/35
 Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists(1000)
jill passed exists(1000)

Verifying Users With Same UID: 1000 ...PASSED

Verifying User Existence: oracle ...PASSED
Verifying User Existence: root ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists(0)
jill passed exists(0)

Verifying Users With Same UID: 0 ...PASSED

Verifying User Existence: root ...PASSED
Verifying Group Existence: oinstall ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists
jill passed exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
jack yes yes yes passed
jill yes yes yes passed
Verifying Group Membership: oinstall ...PASSED
Verifying Run Level ...
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
jack 5 3,5 passed
jill 5 3,5 passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack hard 65536 65536 passed
jill hard 65536 65536 passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 1024 1024 passed
jill soft 1024 1024 passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack hard 16384 16384 passed
jill hard 16384 16384 passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 16384 2047 passed
jill soft 16384 2047 passed
Verifying Soft Limit: maximum user processes ...PASSED

17/35
Verifying Soft Limit: maximum stack size ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 10240 10240 passed
jill soft 10240 10240 passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack x86_64 x86_64 passed
jill x86_64 x86_64 passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 4.1.12-61.1.18.el7uek.x86_64 4.1.12 passed
jill 4.1.12-61.1.18.el7uek.x86_64 4.1.12 passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 250 250 250 passed
jill 250 250 250 passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 32000 32000 32000 passed
jill 32000 32000 32000 passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 100 100 100 passed
jill 100 100 100 passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 128 128 128 passed
jill 128 128 128 passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4398046511104 4398046511104 3053584384 passed
jill 4398046511104 4398046511104 3053584384 passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4096 4096 4096 passed
jill 4096 4096 4096 passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------

18/35
 jack 1073741824 1073741824 1073741824 passed
jill 1073741824 1073741824 1073741824 passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 6815744 6815744 6815744 passed
jill 6815744 6815744 6815744 passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack between 9000 & 65500 between 9000 & 65500 between 9000 & 65535
passed
jill between 9000 & 65500 between 9000 & 65500 between 9000 & 65535
passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 262144 262144 262144 passed
jill 262144 262144 262144 passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4194304 4194304 4194304 passed
jill 4194304 4194304 4194304 passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 262144 262144 262144 passed
jill 262144 262144 262144 passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1048576 1048576 1048576 passed
jill 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1048576 1048576 1048576 passed
jill 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1 1 1 passed
jill 1 1 1 passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack kmod(x86_64)-20-21.el7 kmod(x86_64)-20-21 passed

19/35
 jill kmod(x86_64)-20-21.el7 kmod(x86_64)-20-21 passed
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack kmod-libs(x86_64)-20-21.el7 kmod-libs(x86_64)-20-21 passed
jill kmod-libs(x86_64)-20-21.el7 kmod-libs(x86_64)-20-21 passed
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack binutils-2.25.1-22.base.el7 binutils-2.23.52.0.1 passed
jill binutils-2.25.1-22.base.el7 binutils-2.23.52.0.1 passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
jill compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libgcc(x86_64)-4.8.5-11.el7 libgcc(x86_64)-4.8.2 passed
jill libgcc(x86_64)-4.8.5-11.el7 libgcc(x86_64)-4.8.2 passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libstdc++(x86_64)-4.8.5-11.el7 libstdc++(x86_64)-4.8.2 passed
jill libstdc++(x86_64)-4.8.5-11.el7 libstdc++(x86_64)-4.8.2 passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libstdc++-devel(x86_64)-4.8.5-11.el7 libstdc++-devel(x86_64)-4.8.2
passed
jill libstdc++-devel(x86_64)-4.8.5-11.el7 libstdc++-devel(x86_64)-4.8.2
passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack sysstat-10.1.5-11.el7 sysstat-10.1.5 passed
jill sysstat-10.1.5-11.el7 sysstat-10.1.5 passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack ksh ksh passed
jill ksh ksh passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack make-3.82-23.el7 make-3.82 passed
jill make-3.82-23.el7 make-3.82 passed

20/35
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack glibc(x86_64)-2.17-157.el7 glibc(x86_64)-2.17 passed
jill glibc(x86_64)-2.17-157.el7 glibc(x86_64)-2.17 passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack glibc-devel(x86_64)-2.17-157.el7 glibc-devel(x86_64)-2.17 passed
jill glibc-devel(x86_64)-2.17-157.el7 glibc-devel(x86_64)-2.17 passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
jill libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
jill libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.2.3-15 passed
jill nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.2.3-15 passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack smartmontools-6.2-7.el7 smartmontools-6.2-4 passed
jill smartmontools-6.2-7.el7 smartmontools-6.2-4 passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
jill net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
jill cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED

21/35
Verifying Node Addition ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying '/u01/app/19c/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
Verifying Hosts File ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
Verifying Hosts File ...PASSED

Interface information for node "jack"

Name IP Address Subnet Gateway Def. Gateway HW Address

MTU
------ --------------- --------------- --------------- --------------- -----------------
------
enp0s3 172.168.190.1 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71
1500
enp0s3 172.168.190.20 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71
1500
enp0s3 172.168.190.10 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71
1500
enp0s3 172.168.190.30 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71
1500
enp0s3 172.168.190.3 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71
1500
enp0s8 192.168.190.1 192.168.0.0 0.0.0.0 172.168.190.100 08:00:27:05:B7:1B
1500

Interface information for node "jill"

Name IP Address Subnet Gateway Def. Gateway HW Address

MTU
------ --------------- --------------- --------------- --------------- -----------------
------
enp0s3 172.168.190.2 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:4B:49:CE
1500
enp0s8 192.168.190.2 192.168.0.0 0.0.0.0 172.168.190.100 08:00:27:DE:B5:AE
1500

Check: MTU consistency on the private interfaces of subnet "192.168.0.0"

Node Name IP Address Subnet MTU

---------------- ------------ ------------ ------------ ----------------
jack enp0s8 192.168.190.1 192.168.0.0 1500
jill enp0s8 192.168.190.2 192.168.0.0 1500

Check: MTU consistency of the subnet "172.168.190.0".

Node Name IP Address Subnet MTU

---------------- ------------ ------------ ------------ ----------------
jack enp0s3 172.168.190.1 172.168.190.0 1500
jack enp0s3 172.168.190.20 172.168.190.0 1500

22/35
 jack enp0s3 172.168.190.10 172.168.190.0 1500
jack enp0s3 172.168.190.30 172.168.190.0 1500
jack enp0s3 172.168.190.3 172.168.190.0 1500
jill enp0s3 172.168.190.2 172.168.190.0 1500

Source Destination Connected?

------------------------------ ------------------------------ ----------------
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.20] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.10] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.1] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.10] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.20] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.10] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.10] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.10] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.30] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.30] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.3] jill[enp0s3:172.168.190.2] yes

Source Destination Connected?

------------------------------ ------------------------------ ----------------
jack[enp0s8:192.168.190.1] jill[enp0s8:192.168.190.2] yes
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying subnet mask consistency for subnet "172.168.190.0" ...PASSED
Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.0.0" for multicast communication with multicast group
"224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...Disks
"/dev/oracleasm/disks/DATA1,/dev/oracleasm/disks/DATA2,/dev/oracleasm/disks/DATA3,/dev/orac
are managed by ASM.
Verifying Device Checks for ASM ...PASSED
Verifying Database home availability ...PASSED
Verifying ASMLib installation and configuration verification. ...
Verifying '/etc/init.d/oracleasm' ...PASSED
Verifying '/dev/oracleasm' ...PASSED
Verifying '/etc/sysconfig/oracleasm' ...PASSED

Node Name Status

------------------------------------ ------------------------
jack passed
jill passed
Verifying ASMLib installation and configuration verification. ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Network Time Protocol (NTP) ...
Verifying '/etc/ntp.conf' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes

23/35
 jill yes

Verifying '/etc/ntp.conf' ...PASSED

Verifying '/etc/chrony.conf' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes
jill yes

Verifying '/etc/chrony.conf' ...PASSED

Verifying '/var/run/chronyd.pid' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes
jill yes

Verifying '/var/run/chronyd.pid' ...PASSED

Verifying Daemon 'chronyd' ...
Node Name Running?
------------------------------------ ------------------------
jack yes
jill yes

Verifying Daemon 'chronyd' ...PASSED

Verifying NTP daemon or service using UDP port 123 ...
Node Name Port Open?
------------------------------------ ------------------------
jack yes
jill yes

Verifying NTP daemon or service using UDP port 123 ...PASSED

Verifying chrony daemon is synchronized with at least one external time source ...FAILED
(PRVG-13606)
Verifying Network Time Protocol (NTP) ...FAILED (PRVG-13606)
Verifying User Not In Group "root": oracle ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed does not exist
jill passed does not exist
Verifying User Not In Group "root": oracle ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed

checking response for name "jack" from each of the name servers specified in
"/etc/resolv.conf"

Node Name Source Comment Status

------------ ------------------------ ------------------------ ----------
jack 172.168.190.1 IPv4 passed
jack 172.168.190.2 IPv4 passed

checking response for name "jill" from each of the name servers specified in
"/etc/resolv.conf"

24/35
 Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
jill 172.168.190.2 IPv4 passed
jill 172.168.190.1 IPv4 passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying User Equivalence ...
Node Name Status
------------------------------------ ------------------------
jill passed
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying /boot mount ...PASSED
Verifying zeroconf check ...PASSED

Pre-check for node addition was unsuccessful on all the nodes.

Failures were encountered during execution of CVU verification request "stage -pre
nodeadd".

Verifying Physical Memory ...FAILED

jack: PRVF-7530 : Sufficient physical memory is not available on node "jack"
[Required physical memory = 8GB (8388608.0KB)]

jill: PRVF-7530 : Sufficient physical memory is not available on node "jill"

[Required physical memory = 8GB (8388608.0KB)]

Verifying Network Time Protocol (NTP) ...FAILED

Verifying chrony daemon is synchronized with at least one external time
source ...FAILED
jack: PRVG-13606 : chrony daemon is not synchronized with any external time
source on node "jack".

jill: PRVG-13606 : chrony daemon is not synchronized with any external time
source on node "jill".

CVU operation performed: stage -pre nodeadd

Date: Jul 20, 2021 9:40:56 PM
CVU home: /u01/app/19c/grid/
User: oracle

Above node addition pre-check has to get passed to add nodes to the existing One-Node RAC
environment.

NOTE: The failures over here are due to home setup and less resources. In your work
environment, you should’nt face a single error

Add node

From Jack node,

25/35
For GRID_HOME :

26/35
[oracle@jack addnode]$ cd /u01/app/19c/grid
[oracle@jack grid]$ cd addnode/
[oracle@jack addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={jill}"
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={jill-vip}" -ignorePrereqFailure
[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details.
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log
ACTION: Identify the list of failed prerequisite checks from the log:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log. Then either from
the log file or from installation manual find the appropriate configuration to meet the
prerequisites and fix it manually.

Copy Files to Remote Nodes in progress.

.................................................. 6% Done.
.................................................. 11% Done.
....................
Copy Files to Remote Nodes successful.

Prepare Configuration in progress.

Prepare Configuration successful.

.................................................. 21% Done.
You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log

Instantiate files in progress.

Instantiate files successful.

.................................................. 49% Done.

Saving cluster inventory in progress.

.................................................. 83% Done.

Saving cluster inventory successful.

The Cluster Node Addition of /u01/app/19c/grid was successful.
Please check '/u01/app/19c/oraInventory/logs/silentInstall2021-07-27_12-15-11PM.log' for
more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.

.................................................. 90% Done.

Update Inventory in progress.

You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log

Update Inventory successful.

.................................................. 97% Done.

As a root user, execute the following script(s):

1. /u01/app/19c/oraInventory/orainstRoot.sh
2. /u01/app/19c/grid/root.sh

Execute /u01/app/19c/oraInventory/orainstRoot.sh on the following nodes:

[jill]
Execute /u01/app/19c/grid/root.sh on the following nodes:

27/35
[jill]

The scripts can be executed in parallel on all the nodes.

Successfully Setup Software with warning(s).

.................................................. 100% Done.

As root user, execute orainstRoot.sh and root.sh on jill :

28/35
[root@jill ~]# /u01/app/19c/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/19c/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/19c/oraInventory to oinstall.

The execution of the script is complete.

[root@jill ~]# /u01/app/19c/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/19c/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/19c/gridbase/crsdata/jill/crsconfig/rootcrs_jill_2021-07-27_01-03-30PM.log
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/07/27 13:03:39 CLSRSC-363: User ignored prerequisites during installation
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/07/27 13:03:40 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/07/27 13:03:46 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2021/07/27 13:03:47 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/07/27 13:03:50 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/07/27 13:03:55 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/07/27 13:03:55 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/07/27 13:04:16 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/07/27 13:04:17 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/07/27 13:04:22 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/07/27 13:04:23 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/07/27 13:04:26 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA)
Collector.
2021/07/27 13:05:07 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/07/27 13:05:10 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/07/27 13:05:54 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/07/27 13:05:57 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/07/27 13:06:15 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/07/27 13:08:32 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/07/27 13:08:33 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
clscfg: EXISTING configuration version 19 detected.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2021/07/27 13:09:17 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

29/35
2021/07/27 13:09:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ...
succeeded

Check Clusterware status :-

30/35
[oracle@jill software]$ cd /u01/app/19c/grid/bin/
[oracle@jill bin]$ ./crsctl check cluster -all
**************************************************************
jack:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
jill:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@jill bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.GHCHKPT.advm
OFFLINE OFFLINE jack STABLE
OFFLINE OFFLINE jill STABLE
ora.LISTENER.lsnr
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.chad
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.data.ghchkpt.acfs
OFFLINE OFFLINE jack STABLE
OFFLINE OFFLINE jill STABLE
ora.helper
OFFLINE OFFLINE jack IDLE,STABLE
OFFLINE OFFLINE jill IDLE,STABLE
ora.net1.network
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.ons
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.proxy_advm
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE

31/35
 3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jill STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE jack STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE jack STABLE
ora.MGMTLSNR
1 ONLINE ONLINE jack 169.254.24.18 192.16
8.190.1,STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE jack Started,STABLE
2 ONLINE ONLINE jill Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 ONLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE jack STABLE
ora.infra.db
1 ONLINE ONLINE jack Open,HOME=/u01/app/o
racle/product/19c/db
_1,STABLE
ora.jack.vip
1 ONLINE ONLINE jack STABLE
ora.jill.vip
1 ONLINE ONLINE jill STABLE
ora.mgmtdb
1 ONLINE ONLINE jack Open,STABLE
ora.qosmserver
1 ONLINE ONLINE jill STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE jill STABLE
ora.scan2.vip
1 ONLINE ONLINE jack STABLE
ora.scan3.vip
1 ONLINE ONLINE jack STABLE

For ORACLE_HOME and configure RDBMS software:

32/35
[oracle@jack ~]$ cd $ORACLE_HOME/addnode

[oracle@jack addnode]$ $ORACLE_HOME/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={jill}"

"CLUSTER_NEW_VIRTUAL_HOSTNAMES={jill-vip}"

Prepare Configuration in progress.

Prepare Configuration successful.

.................................................. 7% Done.

Copy Files to Remote Nodes in progress.

.................................................. 12% Done.
.................................................. 18% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_01-23-09PM.log

Instantiate files in progress.

Instantiate files successful.

.................................................. 52% Done.

Saving cluster inventory in progress.

.................................................. 89% Done.

Saving cluster inventory successful.

The Cluster Node Addition of /u01/app/oracle/product/19c/db_1 was successful.
Please check '/u01/app/19c/oraInventory/logs/silentInstall2021-07-27_01-23-09PM.log' for
more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.

.................................................. 96% Done.

As a root user, execute the following script(s):

1. /u01/app/oracle/product/19c/db_1/root.sh

Execute /u01/app/oracle/product/19c/db_1/root.sh on the following nodes:

[jill]

Successfully Setup Software.

.................................................. 100% Done.

Run the root.sh script as instructed above from JILL node :

33/35
[root@jill bin]# /u01/app/oracle/product/19c/db_1/root.sh
Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/19c/db_1
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Add init parameters for infra2 instance from Node 1 Jack

SQL> alter system set undo_tablespace=UNDOTBS2 sid='infra2' scope=spfile;

System altered.

SQL> alter system set instance_number=2 sid='infra2' scope=spfile;

System altered.

SQL> alter system set instance_name='infra2' sid='infra2' scope=spfile;

System altered.

SQL> alter system set thread=2 sid='infra2' scope=spfile;

System altered.

SQL> alter database add logfile thread 2 group 4 ('+DATA','+FRA') size 200m;

Database altered.

SQL> alter database add logfile thread 2 group 5 ('+DATA','+FRA') size 200m;

Database altered.

SQL> alter database enable public thread 2;

Database altered.

Add database to OCR :

34/35
[oracle@jack ~]$ srvctl config database -d infra
Database unique name: infra
Database name: infra
Oracle home: /u01/app/oracle/product/19c/db_1
Oracle user: oracle
Spfile: +DATA/INFRA/PARAMETERFILE/spfile.274.1079014371
Password file: +DATA/INFRA/PASSWORD/pwdinfra.262.1079013137
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: infra1
Configured nodes: jack
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

[oracle@jack ~]$srvctl add instance -db infra -instance infra2 -node jill

[oracle@jack ~]$srvctl start instance -i infra2 -d infra

35/35