hanode1> cf takeover cf: takeover initiated by operator hanode1> Mon Feb 24 17:43:30 GMT [hanode1: cf.misc.

operatorTakeover:warning]: Cluster monitor: takeover initiated by operator Mon Feb 24 17:43:30 GMT [hanode1: cf.fsm.nfo.acceptTakeoverReq:warning]: Negotiated failover: accepting takeover request by partner, reason: operator initiated cf takeover. Asking partner to shutdown gracefully; will takeover in at most 180 seconds. Mon Feb 24 17:43:34 GMT [hanode1: cf.fsm.firmwareStatus:info]: Cluster monitor: partner rebooting Mon Feb 24 17:43:34 GMT [hanode1: cf.fsm.nfo.partnerShutdown:warning]: Negotiated failover: partner has shutdown Mon Feb 24 17:43:34 GMT [hanode1: cf.fsm.takeover.nfo:info]: Cluster monitor: takeover attempted after 'cf takeover'. command Mon Feb 24 17:43:34 GMT [hanode1: cf.fsm.stateTransit:warning]: Cluster monitor: UP --> TAKEOVER Mon Feb 24 17:43:34 GMT [hanode1: cf.fm.takeoverStarted:warning]: Cluster monitor: takeover started Mon Feb 24 17:43:35 GMT [hanode1: nv.partner.disabled:info]: NVRAM takeover: Partner NVRAM was disabled. Replaying takeover WAFL log Mon Feb 24 17:43:36 GMT [hanode2/hanode1: wafl.takeover.nvram.missing:info]: WAFL takeover: No WAFL nvlog records were found to replay. Mon Feb 24 17:43:36 GMT [hanode2/hanode1: wafl.replay.done:info]: WAFL log replay completed, 0 seconds Vdisk Snap Table for host:1 is initialized Mon Feb 24 17:43:47 GMT [hanode2/hanode1: net.ifconfig.noPartner:error]: ifconfig: 'ns0' cannot be configured: Address does not match any partner interface. ifconfig: ns0: no such interface ifconfig: ns1: no such interface add net default: gateway 192.168.56.1: network unreachable Mon Feb 24 17:43:47 GMT [hanode2/hanode1: net.ifconfig.noPartner:error]: ifconfig: 'ns1' cannot be configured: Address does not match any partner interface. hanode1> Mon Feb 24 17:43:58 GMT [hanode2/hanode1: snmp.agent.msg.access.denied:warning]: Permission denied for SNMPv3 requests from root. Reason: Password is too short (SNMPv3 requires at least 8 characters). Mon Feb 24 17:44:00 GMT [hanode1: monitor.globalStatus.ok:info]: This node is attempting to takeover hanode2. hanode1> cf takeoverMon Feb 24 17:44:39 GMT [hanode1: net.ifconfig.takeoverError:warning]: WARNING: 2 errors detected during network takeover processing WARNING: Some network clients may not be able to access the cluster during takeover

Mon Feb 24 17:44:39 GMT [hanode1: cf.rsrc.takeoverOpFail:error]: Cluster monitor: takeover during ifconfig_2 failed; takeover continuing... CIFS partner server is running. Mon Feb 24 17:44:39 GMT [hanode1 (takeover): cf.rsrc.transitTime:notice]: Top Takeover transit times registry_postrc_phase1=41429, rc=19281 {options=10200, hostname=8642, always_do_just_after_etc_rc=170, ifconfig=88, always_do_just_before_etc_rc=80, raid_config_prerc=20, options=19, routed=11, ems_postrc=11}, wafl=980 {prvol_to_done=520, pvvols_to_done=460, paggrs_to_done=0, prvol_mnt_end=0, part_vols_mnt_end=0}, registry_prerc=669, raid=460, wafl_sync=385, wafl_restart=260 {restarters=200, vdisk=60}, raid Mon Feb 24 17:44:39 GMT [hanode1 (takeover): cf.fm.takeoverComplete:warning]: Cluster monitor: takeover completed Mon Feb 24 17:44:39 GMT [hanode1 (takeover): cf.fm.takeoverDuration:warning]: Cluster monitor: takeover duration time is 65 seconds cf: in takeover mode! hanode1(takeover)> vol status Volume State vol0 online no_atime_update=on,

Status raid0, flex

Options root,

create_ucode=on, convert_ucode=on, maxdirsize=2621 hanode1(takeover)> cf giMon Feb 24 17:45:00 GMT [hanode1 (takeover): monitor.globalStatus.critical:CRITICAL]: This node has taken over hanode2. Mon Feb 24 17:45:00 GMT [hanode2/hanode1: monitor.globalStatus.critical:CRITICAL]: hanode1 has taken over this node. hanode1(takeover)> cf give backup cf: Invalid option 'give' Usage: cf [disable|enable|forcetakeover [[d][f]]|forcegiveback|giveback [-f]|hw_assist|monitor|partner|status [-t]|takeover [-f | -n]] hanode1(takeover)> cf givebackup cf: Invalid option 'givebackup' Usage: cf [disable|enable|forcetakeover [[d][f]]|forcegiveback|giveback [-f]|hw_assist|monitor|partner|status [-t]|takeover [-f | -n]] hanode1(takeover)> cMon Feb 24 17:45:29 GMT [hanode2/hanode1: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server. f givebackup cf: Invalid option 'givebackup' Usage: cf [disable|enable|forcetakeover [[d][f]]|forcegiveback|giveback [-f]|hw_assist|monitor|partner|status [-t]|takeover [-f | -n]]

hanode1(takeover)> hanode1(takeover)> partner Login to partner shell: hanode2 hanode2/hanode1> Mon Feb 24 17:45:37 GMT [hanode1 (takeover): cf.partner.login:notice]: Login to partner shell: hanode2 hanode2/hanode1> uptime 5:45pm down 2 mins, taken over by node 'hanode1' hanode2/hanode1> partner Logoff from partner shell: hanode2 hanode1(takeover)> Mon Feb 24 17:45:54 GMT [hanode1 (takeover): cf.partner.logoff:notice]: Logoff from partner shell: hanode2 hanode1(takeover)> cf giveback Mon Feb 24 17:46:27 GMT [hanode1 (takeover): cf.misc.operatorGiveback:info]: Cluster monitor: giveback initiated by operator Mon Feb 24 17:46:27 GMT [hanode1: cf.fm.givebackStarted:warning]: Cluster monitor: giveback started hanode1(takeover)> CIFS partner server is shutting down... CIFS partner server has shut down... Mon Feb 24 17:46:30 GMT [hanode1: cf.rsrc.transitTime:notice]: Top Giveback transit times ndmpd=1897, wafl=581 {sync_clean=290, giveback_sync=247, finish=32, vol_refs=12, mark_abort=0, wait_offline=0, wait_create=0, abort_scans=0, drain_msgs=0, zombie_wait=0}, wafl_gb_sync=388, raid=88, java=50, registry_giveback=50, vdisk=34, cifs=29, snmp=26, nfsd=24 Mon Feb 24 17:46:30 GMT [hanode1: asup.msg.giveback.delayed:info]: giveback AutoSupport delayed 5 minutes (until after the giveback process is complete). Mon Feb 24 17:46:30 GMT [hanode1: cf.fm.givebackComplete:warning]: Cluster monitor: giveback completed Mon Feb 24 17:46:30 GMT [hanode1: cf.fm.givebackDuration:warning]: Cluster monitor: giveback duration time is 3 seconds Mon Feb 24 17:46:30 GMT [hanode1: cf.fsm.stateTransit:warning]: Cluster monitor: TAKEOVER --> UP Mon Feb 24 17:46:31 GMT [hanode1: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of hanode1 by hanode2 disabled (unsynchronized log) Mon Feb 24 17:46:36 GMT [hanode1: cf.fsm.partnerNotResponding:notice]: Cluster monitor: partner not responding Mon Feb 24 17:46:36 GMT [hanode1: cf.fm.timeMasterStatus:info]: Acting as cluster time master Mon Feb 24 17:46:37 GMT [hanode1: cf.fsm.partnerOk:notice]: Cluster monitor: partner ok Mon Feb 24 17:46:37 GMT [hanode1: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of hanode2 enabled

Mon Feb 24 17:46:40 GMT [hanode1: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of hanode2 disabled (unsynchronized log) Mon Feb 24 17:46:41 GMT [hanode1: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of hanode1 by hanode2 enabled Mon Feb 24 17:46:42 GMT [hanode1: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of hanode2 enabled Mon Feb 24 17:47:00 GMT [hanode1: monitor.globalStatus.ok:info]: The system's global status is normal. hanode1> uptime 5:47pm up 14 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops hanode1> hanode1> hanode1> hanode1> version NetApp Release 7.3.6: Thu Jul 7 00:41:19 PDT 2011 hanode1> hanode1> hanode1> nodes nodes not found. Type '?' for a list of commands hanode1> node node not found. Type '?' for a list of commands hanode1> hanode1> hanode1> cf status Cluster enabled, hanode2 is up. hanode1> hanode1> hanode1> hanode1> hanode1> login as: root root@192.168.56.104's password: node1> node1> node1> node1> df -g Filesystem on /vol/vol0/ /vol/vol0/ /vol/vol0/.snapshot /vol/vol0/.snapsh ot /vol/vol1/ /vol/vol1/

total 0GB 0GB 0GB

used 0GB 0GB 0GB

avail capacity 0GB 0GB 0GB 50% ---% 62%

Mounted

/vol/vol1/.snapshot 0GB 0GB 0GB 0% /vol/vol1/.snapsh ot /vol/NFSvol/ 0GB 0GB 0GB 0% /vol/NFSvol/ /vol/NFSvol/.snapshot 0GB 0GB 0GB 1% /vol/NFSvol/.sn apshot /vol/vol2/ 0GB 0GB 0GB 7% /vol/vol2/ /vol/vol2/.snapshot 0GB 0GB 0GB 0% /vol/vol2/.snapsh ot /vol/vfiler1/ 0GB 0GB 0GB 0% /vol/vfiler1/ /vol/vfiler1/.snapshot 0GB 0GB 0GB 0% /vol/vfiler1/. snapshot /vol/src_vol/ 0GB 0GB 0GB 0% /vol/src_vol/ /vol/src_vol/.snapshot 0GB 0GB 0GB 0% /vol/src_vol/. snapshot node1> src_vol src_vol not found. Type '?' for a list of commands node1> node1> node1> sysstat -us 1 CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 1 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 0 0 0 >60 100% 0% 0% 0% 0 0 0 0 8 0 0 >60 100% 0% 2% 0% 0 0 0 32 164 0 0 >60 100% 21% T 22%

0% 0 0 0 0 0 0 0 >60 100% 0% 2% -Summary Statistics ( 11 samples 1 secs/sample) CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util Min 0% 0 0 0 0 0 0 0 >60 100% 0% * 0% Avg 0% 0 0 0 2 15 0 0 >60 100% 1% * 2% Max 0% 0 1 0 32 164 0 0 >60 100% 21% * 22% node1> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache in out read write read write age hit 0% 0 0 0 0 0 1 0 0 0 0 >60 100% 0% 0 0 0 0 0 0 0 0 0 0 >60 100% 0% 0 0 0 0 0 0 0 0 0 0 >60 100% 0% 0 0 0 0 0 0 0 8 0 0 >60 100% 0% 0 0 0 0 0 0 32 170 0 0 >60 100% 0% 0 0 0 0 0 0 0 0 0 0 >60 100% 0% 0 0 0 0 0 0 0 0 0 0 >60 100% 1% 0 0 0 0 0 1 0 0 0 0 >60 100% 0% 0 0 0 0 0 0 0 8 0 0 >60 100% node1> nfsstat -I nfsstat: Invalid option - I usage: nfsstat <interval> [ <count> ] usage: vfiler run <vfname> nfsstat -h [ -n [ <count> ] ] | [ [ <hostname or ip address> ] [ <interval> ] ] usage: vfiler run <vfname> nfsstat -l [ <count> ] usage: nfsstat -z usage: nfsstat -C usage: nfsstat [ -c ] [ -t ] node1> nfsstat -i nfsstat: Invalid option - i

usage: nfsstat <interval> [ <count> ] usage: vfiler run <vfname> nfsstat -h [ -n [ <count> ] ] | [ [ <hostname or ip address> ] [ <interval> ] ] usage: vfiler run <vfname> nfsstat -l [ <count> ] usage: nfsstat -z usage: nfsstat -C usage: nfsstat [ -c ] [ -t ] node1> nfsstat Server rpc: TCP: calls badcalls 303 10 UDP: calls 0 IPv4: calls 303 IPv6: calls 0 badcalls 0 badcalls 10 badcalls 0

nullrecv 0 nullrecv 0 nullrecv 0 nullrecv 0

badlen 0 badlen 0 badlen 0 badlen 0

xdrcall 10 xdrcall 0 xdrcall 10 xdrcall 0

Server nfs: calls badcalls 293 0 Server nfs null 0 0% wrcache symlink 0 0% mkdir 0 0% V2: (0 calls) getattr setattr 0 0% 0 0% write create 0 0% rmdir 0 0% 0 0% readdir 0 0% root 0 0% remove 0 0% statfs 0 0% 2K-4095 0 lookup 0 0% rename 0 0% readlink 0 0% link 0 0% read 0 0% 0 0%

Read request stats (version 2) 0-511 512-1023 1K-2047 32767 32K -65535 64K-131071 > 131071 0 0 0 0 0 0 Write request stats (version 2) 0-511 512-1023 1K-2047 32767 32K -65535 64K-131071 > 131071

4K-8191 0

8K-16383 0

16K0

2K-4095

4K-8191

8K-16383

16K-

0 0 0 Server nfs null 19 6% write rmdir 14 5% rename pathconf 3 1% commit 0 0%

0 0

V3: (293 calls) getattr setattr 51 17% 31 11% create mkdir 24 8% link 0 0% 0 0% readdir 0 0%

lookup 31 11% symlink 0 0% readdir+ 21 7%

access 45 15% mknod 0 0% fsstat 15 5%

readlink 0 0% remove 18 6% fsinfo 20 7%

read 1 0% 0 0% 0 0%

Read request stats (version 3) 0-511 512-1023 1K-2047 32767 32K -65535 64K-131071 > 131071 1 0 0 0 0 0 Write request stats (version 3) 0-511 512-1023 1K-2047 32767 32K -65535 64K-131071 > 131071 3 0 0 0 0 0

2K-4095 0

4K-8191 0

8K-16383 0

16K0

2K-4095 0

4K-8191 9

8K-16383 2

16K0

Misaligned Read request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 0 0 0 0 0 0 0 Misaligned Write request stats BIN-0 BIN-1 BIN-2 BIN-3 BIN-4 BIN-5 BIN-6 11 0 0 0 0 0 0 node1> nfsstat -z 1 1 nfsstat: zeroing stats - ignoring other options specified node1> nfsstat -z 1 nfsstat: zeroing stats - ignoring other options specified node1> nfsstat Server rpc: TCP: calls badcalls 0 0 UDP: calls 0 badcalls 0

BIN-7 0 BIN-7 0

nullrecv 0 nullrecv 0

badlen 0 badlen 0

xdrcall 0 xdrcall 0

IPv4: calls 0 IPv6: calls 0

badcalls 0 badcalls 0

nullrecv 0 nullrecv 0

badlen 0 badlen 0

xdrcall 0 xdrcall 0

Server nfs: calls badcalls 0 0