SyncIQ Failures Checking

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE
SYNCIQ FAILURES
OneFS 7.1.0 - 8.1.0
Abstract
This troubleshooting guide helps you identify and address common issues that
can cause SyncIQ to fail.
August 2017
1 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures
___________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at _________________
http://bit.ly/isi-docfeedback.
Contents and overview
Note
Follow all of these steps, in order, until you reach a resolution.
1. Perform
Start troubleshooting
troubleshooting
Page 3
steps in order.
Sync fails within the first five minutes

Page 4
Sync fails after the first five minutes

Page 9
SyncIQ error connecting to daemon

(bandwidth, throttle, pworker)
Page 11
2. Appendixes Appendix A
Identify the error in the policy report output
Appendix B
Determine whether there is a source pool
restriction
Appendix C
How to use this flowchart
___________________________
Start troubleshooting
Introduction
Start troubleshooting here. If you need help
to understand the flowchart conventions that Start
are used in this guide, see Appendix C: How
to use this flowchart.
Capture the error for the failing policy as follows:
1. Obtain the SyncIQ job ID by running the following command on the source cluster, where
<policy-name> is the policy name. See Appendix
_________ A for an example of the output:
isi sync reports list --policy-name=<policy-name> --sort job_id
2. View the report by running the following command, where <policy-name> is the policy name
and <JobID> is the job ID you obtained in step 1. The output of the command lists the error.
See Appendix
_________ A for an example of the output:
isi sync reports view --policy=<policy-name> <JobID> | less
In the report, find the error that relates to the failure.

Note the start and end times for the policy. See the
examples in Appendix A.
Did the
Within policy fail within After
the first the first five minutes of the first
starting, or after the first
five minutes five minutes? five minutes
Go to Page 4 Go to Page 9
___________________________
Sync fails within the first five minutes
You could have arrived here from: Page

4
Page 3 - Start troubleshooting
Error Tables
Find your error in the tables on this page and the next few
pages. Follow the instructions for your error.
Error Action
FAILED ASSERTION Contact Isilon Technical Support

for assistance.
Unable to update metadata (inode changes)... Unable

to open Lin <lin>: No such file or directory
One of the following operation not permitted errors:

unable to delete
failed to move
unable to rename
Error opening linmap <policyID>: No such file or

directory
Closing sworker (X): Broken pipe/Work item X has been

restarted too many times./SyncIQ policy failed
Failed to diff_range, Failed to diff file, Input/

output error, Failure due to file system errors:
Missing higher snapshot version of a file
SyncIQ is unable to connect to a resource on the

target cluster.
or
Source node could not connect to target cluster.
If your error is not on

this page, go to Page 5
___________________________
Sync fails within the first five minutes (2)
You could have arrived here from:
Page 4 - Sync fails within the first five

minutes Page
5
Error Action
SyncIQ scheduler failed to start

policy. No nodes in cluster have
Go to the following article to resolve your issue. If the
external IPs
problem persists, contact Isilon Technical Support.
or
SyncIQ scheduler failed to start
OneFS: SyncIQ policy fails to run with "failed to start
policy. Policy <name> has an invalid
policy" error, article 471903
subnet:pool restriction of
subnet0:pool1.

Error with these three phrases together:
Unable to delete
OneFS: SyncIQ policy fails with error: Unable to delete
unlink of Lin 0 failed
[path]: unlink of Lin 0 failed: No such file or directory,
No such file or directory
article 471906

These two errors together: problem persists, contact Isilon Technical Support.
Failed to smkchdirfd to target
directory OneFS 7.2: SyncIQ fails with error "Failed to
Operation Not Permitted smkchdirfd to target directory" and "Operation not
permitted," article 469809

SyncIQ detected a problem with policy
configuration. Policy root path
OneFS: SyncIQ fails with error: SyncIQ detected a
directory <directory> does not match
problem with policy configuration. Policy root path
between old and new snapshots
directory does not match between old and new
snapshots, article 471898

___________________________

Page
6
minutes (2)
Error Action

SyncIQ failed to take a snapshot on
source cluster. OneFS: SyncIQ policy fails with error: SyncIQ failed to
Can't find latest snapid <ID> take a snapshot on source cluster. Can't find latest
snapid XXX,
article 463816

SyncIQ failed to take a snapshot on
source cluster. Unable to delete OneFS: Sync fails with error: SyncIQ failed to take a
snapshot from previous run: 44 snapshot on source cluster. Unable to delete snapshot
from previous run,
article 469839

SyncIQ failed to take a snapshot on problem persists, contact Isilon Technical Support.
source cluster. Failed to open and lock
new snapid XXX: OneFS: Sync fails with error: SyncIQ failed to take a
ifs_snap_create_lock_lease() failed: snapshot on source cluster. Failed to open and lock
Operation not permitted. new snapid XXX: ifs_snap_create_lock_lease() failed:
Operation not permitted, article 471907

Failed to open: <file path> (lin
xxxxxxxxx): Permission denied Isilon OneFS: A SyncIQ job fails with "permission
denied" error because AVScan has quarantined a file,
article 477492

___________________________

Page
Page 6 - Sync fails within the first five 7
minutes (3)
Error Action

bad checksum Input/output error due to
network WAN accelerator
Isilon OneFS: A SyncIQ job fails with "bad checksum:
Input/output error", article 477493

Initial or diffsync error "Failed to
open dir 0"
Isilon OneFS: A SyncIQ job fails with error "Failed to
open dir 0" article 477494

Primary authentication fails
SyncIQ job fails with error Primary authentication fails,
article 464224

Failed to move .tmp-working-dir
Isilon OneFS: SyncIQ Error when source directory is
overwritten or deleted and re-created: Failed to move
.tmp-working-dir, article 463450

SyncIQ policy failed. Unexpected non-
I/O error preceded by All syncIQ jobs fail with "Unexpected non-I/O error,"
preceded by "msg_handshake failed" error,
article 466618

___________________________

minutes (4)
Page
8
Error Action

(policy name: <name>) SyncIQ detected a
problem with policy configuration.
OneFS: SyncIQ policy fails with error: SyncIQ detected
Policy has force_interface set, but the
a problem with policy configuration. Policy has
sysctl net.inet.ip.choose_ifa_by_ipsrc
force_interface set, but the sysctl
is not set
net.inet.ip.choose_ifa_by_ipsrc is not set,
article 463852

Go to Page 11.
If your error is not listed in the table,

contact Isilon Technical Support
for assistance.
___________________________
Sync fails after the first five minutes
Page 3 Start troubleshooting

Page
9
Error Tables
Find your error in the tables on this page and the next few
pages. Follow the instructions for your error.
Error Action
Contact Isilon Technical Support

FAILED ASSERTION
for assistance.
Unable to update metadata (inode

changes)... Unable to open Lin <lin>:
No such file or directory
Closing sworker (X): Broken pipe/Work

item X has been restarted too many
times./SyncIQ policy failed

___________________________
Sync fails after the first five minutes (2)
Page 9 - Sync fails after the first five Page

minutes 10
Error Action

SyncIQ failed to take a snapshot on problem persists, contact Isilon Technical Support.
source cluster.
Failed to open and lock new snapid XXX: OneFS: Sync fails with error: SyncIQ failed to take a
ifs_snap_create_lock_lease() failed: snapshot on source cluster. Failed to open and lock
Operation not permitted. new snapid XXX: ifs_snap_create_lock_lease() failed:
Operation not permitted, article 471907

These two errors together: problem persists, contact Isilon Technical Support.
Failed to smkchdirfd to target OneFS 7.0 -7.1: SyncIQ fails with error "Failed to
directory smkchdirfd to target directory" and "Operation not
Operation Not Permitted permitted", article 469809

bad checksum Input/output error due to
network WAN accelerator Isilon OneFS: A SyncIQ job fails with "bad checksum:
Input/output error",
article 477493

Go to Page 11.
If your error is not listed in the table,

contact Isilon Technical Support
for assistance.
___________________________
SyncIQ error connecting to daemon (bandwidth, throttle, pworker)
You could have arrived here

from Page
11
Page 8 - Sync fails within the first
five minutes (5)
Page 10 - SyncIQ fails after the first five
minutes (2)
Read the following notes about this error.

Then continue on to troubleshooting the error.
Notes about this error:

Error:
SyncIQ error connecting to daemon (bandwidth, throttle, pworker)
This error occurs:
When the sync fails at any time during the sync job.
This error appears:
In the sync policy report on the OneFS web administration interface or command-line interface
and in the /var/log/isi_migrate.log file.
Cause of this error:
The source pool does not include node 1. The bandwidth/throttle daemon cannot be reached
because it always runs on node 1.
Example of error:
2012-12-27T18:10:37-06:00 <3.3> cluster1-8(id8) isi_migrate[11771]:
coord[policy1]: siq_create_alert: type: 11 (policy name: policy1
target: cluster1.company.com) SyncIQ error connecting to daemon
(bandwidth, throttle, pworker). Please verify all SyncIQ daemons are
running. Unable to connect to throttle host for last 1080 seconds
Go to Page 12
___________________________
SyncIQ error connecting to daemon (bandwidth, throttle, pworker) (2)
Page
You could have arrived here 12
from
Page 11 - SyncIQ error connecting

to daemon (bandwidth, throttle,
pworker)
Check whether the policy has a source pool restriction by running the
following command on the source cluster, where <policy-name> is the
name of the policy.
See____________
Appendix B for example output
isi sync policies view <policy-name>
In the output, note whether a Source Subnet and Source Pool are listed.
These identify the source pool restriction.
Does the
policy have a source Yes Go to Page 14
pool restriction?
No
Go to Page 13
___________________________
Page
You could have arrived here from: 13
Page 12 - SyncIQ error connecting to

daemon (bandwidth, throttle, pworker) (2)
Does the policy

Contact
have bandwidth/throttle No Isilon Technical Support.
rules configured?
Yes
Use the OneFS web administration interface to add a source pool restriction
as follows:
1. Click Data Protection > SyncIQ > Policies.

2. For the policy that you want to set the restriction on, click View/Edit.
3. Click Edit Policy.
4. In the Source Cluster section, in the Restrict Source Nodes section,
select the radio button for Run the policy only on nodes in the
specified subnet and pool.
5. Select a subnet and pool from the drop-down list.
6. Click Save Changes.
IMPORTANT!
Make sure that the front-end network ports on all
of the nodes in the source pool restriction can
see each other on the LAN.
Go to Page 14
___________________________

Page
14
Are you running OneFS

7.1.x? No Go to Page 16
Yes
Go to Page 15
___________________________
You could have arrived here from: Page

15
OneFS 7.1.x
Run the following command:
isi networks list pools -v
In the output, locate the source pool restriction that you identified in
the previous step (displayed in the form subnet:pool in the output of this
command). Then, for that restriction, check whether node 1 (the node with
LNN 1) is listed in the Pool Membership.
See the box on this page for example output. In this example, nodes 2 and 3
are listed in the Pool Membership, but node 1 is not listed.
Is node 1
listed in the Pool Yes Go to Page 17
Membership?
No
Go to Page 16
Example output of isi networks list pools -v
subnet1:pool0
In Subnet: subnet1
Allocation: Static
Ranges: 1
10.1.1.101-10.1.1.116
Pool Membership: 6
2:10gige-1 (up)
2:10gige-2 (up)
3:10gige-1 (up)
3:10gige-2 (up)
...<truncated>
___________________________
Page
You could have arrived here from: 16
Page 14 - SyncIQ error connecting to daemon

(bandwidth, throttle, pworker) (4)
Page 15 - SyncIQ error connecting to daemon
(bandwidth, throttle, pworker) (5)
OneFS 8.x.x
Run the following command to view a list of the network pools within a groupnet or subnet:
isi network pools list
Extract the groupnet:subnet value for the desired pool name from the output, for example, groupnet1.subnet3 for
pool5, and provide it as an input for the following command to check the nodes within that pool.
isi network pools view <groupnet_name>.<subnet_name>.<pool_name>
See the boxes on this page for example output of both the previously-listed commands.
Is node 1
listed under Ifaces? Yes Go to Page 18
No
Go to Page 17
Example output of isi network pools list Cont...

groupnet1.subnet3
Groupnet: groupnet1
ID SC Zone Allocation Method subnet: subnet3
---------------------------------------------------------------------- Name: pool5
groupnet1.subnet3.pool5 data.company.com Rules: -
static Access Zone: zone3
groupnet1.subnet3.pool7 data.company.com Allocation Method: static
dynamic Aggregation Mode: lacp
---------------------------------------------------------------------- SC Suspended Nodes: -
Example output of isi network pools view Description: -
groupnet1.subnet3.pool5 to display the nodes within Ifaces: 1:ext-2, 2:ext-2, 3:ext-2
pool5 IP Ranges: 203.0.223.12-203.0.223.22
-----------
ID: groupnet0.subnet3.pool5 truncated
___________________________

Page
17
Is it acceptable
for your workflow to Contact
add node 1 to the No Isilon Technical Support.
source pool?
Yes
Add node 1 to the source pool using the OneFS

web administration interface. For instructions, see
the OneFS Web Administration Guide
for your version of OneFS.
Run the following command to try to run the SyncIQ job again,
where <policy-name> is the name of the failed policy.
isi sync jobs start <policy-name>
Does the sync

complete successfully No Go to Page 18
without errors?
Yes
End troubleshooting
___________________________

Page 16 - SyncIQ error connecting to Page
daemon (bandwidth, throttle, pworker) (6) 18
On the source cluster, increase the logging level by running the following
command, where <policy-name> is the name of the sync policy:
isi sync policies modify <policy-name> --log-level=trace
Start the policy by running the following command, where <policy-name>

is the name of the policy:
isi sync jobs start <policy-name>
Let the policy run until it fails.
Go to Page 19
___________________________

Page
19
Upload logs and data to Isilon Technical Support as follows:
isi_gather_info -f /var/crash/isi_migr*
CAUTION!
This step is very important. Failure to do this may cause boot flash drives
to degrade prematurely.
On the source cluster, set the logging level back to normal by running the following
command, where <policy-name> is the name of the policy.
isi sync policies modify <policy-name> --log-level=notice
Contact
Isilon Technical Support.
___________________________
Appendix A - Identify the error in the policy report output
Page 3 - Start troubleshooting
Identify the job ID:
Output for the isi sync reports list --policy-name=policy1 --sort job_id command
cluster-1# isi sync reports list --policy-name=policy1 --sort job_id
Policy Name Job ID Start Time End Time Action State
------------------------------------------------------------------------------------------
policy1 1 2016-02-02T11:06:30 2016-02-02T11:06:38 run finished
policy1 2 2016-02-02T11:06:44 2016-02-02T11:06:53 run finished
policy1 3 2016-02-02T11:18:37 2016-02-02T11:18:41 run failed
Identify the error:

Example output for the isi sync reports view --policy=dottest 1 | less command
cluster-1# isi sync reports view --policy=dottest 1 | less
Policy Name: dottest
Job ID: 1
Start Time: 2016-02-02T17:27:55
End Time: 2016-02-02T17:33:10
Action: run
State: failed
ID: 1-dottest
Policy ID: a12345678b901c23456abc78912d34dc
Sync Type: invalid
Duration: 5m12s
Errors: No node on source cluster was able to
connect to target cluster., Source node
could not connect to target cluster.
....<truncated>
___________________________
Appendix B - Determine whether there is a source pool restriction

A source pool restriction can be found by running the following commands. The bold lines
in the example output identify the restrictions. In these examples, the restrictions are
subnet1:pool0.
Example output for isi sync policies view <policy-name>

cluster-1# isi sync polies view policy1
ID: 1234567891234a5a67890f1234ba8cab
Name: policy1
Path: /ifs/data/backup
Action: sync
Enabled: No
Target: cluster.company.com
Description:
Check Integrity: Yes
Source Include Directories: -
Source Exclude Directories: -
Source Subnet: subnet1
Source Pool: pool0
...<truncated>
___________________________
Appendix C: How to use this flowchart
Introduction
Describes what the section helps you to
accomplish.
Page # - Page title

Page
# Note
Provides context and additional
information. Sometimes a note is linked
Directional arrows indicate
to a process step with a colored dot.
the path through the
process flow.
Yes Decision diamond No
Process step with command:

Process step
CAUTION! command xyz
Caution boxes warn that
a particular step needs
to be performed with
great care, to prevent
serious consequences.
Optional process step Go to Page #
End point Document Shape

Calls out supporting documentation
for a process step. When possible,
these shapes contain links to the
reference document.
Sometimes linked to a process step
with a colored dot.
___________________________
Copyright 2017 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS-IS. DELL MAKES NO

REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL
SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be the property of their respective owners.
EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 in North America 1-866-464-7381
www.EMC.com
___________________________

SyncIQ Failures Checking

Uploaded by

Copyright:

Available Formats

SyncIQ Failures Checking

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SyncIQ Failures Checking

Uploaded by

Copyright:

Available Formats

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

OneFS 7.1.0 - 8.1.0

1 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

Sync fails within the first five minutes

Sync fails after the first five minutes

SyncIQ error connecting to daemon

2 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

Capture the error for the failing policy as follows:

isi sync reports list --policy-name=<policy-name> --sort job_id

isi sync reports view --policy=<policy-name> <JobID> | less

In the report, find the error that relates to the failure.

3 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from: Page

FAILED ASSERTION Contact Isilon Technical Support

Unable to update metadata (inode changes)... Unable

One of the following operation not permitted errors:

Error opening linmap <policyID>: No such file or

Closing sworker (X): Broken pipe/Work item X has been

Failed to diff_range, Failed to diff file, Input/

SyncIQ is unable to connect to a resource on the

If your error is not on

4 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Page 4 - Sync fails within the first five

SyncIQ scheduler failed to start

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

If your error is not on

5 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

If your error is not on

6 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

Go to the following article to resolve your issue. If the

If your error is not on

7 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Page 7 - Sync fails within the first five

Go to the following article to resolve your issue. If the

SyncIQ error connecting to daemon

If your error is not listed in the table,

8 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Page 3 Start troubleshooting

Contact Isilon Technical Support

Unable to update metadata (inode

Closing sworker (X): Broken pipe/Work

If your error is not on

9 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

You could have arrived here from:

Page 9 - Sync fails after the first five Page