Skip to content

Commit 14b395e

Browse files
torvaldstorvalds
committed
Merge branch 'for-2.6.27' of git://linux-nfs.org/~bfields/linux
* 'for-2.6.27' of git://linux-nfs.org/~bfields/linux: (51 commits) nfsd: nfs4xdr.c do-while is not a compound statement nfsd: Use C99 initializers in fs/nfsd/nfs4xdr.c lockd: Pass "struct sockaddr *" to new failover-by-IP function lockd: get host reference in nlmsvc_create_block() instead of callers lockd: minor svclock.c style fixes lockd: eliminate duplicate nlmsvc_lookup_host call from nlmsvc_lock lockd: eliminate duplicate nlmsvc_lookup_host call from nlmsvc_testlock lockd: nlm_release_host() checks for NULL, caller needn't file lock: reorder struct file_lock to save space on 64 bit builds nfsd: take file and mnt write in nfs4_upgrade_open nfsd: document open share bit tracking nfsd: tabulate nfs4 xdr encoding functions nfsd: dprint operation names svcrdma: Change WR context get/put to use the kmem cache svcrdma: Create a kmem cache for the WR contexts svcrdma: Add flush_scheduled_work to module exit function svcrdma: Limit ORD based on client's advertised IRD svcrdma: Remove unused wait q from svcrdma_xprt structure svcrdma: Remove unneeded spin locks from __svc_rdma_free svcrdma: Add dma map count and WARN_ON ...
2 parents 734b397 + 5108b27 commit 14b395e

36 files changed

+1042
-966
lines changed

‎Documentation/filesystems/nfs-rdma.txt

Lines changed: 59 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
################################################################################
66

77
Author: NetApp and Open Grid Computing
8-
Date: April 15, 2008
8+
Date: May 29, 2008
99

1010
Table of Contents
1111
~~~~~~~~~~~~~~~~~
@@ -60,39 +60,52 @@ Installation
6060
The procedures described in this document have been tested with
6161
distributions from Red Hat's Fedora Project (http://fedora.redhat.com/).
6262

63-
- Install nfs-utils-1.1.1 or greater on the client
63+
- Install nfs-utils-1.1.2 or greater on the client
6464

65-
An NFS/RDMA mount point can only be obtained by using the mount.nfs
66-
command in nfs-utils-1.1.1 or greater. To see which version of mount.nfs
67-
you are using, type:
65+
An NFS/RDMA mount point can be obtained by using the mount.nfs command in
66+
nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
67+
version with support for NFS/RDMA mounts, but for various reasons we
68+
recommend using nfs-utils-1.1.2 or greater). To see which version of
69+
mount.nfs you are using, type:
6870

69-
> /sbin/mount.nfs -V
71+
$ /sbin/mount.nfs -V
7072

71-
If the version is less than 1.1.1 or the command does not exist,
72-
then you will need to install the latest version of nfs-utils.
73+
If the version is less than 1.1.2 or the command does not exist,
74+
you should install the latest version of nfs-utils.
7375

7476
Download the latest package from:
7577

7678
http://www.kernel.org/pub/linux/utils/nfs
7779

7880
Uncompress the package and follow the installation instructions.
7981

80-
If you will not be using GSS and NFSv4, the installation process
81-
can be simplified by disabling these features when running configure:
82+
If you will not need the idmapper and gssd executables (you do not need
83+
these to create an NFS/RDMA enabled mount command), the installation
84+
process can be simplified by disabling these features when running
85+
configure:
8286

83-
> ./configure --disable-gss --disable-nfsv4
87+
$ ./configure --disable-gss --disable-nfsv4
8488

85-
For more information on this see the package's README and INSTALL files.
89+
To build nfs-utils you will need the tcp_wrappers package installed. For
90+
more information on this see the package's README and INSTALL files.
8691

8792
After building the nfs-utils package, there will be a mount.nfs binary in
8893
the utils/mount directory. This binary can be used to initiate NFS v2, v3,
89-
or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4.
90-
The standard technique is to create a symlink called mount.nfs4 to mount.nfs.
94+
or v4 mounts. To initiate a v4 mount, the binary must be called
95+
mount.nfs4. The standard technique is to create a symlink called
96+
mount.nfs4 to mount.nfs.
9197

92-
NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed
98+
This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
99+
100+
$ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
101+
102+
In this location, mount.nfs will be invoked automatically for NFS mounts
103+
by the system mount commmand.
104+
105+
NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed
93106
on the NFS client machine. You do not need this specific version of
94107
nfs-utils on the server. Furthermore, only the mount.nfs command from
95-
nfs-utils-1.1.1 is needed on the client.
108+
nfs-utils-1.1.2 is needed on the client.
96109

97110
- Install a Linux kernel with NFS/RDMA
98111

@@ -156,8 +169,8 @@ Check RDMA and NFS Setup
156169
this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
157170
card:
158171

159-
> modprobe ib_mthca
160-
> modprobe ib_ipoib
172+
$ modprobe ib_mthca
173+
$ modprobe ib_ipoib
161174

162175
If you are using InfiniBand, make sure there is a Subnet Manager (SM)
163176
running on the network. If your IB switch has an embedded SM, you can
@@ -166,18 +179,18 @@ Check RDMA and NFS Setup
166179

167180
If an SM is running on your network, you should see the following:
168181

169-
> cat /sys/class/infiniband/driverX/ports/1/state
182+
$ cat /sys/class/infiniband/driverX/ports/1/state
170183
4: ACTIVE
171184

172185
where driverX is mthca0, ipath5, ehca3, etc.
173186

174187
To further test the InfiniBand software stack, use IPoIB (this
175188
assumes you have two IB hosts named host1 and host2):
176189

177-
host1> ifconfig ib0 a.b.c.x
178-
host2> ifconfig ib0 a.b.c.y
179-
host1> ping a.b.c.y
180-
host2> ping a.b.c.x
190+
host1$ ifconfig ib0 a.b.c.x
191+
host2$ ifconfig ib0 a.b.c.y
192+
host1$ ping a.b.c.y
193+
host2$ ping a.b.c.x
181194

182195
For other device types, follow the appropriate procedures.
183196

@@ -202,55 +215,57 @@ NFS/RDMA Setup
202215
/vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
203216
/vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
204217

205-
The IP address(es) is(are) the client's IPoIB address for an InfiniBand HCA or the
206-
cleint's iWARP address(es) for an RNIC.
218+
The IP address(es) is(are) the client's IPoIB address for an InfiniBand
219+
HCA or the cleint's iWARP address(es) for an RNIC.
207220

208-
NOTE: The "insecure" option must be used because the NFS/RDMA client does not
209-
use a reserved port.
221+
NOTE: The "insecure" option must be used because the NFS/RDMA client does
222+
not use a reserved port.
210223

211224
Each time a machine boots:
212225

213226
- Load and configure the RDMA drivers
214227

215228
For InfiniBand using a Mellanox adapter:
216229

217-
> modprobe ib_mthca
218-
> modprobe ib_ipoib
219-
> ifconfig ib0 a.b.c.d
230+
$ modprobe ib_mthca
231+
$ modprobe ib_ipoib
232+
$ ifconfig ib0 a.b.c.d
220233

221234
NOTE: use unique addresses for the client and server
222235

223236
- Start the NFS server
224237

225-
If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
226-
load the RDMA transport module:
238+
If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
239+
kernel config), load the RDMA transport module:
227240

228-
> modprobe svcrdma
241+
$ modprobe svcrdma
229242

230-
Regardless of how the server was built (module or built-in), start the server:
243+
Regardless of how the server was built (module or built-in), start the
244+
server:
231245

232-
> /etc/init.d/nfs start
246+
$ /etc/init.d/nfs start
233247

234248
or
235249

236-
> service nfs start
250+
$ service nfs start
237251

238252
Instruct the server to listen on the RDMA transport:
239253

240-
> echo rdma 2050 > /proc/fs/nfsd/portlist
254+
$ echo rdma 2050 > /proc/fs/nfsd/portlist
241255

242256
- On the client system
243257

244-
If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
245-
load the RDMA client module:
258+
If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
259+
kernel config), load the RDMA client module:
246260

247-
> modprobe xprtrdma.ko
261+
$ modprobe xprtrdma.ko
248262

249-
Regardless of how the client was built (module or built-in), issue the mount.nfs command:
263+
Regardless of how the client was built (module or built-in), use this
264+
command to mount the NFS/RDMA server:
250265

251-
> /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/<export> /mnt -i -o rdma,port=2050
266+
$ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt
252267

253-
To verify that the mount is using RDMA, run "cat /proc/mounts" and check the
254-
"proto" field for the given mount.
268+
To verify that the mount is using RDMA, run "cat /proc/mounts" and check
269+
the "proto" field for the given mount.
255270

256271
Congratulations! You're using NFS/RDMA!

‎fs/lockd/svc.c

Lines changed: 13 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ EXPORT_SYMBOL(nlmsvc_ops);
5050
static DEFINE_MUTEX(nlmsvc_mutex);
5151
static unsigned int nlmsvc_users;
5252
static struct task_struct *nlmsvc_task;
53-
static struct svc_serv *nlmsvc_serv;
53+
static struct svc_rqst *nlmsvc_rqst;
5454
int nlmsvc_grace_period;
5555
unsigned long nlmsvc_timeout;
5656

@@ -194,20 +194,11 @@ lockd(void *vrqstp)
194194

195195
svc_process(rqstp);
196196
}
197-
198197
flush_signals(current);
199198
if (nlmsvc_ops)
200199
nlmsvc_invalidate_all();
201200
nlm_shutdown_hosts();
202-
203201
unlock_kernel();
204-
205-
nlmsvc_task = NULL;
206-
nlmsvc_serv = NULL;
207-
208-
/* Exit the RPC thread */
209-
svc_exit_thread(rqstp);
210-
211202
return 0;
212203
}
213204

@@ -254,16 +245,15 @@ int
254245
lockd_up(int proto) /* Maybe add a 'family' option when IPv6 is supported ?? */
255246
{
256247
struct svc_serv *serv;
257-
struct svc_rqst *rqstp;
258248
int error = 0;
259249

260250
mutex_lock(&nlmsvc_mutex);
261251
/*
262252
* Check whether we're already up and running.
263253
*/
264-
if (nlmsvc_serv) {
254+
if (nlmsvc_rqst) {
265255
if (proto)
266-
error = make_socks(nlmsvc_serv, proto);
256+
error = make_socks(nlmsvc_rqst->rq_server, proto);
267257
goto out;
268258
}
269259

@@ -288,26 +278,26 @@ lockd_up(int proto) /* Maybe add a 'family' option when IPv6 is supported ?? */
288278
/*
289279
* Create the kernel thread and wait for it to start.
290280
*/
291-
rqstp = svc_prepare_thread(serv, &serv->sv_pools[0]);
292-
if (IS_ERR(rqstp)) {
293-
error = PTR_ERR(rqstp);
281+
nlmsvc_rqst = svc_prepare_thread(serv, &serv->sv_pools[0]);
282+
if (IS_ERR(nlmsvc_rqst)) {
283+
error = PTR_ERR(nlmsvc_rqst);
284+
nlmsvc_rqst = NULL;
294285
printk(KERN_WARNING
295286
"lockd_up: svc_rqst allocation failed, error=%d\n",
296287
error);
297288
goto destroy_and_out;
298289
}
299290

300291
svc_sock_update_bufs(serv);
301-
nlmsvc_serv = rqstp->rq_server;
302292

303-
nlmsvc_task = kthread_run(lockd, rqstp, serv->sv_name);
293+
nlmsvc_task = kthread_run(lockd, nlmsvc_rqst, serv->sv_name);
304294
if (IS_ERR(nlmsvc_task)) {
305295
error = PTR_ERR(nlmsvc_task);
296+
svc_exit_thread(nlmsvc_rqst);
306297
nlmsvc_task = NULL;
307-
nlmsvc_serv = NULL;
298+
nlmsvc_rqst = NULL;
308299
printk(KERN_WARNING
309300
"lockd_up: kthread_run failed, error=%d\n", error);
310-
svc_exit_thread(rqstp);
311301
goto destroy_and_out;
312302
}
313303

@@ -346,6 +336,9 @@ lockd_down(void)
346336
BUG();
347337
}
348338
kthread_stop(nlmsvc_task);
339+
svc_exit_thread(nlmsvc_rqst);
340+
nlmsvc_task = NULL;
341+
nlmsvc_rqst = NULL;
349342
out:
350343
mutex_unlock(&nlmsvc_mutex);
351344
}

‎fs/lockd/svc4proc.c

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,7 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
5858
return 0;
5959

6060
no_locks:
61-
if (host)
62-
nlm_release_host(host);
61+
nlm_release_host(host);
6362
if (error)
6463
return error;
6564
return nlm_lck_denied_nolocks;
@@ -100,7 +99,7 @@ nlm4svc_proc_test(struct svc_rqst *rqstp, struct nlm_args *argp,
10099
return resp->status == nlm_drop_reply ? rpc_drop_reply :rpc_success;
101100

102101
/* Now check for conflicting locks */
103-
resp->status = nlmsvc_testlock(rqstp, file, &argp->lock, &resp->lock, &resp->cookie);
102+
resp->status = nlmsvc_testlock(rqstp, file, host, &argp->lock, &resp->lock, &resp->cookie);
104103
if (resp->status == nlm_drop_reply)
105104
rc = rpc_drop_reply;
106105
else
@@ -146,7 +145,7 @@ nlm4svc_proc_lock(struct svc_rqst *rqstp, struct nlm_args *argp,
146145
#endif
147146

148147
/* Now try to lock the file */
149-
resp->status = nlmsvc_lock(rqstp, file, &argp->lock,
148+
resp->status = nlmsvc_lock(rqstp, file, host, &argp->lock,
150149
argp->block, &argp->cookie);
151150
if (resp->status == nlm_drop_reply)
152151
rc = rpc_drop_reply;

0 commit comments

Comments
 (0)