mm/gup.c: teach get_user_pages_unlocked to handle FOLL_NOWAIT

aagit · torvalds · commit 96312e61282a · 2018-03-09T16:40:01.000-08:00
KVM is hanging during postcopy live migration with userfaultfd because get_user_pages_unlocked is not capable to handle FOLL_NOWAIT. Earlier FOLL_NOWAIT was only ever passed to get_user_pages. Specifically faultin_page (the callee of get_user_pages_unlocked caller) doesn't know that if FAULT_FLAG_RETRY_NOWAIT was set in the page fault flags, when VM_FAULT_RETRY is returned, the mmap_sem wasn't actually released (even if nonblocking is not NULL). So it sets *nonblocking to zero and the caller won't release the mmap_sem thinking it was already released, but it wasn't because of FOLL_NOWAIT. Link: http://lkml.kernel.org/r/20180302174343.5421-2-aarcange@redhat.com Fixes: ce53053 ("kvm: switch get_user_page_nowait() to get_user_pages_unlocked()") Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> Reported-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Tested-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
diff --git a/mm/gup.c b/mm/gup.c
@@ -516,7 +516,7 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	}
 
 	if (ret & VM_FAULT_RETRY) {
-		if (nonblocking)
+		if (nonblocking && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
 			*nonblocking = 0;
 		return -EBUSY;
 	}
@@ -890,7 +890,10 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
 				break;
 		}
 		if (*locked) {
-			/* VM_FAULT_RETRY didn't trigger */
+			/*
+			 * VM_FAULT_RETRY didn't trigger or it was a
+			 * FOLL_NOWAIT.
+			 */
 			if (!pages_done)
 				pages_done = ret;
 			break;

Original file line number	Diff line number	Diff line change
`@@ -516,7 +516,7 @@ static int faultin_page(struct task_struct tsk, struct vm_area_struct vma,`
`516`	`516`	`}`
`517`	`517`
`518`	`518`	`if (ret & VM_FAULT_RETRY) {`
`519`		`- if (nonblocking)`
	`519`	`+ if (nonblocking && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))`
`520`	`520`	`*nonblocking = 0;`
`521`	`521`	`return -EBUSY;`
`522`	`522`	`}`
`@@ -890,7 +890,10 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,`
`890`	`890`	`break;`
`891`	`891`	`}`
`892`	`892`	`if (*locked) {`
`893`		`- /* VM_FAULT_RETRY didn't trigger */`
	`893`	`+ /*`
	`894`	`+ * VM_FAULT_RETRY didn't trigger or it was a`
	`895`	`+ * FOLL_NOWAIT.`
	`896`	`+ */`
`894`	`897`	`if (!pages_done)`
`895`	`898`	`pages_done = ret;`
`896`	`899`	`break;`