libFuzzer freezes when ulimit max file sets to a high value (happens in docker environment) #156936
Description
Recently I was trying to test some fuzzers. The build environment requires a lot of dependencies so I decided to go with a docker setup. Everything worked fine until I ran the fuzzer: the moment it prints the newly discovered function names, or a crash happens, it hangs there almost forever.
After spending the whole afternoon debugging the issue, I found the following facts:
- docker sets the "max open files" ulimit to a really high value (1073741816 on my setup)
- libfuzzer tries to close all the possible file handles when a subprocess exits.
- when printing discovered function names, or a crash happens, a subprocess is exiting.
All three together causes the freeze behavior -- it's not freeze but rather waiting for the subprocess to call close syscall for 1073741816 times. After manually change the "max open file" value by running "ulimit -n" the issue disappears.
Here is strace output when the hang happens:
925 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f71a8045000
925 munmap(0x7f71a8045000, 4096) = 0
925 munmap(0x7f71a8046000, 16384) = 0
925 stat("/usr/bin/llvm-symbolizer-14", {st_mode=S_IFREG|0755, st_size=56104, ...}) = 0
925 pipe2([4, 5], 0) = 0
925 pipe2([6, 7], 0) = 0
925 fork() = 986
925 close(6) = 0
925 close(5 <unfinished ...>
986 close(0 <unfinished ...>
925 <... close resumed>) = 0
986 <... close resumed>) = 0
925 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
986 dup2(6, 0) = 0
986 close(6) = 0
986 close(1) = 0
986 dup2(5, 1) = 1
986 close(5) = 0
986 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0
986 close(1073741816) = -1 EBADF (Bad file descriptor)
986 close(1073741815) = -1 EBADF (Bad file descriptor)
986 close(1073741814) = -1 EBADF (Bad file descriptor)
986 close(1073741813) = -1 EBADF (Bad file descriptor)
986 close(1073741812) = -1 EBADF (Bad file descriptor)
986 close(1073741811) = -1 EBADF (Bad file descriptor)
986 close(1073741810) = -1 EBADF (Bad file descriptor)
The offending code is here:
# if SANITIZER_FREEBSD
internal_close_range(3, ~static_cast<fd_t>(0), 0);
# else
for (int fd = sysconf(_SC_OPEN_MAX); fd > 2; fd--) internal_close(fd); <=== HERE
# endif
It doesn't necessary to be a bug in the libfuzzer runtime code, but some improvement (or at least a warning message) can make this issue less annoying/confusing.