[suggest][performance optimization] using file reading ahead to accelerate the file reading speed #770
bamboolight
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
In my case, the computer is run in a virtul machine, and the hard disk(SSD) reading speed limit is 400MB/s.
I use rsync to send a large number of small files(the number of files is 202111) to other computer.
without reading ahead files, the disk reading speed is under 50 MB/s.
I modify the rsync source code, use the posis api posix_fadvise(https://www.man7.org/linux/man-pages/man2/posix_fadvise.2.html) to read ahead files in batches.
posix_fadvise will read file to system page cache,then rsync can read file from page cache。
Test result shows that the disk reading speed is near 400 MB/s,and it significantly reduce the time for file reading.
The optimized rsync run in production environment for several months steadily, without any crash.
Because the company‘s rules, I can not share the optimized source code to github. so i share this idea to all.
Beta Was this translation helpful? Give feedback.
All reactions