Skip to content

shred: detected block size differs from GNU #7870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
BenWiederhake opened this issue May 1, 2025 · 5 comments
Open

shred: detected block size differs from GNU #7870

BenWiederhake opened this issue May 1, 2025 · 5 comments
Labels
U - shred

Comments

@BenWiederhake
Copy link
Collaborator

BenWiederhake commented May 1, 2025 •
edited
Loading

uutils du uses a different block size than GNU du, which leads to files of different sizes:

$ echo a > foo && shred -v -n1 foo && du -b foo
shred: foo: pass 1/1 (random)...
4096	foo
$ echo a > foo && cargo run -q --features shred shred -v -n1 foo && du -b foo
shred: foo: pass 1/1 (random)...
65536	foo

I'm not convinced that this is fine, but there's a good chance. If you, dear reader, know for certain that this is fine, please add a paragraph to https://uutils.github.io/coreutils/docs/extensions.html#shred .

Found while trying to solve #5711.

EDIT: Also, this comment (and perhaps code) in shred.rs needs to be updated:

// This block size seems to match GNU (2^16 = 65536)
const BLOCK_SIZE: usize = 1 << 16;
@alexs-sh
Copy link
Contributor

alexs-sh commented May 5, 2025 •
edited
Loading

@BenWiederhake Good catch. Thanks!

Digging a bit into GNU coreutils, gnulib, and glibc shows that the default block size shouldn't be a constant. The most common values are 512 bytes and 4K bytes, which seem to be chosen to align with default block or page sizes.

The size may depend on both runtime and compile-time factors. For example, GNU Coreutils uses fstat along with the STP_BLKSIZE macro to determine the block size. This macro can either resolve to a constant or depend on the struct stat argument. The macro also could depend on glibc/musl/... over DEV_BSIZE definition(default, glibc).

I'll try using a 4K constant just to experiment with different block sizes, observe how the tests behave, and gather some data. (See PR #7886)

But it seems that the more accurate approach would be to use fstat to retrieve the block size, as this should yield results that are closer to native GNU Coreutils behavior. I'll try this approach as well.

@alexs-sh
Copy link
Contributor

alexs-sh commented May 5, 2025

@tertsdiepraam Hello

I noticed your commit that changed the block size. If you remember any details about why 65K was chosen, that would be helpful and interesting to know. Any additional thoughts or comments would also be appreciated.

Thank you in advance

@tertsdiepraam
Copy link
Member

Sorry, I don't remember. That's a long time ago 😄 Feel free to change it

alexs-sh added a commit to alexs-sh/coreutils that referenced this issue May 5, 2025
@alexs-sh
shred: use 4K block
b5dd16f 
This commit changes the default block size for the shred utility to make
it more consistent with the GNU coreutils version.

Issue uutils#7870
@alexs-sh
Copy link
Contributor

alexs-sh commented May 5, 2025

OK. Thx for the fast response 😄

@alexs-sh alexs-sh mentioned this issue May 5, 2025
Open
@jtracey
Copy link
Contributor

jtracey commented May 5, 2025

Please don't look at the GNU coreutils source code or link it here, the license isn't compatible. See the warning in CONTRIBUTING.md for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
U - shred
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@BenWiederhake @jtracey @tertsdiepraam @alexs-sh