• Add a --random-seed option that allows using a string as a seed for deterministic output. For example: shuf --random-seed=🦀 -i1-10.

  This comes with a forward compatibility commitment: we have to keep using SHA-3 and ChaCha12 and the current (in-tree) algorithms whenever --random-seed is passed. Details and motivation:

  coreutils/src/uu/shuf/src/random_seed.rs

  Lines 12 to 61 in e89aaf8

  	/// Reproducible seeded random number generation.
  	///
  	/// The behavior should stay the same between releases, so don't change it without
  	/// a very good reason.
  	///
  	/// # How it works
  	///
  	/// - Take a Unicode string as the seed.
  	///
  	/// - Encode this seed as UTF-8.
  	///
  	/// - Take the SHA3-256 hash of the encoded seed.
  	///
  	/// - Use that hash as the input for a [`rand_chacha`] ChaCha12 RNG.
  	/// (We don't touch the nonce, so that's probably zero.)
  	///
  	/// - Take 64-bit samples from the RNG.
  	///
  	/// - Use Lemire's method to generate uniformly distributed integers and:
  	///
  	/// - With --repeat, use these to pick elements from ranges.
  	///
  	/// - Without --repeat, use these to do left-to-right modern Fisher-Yates.
  	///
  	/// # Why it works like this
  	///
  	/// - Unicode string: Greatest common denominator between platforms. Windows doesn't
  	/// let you pass raw bytes as a CLI argument and that would be bad practice anyway.
  	/// A decimal or hex number would work but this is much more flexible without being
  	/// unmanageable.
  	///
  	/// (Footgun: if the user passes a filename we won't read from the file but the
  	/// command will run anyway.)
  	///
  	/// - UTF-8: That's what Rust likes and it's the least unreasonable Unicode encoding.
  	///
  	/// - SHA3-256: We want to make good use of the entire user input and SHA-3 is
  	/// state of the art. ChaCha12 takes a 256-bit seed.
  	///
  	/// - ChaCha12: [`rand`]'s default rng as of writing. Seems state of the art.
  	///
  	/// - 64-bit samples: We could often get away with 32-bit samples but let's keep things
  	/// simple and only use one width. (There doesn't seem to be much of a performance hit.)
  	///
  	/// - Lemire, Fisher-Yates: These are very easy to implement and maintain ourselves.
  	/// `rand` provides fancier implementations but only promises reproducibility within
  	/// patch releases: https://rust-random.github.io/book/crate-reprod.html
  	///
  	/// Strictly speaking even `ChaCha12` is subject to breakage. But since it's a very
  	/// specific algorithm I assume it's safe in practice.

  Resolves Implementation of shuf --random-source #2528.

  I'd like some input on the design. With this interface it's possible that a user passes a filename as an argument without noticing that it uses the filename itself as the seed rather than the contents of the file. Are we OK with that? I tried to make the help text as clear as possible:

  --random-seed <STRING> seed with STRING for reproducible output

• Make the behavior of --random-source mostly compatible with GNU. I managed to black box reverse engineer their RNG (which is luckily very simple and similar to our old implementation). So now people who use this trick for reproducibility will be able to switch to uutils and still get the old output they're relying on. (Except for one weirdly specific invocation for which I've added a #[ignore] test.)

• Use u64 instead of usize for --input-range and --head-count. It's now possible to generate integers above u32::MAX on 32-bit systems.

• Replace the nonrepeating integer generator. The new one has better reproducibility: it matches GNU and gives identical output to seq | shuf. (The performance is practically the same.)

• Handle --random-source I/O errors properly. They used to cause a panic (which due to buffering quirks would get printed above the output rather than at the end). Now they're reported normally.

• Flush the BufWriter at the end. If we don't do this then write errors get ignored.

• Use a larger write buffer (8K → 64K).

• Use the itoa crate to format integers. With --repeat and --input-range this was a bottleneck.

I also experimented with I/O tricks like vectored writes but they turned out to be too situational, so not a lot of performance work. shuf -r -n10000000 -i0-10 now runs twice as fast though.