Skip to content

Commit 8adc048

Browse files
torvaldstorvalds
committed
Merge tag 'random-6.1-rc1-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/crng/random
Pull random number generator updates from Jason Donenfeld: - Huawei reported that when they updated their kernel from 4.4 to something much newer, some userspace code they had broke, the culprit being the accidental removal of O_NONBLOCK from /dev/random way back in 5.6. It's been gone for over 2 years now and this is the first we've heard of it, but userspace breakage is userspace breakage, so O_NONBLOCK is now back. - Use randomness from hardware RNGs much more often during early boot, at the same interval that crng reseeds are done, from Dominik. - A semantic change in hardware RNG throttling, so that the hwrng framework can properly feed random.c with randomness from hardware RNGs that aren't specifically marked as creditable. A related patch coming to you via Herbert's hwrng tree depends on this one, not to compile, but just to function properly, so you may want to merge this PULL before that one. - A fix to clamp credited bits from the interrupts pool to the size of the pool sample. This is mainly just a theoretical fix, as it'd be pretty hard to exceed it in practice. - Oracle reported that InfiniBand TCP latency regressed by around 10-15% after a change a few cycles ago made at the request of the RT folks, in which we hoisted a somewhat rare operation (1 in 1024 times) out of the hard IRQ handler and into a workqueue, a pretty common and boring pattern. It turns out, though, that scheduling a worker from there has overhead of its own, whereas scheduling a timer on that same CPU for the next jiffy amortizes better and doesn't incur the same overhead. I also eliminated a cache miss by moving the work_struct (and subsequently, the timer_list) to below a critical cache line, so that the more critical members that are accessed on every hard IRQ aren't split between two cache lines. - The boot-time initialization of the RNG has been split into two approximate phases: what we can accomplish before timekeeping is possible and what we can accomplish after. This winds up being useful so that we can use RDRAND to seed the RNG before CONFIG_SLAB_FREELIST_RANDOM=y systems initialize slabs, in addition to other early uses of randomness. The effect is that systems with RDRAND (or a bootloader seed) will never see any warnings at all when setting CONFIG_WARN_ALL_UNSEEDED_RANDOM=y. And kfence benefits from getting a better seed of its own. - Small systems without much entropy sometimes wind up putting some truncated serial number read from flash into hostname, so contribute utsname changes to the RNG, without crediting. - Add smaller batches to serve requests for smaller integers, and make use of them when people ask for random numbers bounded by a given compile-time constant. This has positive effects all over the tree, most notably in networking and kfence. - The original jitter algorithm intended (I believe) to schedule the timer for the next jiffy, not the next-next jiffy, yet it used mod_timer(jiffies + 1), which will fire on the next-next jiffy, instead of what I believe was intended, mod_timer(jiffies), which will fire on the next jiffy. So fix that. - Fix a comment typo, from William. * tag 'random-6.1-rc1-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/crng/random: random: clear new batches when bringing new CPUs online random: fix typos in get_random_bytes() comment random: schedule jitter credit for next jiffy, not in two jiffies prandom: make use of smaller types in prandom_u32_max random: add 8-bit and 16-bit batches utsname: contribute changes to RNG random: use init_utsname() instead of utsname() kfence: use better stack hash seed random: split initialization into early step and later step random: use expired timer rather than wq for mixing fast pool random: avoid reading two cache lines on irq randomness random: clamp credited irq bits to maximum mixed random: throttle hwrng writes if no entropy is credited random: use hwgenerator randomness more frequently at early boot random: restore O_NONBLOCK support
2 parents 52abb27 + a890d1c commit 8adc048

File tree

8 files changed

+114
-77
lines changed

8 files changed

+114
-77
lines changed

‎drivers/char/mem.c

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -712,8 +712,8 @@ static const struct memdev {
712712
#endif
713713
[5] = { "zero", 0666, &zero_fops, FMODE_NOWAIT },
714714
[7] = { "full", 0666, &full_fops, 0 },
715-
[8] = { "random", 0666, &random_fops, 0 },
716-
[9] = { "urandom", 0666, &urandom_fops, 0 },
715+
[8] = { "random", 0666, &random_fops, FMODE_NOWAIT },
716+
[9] = { "urandom", 0666, &urandom_fops, FMODE_NOWAIT },
717717
#ifdef CONFIG_PRINTK
718718
[11] = { "kmsg", 0644, &kmsg_fops, 0 },
719719
#endif

‎drivers/char/random.c

Lines changed: 81 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -96,8 +96,8 @@ MODULE_PARM_DESC(ratelimit_disable, "Disable random ratelimit suppression");
9696
/*
9797
* Returns whether or not the input pool has been seeded and thus guaranteed
9898
* to supply cryptographically secure random numbers. This applies to: the
99-
* /dev/urandom device, the get_random_bytes function, and the get_random_{u32,
100-
* ,u64,int,long} family of functions.
99+
* /dev/urandom device, the get_random_bytes function, and the get_random_{u8,
100+
* u16,u32,u64,int,long} family of functions.
101101
*
102102
* Returns: true if the input pool has been seeded.
103103
* false if the input pool has not been seeded.
@@ -119,9 +119,9 @@ static void try_to_generate_entropy(void);
119119
/*
120120
* Wait for the input pool to be seeded and thus guaranteed to supply
121121
* cryptographically secure random numbers. This applies to: the /dev/urandom
122-
* device, the get_random_bytes function, and the get_random_{u32,u64,int,long}
123-
* family of functions. Using any of these functions without first calling
124-
* this function forfeits the guarantee of security.
122+
* device, the get_random_bytes function, and the get_random_{u8,u16,u32,u64,
123+
* int,long} family of functions. Using any of these functions without first
124+
* calling this function forfeits the guarantee of security.
125125
*
126126
* Returns: 0 if the input pool has been seeded.
127127
* -ERESTARTSYS if the function was interrupted by a signal.
@@ -157,17 +157,19 @@ EXPORT_SYMBOL(wait_for_random_bytes);
157157
* There are a few exported interfaces for use by other drivers:
158158
*
159159
* void get_random_bytes(void *buf, size_t len)
160+
* u8 get_random_u8()
161+
* u16 get_random_u16()
160162
* u32 get_random_u32()
161163
* u64 get_random_u64()
162164
* unsigned int get_random_int()
163165
* unsigned long get_random_long()
164166
*
165167
* These interfaces will return the requested number of random bytes
166168
* into the given buffer or as a return value. This is equivalent to
167-
* a read from /dev/urandom. The u32, u64, int, and long family of
168-
* functions may be higher performance for one-off random integers,
169-
* because they do a bit of buffering and do not invoke reseeding
170-
* until the buffer is emptied.
169+
* a read from /dev/urandom. The u8, u16, u32, u64, int, and long
170+
* family of functions may be higher performance for one-off random
171+
* integers, because they do a bit of buffering and do not invoke
172+
* reseeding until the buffer is emptied.
171173
*
172174
*********************************************************************/
173175

@@ -260,25 +262,23 @@ static void crng_fast_key_erasure(u8 key[CHACHA_KEY_SIZE],
260262
}
261263

262264
/*
263-
* Return whether the crng seed is considered to be sufficiently old
264-
* that a reseeding is needed. This happens if the last reseeding
265-
* was CRNG_RESEED_INTERVAL ago, or during early boot, at an interval
265+
* Return the interval until the next reseeding, which is normally
266+
* CRNG_RESEED_INTERVAL, but during early boot, it is at an interval
266267
* proportional to the uptime.
267268
*/
268-
static bool crng_has_old_seed(void)
269+
static unsigned int crng_reseed_interval(void)
269270
{
270271
static bool early_boot = true;
271-
unsigned long interval = CRNG_RESEED_INTERVAL;
272272

273273
if (unlikely(READ_ONCE(early_boot))) {
274274
time64_t uptime = ktime_get_seconds();
275275
if (uptime >= CRNG_RESEED_INTERVAL / HZ * 2)
276276
WRITE_ONCE(early_boot, false);
277277
else
278-
interval = max_t(unsigned int, CRNG_RESEED_START_INTERVAL,
279-
(unsigned int)uptime / 2 * HZ);
278+
return max_t(unsigned int, CRNG_RESEED_START_INTERVAL,
279+
(unsigned int)uptime / 2 * HZ);
280280
}
281-
return time_is_before_jiffies(READ_ONCE(base_crng.birth) + interval);
281+
return CRNG_RESEED_INTERVAL;
282282
}
283283

284284
/*
@@ -320,7 +320,7 @@ static void crng_make_state(u32 chacha_state[CHACHA_STATE_WORDS],
320320
* If the base_crng is old enough, we reseed, which in turn bumps the
321321
* generation counter that we check below.
322322
*/
323-
if (unlikely(crng_has_old_seed()))
323+
if (unlikely(time_is_before_jiffies(READ_ONCE(base_crng.birth) + crng_reseed_interval())))
324324
crng_reseed();
325325

326326
local_lock_irqsave(&crngs.lock, flags);
@@ -384,11 +384,11 @@ static void _get_random_bytes(void *buf, size_t len)
384384
}
385385

386386
/*
387-
* This function is the exported kernel interface. It returns some
388-
* number of good random numbers, suitable for key generation, seeding
389-
* TCP sequence numbers, etc. In order to ensure that the randomness
390-
* by this function is okay, the function wait_for_random_bytes()
391-
* should be called and return 0 at least once at any point prior.
387+
* This function is the exported kernel interface. It returns some number of
388+
* good random numbers, suitable for key generation, seeding TCP sequence
389+
* numbers, etc. In order to ensure that the randomness returned by this
390+
* function is okay, the function wait_for_random_bytes() should be called and
391+
* return 0 at least once at any point prior.
392392
*/
393393
void get_random_bytes(void *buf, size_t len)
394394
{
@@ -506,8 +506,10 @@ type get_random_ ##type(void) \
506506
} \
507507
EXPORT_SYMBOL(get_random_ ##type);
508508

509-
DEFINE_BATCHED_ENTROPY(u64)
509+
DEFINE_BATCHED_ENTROPY(u8)
510+
DEFINE_BATCHED_ENTROPY(u16)
510511
DEFINE_BATCHED_ENTROPY(u32)
512+
DEFINE_BATCHED_ENTROPY(u64)
511513

512514
#ifdef CONFIG_SMP
513515
/*
@@ -522,6 +524,8 @@ int __cold random_prepare_cpu(unsigned int cpu)
522524
* randomness.
523525
*/
524526
per_cpu_ptr(&crngs, cpu)->generation = ULONG_MAX;
527+
per_cpu_ptr(&batched_entropy_u8, cpu)->position = UINT_MAX;
528+
per_cpu_ptr(&batched_entropy_u16, cpu)->position = UINT_MAX;
525529
per_cpu_ptr(&batched_entropy_u32, cpu)->position = UINT_MAX;
526530
per_cpu_ptr(&batched_entropy_u64, cpu)->position = UINT_MAX;
527531
return 0;
@@ -774,18 +778,13 @@ static int random_pm_notification(struct notifier_block *nb, unsigned long actio
774778
static struct notifier_block pm_notifier = { .notifier_call = random_pm_notification };
775779

776780
/*
777-
* The first collection of entropy occurs at system boot while interrupts
778-
* are still turned off. Here we push in latent entropy, RDSEED, a timestamp,
779-
* utsname(), and the command line. Depending on the above configuration knob,
780-
* RDSEED may be considered sufficient for initialization. Note that much
781-
* earlier setup may already have pushed entropy into the input pool by the
782-
* time we get here.
781+
* This is called extremely early, before time keeping functionality is
782+
* available, but arch randomness is. Interrupts are not yet enabled.
783783
*/
784-
int __init random_init(const char *command_line)
784+
void __init random_init_early(const char *command_line)
785785
{
786-
ktime_t now = ktime_get_real();
787-
size_t i, longs, arch_bits;
788786
unsigned long entropy[BLAKE2S_BLOCK_SIZE / sizeof(long)];
787+
size_t i, longs, arch_bits;
789788

790789
#if defined(LATENT_ENTROPY_PLUGIN)
791790
static const u8 compiletime_seed[BLAKE2S_BLOCK_SIZE] __initconst __latent_entropy;
@@ -805,34 +804,49 @@ int __init random_init(const char *command_line)
805804
i += longs;
806805
continue;
807806
}
808-
entropy[0] = random_get_entropy();
809-
_mix_pool_bytes(entropy, sizeof(*entropy));
810807
arch_bits -= sizeof(*entropy) * 8;
811808
++i;
812809
}
813-
_mix_pool_bytes(&now, sizeof(now));
814-
_mix_pool_bytes(utsname(), sizeof(*(utsname())));
810+
811+
_mix_pool_bytes(init_utsname(), sizeof(*(init_utsname())));
815812
_mix_pool_bytes(command_line, strlen(command_line));
813+
814+
/* Reseed if already seeded by earlier phases. */
815+
if (crng_ready())
816+
crng_reseed();
817+
else if (trust_cpu)
818+
_credit_init_bits(arch_bits);
819+
}
820+
821+
/*
822+
* This is called a little bit after the prior function, and now there is
823+
* access to timestamps counters. Interrupts are not yet enabled.
824+
*/
825+
void __init random_init(void)
826+
{
827+
unsigned long entropy = random_get_entropy();
828+
ktime_t now = ktime_get_real();
829+
830+
_mix_pool_bytes(&now, sizeof(now));
831+
_mix_pool_bytes(&entropy, sizeof(entropy));
816832
add_latent_entropy();
817833

818834
/*
819-
* If we were initialized by the bootloader before jump labels are
820-
* initialized, then we should enable the static branch here, where
835+
* If we were initialized by the cpu or bootloader before jump labels
836+
* are initialized, then we should enable the static branch here, where
821837
* it's guaranteed that jump labels have been initialized.
822838
*/
823839
if (!static_branch_likely(&crng_is_ready) && crng_init >= CRNG_READY)
824840
crng_set_ready(NULL);
825841

842+
/* Reseed if already seeded by earlier phases. */
826843
if (crng_ready())
827844
crng_reseed();
828-
else if (trust_cpu)
829-
_credit_init_bits(arch_bits);
830845

831846
WARN_ON(register_pm_notifier(&pm_notifier));
832847

833-
WARN(!random_get_entropy(), "Missing cycle counter and fallback timer; RNG "
834-
"entropy collection will consequently suffer.");
835-
return 0;
848+
WARN(!entropy, "Missing cycle counter and fallback timer; RNG "
849+
"entropy collection will consequently suffer.");
836850
}
837851

838852
/*
@@ -866,11 +880,11 @@ void add_hwgenerator_randomness(const void *buf, size_t len, size_t entropy)
866880
credit_init_bits(entropy);
867881

868882
/*
869-
* Throttle writing to once every CRNG_RESEED_INTERVAL, unless
870-
* we're not yet initialized.
883+
* Throttle writing to once every reseed interval, unless we're not yet
884+
* initialized or no entropy is credited.
871885
*/
872-
if (!kthread_should_stop() && crng_ready())
873-
schedule_timeout_interruptible(CRNG_RESEED_INTERVAL);
886+
if (!kthread_should_stop() && (crng_ready() || !entropy))
887+
schedule_timeout_interruptible(crng_reseed_interval());
874888
}
875889
EXPORT_SYMBOL_GPL(add_hwgenerator_randomness);
876890

@@ -920,20 +934,23 @@ EXPORT_SYMBOL_GPL(unregister_random_vmfork_notifier);
920934
#endif
921935

922936
struct fast_pool {
923-
struct work_struct mix;
924937
unsigned long pool[4];
925938
unsigned long last;
926939
unsigned int count;
940+
struct timer_list mix;
927941
};
928942

943+
static void mix_interrupt_randomness(struct timer_list *work);
944+
929945
static DEFINE_PER_CPU(struct fast_pool, irq_randomness) = {
930946
#ifdef CONFIG_64BIT
931947
#define FASTMIX_PERM SIPHASH_PERMUTATION
932-
.pool = { SIPHASH_CONST_0, SIPHASH_CONST_1, SIPHASH_CONST_2, SIPHASH_CONST_3 }
948+
.pool = { SIPHASH_CONST_0, SIPHASH_CONST_1, SIPHASH_CONST_2, SIPHASH_CONST_3 },
933949
#else
934950
#define FASTMIX_PERM HSIPHASH_PERMUTATION
935-
.pool = { HSIPHASH_CONST_0, HSIPHASH_CONST_1, HSIPHASH_CONST_2, HSIPHASH_CONST_3 }
951+
.pool = { HSIPHASH_CONST_0, HSIPHASH_CONST_1, HSIPHASH_CONST_2, HSIPHASH_CONST_3 },
936952
#endif
953+
.mix = __TIMER_INITIALIZER(mix_interrupt_randomness, 0)
937954
};
938955

939956
/*
@@ -975,7 +992,7 @@ int __cold random_online_cpu(unsigned int cpu)
975992
}
976993
#endif
977994

978-
static void mix_interrupt_randomness(struct work_struct *work)
995+
static void mix_interrupt_randomness(struct timer_list *work)
979996
{
980997
struct fast_pool *fast_pool = container_of(work, struct fast_pool, mix);
981998
/*
@@ -1006,7 +1023,7 @@ static void mix_interrupt_randomness(struct work_struct *work)
10061023
local_irq_enable();
10071024

10081025
mix_pool_bytes(pool, sizeof(pool));
1009-
credit_init_bits(max(1u, (count & U16_MAX) / 64));
1026+
credit_init_bits(clamp_t(unsigned int, (count & U16_MAX) / 64, 1, sizeof(pool) * 8));
10101027

10111028
memzero_explicit(pool, sizeof(pool));
10121029
}
@@ -1029,10 +1046,11 @@ void add_interrupt_randomness(int irq)
10291046
if (new_count < 1024 && !time_is_before_jiffies(fast_pool->last + HZ))
10301047
return;
10311048

1032-
if (unlikely(!fast_pool->mix.func))
1033-
INIT_WORK(&fast_pool->mix, mix_interrupt_randomness);
10341049
fast_pool->count |= MIX_INFLIGHT;
1035-
queue_work_on(raw_smp_processor_id(), system_highpri_wq, &fast_pool->mix);
1050+
if (!timer_pending(&fast_pool->mix)) {
1051+
fast_pool->mix.expires = jiffies;
1052+
add_timer_on(&fast_pool->mix, raw_smp_processor_id());
1053+
}
10361054
}
10371055
EXPORT_SYMBOL_GPL(add_interrupt_randomness);
10381056

@@ -1191,7 +1209,7 @@ static void __cold entropy_timer(struct timer_list *timer)
11911209
*/
11921210
static void __cold try_to_generate_entropy(void)
11931211
{
1194-
enum { NUM_TRIAL_SAMPLES = 8192, MAX_SAMPLES_PER_BIT = HZ / 30 };
1212+
enum { NUM_TRIAL_SAMPLES = 8192, MAX_SAMPLES_PER_BIT = HZ / 15 };
11951213
struct entropy_timer_state stack;
11961214
unsigned int i, num_different = 0;
11971215
unsigned long last = random_get_entropy();
@@ -1210,7 +1228,7 @@ static void __cold try_to_generate_entropy(void)
12101228
timer_setup_on_stack(&stack.timer, entropy_timer, 0);
12111229
while (!crng_ready() && !signal_pending(current)) {
12121230
if (!timer_pending(&stack.timer))
1213-
mod_timer(&stack.timer, jiffies + 1);
1231+
mod_timer(&stack.timer, jiffies);
12141232
mix_pool_bytes(&stack.entropy, sizeof(stack.entropy));
12151233
schedule();
12161234
stack.entropy = random_get_entropy();
@@ -1347,6 +1365,11 @@ static ssize_t random_read_iter(struct kiocb *kiocb, struct iov_iter *iter)
13471365
{
13481366
int ret;
13491367

1368+
if (!crng_ready() &&
1369+
((kiocb->ki_flags & (IOCB_NOWAIT | IOCB_NOIO)) ||
1370+
(kiocb->ki_filp->f_flags & O_NONBLOCK)))
1371+
return -EAGAIN;
1372+
13501373
ret = wait_for_random_bytes();
13511374
if (ret != 0)
13521375
return ret;

‎include/linux/prandom.h

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,13 @@
1212
#include <linux/percpu.h>
1313
#include <linux/random.h>
1414

15+
/* Deprecated: use get_random_u32 instead. */
1516
static inline u32 prandom_u32(void)
1617
{
1718
return get_random_u32();
1819
}
1920

21+
/* Deprecated: use get_random_bytes instead. */
2022
static inline void prandom_bytes(void *buf, size_t nbytes)
2123
{
2224
return get_random_bytes(buf, nbytes);
@@ -37,17 +39,20 @@ void prandom_seed_full_state(struct rnd_state __percpu *pcpu_state);
3739
* prandom_u32_max - returns a pseudo-random number in interval [0, ep_ro)
3840
* @ep_ro: right open interval endpoint
3941
*
40-
* Returns a pseudo-random number that is in interval [0, ep_ro). Note
41-
* that the result depends on PRNG being well distributed in [0, ~0U]
42-
* u32 space. Here we use maximally equidistributed combined Tausworthe
43-
* generator, that is, prandom_u32(). This is useful when requesting a
44-
* random index of an array containing ep_ro elements, for example.
42+
* Returns a pseudo-random number that is in interval [0, ep_ro). This is
43+
* useful when requesting a random index of an array containing ep_ro elements,
44+
* for example. The result is somewhat biased when ep_ro is not a power of 2,
45+
* so do not use this for cryptographic purposes.
4546
*
4647
* Returns: pseudo-random number in interval [0, ep_ro)
4748
*/
4849
static inline u32 prandom_u32_max(u32 ep_ro)
4950
{
50-
return (u32)(((u64) prandom_u32() * ep_ro) >> 32);
51+
if (__builtin_constant_p(ep_ro <= 1U << 8) && ep_ro <= 1U << 8)
52+
return (get_random_u8() * ep_ro) >> 8;
53+
if (__builtin_constant_p(ep_ro <= 1U << 16) && ep_ro <= 1U << 16)
54+
return (get_random_u16() * ep_ro) >> 16;
55+
return ((u64)get_random_u32() * ep_ro) >> 32;
5156
}
5257

5358
/*

‎include/linux/random.h

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ static inline int unregister_random_vmfork_notifier(struct notifier_block *nb) {
3838
#endif
3939

4040
void get_random_bytes(void *buf, size_t len);
41+
u8 get_random_u8(void);
42+
u16 get_random_u16(void);
4143
u32 get_random_u32(void);
4244
u64 get_random_u64(void);
4345
static inline unsigned int get_random_int(void)
@@ -72,7 +74,8 @@ static inline unsigned long get_random_canary(void)
7274
return get_random_long() & CANARY_MASK;
7375
}
7476

75-
int __init random_init(const char *command_line);
77+
void __init random_init_early(const char *command_line);
78+
void __init random_init(void);
7679
bool rng_is_initialized(void);
7780
int wait_for_random_bytes(void);
7881

@@ -93,6 +96,8 @@ static inline int get_random_bytes_wait(void *buf, size_t nbytes)
9396
*out = get_random_ ## name(); \
9497
return 0; \
9598
}
99+
declare_get_random_var_wait(u8, u8)
100+
declare_get_random_var_wait(u16, u16)
96101
declare_get_random_var_wait(u32, u32)
97102
declare_get_random_var_wait(u64, u32)
98103
declare_get_random_var_wait(int, unsigned int)

0 commit comments

Comments
 (0)