Skip to content

Commit b18d628

Browse files
torvaldstorvalds
committed
Merge branch 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 APIC updates from Thomas Gleixner: "This update provides a major overhaul of the APIC initialization and vector allocation code: - Unification of the APIC and interrupt mode setup which was scattered all over the place and was hard to follow. This also distangles the timer setup from the APIC initialization which brings a clear separation of functionality. Great detective work from Dou Lyiang! - Refactoring of the x86 vector allocation mechanism. The existing code was based on nested loops and rather convoluted APIC callbacks which had a horrible worst case behaviour and tried to serve all different use cases in one go. This led to quite odd hacks when supporting the new managed interupt facility for multiqueue devices and made it more or less impossible to deal with the vector space exhaustion which was a major roadblock for server hibernation. Aside of that the code dealing with cpu hotplug and the system vectors was disconnected from the actual vector management and allocation code, which made it hard to follow and maintain. Utilizing the new bitmap matrix allocator core mechanism, the new allocator and management code consolidates the handling of system vectors, legacy vectors, cpu hotplug mechanisms and the actual allocation which needs to be aware of system and legacy vectors and hotplug constraints into a single consistent entity. This has one visible change: The support for multi CPU targets of interrupts, which is only available on a certain subset of CPUs/APIC variants has been removed in favour of single interrupt targets. A proper analysis of the multi CPU target feature revealed that there is no real advantage as the vast majority of interrupts end up on the CPU with the lowest APIC id in the set of target CPUs anyway. That change was agreed on by the relevant folks and allowed to simplify the implementation significantly and to replace rather fragile constructs like the vector cleanup IPI with straight forward and solid code. Furthermore this allowed to cleanly separate the allocation details for legacy, normal and managed interrupts: * Legacy interrupts are not longer wasting 16 vectors unconditionally * Managed interrupts have now a guaranteed vector reservation, but the actual vector assignment happens when the interrupt is requested. It's guaranteed not to fail. * Normal interrupts no longer allocate vectors unconditionally when the interrupt is set up (IO/APIC init or MSI(X) enable). The mechanism has been switched to a best effort reservation mode. The actual allocation happens when the interrupt is requested. Contrary to managed interrupts the request can fail due to vector space exhaustion, but drivers must handle a fail of request_irq() anyway. When the interrupt is freed, the vector is handed back as well. This solves a long standing problem with large unconditional vector allocations for a certain class of enterprise devices which prevented server hibernation due to vector space exhaustion when the unused allocated vectors had to be migrated to CPU0 while unplugging all non boot CPUs. The code has been equipped with trace points and detailed debugfs information to aid analysis of the vector space" * 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (60 commits) x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code genirq: Add config option for reservation mode x86/vector: Use correct per cpu variable in free_moved_vector() x86/apic/vector: Ignore set_affinity call for inactive interrupts x86/apic: Fix spelling mistake: "symmectic" -> "symmetric" x86/apic: Use dead_cpu instead of current CPU when cleaning up ACPI/init: Invoke early ACPI initialization earlier x86/vector: Respect affinity mask in irq descriptor x86/irq: Simplify hotplug vector accounting x86/vector: Switch IOAPIC to global reservation mode x86/vector/msi: Switch to global reservation mode x86/vector: Handle managed interrupts proper x86/io_apic: Reevaluate vector configuration on activate() iommu/amd: Reevaluate vector configuration on activate() iommu/vt-d: Reevaluate vector configuration on activate() x86/apic/msi: Force reactivation of interrupts at startup time x86/vector: Untangle internal state from irq_cfg x86/vector: Compile SMP only code conditionally x86/apic: Remove unused callbacks ...
2 parents 7d58e1c + 141d3b1 commit b18d628

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+1493
-1317
lines changed

‎arch/x86/Kconfig

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,10 @@ config X86
9393
select GENERIC_FIND_FIRST_BIT
9494
select GENERIC_IOMAP
9595
select GENERIC_IRQ_EFFECTIVE_AFF_MASK if SMP
96+
select GENERIC_IRQ_MATRIX_ALLOCATOR if X86_LOCAL_APIC
9697
select GENERIC_IRQ_MIGRATION if SMP
9798
select GENERIC_IRQ_PROBE
99+
select GENERIC_IRQ_RESERVATION_MODE
98100
select GENERIC_IRQ_SHOW
99101
select GENERIC_PENDING_IRQ if SMP
100102
select GENERIC_SMP_IDLE_THREAD

‎arch/x86/include/asm/apic.h

Lines changed: 86 additions & 184 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,15 @@ extern int local_apic_timer_c2_ok;
5353
extern int disable_apic;
5454
extern unsigned int lapic_timer_frequency;
5555

56+
extern enum apic_intr_mode_id apic_intr_mode;
57+
enum apic_intr_mode_id {
58+
APIC_PIC,
59+
APIC_VIRTUAL_WIRE,
60+
APIC_VIRTUAL_WIRE_NO_CONFIG,
61+
APIC_SYMMETRIC_IO,
62+
APIC_SYMMETRIC_IO_NO_ROUTING
63+
};
64+
5665
#ifdef CONFIG_SMP
5766
extern void __inquire_remote_apic(int apicid);
5867
#else /* CONFIG_SMP */
@@ -127,14 +136,13 @@ extern void disconnect_bsp_APIC(int virt_wire_setup);
127136
extern void disable_local_APIC(void);
128137
extern void lapic_shutdown(void);
129138
extern void sync_Arb_IDs(void);
130-
extern void init_bsp_APIC(void);
139+
extern void apic_intr_mode_init(void);
131140
extern void setup_local_APIC(void);
132141
extern void init_apic_mappings(void);
133142
void register_lapic_address(unsigned long address);
134143
extern void setup_boot_APIC_clock(void);
135144
extern void setup_secondary_APIC_clock(void);
136145
extern void lapic_update_tsc_freq(void);
137-
extern int APIC_init_uniprocessor(void);
138146

139147
#ifdef CONFIG_X86_64
140148
static inline int apic_force_enable(unsigned long addr)
@@ -145,7 +153,7 @@ static inline int apic_force_enable(unsigned long addr)
145153
extern int apic_force_enable(unsigned long addr);
146154
#endif
147155

148-
extern int apic_bsp_setup(bool upmode);
156+
extern void apic_bsp_setup(bool upmode);
149157
extern void apic_ap_setup(void);
150158

151159
/*
@@ -161,6 +169,10 @@ static inline int apic_is_clustered_box(void)
161169
#endif
162170

163171
extern int setup_APIC_eilvt(u8 lvt_off, u8 vector, u8 msg_type, u8 mask);
172+
extern void lapic_assign_system_vectors(void);
173+
extern void lapic_assign_legacy_vector(unsigned int isairq, bool replace);
174+
extern void lapic_online(void);
175+
extern void lapic_offline(void);
164176

165177
#else /* !CONFIG_X86_LOCAL_APIC */
166178
static inline void lapic_shutdown(void) { }
@@ -170,6 +182,9 @@ static inline void disable_local_APIC(void) { }
170182
# define setup_boot_APIC_clock x86_init_noop
171183
# define setup_secondary_APIC_clock x86_init_noop
172184
static inline void lapic_update_tsc_freq(void) { }
185+
static inline void apic_intr_mode_init(void) { }
186+
static inline void lapic_assign_system_vectors(void) { }
187+
static inline void lapic_assign_legacy_vector(unsigned int i, bool r) { }
173188
#endif /* !CONFIG_X86_LOCAL_APIC */
174189

175190
#ifdef CONFIG_X86_X2APIC
@@ -265,73 +280,63 @@ struct irq_data;
265280
* James Cleverdon.
266281
*/
267282
struct apic {
268-
char *name;
269-
270-
int (*probe)(void);
271-
int (*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
272-
int (*apic_id_valid)(int apicid);
273-
int (*apic_id_registered)(void);
274-
275-
u32 irq_delivery_mode;
276-
u32 irq_dest_mode;
277-
278-
const struct cpumask *(*target_cpus)(void);
279-
280-
int disable_esr;
281-
282-
int dest_logical;
283-
unsigned long (*check_apicid_used)(physid_mask_t *map, int apicid);
284-
285-
void (*vector_allocation_domain)(int cpu, struct cpumask *retmask,
286-
const struct cpumask *mask);
287-
void (*init_apic_ldr)(void);
288-
289-
void (*ioapic_phys_id_map)(physid_mask_t *phys_map, physid_mask_t *retmap);
290-
291-
void (*setup_apic_routing)(void);
292-
int (*cpu_present_to_apicid)(int mps_cpu);
293-
void (*apicid_to_cpu_present)(int phys_apicid, physid_mask_t *retmap);
294-
int (*check_phys_apicid_present)(int phys_apicid);
295-
int (*phys_pkg_id)(int cpuid_apic, int index_msb);
296-
297-
unsigned int (*get_apic_id)(unsigned long x);
298-
/* Can't be NULL on 64-bit */
299-
unsigned long (*set_apic_id)(unsigned int id);
300-
301-
int (*cpu_mask_to_apicid)(const struct cpumask *cpumask,
302-
struct irq_data *irqdata,
303-
unsigned int *apicid);
304-
305-
/* ipi */
306-
void (*send_IPI)(int cpu, int vector);
307-
void (*send_IPI_mask)(const struct cpumask *mask, int vector);
308-
void (*send_IPI_mask_allbutself)(const struct cpumask *mask,
309-
int vector);
310-
void (*send_IPI_allbutself)(int vector);
311-
void (*send_IPI_all)(int vector);
312-
void (*send_IPI_self)(int vector);
283+
/* Hotpath functions first */
284+
void (*eoi_write)(u32 reg, u32 v);
285+
void (*native_eoi_write)(u32 reg, u32 v);
286+
void (*write)(u32 reg, u32 v);
287+
u32 (*read)(u32 reg);
288+
289+
/* IPI related functions */
290+
void (*wait_icr_idle)(void);
291+
u32 (*safe_wait_icr_idle)(void);
292+
293+
void (*send_IPI)(int cpu, int vector);
294+
void (*send_IPI_mask)(const struct cpumask *mask, int vector);
295+
void (*send_IPI_mask_allbutself)(const struct cpumask *msk, int vec);
296+
void (*send_IPI_allbutself)(int vector);
297+
void (*send_IPI_all)(int vector);
298+
void (*send_IPI_self)(int vector);
299+
300+
/* dest_logical is used by the IPI functions */
301+
u32 dest_logical;
302+
u32 disable_esr;
303+
u32 irq_delivery_mode;
304+
u32 irq_dest_mode;
305+
306+
/* Functions and data related to vector allocation */
307+
void (*vector_allocation_domain)(int cpu, struct cpumask *retmask,
308+
const struct cpumask *mask);
309+
int (*cpu_mask_to_apicid)(const struct cpumask *cpumask,
310+
struct irq_data *irqdata,
311+
unsigned int *apicid);
312+
u32 (*calc_dest_apicid)(unsigned int cpu);
313+
314+
/* ICR related functions */
315+
u64 (*icr_read)(void);
316+
void (*icr_write)(u32 low, u32 high);
317+
318+
/* Probe, setup and smpboot functions */
319+
int (*probe)(void);
320+
int (*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
321+
int (*apic_id_valid)(int apicid);
322+
int (*apic_id_registered)(void);
323+
324+
bool (*check_apicid_used)(physid_mask_t *map, int apicid);
325+
void (*init_apic_ldr)(void);
326+
void (*ioapic_phys_id_map)(physid_mask_t *phys_map, physid_mask_t *retmap);
327+
void (*setup_apic_routing)(void);
328+
int (*cpu_present_to_apicid)(int mps_cpu);
329+
void (*apicid_to_cpu_present)(int phys_apicid, physid_mask_t *retmap);
330+
int (*check_phys_apicid_present)(int phys_apicid);
331+
int (*phys_pkg_id)(int cpuid_apic, int index_msb);
332+
333+
u32 (*get_apic_id)(unsigned long x);
334+
u32 (*set_apic_id)(unsigned int id);
313335

314336
/* wakeup_secondary_cpu */
315-
int (*wakeup_secondary_cpu)(int apicid, unsigned long start_eip);
337+
int (*wakeup_secondary_cpu)(int apicid, unsigned long start_eip);
316338

317-
void (*inquire_remote_apic)(int apicid);
318-
319-
/* apic ops */
320-
u32 (*read)(u32 reg);
321-
void (*write)(u32 reg, u32 v);
322-
/*
323-
* ->eoi_write() has the same signature as ->write().
324-
*
325-
* Drivers can support both ->eoi_write() and ->write() by passing the same
326-
* callback value. Kernel can override ->eoi_write() and fall back
327-
* on write for EOI.
328-
*/
329-
void (*eoi_write)(u32 reg, u32 v);
330-
void (*native_eoi_write)(u32 reg, u32 v);
331-
u64 (*icr_read)(void);
332-
void (*icr_write)(u32 low, u32 high);
333-
void (*wait_icr_idle)(void);
334-
u32 (*safe_wait_icr_idle)(void);
339+
void (*inquire_remote_apic)(int apicid);
335340

336341
#ifdef CONFIG_X86_32
337342
/*
@@ -346,6 +351,7 @@ struct apic {
346351
*/
347352
int (*x86_32_early_logical_apicid)(int cpu);
348353
#endif
354+
char *name;
349355
};
350356

351357
/*
@@ -380,6 +386,7 @@ extern struct apic *__apicdrivers[], *__apicdrivers_end[];
380386
*/
381387
#ifdef CONFIG_SMP
382388
extern int wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip);
389+
extern int lapic_can_unplug_cpu(void);
383390
#endif
384391

385392
#ifdef CONFIG_X86_LOCAL_APIC
@@ -463,156 +470,51 @@ static inline unsigned default_get_apic_id(unsigned long x)
463470
extern void apic_send_IPI_self(int vector);
464471

465472
DECLARE_PER_CPU(int, x2apic_extra_bits);
466-
467-
extern int default_cpu_present_to_apicid(int mps_cpu);
468-
extern int default_check_phys_apicid_present(int phys_apicid);
469473
#endif
470474

471475
extern void generic_bigsmp_probe(void);
472476

473-
474477
#ifdef CONFIG_X86_LOCAL_APIC
475478

476479
#include <asm/smp.h>
477480

478481
#define APIC_DFR_VALUE (APIC_DFR_FLAT)
479482

480-
static inline const struct cpumask *default_target_cpus(void)
481-
{
482-
#ifdef CONFIG_SMP
483-
return cpu_online_mask;
484-
#else
485-
return cpumask_of(0);
486-
#endif
487-
}
488-
489-
static inline const struct cpumask *online_target_cpus(void)
490-
{
491-
return cpu_online_mask;
492-
}
493-
494483
DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_bios_cpu_apicid);
495484

485+
extern struct apic apic_noop;
496486

497487
static inline unsigned int read_apic_id(void)
498488
{
499-
unsigned int reg;
500-
501-
reg = apic_read(APIC_ID);
489+
unsigned int reg = apic_read(APIC_ID);
502490

503491
return apic->get_apic_id(reg);
504492
}
505493

506-
static inline int default_apic_id_valid(int apicid)
507-
{
508-
return (apicid < 255);
509-
}
510-
494+
extern int default_apic_id_valid(int apicid);
511495
extern int default_acpi_madt_oem_check(char *, char *);
512-
513496
extern void default_setup_apic_routing(void);
514497

515-
extern struct apic apic_noop;
516-
517-
#ifdef CONFIG_X86_32
518-
519-
static inline int noop_x86_32_early_logical_apicid(int cpu)
520-
{
521-
return BAD_APICID;
522-
}
523-
524-
/*
525-
* Set up the logical destination ID.
526-
*
527-
* Intel recommends to set DFR, LDR and TPR before enabling
528-
* an APIC. See e.g. "AP-388 82489DX User's Manual" (Intel
529-
* document number 292116). So here it goes...
530-
*/
531-
extern void default_init_apic_ldr(void);
532-
533-
static inline int default_apic_id_registered(void)
534-
{
535-
return physid_isset(read_apic_id(), phys_cpu_present_map);
536-
}
537-
538-
static inline int default_phys_pkg_id(int cpuid_apic, int index_msb)
539-
{
540-
return cpuid_apic >> index_msb;
541-
}
542-
543-
#endif
498+
extern u32 apic_default_calc_apicid(unsigned int cpu);
499+
extern u32 apic_flat_calc_apicid(unsigned int cpu);
544500

545501
extern int flat_cpu_mask_to_apicid(const struct cpumask *cpumask,
546502
struct irq_data *irqdata,
547503
unsigned int *apicid);
548504
extern int default_cpu_mask_to_apicid(const struct cpumask *cpumask,
549505
struct irq_data *irqdata,
550506
unsigned int *apicid);
551-
552-
static inline void
553-
flat_vector_allocation_domain(int cpu, struct cpumask *retmask,
554-
const struct cpumask *mask)
555-
{
556-
/* Careful. Some cpus do not strictly honor the set of cpus
557-
* specified in the interrupt destination when using lowest
558-
* priority interrupt delivery mode.
559-
*
560-
* In particular there was a hyperthreading cpu observed to
561-
* deliver interrupts to the wrong hyperthread when only one
562-
* hyperthread was specified in the interrupt desitination.
563-
*/
564-
cpumask_clear(retmask);
565-
cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
566-
}
567-
568-
static inline void
569-
default_vector_allocation_domain(int cpu, struct cpumask *retmask,
570-
const struct cpumask *mask)
571-
{
572-
cpumask_copy(retmask, cpumask_of(cpu));
573-
}
574-
575-
static inline unsigned long default_check_apicid_used(physid_mask_t *map, int apicid)
576-
{
577-
return physid_isset(apicid, *map);
578-
}
579-
580-
static inline void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap)
581-
{
582-
*retmap = *phys_map;
583-
}
584-
585-
static inline int __default_cpu_present_to_apicid(int mps_cpu)
586-
{
587-
if (mps_cpu < nr_cpu_ids && cpu_present(mps_cpu))
588-
return (int)per_cpu(x86_bios_cpu_apicid, mps_cpu);
589-
else
590-
return BAD_APICID;
591-
}
592-
593-
static inline int
594-
__default_check_phys_apicid_present(int phys_apicid)
595-
{
596-
return physid_isset(phys_apicid, phys_cpu_present_map);
597-
}
598-
599-
#ifdef CONFIG_X86_32
600-
static inline int default_cpu_present_to_apicid(int mps_cpu)
601-
{
602-
return __default_cpu_present_to_apicid(mps_cpu);
603-
}
604-
605-
static inline int
606-
default_check_phys_apicid_present(int phys_apicid)
607-
{
608-
return __default_check_phys_apicid_present(phys_apicid);
609-
}
610-
#else
507+
extern bool default_check_apicid_used(physid_mask_t *map, int apicid);
508+
extern void flat_vector_allocation_domain(int cpu, struct cpumask *retmask,
509+
const struct cpumask *mask);
510+
extern void default_vector_allocation_domain(int cpu, struct cpumask *retmask,
511+
const struct cpumask *mask);
512+
extern void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap);
611513
extern int default_cpu_present_to_apicid(int mps_cpu);
612514
extern int default_check_phys_apicid_present(int phys_apicid);
613-
#endif
614515

615516
#endif /* CONFIG_X86_LOCAL_APIC */
517+
616518
extern void irq_enter(void);
617519
extern void irq_exit(void);
618520

‎arch/x86/include/asm/desc.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -393,7 +393,7 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)
393393
void update_intr_gate(unsigned int n, const void *addr);
394394
void alloc_intr_gate(unsigned int n, const void *addr);
395395

396-
extern unsigned long used_vectors[];
396+
extern unsigned long system_vectors[];
397397

398398
#ifdef CONFIG_X86_64
399399
DECLARE_PER_CPU(u32, debug_idt_ctr);

0 commit comments

Comments
 (0)