openkylin
/
linux
mirror of https://gitee.com/openkylin/linux.git
9
0
Fork 0
Code Issues Projects Releases Wiki Activity
linux/arch/arm/kernel/module-plts.c

262 lines
7.5 KiB
C
Raw Normal View History

ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
/*
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
* Copyright (C) 2014-2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#include <linux/elf.h>
#include <linux/kernel.h>
#include <linux/module.h>
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
#include <linux/sort.h>
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
#include <asm/cache.h>
#include <asm/opcodes.h>
#define PLT_ENT_STRIDE L1_CACHE_BYTES
#define PLT_ENT_COUNT (PLT_ENT_STRIDE / sizeof(u32))
#define PLT_ENT_SIZE (sizeof(struct plt_entries) / PLT_ENT_COUNT)
#ifdef CONFIG_THUMB2_KERNEL
#define PLT_ENT_LDR __opcode_to_mem_thumb32(0xf8dff000 | \
(PLT_ENT_STRIDE - 4))
#else
#define PLT_ENT_LDR __opcode_to_mem_arm(0xe59ff000 | \
(PLT_ENT_STRIDE - 8))
#endif
struct plt_entries {
u32 ldr[PLT_ENT_COUNT];
u32 lit[PLT_ENT_COUNT];
};
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
static bool in_init(const struct module *mod, unsigned long loc)
{
return loc - (u32)mod->init_layout.base < mod->init_layout.size;
}
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
u32 get_module_plt(struct module *mod, unsigned long loc, Elf32_Addr val)
{
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
struct mod_plt_sec *pltsec = !in_init(mod, loc) ? &mod->arch.core :
&mod->arch.init;
struct plt_entries *plt = (struct plt_entries *)pltsec->plt->sh_addr;
ARM: kernel: avoid brute force search on PLT generation Given that we now sort the relocation sections in a way that guarantees that entries that can share a single PLT entry end up adjacently, there is no a longer a need to go over the entire list to look for an existing entry that matches our jump target. If such a match exists, it was the last one to be emitted, so we can simply check the preceding slot. Note that this will still work correctly in the [theoretical] presence of call/jump relocations against SHN_UNDEF symbols with non-zero addends, although not optimally. Since the relocations are presented in the same order that we checked them for duplicates, any duplicates that we failed to spot the first time around will be accounted for in the PLT allocation so there is guaranteed to be sufficient space for them when actually emitting the PLT. For instance, the following sequence of relocations: 000004d8 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000004fc 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000050e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000520 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000532 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000544 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000556 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000568 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000057a 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000058c 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000059e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005b0 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005c2 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005d4 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null may result in several PLT entries to be allocated, and also emitted, if any of the entries in the middle refer to a Place that contains a non-zero addend (i.e., one for all the preceding zero-addend relocations, one for all the following zero-addend relocations, and one for the non-zero addend relocation itself) Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-18 16:58:49 +08:00
int idx = 0;
/*
* Look for an existing entry pointing to 'val'. Given that the
* relocations are sorted, this will be the last entry we allocated.
* (if one exists).
*/
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
if (pltsec->plt_count > 0) {
plt += (pltsec->plt_count - 1) / PLT_ENT_COUNT;
idx = (pltsec->plt_count - 1) % PLT_ENT_COUNT;
ARM: kernel: avoid brute force search on PLT generation Given that we now sort the relocation sections in a way that guarantees that entries that can share a single PLT entry end up adjacently, there is no a longer a need to go over the entire list to look for an existing entry that matches our jump target. If such a match exists, it was the last one to be emitted, so we can simply check the preceding slot. Note that this will still work correctly in the [theoretical] presence of call/jump relocations against SHN_UNDEF symbols with non-zero addends, although not optimally. Since the relocations are presented in the same order that we checked them for duplicates, any duplicates that we failed to spot the first time around will be accounted for in the PLT allocation so there is guaranteed to be sufficient space for them when actually emitting the PLT. For instance, the following sequence of relocations: 000004d8 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000004fc 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000050e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000520 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000532 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000544 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000556 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000568 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000057a 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000058c 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000059e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005b0 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005c2 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005d4 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null may result in several PLT entries to be allocated, and also emitted, if any of the entries in the middle refer to a Place that contains a non-zero addend (i.e., one for all the preceding zero-addend relocations, one for all the following zero-addend relocations, and one for the non-zero addend relocation itself) Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-18 16:58:49 +08:00
if (plt->lit[idx] == val)
return (u32)&plt->ldr[idx];
idx = (idx + 1) % PLT_ENT_COUNT;
if (!idx)
plt++;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
}
ARM: kernel: avoid brute force search on PLT generation Given that we now sort the relocation sections in a way that guarantees that entries that can share a single PLT entry end up adjacently, there is no a longer a need to go over the entire list to look for an existing entry that matches our jump target. If such a match exists, it was the last one to be emitted, so we can simply check the preceding slot. Note that this will still work correctly in the [theoretical] presence of call/jump relocations against SHN_UNDEF symbols with non-zero addends, although not optimally. Since the relocations are presented in the same order that we checked them for duplicates, any duplicates that we failed to spot the first time around will be accounted for in the PLT allocation so there is guaranteed to be sufficient space for them when actually emitting the PLT. For instance, the following sequence of relocations: 000004d8 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000004fc 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000050e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000520 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000532 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000544 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000556 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000568 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000057a 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000058c 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000059e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005b0 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005c2 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005d4 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null may result in several PLT entries to be allocated, and also emitted, if any of the entries in the middle refer to a Place that contains a non-zero addend (i.e., one for all the preceding zero-addend relocations, one for all the following zero-addend relocations, and one for the non-zero addend relocation itself) Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-18 16:58:49 +08:00
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
pltsec->plt_count++;
BUG_ON(pltsec->plt_count * PLT_ENT_SIZE > pltsec->plt->sh_size);
ARM: kernel: avoid brute force search on PLT generation Given that we now sort the relocation sections in a way that guarantees that entries that can share a single PLT entry end up adjacently, there is no a longer a need to go over the entire list to look for an existing entry that matches our jump target. If such a match exists, it was the last one to be emitted, so we can simply check the preceding slot. Note that this will still work correctly in the [theoretical] presence of call/jump relocations against SHN_UNDEF symbols with non-zero addends, although not optimally. Since the relocations are presented in the same order that we checked them for duplicates, any duplicates that we failed to spot the first time around will be accounted for in the PLT allocation so there is guaranteed to be sufficient space for them when actually emitting the PLT. For instance, the following sequence of relocations: 000004d8 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000004fc 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000050e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000520 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000532 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000544 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000556 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 00000568 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000057a 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000058c 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 0000059e 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005b0 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005c2 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null 000005d4 00058b0a R_ARM_THM_CALL 00000000 warn_slowpath_null may result in several PLT entries to be allocated, and also emitted, if any of the entries in the middle refer to a Place that contains a non-zero addend (i.e., one for all the preceding zero-addend relocations, one for all the following zero-addend relocations, and one for the non-zero addend relocation itself) Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-18 16:58:49 +08:00
if (!idx)
/* Populate a new set of entries */
*plt = (struct plt_entries){
{ [0 ... PLT_ENT_COUNT - 1] = PLT_ENT_LDR, },
{ val, }
};
else
plt->lit[idx] = val;
return (u32)&plt->ldr[idx];
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
}
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
#define cmp_3way(a,b) ((a) < (b) ? -1 : (a) > (b))
static int cmp_rel(const void *a, const void *b)
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
{
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
const Elf32_Rel *x = a, *y = b;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
int i;
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
/* sort by type and symbol index */
i = cmp_3way(ELF32_R_TYPE(x->r_info), ELF32_R_TYPE(y->r_info));
if (i == 0)
i = cmp_3way(ELF32_R_SYM(x->r_info), ELF32_R_SYM(y->r_info));
return i;
}
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
static bool is_zero_addend_relocation(Elf32_Addr base, const Elf32_Rel *rel)
{
u32 *tval = (u32 *)(base + rel->r_offset);
/*
* Do a bitwise compare on the raw addend rather than fully decoding
* the offset and doing an arithmetic comparison.
* Note that a zero-addend jump/call relocation is encoded taking the
* PC bias into account, i.e., -8 for ARM and -4 for Thumb2.
*/
switch (ELF32_R_TYPE(rel->r_info)) {
u16 upper, lower;
case R_ARM_THM_CALL:
case R_ARM_THM_JUMP24:
upper = __mem_to_opcode_thumb16(((u16 *)tval)[0]);
lower = __mem_to_opcode_thumb16(((u16 *)tval)[1]);
return (upper & 0x7ff) == 0x7ff && (lower & 0x2fff) == 0x2ffe;
case R_ARM_CALL:
case R_ARM_PC24:
case R_ARM_JUMP24:
return (__mem_to_opcode_arm(*tval) & 0xffffff) == 0xfffffe;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
}
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
BUG();
}
static bool duplicate_rel(Elf32_Addr base, const Elf32_Rel *rel, int num)
{
const Elf32_Rel *prev;
/*
* Entries are sorted by type and symbol index. That means that,
* if a duplicate entry exists, it must be in the preceding
* slot.
*/
if (!num)
return false;
prev = rel + num - 1;
return cmp_rel(rel + num, prev) == 0 &&
is_zero_addend_relocation(base, prev);
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
}
/* Count how many PLT entries we may need */
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
static unsigned int count_plts(const Elf32_Sym *syms, Elf32_Addr base,
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
const Elf32_Rel *rel, int num, Elf32_Word dstidx)
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
{
unsigned int ret = 0;
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
const Elf32_Sym *s;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
int i;
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
for (i = 0; i < num; i++) {
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
switch (ELF32_R_TYPE(rel[i].r_info)) {
case R_ARM_CALL:
case R_ARM_PC24:
case R_ARM_JUMP24:
case R_ARM_THM_CALL:
case R_ARM_THM_JUMP24:
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
/*
* We only have to consider branch targets that resolve
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
* to symbols that are defined in a different section.
* This is not simply a heuristic, it is a fundamental
* limitation, since there is no guaranteed way to emit
* PLT entries sufficiently close to the branch if the
* section size exceeds the range of a branch
* instruction. So ignore relocations against defined
* symbols if they live in the same section as the
* relocation target.
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
*/
s = syms + ELF32_R_SYM(rel[i].r_info);
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
if (s->st_shndx == dstidx)
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
break;
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
/*
* Jump relocations with non-zero addends against
* undefined symbols are supported by the ELF spec, but
* do not occur in practice (e.g., 'jump n bytes past
* the entry point of undefined function symbol f').
* So we need to support them, but there is no need to
* take them into consideration when trying to optimize
* this code. So let's only check for duplicates when
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
* the addend is zero. (Note that calls into the core
* module via init PLT entries could involve section
* relative symbol references with non-zero addends, for
* which we may end up emitting duplicates, but the init
* PLT is released along with the rest of the .init
* region as soon as module loading completes.)
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
*/
if (!is_zero_addend_relocation(base, rel + i) ||
!duplicate_rel(base, rel, i))
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
ret++;
}
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
}
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
return ret;
}
int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
char *secstrings, struct module *mod)
{
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
unsigned long core_plts = 0;
unsigned long init_plts = 0;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
Elf32_Shdr *s, *sechdrs_end = sechdrs + ehdr->e_shnum;
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
Elf32_Sym *syms = NULL;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
/*
* To store the PLTs, we expand the .text section for core module code
ARM: kernel: merge core and init PLTs The PLT code uses a separate .init.plt section to allocate PLT entries for jump and call instructions in __init code. However, even for fairly sizable modules like mac80211.ko, we only end up with a couple of PLT entries in the .init section, and so we can simplify the code significantly by emitting all PLT entries into the same section. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 23:21:02 +08:00
* and for initialization code.
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
*/
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
for (s = sechdrs; s < sechdrs_end; ++s) {
ARM: kernel: merge core and init PLTs The PLT code uses a separate .init.plt section to allocate PLT entries for jump and call instructions in __init code. However, even for fairly sizable modules like mac80211.ko, we only end up with a couple of PLT entries in the .init section, and so we can simplify the code significantly by emitting all PLT entries into the same section. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 23:21:02 +08:00
if (strcmp(".plt", secstrings + s->sh_name) == 0)
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
mod->arch.core.plt = s;
else if (strcmp(".init.plt", secstrings + s->sh_name) == 0)
mod->arch.init.plt = s;
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
else if (s->sh_type == SHT_SYMTAB)
syms = (Elf32_Sym *)s->sh_addr;
}
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
if (!mod->arch.core.plt || !mod->arch.init.plt) {
pr_err("%s: module PLT section(s) missing\n", mod->name);
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
return -ENOEXEC;
}
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
if (!syms) {
pr_err("%s: module symtab section missing\n", mod->name);
return -ENOEXEC;
}
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
for (s = sechdrs + 1; s < sechdrs_end; ++s) {
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
Elf32_Rel *rels = (void *)ehdr + s->sh_offset;
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
int numrels = s->sh_size / sizeof(Elf32_Rel);
Elf32_Shdr *dstsec = sechdrs + s->sh_info;
if (s->sh_type != SHT_REL)
continue;
ARM: kernel: allocate PLT entries only for external symbols When CONFIG_ARM_MODULE_PLTS is enabled, jump and call instructions in modules no longer need to be within 16 MB (8 MB for Thumb2) of their targets. If they are further away, a PLT entry will be generated on the fly for each of them, which extends the range to the entire 32-bit address space. However, since these PLT entries will become the branch targets of the original jump and call instructions, the PLT itself needs to be in range, or we end up in the same situation we started in. Since the PLT is in a separate section, this essentially means that all jumps and calls inside the same module must be resolvable without PLT entries. The PLT allocation code executes before the module itself is loaded in its final location, and so it has to use a worst-case estimate for which jumps and calls will require an entry in the PLT at relocation time. As an optimization, this code deduplicates entries pointing to the same symbol, using a O(n^2) algorithm. However, it does not take the above into account, i.e., that PLT entries will only be needed for jump and call relocations against symbols that are not defined in the module. So disregard relocations against symbols that are defined in the module itself. As an additional minor optimization, ignore input sections that lack the SHF_EXECINSTR flag. Since jump and call relocations operate on executable instructions only, there is no need to look in sections that do not contain executable code. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-16 22:49:56 +08:00
/* ignore relocations that operate on non-exec sections */
if (!(dstsec->sh_flags & SHF_EXECINSTR))
continue;
ARM: kernel: sort relocation sections before allocating PLTs The PLT allocation routines try to establish an upper bound on the number of PLT entries that will be required at relocation time, and optimize this by disregarding duplicates (i.e., PLT entries that will end up pointing to the same function). This is currently a O(n^2) algorithm, but we can greatly simplify this by - sorting the relocation section so that relocations that can use the same PLT entry will be listed adjacently, - disregard jump/call relocations with addends; these are highly unusual, for relocations against SHN_UNDEF symbols, and so we can simply allocate a PLT entry for each one we encounter, without trying to optimize away duplicates. Tested-by: Jongsung Kim <neidhard.kim@lge.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2016-08-17 19:45:21 +08:00
/* sort by type and symbol index */
sort(rels, numrels, sizeof(Elf32_Rel), cmp_rel, NULL);
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
if (strncmp(secstrings + dstsec->sh_name, ".init", 5) != 0)
core_plts += count_plts(syms, dstsec->sh_addr, rels,
numrels, s->sh_info);
else
init_plts += count_plts(syms, dstsec->sh_addr, rels,
numrels, s->sh_info);
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
}
ARM: 8662/1: module: split core and init PLT sections Since commit 35fa91eed817 ("ARM: kernel: merge core and init PLTs"), the ARM module PLT code allocates all PLT entries in a single core section, since the overhead of having a separate init PLT section is not justified by the small number of PLT entries usually required for init code. However, the core and init module regions are allocated independently, and there is a corner case where the core region may be allocated from the VMALLOC region if the dedicated module region is exhausted, but the init region, being much smaller, can still be allocated from the module region. This puts the PLT entries out of reach of the relocated branch instructions, defeating the whole purpose of PLTs. So split the core and init PLT regions, and name the latter ".init.plt" so it gets allocated along with (and sufficiently close to) the .init sections that it serves. Also, given that init PLT entries may need to be emitted for branches that target the core module, modify the logic that disregards defined symbols to only disregard symbols that are defined in the same section. Fixes: 35fa91eed817 ("ARM: kernel: merge core and init PLTs") Cc: <stable@vger.kernel.org> # v4.9+ Reported-by: Angus Clark <angus@angusclark.org> Tested-by: Angus Clark <angus@angusclark.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-02-23 02:40:12 +08:00
mod->arch.core.plt->sh_type = SHT_NOBITS;
mod->arch.core.plt->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
mod->arch.core.plt->sh_addralign = L1_CACHE_BYTES;
mod->arch.core.plt->sh_size = round_up(core_plts * PLT_ENT_SIZE,
sizeof(struct plt_entries));
mod->arch.core.plt_count = 0;
mod->arch.init.plt->sh_type = SHT_NOBITS;
mod->arch.init.plt->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
mod->arch.init.plt->sh_addralign = L1_CACHE_BYTES;
mod->arch.init.plt->sh_size = round_up(init_plts * PLT_ENT_SIZE,
sizeof(struct plt_entries));
mod->arch.init.plt_count = 0;
pr_debug("%s: plt=%x, init.plt=%x\n", __func__,
mod->arch.core.plt->sh_size, mod->arch.init.plt->sh_size);
ARM: 8220/1: allow modules outside of bl range Loading modules far away from the kernel in memory is problematic because the 'bl' instruction only has limited reach, and modules are not built with PLTs. Instead of using the -mlong-calls option (which affects all compiler emitted bl instructions, but not the ones in assembler), this patch allocates some additional space at module load time, and populates it with PLT like veneers when encountering relocations that are out of range. This should work with all relocations against symbols exported by the kernel, including those resulting from GCC generated implicit function calls for ftrace etc. The module memory size increases by about 5% on average, regardless of whether any PLT entries were actually needed. However, due to the page based rounding that occurs when allocating module memory, the average memory footprint increase is negligible. Reviewed-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-11-24 23:54:35 +08:00
return 0;
}