License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 22:07:57 +08:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2008-03-18 07:36:55 +08:00
|
|
|
#include <linux/mm.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/gfp.h>
|
2008-03-18 07:36:55 +08:00
|
|
|
#include <asm/pgalloc.h>
|
2008-03-18 07:37:03 +08:00
|
|
|
#include <asm/pgtable.h>
|
2008-03-18 07:36:55 +08:00
|
|
|
#include <asm/tlb.h>
|
2008-06-20 21:34:46 +08:00
|
|
|
#include <asm/fixmap.h>
|
2015-04-15 06:47:32 +08:00
|
|
|
#include <asm/mtrr.h>
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2016-07-27 06:24:30 +08:00
|
|
|
#define PGALLOC_GFP (GFP_KERNEL_ACCOUNT | __GFP_NOTRACK | __GFP_ZERO)
|
2009-02-22 18:28:25 +08:00
|
|
|
|
2010-02-17 18:38:10 +08:00
|
|
|
#ifdef CONFIG_HIGHPTE
|
|
|
|
#define PGALLOC_USER_GFP __GFP_HIGHMEM
|
|
|
|
#else
|
|
|
|
#define PGALLOC_USER_GFP 0
|
|
|
|
#endif
|
|
|
|
|
|
|
|
gfp_t __userpte_alloc_gfp = PGALLOC_GFP | PGALLOC_USER_GFP;
|
|
|
|
|
2008-03-18 07:36:55 +08:00
|
|
|
pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
|
|
|
|
{
|
2016-07-27 06:24:30 +08:00
|
|
|
return (pte_t *)__get_free_page(PGALLOC_GFP & ~__GFP_ACCOUNT);
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
|
|
|
|
{
|
|
|
|
struct page *pte;
|
|
|
|
|
2010-02-17 18:38:10 +08:00
|
|
|
pte = alloc_pages(__userpte_alloc_gfp, 0);
|
2013-11-15 06:31:47 +08:00
|
|
|
if (!pte)
|
|
|
|
return NULL;
|
|
|
|
if (!pgtable_page_ctor(pte)) {
|
|
|
|
__free_page(pte);
|
|
|
|
return NULL;
|
|
|
|
}
|
2008-03-18 07:36:55 +08:00
|
|
|
return pte;
|
|
|
|
}
|
|
|
|
|
2010-02-17 18:38:10 +08:00
|
|
|
static int __init setup_userpte(char *arg)
|
|
|
|
{
|
|
|
|
if (!arg)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* "userpte=nohigh" disables allocation of user pagetables in
|
|
|
|
* high memory.
|
|
|
|
*/
|
|
|
|
if (strcmp(arg, "nohigh") == 0)
|
|
|
|
__userpte_alloc_gfp &= ~__GFP_HIGHMEM;
|
|
|
|
else
|
|
|
|
return -EINVAL;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
early_param("userpte", setup_userpte);
|
|
|
|
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 13:44:28 +08:00
|
|
|
void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
|
2008-03-18 07:36:57 +08:00
|
|
|
{
|
|
|
|
pgtable_page_dtor(pte);
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_release_pte(page_to_pfn(pte));
|
2017-08-28 16:22:51 +08:00
|
|
|
tlb_remove_table(tlb, pte);
|
2008-03-18 07:36:57 +08:00
|
|
|
}
|
|
|
|
|
2015-04-15 06:46:14 +08:00
|
|
|
#if CONFIG_PGTABLE_LEVELS > 2
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 13:44:28 +08:00
|
|
|
void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
|
2008-03-18 07:36:58 +08:00
|
|
|
{
|
2013-11-22 06:32:09 +08:00
|
|
|
struct page *page = virt_to_page(pmd);
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT);
|
2013-04-13 07:23:54 +08:00
|
|
|
/*
|
|
|
|
* NOTE! For PAE, any changes to the top page-directory-pointer-table
|
|
|
|
* entries need a full cr3 reload to flush.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_X86_PAE
|
|
|
|
tlb->need_flush_all = 1;
|
|
|
|
#endif
|
2013-11-22 06:32:09 +08:00
|
|
|
pgtable_pmd_page_dtor(page);
|
2017-08-28 16:22:51 +08:00
|
|
|
tlb_remove_table(tlb, page);
|
2008-03-18 07:36:58 +08:00
|
|
|
}
|
2008-03-18 07:36:59 +08:00
|
|
|
|
2015-04-15 06:46:14 +08:00
|
|
|
#if CONFIG_PGTABLE_LEVELS > 3
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 13:44:28 +08:00
|
|
|
void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)
|
2008-03-18 07:36:59 +08:00
|
|
|
{
|
2008-03-18 07:37:02 +08:00
|
|
|
paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
|
2017-08-28 16:22:51 +08:00
|
|
|
tlb_remove_table(tlb, virt_to_page(pud));
|
2008-03-18 07:36:59 +08:00
|
|
|
}
|
2017-03-30 16:07:29 +08:00
|
|
|
|
|
|
|
#if CONFIG_PGTABLE_LEVELS > 4
|
|
|
|
void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
|
|
|
|
{
|
|
|
|
paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT);
|
2017-08-28 16:22:51 +08:00
|
|
|
tlb_remove_table(tlb, virt_to_page(p4d));
|
2017-03-30 16:07:29 +08:00
|
|
|
}
|
|
|
|
#endif /* CONFIG_PGTABLE_LEVELS > 4 */
|
2015-04-15 06:46:14 +08:00
|
|
|
#endif /* CONFIG_PGTABLE_LEVELS > 3 */
|
|
|
|
#endif /* CONFIG_PGTABLE_LEVELS > 2 */
|
2008-03-18 07:36:58 +08:00
|
|
|
|
2008-03-18 07:36:55 +08:00
|
|
|
static inline void pgd_list_add(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
struct page *page = virt_to_page(pgd);
|
|
|
|
|
|
|
|
list_add(&page->lru, &pgd_list);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void pgd_list_del(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
struct page *page = virt_to_page(pgd);
|
|
|
|
|
|
|
|
list_del(&page->lru);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define UNSHARED_PTRS_PER_PGD \
|
2008-03-18 07:37:13 +08:00
|
|
|
(SHARED_KERNEL_PMD ? KERNEL_PGD_BOUNDARY : PTRS_PER_PGD)
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2010-09-22 03:01:51 +08:00
|
|
|
|
|
|
|
static void pgd_set_mm(pgd_t *pgd, struct mm_struct *mm)
|
|
|
|
{
|
|
|
|
BUILD_BUG_ON(sizeof(virt_to_page(pgd)->index) < sizeof(mm));
|
|
|
|
virt_to_page(pgd)->index = (pgoff_t)mm;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct mm_struct *pgd_page_get_mm(struct page *page)
|
|
|
|
{
|
|
|
|
return (struct mm_struct *)page->index;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void pgd_ctor(struct mm_struct *mm, pgd_t *pgd)
|
2008-03-18 07:36:55 +08:00
|
|
|
{
|
|
|
|
/* If the pgd points to a shared pagetable level (either the
|
|
|
|
ptes in non-PAE, or shared PMD in PAE), then just copy the
|
|
|
|
references from swapper_pg_dir. */
|
2015-04-15 06:46:14 +08:00
|
|
|
if (CONFIG_PGTABLE_LEVELS == 2 ||
|
|
|
|
(CONFIG_PGTABLE_LEVELS == 3 && SHARED_KERNEL_PMD) ||
|
2017-03-30 16:07:29 +08:00
|
|
|
CONFIG_PGTABLE_LEVELS >= 4) {
|
2008-03-18 07:37:13 +08:00
|
|
|
clone_pgd_range(pgd + KERNEL_PGD_BOUNDARY,
|
|
|
|
swapper_pg_dir + KERNEL_PGD_BOUNDARY,
|
2008-03-18 07:36:55 +08:00
|
|
|
KERNEL_PGD_PTRS);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* list required to sync kernel mapping updates */
|
2010-09-22 03:01:51 +08:00
|
|
|
if (!SHARED_KERNEL_PMD) {
|
|
|
|
pgd_set_mm(pgd, mm);
|
2008-03-18 07:36:55 +08:00
|
|
|
pgd_list_add(pgd);
|
2010-09-22 03:01:51 +08:00
|
|
|
}
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
|
|
|
|
2008-08-29 19:51:32 +08:00
|
|
|
static void pgd_dtor(pgd_t *pgd)
|
2008-03-18 07:36:55 +08:00
|
|
|
{
|
|
|
|
if (SHARED_KERNEL_PMD)
|
|
|
|
return;
|
|
|
|
|
2011-02-17 07:45:22 +08:00
|
|
|
spin_lock(&pgd_lock);
|
2008-03-18 07:36:55 +08:00
|
|
|
pgd_list_del(pgd);
|
2011-02-17 07:45:22 +08:00
|
|
|
spin_unlock(&pgd_lock);
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
|
|
|
|
2008-03-18 07:37:14 +08:00
|
|
|
/*
|
|
|
|
* List of all pgd's needed for non-PAE so it can invalidate entries
|
|
|
|
* in both cached and uncached pgd's; not needed for PAE since the
|
|
|
|
* kernel pmd is shared. If PAE were not to share the pmd a similar
|
|
|
|
* tactic would be needed. This is essentially codepath-based locking
|
|
|
|
* against pageattr.c; it is the unique case in which a valid change
|
|
|
|
* of kernel pagetables can't be lazily synchronized by vmalloc faults.
|
|
|
|
* vmalloc faults work because attached pagetables are never freed.
|
2012-12-06 17:39:54 +08:00
|
|
|
* -- nyc
|
2008-03-18 07:37:14 +08:00
|
|
|
*/
|
|
|
|
|
2008-03-18 07:36:55 +08:00
|
|
|
#ifdef CONFIG_X86_PAE
|
2008-06-25 12:19:13 +08:00
|
|
|
/*
|
|
|
|
* In PAE mode, we need to do a cr3 reload (=tlb flush) when
|
|
|
|
* updating the top-level pagetable entries to guarantee the
|
|
|
|
* processor notices the update. Since this is expensive, and
|
|
|
|
* all 4 top-level entries are used almost immediately in a
|
|
|
|
* new process's life, we just pre-populate them here.
|
|
|
|
*
|
|
|
|
* Also, if we're in a paravirt environment where the kernel pmd is
|
|
|
|
* not shared between pagetables (!SHARED_KERNEL_PMDS), we allocate
|
|
|
|
* and initialize the kernel pmds here.
|
|
|
|
*/
|
|
|
|
#define PREALLOCATED_PMDS UNSHARED_PTRS_PER_PGD
|
|
|
|
|
|
|
|
void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmd)
|
|
|
|
{
|
|
|
|
paravirt_alloc_pmd(mm, __pa(pmd) >> PAGE_SHIFT);
|
|
|
|
|
|
|
|
/* Note: almost everything apart from _PAGE_PRESENT is
|
|
|
|
reserved at the pmd (PDPT) level. */
|
|
|
|
set_pud(pudp, __pud(__pa(pmd) | _PAGE_PRESENT));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* According to Intel App note "TLBs, Paging-Structure Caches,
|
|
|
|
* and Their Invalidation", April 2007, document 317080-001,
|
|
|
|
* section 8.1: in PAE mode we explicitly have to flush the
|
|
|
|
* TLB via cr3 if the top-level pgd is changed...
|
|
|
|
*/
|
2011-03-16 11:37:29 +08:00
|
|
|
flush_tlb_mm(mm);
|
2008-06-25 12:19:13 +08:00
|
|
|
}
|
|
|
|
#else /* !CONFIG_X86_PAE */
|
|
|
|
|
|
|
|
/* No need to prepopulate any pagetable entries in non-PAE modes. */
|
|
|
|
#define PREALLOCATED_PMDS 0
|
|
|
|
|
|
|
|
#endif /* CONFIG_X86_PAE */
|
|
|
|
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
static void free_pmds(struct mm_struct *mm, pmd_t *pmds[])
|
2008-06-25 12:19:13 +08:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for(i = 0; i < PREALLOCATED_PMDS; i++)
|
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel. So we disable split page table lock, if spinlock_t is
too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
This patch (of 34):
I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled. Let's add missed constructor/destructor calls.
I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl. It's effectively equal to
spin_lock_init(&page->ptl).
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chen Liqin <liqin.chen@sunplusct.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
Cc: Lennox Wu <lennox.wu@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Richard Kuo <rkuo@codeaurora.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-15 06:31:13 +08:00
|
|
|
if (pmds[i]) {
|
|
|
|
pgtable_pmd_page_dtor(virt_to_page(pmds[i]));
|
2008-06-25 12:19:13 +08:00
|
|
|
free_page((unsigned long)pmds[i]);
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
mm_dec_nr_pmds(mm);
|
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel. So we disable split page table lock, if spinlock_t is
too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
This patch (of 34):
I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled. Let's add missed constructor/destructor calls.
I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl. It's effectively equal to
spin_lock_init(&page->ptl).
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chen Liqin <liqin.chen@sunplusct.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
Cc: Lennox Wu <lennox.wu@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Richard Kuo <rkuo@codeaurora.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-15 06:31:13 +08:00
|
|
|
}
|
2008-06-25 12:19:13 +08:00
|
|
|
}
|
|
|
|
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[])
|
2008-06-25 12:19:13 +08:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
bool failed = false;
|
2016-07-27 06:24:30 +08:00
|
|
|
gfp_t gfp = PGALLOC_GFP;
|
|
|
|
|
|
|
|
if (mm == &init_mm)
|
|
|
|
gfp &= ~__GFP_ACCOUNT;
|
2008-06-25 12:19:13 +08:00
|
|
|
|
|
|
|
for(i = 0; i < PREALLOCATED_PMDS; i++) {
|
2016-07-27 06:24:30 +08:00
|
|
|
pmd_t *pmd = (pmd_t *)__get_free_page(gfp);
|
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel. So we disable split page table lock, if spinlock_t is
too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
This patch (of 34):
I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled. Let's add missed constructor/destructor calls.
I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl. It's effectively equal to
spin_lock_init(&page->ptl).
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chen Liqin <liqin.chen@sunplusct.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
Cc: Lennox Wu <lennox.wu@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Richard Kuo <rkuo@codeaurora.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-15 06:31:13 +08:00
|
|
|
if (!pmd)
|
2008-06-25 12:19:13 +08:00
|
|
|
failed = true;
|
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel. So we disable split page table lock, if spinlock_t is
too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
This patch (of 34):
I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled. Let's add missed constructor/destructor calls.
I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl. It's effectively equal to
spin_lock_init(&page->ptl).
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chen Liqin <liqin.chen@sunplusct.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
Cc: Lennox Wu <lennox.wu@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Richard Kuo <rkuo@codeaurora.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-15 06:31:13 +08:00
|
|
|
if (pmd && !pgtable_pmd_page_ctor(virt_to_page(pmd))) {
|
2013-11-21 06:16:36 +08:00
|
|
|
free_page((unsigned long)pmd);
|
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel. So we disable split page table lock, if spinlock_t is
too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
This patch (of 34):
I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled. Let's add missed constructor/destructor calls.
I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl. It's effectively equal to
spin_lock_init(&page->ptl).
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chen Liqin <liqin.chen@sunplusct.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
Cc: Lennox Wu <lennox.wu@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Richard Kuo <rkuo@codeaurora.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-15 06:31:13 +08:00
|
|
|
pmd = NULL;
|
|
|
|
failed = true;
|
|
|
|
}
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
if (pmd)
|
|
|
|
mm_inc_nr_pmds(mm);
|
2008-06-25 12:19:13 +08:00
|
|
|
pmds[i] = pmd;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (failed) {
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
free_pmds(mm, pmds);
|
2008-06-25 12:19:13 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-03-18 07:36:55 +08:00
|
|
|
/*
|
|
|
|
* Mop up any pmd pages which may still be attached to the pgd.
|
|
|
|
* Normally they will be freed by munmap/exit_mmap, but any pmd we
|
|
|
|
* preallocate which never got a corresponding vma will need to be
|
|
|
|
* freed manually.
|
|
|
|
*/
|
|
|
|
static void pgd_mop_up_pmds(struct mm_struct *mm, pgd_t *pgdp)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2008-06-25 12:19:13 +08:00
|
|
|
for(i = 0; i < PREALLOCATED_PMDS; i++) {
|
2008-03-18 07:36:55 +08:00
|
|
|
pgd_t pgd = pgdp[i];
|
|
|
|
|
|
|
|
if (pgd_val(pgd) != 0) {
|
|
|
|
pmd_t *pmd = (pmd_t *)pgd_page_vaddr(pgd);
|
|
|
|
|
|
|
|
pgdp[i] = native_make_pgd(0);
|
|
|
|
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_release_pmd(pgd_val(pgd) >> PAGE_SHIFT);
|
2008-03-18 07:36:55 +08:00
|
|
|
pmd_free(mm, pmd);
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
mm_dec_nr_pmds(mm);
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-06-25 12:19:13 +08:00
|
|
|
static void pgd_prepopulate_pmd(struct mm_struct *mm, pgd_t *pgd, pmd_t *pmds[])
|
2008-03-18 07:36:55 +08:00
|
|
|
{
|
2017-03-13 22:33:05 +08:00
|
|
|
p4d_t *p4d;
|
2008-03-18 07:36:55 +08:00
|
|
|
pud_t *pud;
|
|
|
|
int i;
|
|
|
|
|
2008-08-09 04:46:07 +08:00
|
|
|
if (PREALLOCATED_PMDS == 0) /* Work around gcc-3.4.x bug */
|
|
|
|
return;
|
|
|
|
|
2017-03-13 22:33:05 +08:00
|
|
|
p4d = p4d_offset(pgd, 0);
|
|
|
|
pud = pud_offset(p4d, 0);
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2013-07-09 07:00:17 +08:00
|
|
|
for (i = 0; i < PREALLOCATED_PMDS; i++, pud++) {
|
2008-06-25 12:19:13 +08:00
|
|
|
pmd_t *pmd = pmds[i];
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2008-03-18 07:37:13 +08:00
|
|
|
if (i >= KERNEL_PGD_BOUNDARY)
|
2008-03-18 07:36:55 +08:00
|
|
|
memcpy(pmd, (pmd_t *)pgd_page_vaddr(swapper_pg_dir[i]),
|
|
|
|
sizeof(pmd_t) * PTRS_PER_PMD);
|
|
|
|
|
|
|
|
pud_populate(mm, pud, pmd);
|
|
|
|
}
|
|
|
|
}
|
2008-03-20 03:30:40 +08:00
|
|
|
|
2015-01-16 12:30:01 +08:00
|
|
|
/*
|
|
|
|
* Xen paravirt assumes pgd table should be in one page. 64 bit kernel also
|
|
|
|
* assumes that pgd should be in one page.
|
|
|
|
*
|
|
|
|
* But kernel with PAE paging that is not running as a Xen domain
|
|
|
|
* only needs to allocate 32 bytes for pgd instead of one page.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_X86_PAE
|
|
|
|
|
|
|
|
#include <linux/slab.h>
|
|
|
|
|
|
|
|
#define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t))
|
|
|
|
#define PGD_ALIGN 32
|
|
|
|
|
|
|
|
static struct kmem_cache *pgd_cache;
|
|
|
|
|
|
|
|
static int __init pgd_cache_init(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* When PAE kernel is running as a Xen domain, it does not use
|
|
|
|
* shared kernel pmd. And this requires a whole page for pgd.
|
|
|
|
*/
|
|
|
|
if (!SHARED_KERNEL_PMD)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* when PAE kernel is not running as a Xen domain, it uses
|
|
|
|
* shared kernel pmd. Shared kernel pmd does not require a whole
|
|
|
|
* page for pgd. We are able to just allocate a 32-byte for pgd.
|
|
|
|
* During boot time, we create a 32-byte slab for pgd table allocation.
|
|
|
|
*/
|
|
|
|
pgd_cache = kmem_cache_create("pgd_cache", PGD_SIZE, PGD_ALIGN,
|
|
|
|
SLAB_PANIC, NULL);
|
|
|
|
if (!pgd_cache)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
core_initcall(pgd_cache_init);
|
|
|
|
|
|
|
|
static inline pgd_t *_pgd_alloc(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If no SHARED_KERNEL_PMD, PAE kernel is running as a Xen domain.
|
|
|
|
* We allocate one page for pgd.
|
|
|
|
*/
|
|
|
|
if (!SHARED_KERNEL_PMD)
|
|
|
|
return (pgd_t *)__get_free_page(PGALLOC_GFP);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Now PAE kernel is not running as a Xen domain. We can allocate
|
|
|
|
* a 32-byte slab for pgd to save memory space.
|
|
|
|
*/
|
|
|
|
return kmem_cache_alloc(pgd_cache, PGALLOC_GFP);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void _pgd_free(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
if (!SHARED_KERNEL_PMD)
|
|
|
|
free_page((unsigned long)pgd);
|
|
|
|
else
|
|
|
|
kmem_cache_free(pgd_cache, pgd);
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
static inline pgd_t *_pgd_alloc(void)
|
|
|
|
{
|
|
|
|
return (pgd_t *)__get_free_page(PGALLOC_GFP);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void _pgd_free(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
free_page((unsigned long)pgd);
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_X86_PAE */
|
|
|
|
|
2008-06-25 12:19:13 +08:00
|
|
|
pgd_t *pgd_alloc(struct mm_struct *mm)
|
2008-03-20 03:30:40 +08:00
|
|
|
{
|
2008-06-25 12:19:13 +08:00
|
|
|
pgd_t *pgd;
|
|
|
|
pmd_t *pmds[PREALLOCATED_PMDS];
|
2008-03-20 03:30:40 +08:00
|
|
|
|
2015-01-16 12:30:01 +08:00
|
|
|
pgd = _pgd_alloc();
|
2008-06-25 12:19:13 +08:00
|
|
|
|
|
|
|
if (pgd == NULL)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
mm->pgd = pgd;
|
|
|
|
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
if (preallocate_pmds(mm, pmds) != 0)
|
2008-06-25 12:19:13 +08:00
|
|
|
goto out_free_pgd;
|
|
|
|
|
|
|
|
if (paravirt_pgd_alloc(mm) != 0)
|
|
|
|
goto out_free_pmds;
|
2008-03-20 03:30:40 +08:00
|
|
|
|
|
|
|
/*
|
2008-06-25 12:19:13 +08:00
|
|
|
* Make sure that pre-populating the pmds is atomic with
|
|
|
|
* respect to anything walking the pgd_list, so that they
|
|
|
|
* never see a partially populated pgd.
|
2008-03-20 03:30:40 +08:00
|
|
|
*/
|
2011-02-17 07:45:22 +08:00
|
|
|
spin_lock(&pgd_lock);
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2010-09-22 03:01:51 +08:00
|
|
|
pgd_ctor(mm, pgd);
|
2008-06-25 12:19:13 +08:00
|
|
|
pgd_prepopulate_pmd(mm, pgd, pmds);
|
2008-03-18 07:36:55 +08:00
|
|
|
|
2011-02-17 07:45:22 +08:00
|
|
|
spin_unlock(&pgd_lock);
|
2008-03-18 07:36:55 +08:00
|
|
|
|
|
|
|
return pgd;
|
2008-06-25 12:19:13 +08:00
|
|
|
|
|
|
|
out_free_pmds:
|
mm: account pmd page tables to the process
Dave noticed that unprivileged process can allocate significant amount of
memory -- >500 MiB on x86_64 -- and stay unnoticed by oom-killer and
memory cgroup. The trick is to allocate a lot of PMD page tables. Linux
kernel doesn't account PMD tables to the process, only PTE.
The use-cases below use few tricks to allocate a lot of PMD page tables
while keeping VmRSS and VmPTE low. oom_score for the process will be 0.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#define PUD_SIZE (1UL << 30)
#define PMD_SIZE (1UL << 21)
#define NR_PUD 130000
int main(void)
{
char *addr = NULL;
unsigned long i;
prctl(PR_SET_THP_DISABLE);
for (i = 0; i < NR_PUD ; i++) {
addr = mmap(addr + PUD_SIZE, PUD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
break;
}
*addr = 'x';
munmap(addr, PMD_SIZE);
mmap(addr, PMD_SIZE, PROT_WRITE|PROT_READ,
MAP_ANONYMOUS|MAP_PRIVATE|MAP_FIXED, -1, 0);
if (addr == MAP_FAILED)
perror("re-mmap"), exit(1);
}
printf("PID %d consumed %lu KiB in PMD page tables\n",
getpid(), i * 4096 >> 10);
return pause();
}
The patch addresses the issue by account PMD tables to the process the
same way we account PTE.
The main place where PMD tables is accounted is __pmd_alloc() and
free_pmd_range(). But there're few corner cases:
- HugeTLB can share PMD page tables. The patch handles by accounting
the table to all processes who share it.
- x86 PAE pre-allocates few PMD tables on fork.
- Architectures with FIRST_USER_ADDRESS > 0. We need to adjust sanity
check on exit(2).
Accounting only happens on configuration where PMD page table's level is
present (PMD is not folded). As with nr_ptes we use per-mm counter. The
counter value is used to calculate baseline for badness score by
oom-killer.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Reviewed-by: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@openvz.org>
Cc: David Rientjes <rientjes@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 07:26:50 +08:00
|
|
|
free_pmds(mm, pmds);
|
2008-06-25 12:19:13 +08:00
|
|
|
out_free_pgd:
|
2015-01-16 12:30:01 +08:00
|
|
|
_pgd_free(pgd);
|
2008-06-25 12:19:13 +08:00
|
|
|
out:
|
|
|
|
return NULL;
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void pgd_free(struct mm_struct *mm, pgd_t *pgd)
|
|
|
|
{
|
|
|
|
pgd_mop_up_pmds(mm, pgd);
|
|
|
|
pgd_dtor(pgd);
|
2008-06-25 12:19:12 +08:00
|
|
|
paravirt_pgd_free(mm, pgd);
|
2015-01-16 12:30:01 +08:00
|
|
|
_pgd_free(pgd);
|
2008-03-18 07:36:55 +08:00
|
|
|
}
|
2008-03-18 07:37:03 +08:00
|
|
|
|
2012-11-06 17:54:47 +08:00
|
|
|
/*
|
|
|
|
* Used to set accessed or dirty bits in the page table entries
|
|
|
|
* on other architectures. On x86, the accessed and dirty bits
|
|
|
|
* are tracked by hardware. However, do_wp_page calls this function
|
|
|
|
* to also make the pte writeable at the same time the dirty bit is
|
|
|
|
* set. In that case we do actually need to write the PTE.
|
|
|
|
*/
|
2008-03-18 07:37:03 +08:00
|
|
|
int ptep_set_access_flags(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, pte_t *ptep,
|
|
|
|
pte_t entry, int dirty)
|
|
|
|
{
|
|
|
|
int changed = !pte_same(*ptep, entry);
|
|
|
|
|
2017-09-04 18:25:27 +08:00
|
|
|
if (changed && dirty)
|
2008-03-18 07:37:03 +08:00
|
|
|
*ptep = entry;
|
|
|
|
|
|
|
|
return changed;
|
|
|
|
}
|
2008-03-18 07:37:04 +08:00
|
|
|
|
2011-01-14 07:46:41 +08:00
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
|
|
int pmdp_set_access_flags(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, pmd_t *pmdp,
|
|
|
|
pmd_t entry, int dirty)
|
|
|
|
{
|
|
|
|
int changed = !pmd_same(*pmdp, entry);
|
|
|
|
|
|
|
|
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
if (changed && dirty) {
|
|
|
|
*pmdp = entry;
|
2012-11-20 20:02:51 +08:00
|
|
|
/*
|
|
|
|
* We had a write-protection fault here and changed the pmd
|
|
|
|
* to to more permissive. No need to flush the TLB for that,
|
|
|
|
* #PF is architecturally guaranteed to do that and in the
|
|
|
|
* worst-case we'll generate a spurious fault.
|
|
|
|
*/
|
2011-01-14 07:46:41 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return changed;
|
|
|
|
}
|
2017-02-25 06:57:02 +08:00
|
|
|
|
|
|
|
int pudp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
|
|
|
|
pud_t *pudp, pud_t entry, int dirty)
|
|
|
|
{
|
|
|
|
int changed = !pud_same(*pudp, entry);
|
|
|
|
|
|
|
|
VM_BUG_ON(address & ~HPAGE_PUD_MASK);
|
|
|
|
|
|
|
|
if (changed && dirty) {
|
|
|
|
*pudp = entry;
|
|
|
|
/*
|
|
|
|
* We had a write-protection fault here and changed the pud
|
|
|
|
* to to more permissive. No need to flush the TLB for that,
|
|
|
|
* #PF is architecturally guaranteed to do that and in the
|
|
|
|
* worst-case we'll generate a spurious fault.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
return changed;
|
|
|
|
}
|
2011-01-14 07:46:41 +08:00
|
|
|
#endif
|
|
|
|
|
2008-03-18 07:37:04 +08:00
|
|
|
int ptep_test_and_clear_young(struct vm_area_struct *vma,
|
|
|
|
unsigned long addr, pte_t *ptep)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
if (pte_young(*ptep))
|
|
|
|
ret = test_and_clear_bit(_PAGE_BIT_ACCESSED,
|
2008-05-24 23:24:34 +08:00
|
|
|
(unsigned long *) &ptep->pte);
|
2008-03-18 07:37:04 +08:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
2008-03-18 07:37:05 +08:00
|
|
|
|
2011-01-14 07:46:41 +08:00
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
|
|
int pmdp_test_and_clear_young(struct vm_area_struct *vma,
|
|
|
|
unsigned long addr, pmd_t *pmdp)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
if (pmd_young(*pmdp))
|
|
|
|
ret = test_and_clear_bit(_PAGE_BIT_ACCESSED,
|
2011-01-14 07:47:01 +08:00
|
|
|
(unsigned long *)pmdp);
|
2011-01-14 07:46:41 +08:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
2017-02-25 06:57:02 +08:00
|
|
|
int pudp_test_and_clear_young(struct vm_area_struct *vma,
|
|
|
|
unsigned long addr, pud_t *pudp)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
if (pud_young(*pudp))
|
|
|
|
ret = test_and_clear_bit(_PAGE_BIT_ACCESSED,
|
|
|
|
(unsigned long *)pudp);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
2011-01-14 07:46:41 +08:00
|
|
|
#endif
|
|
|
|
|
2008-03-18 07:37:05 +08:00
|
|
|
int ptep_clear_flush_young(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, pte_t *ptep)
|
|
|
|
{
|
2014-04-08 15:58:09 +08:00
|
|
|
/*
|
|
|
|
* On x86 CPUs, clearing the accessed bit without a TLB flush
|
|
|
|
* doesn't cause data corruption. [ It could cause incorrect
|
|
|
|
* page aging and the (mistaken) reclaim of hot pages, but the
|
|
|
|
* chance of that should be relatively low. ]
|
|
|
|
*
|
|
|
|
* So as a performance optimization don't flush the TLB when
|
|
|
|
* clearing the accessed bit, it will eventually be flushed by
|
|
|
|
* a context switch or a VM operation anyway. [ In the rare
|
|
|
|
* event of it not getting flushed for a long time the delay
|
|
|
|
* shouldn't really matter because there's no real memory
|
|
|
|
* pressure for swapout to react to. ]
|
|
|
|
*/
|
|
|
|
return ptep_test_and_clear_young(vma, address, ptep);
|
2008-03-18 07:37:05 +08:00
|
|
|
}
|
2008-06-18 02:41:54 +08:00
|
|
|
|
2011-01-14 07:46:41 +08:00
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
|
|
int pmdp_clear_flush_young(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, pmd_t *pmdp)
|
|
|
|
{
|
|
|
|
int young;
|
|
|
|
|
|
|
|
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
young = pmdp_test_and_clear_young(vma, address, pmdp);
|
|
|
|
if (young)
|
|
|
|
flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
|
|
|
|
|
|
|
|
return young;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2009-02-16 08:48:54 +08:00
|
|
|
/**
|
|
|
|
* reserve_top_address - reserves a hole in the top of kernel address space
|
|
|
|
* @reserve - size of hole to reserve
|
|
|
|
*
|
|
|
|
* Can be used to relocate the fixmap area and poke a hole in the top
|
|
|
|
* of kernel address space to make room for a hypervisor.
|
|
|
|
*/
|
|
|
|
void __init reserve_top_address(unsigned long reserve)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
BUG_ON(fixmaps_set > 0);
|
2014-05-06 03:19:31 +08:00
|
|
|
__FIXADDR_TOP = round_down(-reserve, 1 << PMD_SHIFT) - PAGE_SIZE;
|
|
|
|
printk(KERN_INFO "Reserving virtual address space above 0x%08lx (rounded to 0x%08lx)\n",
|
|
|
|
-reserve, __FIXADDR_TOP + PAGE_SIZE);
|
2009-02-16 08:48:54 +08:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2008-06-18 02:41:54 +08:00
|
|
|
int fixmaps_set;
|
|
|
|
|
2008-06-18 02:42:01 +08:00
|
|
|
void __native_set_fixmap(enum fixed_addresses idx, pte_t pte)
|
2008-06-18 02:41:54 +08:00
|
|
|
{
|
|
|
|
unsigned long address = __fix_to_virt(idx);
|
|
|
|
|
|
|
|
if (idx >= __end_of_fixed_addresses) {
|
|
|
|
BUG();
|
|
|
|
return;
|
|
|
|
}
|
2008-06-18 02:42:01 +08:00
|
|
|
set_pte_vaddr(address, pte);
|
2008-06-18 02:41:54 +08:00
|
|
|
fixmaps_set++;
|
|
|
|
}
|
2008-06-18 02:42:01 +08:00
|
|
|
|
2009-04-10 01:55:33 +08:00
|
|
|
void native_set_fixmap(enum fixed_addresses idx, phys_addr_t phys,
|
|
|
|
pgprot_t flags)
|
2008-06-18 02:42:01 +08:00
|
|
|
{
|
|
|
|
__native_set_fixmap(idx, pfn_pte(phys >> PAGE_SHIFT, flags));
|
|
|
|
}
|
2015-04-15 06:47:32 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP
|
2017-03-30 16:07:29 +08:00
|
|
|
#ifdef CONFIG_X86_5LEVEL
|
|
|
|
/**
|
|
|
|
* p4d_set_huge - setup kernel P4D mapping
|
|
|
|
*
|
|
|
|
* No 512GB pages yet -- always return 0
|
|
|
|
*/
|
|
|
|
int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* p4d_clear_huge - clear kernel P4D mapping when it is set
|
|
|
|
*
|
|
|
|
* No 512GB pages yet -- always return 0
|
|
|
|
*/
|
|
|
|
int p4d_clear_huge(p4d_t *p4d)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2015-05-26 16:28:07 +08:00
|
|
|
/**
|
|
|
|
* pud_set_huge - setup kernel PUD mapping
|
|
|
|
*
|
2015-05-26 16:28:10 +08:00
|
|
|
* MTRRs can override PAT memory types with 4KiB granularity. Therefore, this
|
|
|
|
* function sets up a huge page only if any of the following conditions are met:
|
|
|
|
*
|
|
|
|
* - MTRRs are disabled, or
|
|
|
|
*
|
|
|
|
* - MTRRs are enabled and the range is completely covered by a single MTRR, or
|
|
|
|
*
|
|
|
|
* - MTRRs are enabled and the corresponding MTRR memory type is WB, which
|
|
|
|
* has no effect on the requested PAT memory type.
|
|
|
|
*
|
|
|
|
* Callers should try to decrease page size (1GB -> 2MB -> 4K) if the bigger
|
|
|
|
* page mapping attempt fails.
|
2015-05-26 16:28:07 +08:00
|
|
|
*
|
|
|
|
* Returns 1 on success and 0 on failure.
|
|
|
|
*/
|
2015-04-15 06:47:32 +08:00
|
|
|
int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot)
|
|
|
|
{
|
2015-05-26 16:28:10 +08:00
|
|
|
u8 mtrr, uniform;
|
2015-04-15 06:47:32 +08:00
|
|
|
|
2015-05-26 16:28:10 +08:00
|
|
|
mtrr = mtrr_type_lookup(addr, addr + PUD_SIZE, &uniform);
|
|
|
|
if ((mtrr != MTRR_TYPE_INVALID) && (!uniform) &&
|
|
|
|
(mtrr != MTRR_TYPE_WRBACK))
|
2015-04-15 06:47:32 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
prot = pgprot_4k_2_large(prot);
|
|
|
|
|
|
|
|
set_pte((pte_t *)pud, pfn_pte(
|
|
|
|
(u64)addr >> PAGE_SHIFT,
|
|
|
|
__pgprot(pgprot_val(prot) | _PAGE_PSE)));
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2015-05-26 16:28:07 +08:00
|
|
|
/**
|
|
|
|
* pmd_set_huge - setup kernel PMD mapping
|
|
|
|
*
|
2015-05-26 16:28:10 +08:00
|
|
|
* See text over pud_set_huge() above.
|
2015-05-26 16:28:07 +08:00
|
|
|
*
|
|
|
|
* Returns 1 on success and 0 on failure.
|
|
|
|
*/
|
2015-04-15 06:47:32 +08:00
|
|
|
int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot)
|
|
|
|
{
|
2015-05-26 16:28:10 +08:00
|
|
|
u8 mtrr, uniform;
|
2015-04-15 06:47:32 +08:00
|
|
|
|
2015-05-26 16:28:10 +08:00
|
|
|
mtrr = mtrr_type_lookup(addr, addr + PMD_SIZE, &uniform);
|
|
|
|
if ((mtrr != MTRR_TYPE_INVALID) && (!uniform) &&
|
|
|
|
(mtrr != MTRR_TYPE_WRBACK)) {
|
|
|
|
pr_warn_once("%s: Cannot satisfy [mem %#010llx-%#010llx] with a huge-page mapping due to MTRR override.\n",
|
|
|
|
__func__, addr, addr + PMD_SIZE);
|
2015-04-15 06:47:32 +08:00
|
|
|
return 0;
|
2015-05-26 16:28:10 +08:00
|
|
|
}
|
2015-04-15 06:47:32 +08:00
|
|
|
|
|
|
|
prot = pgprot_4k_2_large(prot);
|
|
|
|
|
|
|
|
set_pte((pte_t *)pmd, pfn_pte(
|
|
|
|
(u64)addr >> PAGE_SHIFT,
|
|
|
|
__pgprot(pgprot_val(prot) | _PAGE_PSE)));
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2015-05-26 16:28:07 +08:00
|
|
|
/**
|
|
|
|
* pud_clear_huge - clear kernel PUD mapping when it is set
|
|
|
|
*
|
|
|
|
* Returns 1 on success and 0 on failure (no PUD map is found).
|
|
|
|
*/
|
2015-04-15 06:47:32 +08:00
|
|
|
int pud_clear_huge(pud_t *pud)
|
|
|
|
{
|
|
|
|
if (pud_large(*pud)) {
|
|
|
|
pud_clear(pud);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-05-26 16:28:07 +08:00
|
|
|
/**
|
|
|
|
* pmd_clear_huge - clear kernel PMD mapping when it is set
|
|
|
|
*
|
|
|
|
* Returns 1 on success and 0 on failure (no PMD map is found).
|
|
|
|
*/
|
2015-04-15 06:47:32 +08:00
|
|
|
int pmd_clear_huge(pmd_t *pmd)
|
|
|
|
{
|
|
|
|
if (pmd_large(*pmd)) {
|
|
|
|
pmd_clear(pmd);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */
|