2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
*
|
|
|
|
* Copyright (C) 1995 Linus Torvalds
|
|
|
|
*
|
|
|
|
* Support of BIGMEM added by Gerhard Wichert, Siemens AG, July 1999
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/signal.h>
|
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/errno.h>
|
|
|
|
#include <linux/string.h>
|
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/ptrace.h>
|
|
|
|
#include <linux/mman.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/hugetlb.h>
|
|
|
|
#include <linux/swap.h>
|
|
|
|
#include <linux/smp.h>
|
|
|
|
#include <linux/init.h>
|
|
|
|
#include <linux/highmem.h>
|
|
|
|
#include <linux/pagemap.h>
|
2008-12-17 04:17:36 +08:00
|
|
|
#include <linux/pci.h>
|
[PATCH] x86: tighten kernel image page access rights
On x86-64, kernel memory freed after init can be entirely unmapped instead
of just getting 'poisoned' by overwriting with a debug pattern.
On i386 and x86-64 (under CONFIG_DEBUG_RODATA), kernel text and bug table
can also be write-protected.
Compared to the first version, this one prevents re-creating deleted
mappings in the kernel image range on x86-64, if those got removed
previously. This, together with the original changes, prevents temporarily
having inconsistent mappings when cacheability attributes are being
changed on such pages (e.g. from AGP code). While on i386 such duplicate
mappings don't exist, the same change is done there, too, both for
consistency and because checking pte_present() before using various other
pte_XXX functions is a requirement anyway. At once, i386 code gets
adjusted to use pte_huge() instead of open coding this.
AK: split out cpa() changes
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Signed-off-by: Andi Kleen <ak@suse.de>
2007-05-03 01:27:10 +08:00
|
|
|
#include <linux/pfn.h>
|
2006-06-27 17:53:52 +08:00
|
|
|
#include <linux/poison.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <linux/bootmem.h>
|
2010-08-26 04:39:17 +08:00
|
|
|
#include <linux/memblock.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <linux/proc_fs.h>
|
2005-10-30 09:16:57 +08:00
|
|
|
#include <linux/memory_hotplug.h>
|
2005-11-14 08:06:51 +08:00
|
|
|
#include <linux/initrd.h>
|
2006-06-23 17:04:49 +08:00
|
|
|
#include <linux/cpumask.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/gfp.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-02-04 23:47:58 +08:00
|
|
|
#include <asm/asm.h>
|
2008-10-12 21:06:29 +08:00
|
|
|
#include <asm/bios_ebda.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <asm/processor.h>
|
|
|
|
#include <asm/uaccess.h>
|
|
|
|
#include <asm/pgtable.h>
|
|
|
|
#include <asm/dma.h>
|
|
|
|
#include <asm/fixmap.h>
|
|
|
|
#include <asm/e820.h>
|
|
|
|
#include <asm/apic.h>
|
2008-01-30 20:34:10 +08:00
|
|
|
#include <asm/bugs.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <asm/tlb.h>
|
|
|
|
#include <asm/tlbflush.h>
|
2010-11-17 14:09:52 +08:00
|
|
|
#include <asm/olpc_ofw.h>
|
2008-01-30 20:33:39 +08:00
|
|
|
#include <asm/pgalloc.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <asm/sections.h>
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
#include <asm/paravirt.h>
|
2008-02-10 06:24:09 +08:00
|
|
|
#include <asm/setup.h>
|
2008-02-13 04:12:01 +08:00
|
|
|
#include <asm/cacheflush.h>
|
2009-04-28 21:00:49 +08:00
|
|
|
#include <asm/page_types.h>
|
2009-03-05 20:55:08 +08:00
|
|
|
#include <asm/init.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-11-17 11:39:01 +08:00
|
|
|
#include "mm_internal.h"
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
unsigned long highstart_pfn, highend_pfn;
|
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
static noinline int do_test_wp_bit(void);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-03-05 08:10:44 +08:00
|
|
|
bool __read_mostly __vmalloc_start_set = false;
|
2008-06-25 03:18:14 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Creates a middle page table and puts a pointer to it in the
|
|
|
|
* given global directory entry. This only returns the gd entry
|
|
|
|
* in non-PAE compilation mode, since the middle layer is folded.
|
|
|
|
*/
|
|
|
|
static pmd_t * __init one_md_table_init(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
pud_t *pud;
|
|
|
|
pmd_t *pmd_table;
|
2008-01-30 20:34:10 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
#ifdef CONFIG_X86_PAE
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
if (!(pgd_val(*pgd) & _PAGE_PRESENT)) {
|
2012-11-17 11:39:16 +08:00
|
|
|
pmd_table = (pmd_t *)alloc_low_page();
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_alloc_pmd(&init_mm, __pa(pmd_table) >> PAGE_SHIFT);
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));
|
|
|
|
pud = pud_offset(pgd, 0);
|
2008-01-30 20:34:10 +08:00
|
|
|
BUG_ON(pmd_table != pmd_offset(pud, 0));
|
2008-10-31 17:43:04 +08:00
|
|
|
|
|
|
|
return pmd_table;
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
}
|
|
|
|
#endif
|
2005-04-17 06:20:36 +08:00
|
|
|
pud = pud_offset(pgd, 0);
|
|
|
|
pmd_table = pmd_offset(pud, 0);
|
2008-01-30 20:34:10 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
return pmd_table;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a page table and place a pointer to it in a middle page
|
2008-01-30 20:34:10 +08:00
|
|
|
* directory entry:
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
static pte_t * __init one_page_table_init(pmd_t *pmd)
|
|
|
|
{
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
if (!(pmd_val(*pmd) & _PAGE_PRESENT)) {
|
2012-11-17 11:39:16 +08:00
|
|
|
pte_t *page_table = (pte_t *)alloc_low_page();
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_alloc_pte(&init_mm, __pa(page_table) >> PAGE_SHIFT);
|
2005-04-17 06:20:36 +08:00
|
|
|
set_pmd(pmd, __pmd(__pa(page_table) | _PAGE_TABLE));
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
BUG_ON(page_table != pte_offset_kernel(pmd, 0));
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2007-10-18 00:04:34 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
return pte_offset_kernel(pmd, 0);
|
|
|
|
}
|
|
|
|
|
2009-02-24 10:57:21 +08:00
|
|
|
pmd_t * __init populate_extra_pmd(unsigned long vaddr)
|
2009-02-20 15:29:09 +08:00
|
|
|
{
|
|
|
|
int pgd_idx = pgd_index(vaddr);
|
|
|
|
int pmd_idx = pmd_index(vaddr);
|
2009-02-24 10:57:21 +08:00
|
|
|
|
|
|
|
return one_md_table_init(swapper_pg_dir + pgd_idx) + pmd_idx;
|
|
|
|
}
|
|
|
|
|
|
|
|
pte_t * __init populate_extra_pte(unsigned long vaddr)
|
|
|
|
{
|
|
|
|
int pte_idx = pte_index(vaddr);
|
2009-02-20 15:29:09 +08:00
|
|
|
pmd_t *pmd;
|
|
|
|
|
2009-02-24 10:57:21 +08:00
|
|
|
pmd = populate_extra_pmd(vaddr);
|
|
|
|
return one_page_table_init(pmd) + pte_idx;
|
2009-02-20 15:29:09 +08:00
|
|
|
}
|
|
|
|
|
2012-11-17 11:39:06 +08:00
|
|
|
static unsigned long __init
|
|
|
|
page_table_range_init_count(unsigned long start, unsigned long end)
|
|
|
|
{
|
|
|
|
unsigned long count = 0;
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
int pmd_idx_kmap_begin = fix_to_virt(FIX_KMAP_END) >> PMD_SHIFT;
|
|
|
|
int pmd_idx_kmap_end = fix_to_virt(FIX_KMAP_BEGIN) >> PMD_SHIFT;
|
|
|
|
int pgd_idx, pmd_idx;
|
|
|
|
unsigned long vaddr;
|
|
|
|
|
|
|
|
if (pmd_idx_kmap_begin == pmd_idx_kmap_end)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
vaddr = start;
|
|
|
|
pgd_idx = pgd_index(vaddr);
|
2015-07-12 20:18:42 +08:00
|
|
|
pmd_idx = pmd_index(vaddr);
|
2012-11-17 11:39:06 +08:00
|
|
|
|
|
|
|
for ( ; (pgd_idx < PTRS_PER_PGD) && (vaddr != end); pgd_idx++) {
|
|
|
|
for (; (pmd_idx < PTRS_PER_PMD) && (vaddr != end);
|
|
|
|
pmd_idx++) {
|
|
|
|
if ((vaddr >> PMD_SHIFT) >= pmd_idx_kmap_begin &&
|
|
|
|
(vaddr >> PMD_SHIFT) <= pmd_idx_kmap_end)
|
|
|
|
count++;
|
|
|
|
vaddr += PMD_SIZE;
|
|
|
|
}
|
|
|
|
pmd_idx = 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
|
2009-01-16 19:59:33 +08:00
|
|
|
static pte_t *__init page_table_kmap_check(pte_t *pte, pmd_t *pmd,
|
2012-11-17 11:39:06 +08:00
|
|
|
unsigned long vaddr, pte_t *lastpte,
|
|
|
|
void **adr)
|
2009-01-16 19:59:33 +08:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
/*
|
|
|
|
* Something (early fixmap) may already have put a pte
|
|
|
|
* page here, which causes the page table allocation
|
|
|
|
* to become nonlinear. Attempt to fix it, and if it
|
|
|
|
* is still nonlinear then we have to bug.
|
|
|
|
*/
|
|
|
|
int pmd_idx_kmap_begin = fix_to_virt(FIX_KMAP_END) >> PMD_SHIFT;
|
|
|
|
int pmd_idx_kmap_end = fix_to_virt(FIX_KMAP_BEGIN) >> PMD_SHIFT;
|
|
|
|
|
|
|
|
if (pmd_idx_kmap_begin != pmd_idx_kmap_end
|
|
|
|
&& (vaddr >> PMD_SHIFT) >= pmd_idx_kmap_begin
|
2012-11-17 11:39:06 +08:00
|
|
|
&& (vaddr >> PMD_SHIFT) <= pmd_idx_kmap_end) {
|
2009-01-16 19:59:33 +08:00
|
|
|
pte_t *newpte;
|
|
|
|
int i;
|
|
|
|
|
2009-03-05 20:54:58 +08:00
|
|
|
BUG_ON(after_bootmem);
|
2012-11-17 11:39:06 +08:00
|
|
|
newpte = *adr;
|
2009-01-16 19:59:33 +08:00
|
|
|
for (i = 0; i < PTRS_PER_PTE; i++)
|
|
|
|
set_pte(newpte + i, pte[i]);
|
2012-11-17 11:39:06 +08:00
|
|
|
*adr = (void *)(((unsigned long)(*adr)) + PAGE_SIZE);
|
2009-01-16 19:59:33 +08:00
|
|
|
|
|
|
|
paravirt_alloc_pte(&init_mm, __pa(newpte) >> PAGE_SHIFT);
|
|
|
|
set_pmd(pmd, __pmd(__pa(newpte)|_PAGE_TABLE));
|
|
|
|
BUG_ON(newpte != pte_offset_kernel(pmd, 0));
|
|
|
|
__flush_tlb_all();
|
|
|
|
|
|
|
|
paravirt_release_pte(__pa(pte) >> PAGE_SHIFT);
|
|
|
|
pte = newpte;
|
|
|
|
}
|
|
|
|
BUG_ON(vaddr < fix_to_virt(FIX_KMAP_BEGIN - 1)
|
|
|
|
&& vaddr > fix_to_virt(FIX_KMAP_END)
|
|
|
|
&& lastpte && lastpte + PTRS_PER_PTE != pte);
|
|
|
|
#endif
|
|
|
|
return pte;
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
2008-01-30 20:34:10 +08:00
|
|
|
* This function initializes a certain range of kernel virtual memory
|
2005-04-17 06:20:36 +08:00
|
|
|
* with new bootmem page tables, everywhere page tables are missing in
|
|
|
|
* the given range.
|
2008-01-30 20:34:10 +08:00
|
|
|
*
|
|
|
|
* NOTE: The pagetables are allocated contiguous on the physical space
|
|
|
|
* so we can cache the place of the first one and move around without
|
2005-04-17 06:20:36 +08:00
|
|
|
* checking the pgd every time.
|
|
|
|
*/
|
2008-01-30 20:34:10 +08:00
|
|
|
static void __init
|
|
|
|
page_table_range_init(unsigned long start, unsigned long end, pgd_t *pgd_base)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
int pgd_idx, pmd_idx;
|
|
|
|
unsigned long vaddr;
|
2008-01-30 20:34:10 +08:00
|
|
|
pgd_t *pgd;
|
|
|
|
pmd_t *pmd;
|
2009-01-16 19:59:33 +08:00
|
|
|
pte_t *pte = NULL;
|
2012-11-17 11:39:06 +08:00
|
|
|
unsigned long count = page_table_range_init_count(start, end);
|
|
|
|
void *adr = NULL;
|
|
|
|
|
|
|
|
if (count)
|
|
|
|
adr = alloc_low_pages(count);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
vaddr = start;
|
|
|
|
pgd_idx = pgd_index(vaddr);
|
|
|
|
pmd_idx = pmd_index(vaddr);
|
|
|
|
pgd = pgd_base + pgd_idx;
|
|
|
|
|
|
|
|
for ( ; (pgd_idx < PTRS_PER_PGD) && (vaddr != end); pgd++, pgd_idx++) {
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
pmd = one_md_table_init(pgd);
|
|
|
|
pmd = pmd + pmd_index(vaddr);
|
2008-01-30 20:34:10 +08:00
|
|
|
for (; (pmd_idx < PTRS_PER_PMD) && (vaddr != end);
|
|
|
|
pmd++, pmd_idx++) {
|
2009-01-16 19:59:33 +08:00
|
|
|
pte = page_table_kmap_check(one_page_table_init(pmd),
|
2012-11-17 11:39:06 +08:00
|
|
|
pmd, vaddr, pte, &adr);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
vaddr += PMD_SIZE;
|
|
|
|
}
|
|
|
|
pmd_idx = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int is_kernel_text(unsigned long addr)
|
|
|
|
{
|
2010-11-17 05:31:26 +08:00
|
|
|
if (addr >= (unsigned long)_text && addr <= (unsigned long)__init_end)
|
2005-04-17 06:20:36 +08:00
|
|
|
return 1;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2008-01-30 20:34:10 +08:00
|
|
|
* This maps the physical memory to kernel virtual address space, a total
|
|
|
|
* of max_low_pfn pages, by creating page tables starting from address
|
|
|
|
* PAGE_OFFSET:
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
2009-03-05 20:55:07 +08:00
|
|
|
unsigned long __init
|
|
|
|
kernel_physical_mapping_init(unsigned long start,
|
|
|
|
unsigned long end,
|
|
|
|
unsigned long page_size_mask)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2009-03-05 20:55:07 +08:00
|
|
|
int use_pse = page_size_mask == (1<<PG_LEVEL_2M);
|
2010-02-24 23:04:47 +08:00
|
|
|
unsigned long last_map_addr = end;
|
2009-03-05 20:55:07 +08:00
|
|
|
unsigned long start_pfn, end_pfn;
|
2009-03-05 20:54:55 +08:00
|
|
|
pgd_t *pgd_base = swapper_pg_dir;
|
2008-01-30 20:34:10 +08:00
|
|
|
int pgd_idx, pmd_idx, pte_ofs;
|
2005-04-17 06:20:36 +08:00
|
|
|
unsigned long pfn;
|
|
|
|
pgd_t *pgd;
|
|
|
|
pmd_t *pmd;
|
|
|
|
pte_t *pte;
|
2008-09-24 05:00:38 +08:00
|
|
|
unsigned pages_2m, pages_4k;
|
|
|
|
int mapping_iter;
|
|
|
|
|
2009-03-05 20:55:07 +08:00
|
|
|
start_pfn = start >> PAGE_SHIFT;
|
|
|
|
end_pfn = end >> PAGE_SHIFT;
|
|
|
|
|
2008-09-24 05:00:38 +08:00
|
|
|
/*
|
|
|
|
* First iteration will setup identity mapping using large/small pages
|
|
|
|
* based on use_pse, with other attributes same as set by
|
|
|
|
* the early code in head_32.S
|
|
|
|
*
|
|
|
|
* Second iteration will setup the appropriate attributes (NX, GLOBAL..)
|
|
|
|
* as desired for the kernel identity mapping.
|
|
|
|
*
|
|
|
|
* This two pass mechanism conforms to the TLB app note which says:
|
|
|
|
*
|
|
|
|
* "Software should not write to a paging-structure entry in a way
|
|
|
|
* that would change, for any linear address, both the page size
|
|
|
|
* and either the page frame or attributes."
|
|
|
|
*/
|
|
|
|
mapping_iter = 1;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2016-03-29 23:42:03 +08:00
|
|
|
if (!boot_cpu_has(X86_FEATURE_PSE))
|
2008-06-29 15:39:06 +08:00
|
|
|
use_pse = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-09-24 05:00:38 +08:00
|
|
|
repeat:
|
|
|
|
pages_2m = pages_4k = 0;
|
2008-06-29 15:39:06 +08:00
|
|
|
pfn = start_pfn;
|
|
|
|
pgd_idx = pgd_index((pfn<<PAGE_SHIFT) + PAGE_OFFSET);
|
|
|
|
pgd = pgd_base + pgd_idx;
|
2005-04-17 06:20:36 +08:00
|
|
|
for (; pgd_idx < PTRS_PER_PGD; pgd++, pgd_idx++) {
|
|
|
|
pmd = one_md_table_init(pgd);
|
2008-01-30 20:34:10 +08:00
|
|
|
|
2008-06-29 15:39:06 +08:00
|
|
|
if (pfn >= end_pfn)
|
|
|
|
continue;
|
|
|
|
#ifdef CONFIG_X86_PAE
|
|
|
|
pmd_idx = pmd_index((pfn<<PAGE_SHIFT) + PAGE_OFFSET);
|
|
|
|
pmd += pmd_idx;
|
|
|
|
#else
|
|
|
|
pmd_idx = 0;
|
|
|
|
#endif
|
|
|
|
for (; pmd_idx < PTRS_PER_PMD && pfn < end_pfn;
|
2008-01-30 20:31:09 +08:00
|
|
|
pmd++, pmd_idx++) {
|
2008-01-30 20:34:10 +08:00
|
|
|
unsigned int addr = pfn * PAGE_SIZE + PAGE_OFFSET;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
/*
|
|
|
|
* Map with big pages if possible, otherwise
|
|
|
|
* create normal page tables:
|
|
|
|
*/
|
2008-06-29 15:39:06 +08:00
|
|
|
if (use_pse) {
|
2008-01-30 20:34:10 +08:00
|
|
|
unsigned int addr2;
|
2008-01-30 20:31:09 +08:00
|
|
|
pgprot_t prot = PAGE_KERNEL_LARGE;
|
2008-09-24 05:00:38 +08:00
|
|
|
/*
|
|
|
|
* first pass will use the same initial
|
|
|
|
* identity mapping attribute + _PAGE_PSE.
|
|
|
|
*/
|
|
|
|
pgprot_t init_prot =
|
|
|
|
__pgprot(PTE_IDENT_ATTR |
|
|
|
|
_PAGE_PSE);
|
2008-01-30 20:31:09 +08:00
|
|
|
|
2012-11-17 11:38:54 +08:00
|
|
|
pfn &= PMD_MASK >> PAGE_SHIFT;
|
2008-01-30 20:34:10 +08:00
|
|
|
addr2 = (pfn + PTRS_PER_PTE-1) * PAGE_SIZE +
|
2008-01-30 20:31:09 +08:00
|
|
|
PAGE_OFFSET + PAGE_SIZE-1;
|
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
if (is_kernel_text(addr) ||
|
|
|
|
is_kernel_text(addr2))
|
2008-01-30 20:31:09 +08:00
|
|
|
prot = PAGE_KERNEL_LARGE_EXEC;
|
|
|
|
|
2008-05-02 17:46:49 +08:00
|
|
|
pages_2m++;
|
2008-09-24 05:00:38 +08:00
|
|
|
if (mapping_iter == 1)
|
|
|
|
set_pmd(pmd, pfn_pmd(pfn, init_prot));
|
|
|
|
else
|
|
|
|
set_pmd(pmd, pfn_pmd(pfn, prot));
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
pfn += PTRS_PER_PTE;
|
2008-01-30 20:34:10 +08:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
pte = one_page_table_init(pmd);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-06-29 15:39:06 +08:00
|
|
|
pte_ofs = pte_index((pfn<<PAGE_SHIFT) + PAGE_OFFSET);
|
|
|
|
pte += pte_ofs;
|
|
|
|
for (; pte_ofs < PTRS_PER_PTE && pfn < end_pfn;
|
2008-01-30 20:34:10 +08:00
|
|
|
pte++, pfn++, pte_ofs++, addr += PAGE_SIZE) {
|
|
|
|
pgprot_t prot = PAGE_KERNEL;
|
2008-09-24 05:00:38 +08:00
|
|
|
/*
|
|
|
|
* first pass will use the same initial
|
|
|
|
* identity mapping attribute.
|
|
|
|
*/
|
|
|
|
pgprot_t init_prot = __pgprot(PTE_IDENT_ATTR);
|
2008-01-30 20:31:09 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
if (is_kernel_text(addr))
|
|
|
|
prot = PAGE_KERNEL_EXEC;
|
2008-01-30 20:31:09 +08:00
|
|
|
|
2008-05-02 17:46:49 +08:00
|
|
|
pages_4k++;
|
2010-02-24 23:04:47 +08:00
|
|
|
if (mapping_iter == 1) {
|
2008-09-24 05:00:38 +08:00
|
|
|
set_pte(pte, pfn_pte(pfn, init_prot));
|
2010-02-24 23:04:47 +08:00
|
|
|
last_map_addr = (pfn << PAGE_SHIFT) + PAGE_SIZE;
|
|
|
|
} else
|
2008-09-24 05:00:38 +08:00
|
|
|
set_pte(pte, pfn_pte(pfn, prot));
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2008-09-24 05:00:38 +08:00
|
|
|
if (mapping_iter == 1) {
|
|
|
|
/*
|
|
|
|
* update direct mapping page count only in the first
|
|
|
|
* iteration.
|
|
|
|
*/
|
|
|
|
update_page_count(PG_LEVEL_2M, pages_2m);
|
|
|
|
update_page_count(PG_LEVEL_4K, pages_4k);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* local global flush tlb, which will flush the previous
|
|
|
|
* mappings present in both small and large page TLB's.
|
|
|
|
*/
|
|
|
|
__flush_tlb_all();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Second iteration will set the actual desired PTE attributes.
|
|
|
|
*/
|
|
|
|
mapping_iter = 2;
|
|
|
|
goto repeat;
|
|
|
|
}
|
2010-02-24 23:04:47 +08:00
|
|
|
return last_map_addr;
|
2008-04-25 05:40:47 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
pte_t *kmap_pte;
|
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
static inline pte_t *kmap_get_fixmap_pte(unsigned long vaddr)
|
|
|
|
{
|
|
|
|
return pte_offset_kernel(pmd_offset(pud_offset(pgd_offset_k(vaddr),
|
|
|
|
vaddr), vaddr), vaddr);
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
static void __init kmap_init(void)
|
|
|
|
{
|
|
|
|
unsigned long kmap_vstart;
|
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
/*
|
|
|
|
* Cache the first kmap pte:
|
|
|
|
*/
|
2005-04-17 06:20:36 +08:00
|
|
|
kmap_vstart = __fix_to_virt(FIX_KMAP_BEGIN);
|
|
|
|
kmap_pte = kmap_get_fixmap_pte(kmap_vstart);
|
|
|
|
}
|
|
|
|
|
2008-10-31 10:37:09 +08:00
|
|
|
#ifdef CONFIG_HIGHMEM
|
2005-04-17 06:20:36 +08:00
|
|
|
static void __init permanent_kmaps_init(pgd_t *pgd_base)
|
|
|
|
{
|
2008-01-30 20:34:10 +08:00
|
|
|
unsigned long vaddr;
|
2005-04-17 06:20:36 +08:00
|
|
|
pgd_t *pgd;
|
|
|
|
pud_t *pud;
|
|
|
|
pmd_t *pmd;
|
|
|
|
pte_t *pte;
|
|
|
|
|
|
|
|
vaddr = PKMAP_BASE;
|
|
|
|
page_table_range_init(vaddr, vaddr + PAGE_SIZE*LAST_PKMAP, pgd_base);
|
|
|
|
|
|
|
|
pgd = swapper_pg_dir + pgd_index(vaddr);
|
|
|
|
pud = pud_offset(pgd, vaddr);
|
|
|
|
pmd = pmd_offset(pud, vaddr);
|
|
|
|
pte = pte_offset_kernel(pmd, vaddr);
|
2008-01-30 20:34:10 +08:00
|
|
|
pkmap_page_table = pte;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2010-10-06 07:15:15 +08:00
|
|
|
void __init add_highpages_with_active_regions(int nid,
|
|
|
|
unsigned long start_pfn, unsigned long end_pfn)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2011-07-12 17:16:02 +08:00
|
|
|
phys_addr_t start, end;
|
|
|
|
u64 i;
|
|
|
|
|
2015-06-25 07:58:09 +08:00
|
|
|
for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &start, &end, NULL) {
|
2011-07-12 17:16:02 +08:00
|
|
|
unsigned long pfn = clamp_t(unsigned long, PFN_UP(start),
|
|
|
|
start_pfn, end_pfn);
|
|
|
|
unsigned long e_pfn = clamp_t(unsigned long, PFN_DOWN(end),
|
|
|
|
start_pfn, end_pfn);
|
|
|
|
for ( ; pfn < e_pfn; pfn++)
|
|
|
|
if (pfn_valid(pfn))
|
2013-04-30 06:07:12 +08:00
|
|
|
free_highmem_page(pfn_to_page(pfn));
|
2008-01-15 23:44:37 +08:00
|
|
|
}
|
2008-06-15 09:32:52 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
#else
|
2009-01-02 21:42:00 +08:00
|
|
|
static inline void permanent_kmaps_init(pgd_t *pgd_base)
|
|
|
|
{
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
#endif /* CONFIG_HIGHMEM */
|
|
|
|
|
2012-08-22 04:22:38 +08:00
|
|
|
void __init native_pagetable_init(void)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2008-02-10 06:24:09 +08:00
|
|
|
unsigned long pfn, va;
|
2012-08-22 04:22:37 +08:00
|
|
|
pgd_t *pgd, *base = swapper_pg_dir;
|
2008-02-10 06:24:09 +08:00
|
|
|
pud_t *pud;
|
|
|
|
pmd_t *pmd;
|
|
|
|
pte_t *pte;
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
|
|
|
|
/*
|
2008-02-10 06:24:09 +08:00
|
|
|
* Remove any mappings which extend past the end of physical
|
2012-11-17 11:39:11 +08:00
|
|
|
* memory from the boot time page table.
|
|
|
|
* In virtual address space, we should have at least two pages
|
|
|
|
* from VMALLOC_END to pkmap or fixmap according to VMALLOC_END
|
|
|
|
* definition. And max_low_pfn is set to VMALLOC_END physical
|
|
|
|
* address. If initial memory mapping is doing right job, we
|
|
|
|
* should have pte used near max_low_pfn or one pmd is not present.
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
*/
|
2012-11-17 11:39:11 +08:00
|
|
|
for (pfn = max_low_pfn; pfn < 1<<(32-PAGE_SHIFT); pfn++) {
|
2008-02-10 06:24:09 +08:00
|
|
|
va = PAGE_OFFSET + (pfn<<PAGE_SHIFT);
|
|
|
|
pgd = base + pgd_index(va);
|
|
|
|
if (!pgd_present(*pgd))
|
|
|
|
break;
|
|
|
|
|
|
|
|
pud = pud_offset(pgd, va);
|
|
|
|
pmd = pmd_offset(pud, va);
|
|
|
|
if (!pmd_present(*pmd))
|
|
|
|
break;
|
|
|
|
|
2012-11-17 11:39:11 +08:00
|
|
|
/* should not be large page here */
|
|
|
|
if (pmd_large(*pmd)) {
|
|
|
|
pr_warn("try to clear pte for ram above max_low_pfn: pfn: %lx pmd: %p pmd phys: %lx, but pmd is big page and is not using pte !\n",
|
|
|
|
pfn, pmd, __pa(pmd));
|
|
|
|
BUG_ON(1);
|
|
|
|
}
|
|
|
|
|
2008-02-10 06:24:09 +08:00
|
|
|
pte = pte_offset_kernel(pmd, va);
|
|
|
|
if (!pte_present(*pte))
|
|
|
|
break;
|
|
|
|
|
2012-11-17 11:39:11 +08:00
|
|
|
printk(KERN_DEBUG "clearing pte for ram above max_low_pfn: pfn: %lx pmd: %p pmd phys: %lx pte: %p pte phys: %lx\n",
|
|
|
|
pfn, pmd, __pa(pmd), pte, __pa(pte));
|
2008-02-10 06:24:09 +08:00
|
|
|
pte_clear(NULL, va, pte);
|
|
|
|
}
|
2008-03-18 07:37:01 +08:00
|
|
|
paravirt_alloc_pmd(&init_mm, __pa(base) >> PAGE_SHIFT);
|
2012-08-22 04:22:39 +08:00
|
|
|
paging_init();
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Build a proper pagetable for the kernel mappings. Up until this
|
|
|
|
* point, we've been running on some set of pagetables constructed by
|
|
|
|
* the boot process.
|
|
|
|
*
|
|
|
|
* If we're booting on native hardware, this will be a pagetable
|
2008-02-10 06:24:09 +08:00
|
|
|
* constructed in arch/x86/kernel/head_32.S. The root of the
|
|
|
|
* pagetable will be swapper_pg_dir.
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
*
|
|
|
|
* If we're booting paravirtualized under a hypervisor, then there are
|
|
|
|
* more options: we may already be running PAE, and the pagetable may
|
|
|
|
* or may not be based in swapper_pg_dir. In any case,
|
2012-08-22 04:22:38 +08:00
|
|
|
* paravirt_pagetable_init() will set up swapper_pg_dir
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
* appropriately for the rest of the initialization to work.
|
|
|
|
*
|
|
|
|
* In general, pagetable_init() assumes that the pagetable may already
|
|
|
|
* be partially populated, and so it avoids stomping on any existing
|
|
|
|
* mappings.
|
|
|
|
*/
|
2009-03-05 20:55:05 +08:00
|
|
|
void __init early_ioremap_page_table_range_init(void)
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
{
|
2009-03-05 20:54:55 +08:00
|
|
|
pgd_t *pgd_base = swapper_pg_dir;
|
2008-01-30 20:34:10 +08:00
|
|
|
unsigned long vaddr, end;
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Fixed mappings, only the page table structure has to be
|
|
|
|
* created - mappings will be set by set_fixmap():
|
|
|
|
*/
|
|
|
|
vaddr = __fix_to_virt(__end_of_fixed_addresses - 1) & PMD_MASK;
|
[PATCH] i386: PARAVIRT: Hooks to set up initial pagetable
This patch introduces paravirt_ops hooks to control how the kernel's
initial pagetable is set up.
In the case of a native boot, the very early bootstrap code creates a
simple non-PAE pagetable to map the kernel and physical memory. When
the VM subsystem is initialized, it creates a proper pagetable which
respects the PAE mode, large pages, etc.
When booting under a hypervisor, there are many possibilities for what
paging environment the hypervisor establishes for the guest kernel, so
the constructon of the kernel's pagetable depends on the hypervisor.
In the case of Xen, the hypervisor boots the kernel with a fully
constructed pagetable, which is already using PAE if necessary. Also,
Xen requires particular care when constructing pagetables to make sure
all pagetables are always mapped read-only.
In order to make this easier, kernel's initial pagetable construction
has been changed to only allocate and initialize a pagetable page if
there's no page already present in the pagetable. This allows the Xen
paravirt backend to make a copy of the hypervisor-provided pagetable,
allowing the kernel to establish any more mappings it needs while
keeping the existing ones.
A slightly subtle point which is worth highlighting here is that Xen
requires all kernel mappings to share the same pte_t pages between all
pagetables, so that updating a kernel page's mapping in one pagetable
is reflected in all other pagetables. This makes it possible to
allocate a page and attach it to a pagetable without having to
explicitly enumerate that page's mapping in all pagetables.
And:
+From: "Eric W. Biederman" <ebiederm@xmission.com>
If we don't set the leaf page table entries it is quite possible that
will inherit and incorrect page table entry from the initial boot
page table setup in head.S. So we need to redo the effort here,
so we pick up PSE, PGE and the like.
Hypervisors like Xen require that their page tables be read-only,
which is slightly incompatible with our low identity mappings, however
I discussed this with Jeremy he has modified the Xen early set_pte
function to avoid problems in this area.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
2007-05-03 01:27:13 +08:00
|
|
|
end = (FIXADDR_TOP + PMD_SIZE - 1) & PMD_MASK;
|
|
|
|
page_table_range_init(vaddr, end, pgd_base);
|
2008-01-30 20:33:44 +08:00
|
|
|
early_ioremap_reset();
|
2008-06-26 12:51:28 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void __init pagetable_init(void)
|
|
|
|
{
|
|
|
|
pgd_t *pgd_base = swapper_pg_dir;
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
permanent_kmaps_init(pgd_base);
|
|
|
|
}
|
|
|
|
|
2014-01-08 01:03:06 +08:00
|
|
|
pteval_t __supported_pte_mask __read_mostly = ~(_PAGE_NX | _PAGE_GLOBAL);
|
2008-01-30 20:32:57 +08:00
|
|
|
EXPORT_SYMBOL_GPL(__supported_pte_mask);
|
|
|
|
|
2008-06-24 03:00:45 +08:00
|
|
|
/* user-defined highmem size */
|
|
|
|
static unsigned int highmem_pages = -1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* highmem=size forces highmem to be exactly 'size' bytes.
|
|
|
|
* This works even on boxes that have no highmem otherwise.
|
|
|
|
* This also works to reduce highmem size on bigger boxes.
|
|
|
|
*/
|
|
|
|
static int __init parse_highmem(char *arg)
|
|
|
|
{
|
|
|
|
if (!arg)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
highmem_pages = memparse(arg, &arg) >> PAGE_SHIFT;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
early_param("highmem", parse_highmem);
|
|
|
|
|
2009-02-12 20:31:41 +08:00
|
|
|
#define MSG_HIGHMEM_TOO_BIG \
|
|
|
|
"highmem size (%luMB) is bigger than pages available (%luMB)!\n"
|
|
|
|
|
|
|
|
#define MSG_LOWMEM_TOO_SMALL \
|
|
|
|
"highmem size (%luMB) results in <64MB lowmem, ignoring it!\n"
|
2008-06-24 03:00:45 +08:00
|
|
|
/*
|
2009-02-12 20:31:41 +08:00
|
|
|
* All of RAM fits into lowmem - but if user wants highmem
|
|
|
|
* artificially via the highmem=x boot parameter then create
|
|
|
|
* it:
|
2008-06-24 03:00:45 +08:00
|
|
|
*/
|
2012-11-17 11:39:09 +08:00
|
|
|
static void __init lowmem_pfn_init(void)
|
2008-06-24 03:00:45 +08:00
|
|
|
{
|
2008-06-23 18:06:14 +08:00
|
|
|
/* max_low_pfn is 0, we already have early_res support */
|
2008-06-24 03:00:45 +08:00
|
|
|
max_low_pfn = max_pfn;
|
2009-02-12 22:16:03 +08:00
|
|
|
|
2009-02-12 20:31:41 +08:00
|
|
|
if (highmem_pages == -1)
|
|
|
|
highmem_pages = 0;
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
if (highmem_pages >= max_pfn) {
|
|
|
|
printk(KERN_ERR MSG_HIGHMEM_TOO_BIG,
|
|
|
|
pages_to_mb(highmem_pages), pages_to_mb(max_pfn));
|
|
|
|
highmem_pages = 0;
|
|
|
|
}
|
|
|
|
if (highmem_pages) {
|
|
|
|
if (max_low_pfn - highmem_pages < 64*1024*1024/PAGE_SIZE) {
|
|
|
|
printk(KERN_ERR MSG_LOWMEM_TOO_SMALL,
|
2008-06-24 03:00:45 +08:00
|
|
|
pages_to_mb(highmem_pages));
|
|
|
|
highmem_pages = 0;
|
|
|
|
}
|
2009-02-12 20:31:41 +08:00
|
|
|
max_low_pfn -= highmem_pages;
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
if (highmem_pages)
|
|
|
|
printk(KERN_ERR "ignoring highmem size on non-highmem kernel!\n");
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
#define MSG_HIGHMEM_TOO_SMALL \
|
|
|
|
"only %luMB highmem pages available, ignoring highmem size of %luMB!\n"
|
|
|
|
|
|
|
|
#define MSG_HIGHMEM_TRIMMED \
|
|
|
|
"Warning: only 4GB will be used. Use a HIGHMEM64G enabled kernel!\n"
|
|
|
|
/*
|
|
|
|
* We have more RAM than fits into lowmem - we try to put it into
|
|
|
|
* highmem, also taking the highmem=x boot parameter into account:
|
|
|
|
*/
|
2012-11-17 11:39:09 +08:00
|
|
|
static void __init highmem_pfn_init(void)
|
2009-02-12 20:31:41 +08:00
|
|
|
{
|
2009-02-12 22:16:03 +08:00
|
|
|
max_low_pfn = MAXMEM_PFN;
|
|
|
|
|
2009-02-12 20:31:41 +08:00
|
|
|
if (highmem_pages == -1)
|
|
|
|
highmem_pages = max_pfn - MAXMEM_PFN;
|
|
|
|
|
|
|
|
if (highmem_pages + MAXMEM_PFN < max_pfn)
|
|
|
|
max_pfn = MAXMEM_PFN + highmem_pages;
|
|
|
|
|
|
|
|
if (highmem_pages + MAXMEM_PFN > max_pfn) {
|
|
|
|
printk(KERN_WARNING MSG_HIGHMEM_TOO_SMALL,
|
|
|
|
pages_to_mb(max_pfn - MAXMEM_PFN),
|
|
|
|
pages_to_mb(highmem_pages));
|
|
|
|
highmem_pages = 0;
|
|
|
|
}
|
2008-06-24 03:00:45 +08:00
|
|
|
#ifndef CONFIG_HIGHMEM
|
2009-02-12 20:31:41 +08:00
|
|
|
/* Maximum memory usable is what is directly addressable */
|
|
|
|
printk(KERN_WARNING "Warning only %ldMB will be used.\n", MAXMEM>>20);
|
|
|
|
if (max_pfn > MAX_NONPAE_PFN)
|
|
|
|
printk(KERN_WARNING "Use a HIGHMEM64G enabled kernel.\n");
|
|
|
|
else
|
|
|
|
printk(KERN_WARNING "Use a HIGHMEM enabled kernel.\n");
|
|
|
|
max_pfn = MAXMEM_PFN;
|
2008-06-24 03:00:45 +08:00
|
|
|
#else /* !CONFIG_HIGHMEM */
|
|
|
|
#ifndef CONFIG_HIGHMEM64G
|
2009-02-12 20:31:41 +08:00
|
|
|
if (max_pfn > MAX_NONPAE_PFN) {
|
|
|
|
max_pfn = MAX_NONPAE_PFN;
|
|
|
|
printk(KERN_WARNING MSG_HIGHMEM_TRIMMED);
|
|
|
|
}
|
2008-06-24 03:00:45 +08:00
|
|
|
#endif /* !CONFIG_HIGHMEM64G */
|
|
|
|
#endif /* !CONFIG_HIGHMEM */
|
2009-02-12 20:31:41 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Determine low and high memory ranges:
|
|
|
|
*/
|
|
|
|
void __init find_low_pfn_range(void)
|
|
|
|
{
|
|
|
|
/* it could update max_pfn */
|
|
|
|
|
2009-02-12 22:16:03 +08:00
|
|
|
if (max_pfn <= MAXMEM_PFN)
|
2009-02-12 20:31:41 +08:00
|
|
|
lowmem_pfn_init();
|
2009-02-12 22:16:03 +08:00
|
|
|
else
|
|
|
|
highmem_pfn_init();
|
2008-06-24 03:00:45 +08:00
|
|
|
}
|
|
|
|
|
2008-06-22 17:45:39 +08:00
|
|
|
#ifndef CONFIG_NEED_MULTIPLE_NODES
|
2011-02-16 19:13:06 +08:00
|
|
|
void __init initmem_init(void)
|
2008-06-22 17:45:39 +08:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
highstart_pfn = highend_pfn = max_pfn;
|
|
|
|
if (max_pfn > max_low_pfn)
|
|
|
|
highstart_pfn = max_low_pfn;
|
|
|
|
printk(KERN_NOTICE "%ldMB HIGHMEM available.\n",
|
|
|
|
pages_to_mb(highend_pfn - highstart_pfn));
|
|
|
|
high_memory = (void *) __va(highstart_pfn * PAGE_SIZE - 1) + 1;
|
|
|
|
#else
|
|
|
|
high_memory = (void *) __va(max_low_pfn * PAGE_SIZE - 1) + 1;
|
|
|
|
#endif
|
2011-07-14 17:44:23 +08:00
|
|
|
|
2014-01-22 07:49:26 +08:00
|
|
|
memblock_set_node(0, (phys_addr_t)ULLONG_MAX, &memblock.memory, 0);
|
2011-07-14 17:44:23 +08:00
|
|
|
sparse_memory_present_with_active_regions(0);
|
|
|
|
|
2008-06-22 17:45:39 +08:00
|
|
|
#ifdef CONFIG_FLATMEM
|
2013-07-04 06:04:19 +08:00
|
|
|
max_mapnr = IS_ENABLED(CONFIG_HIGHMEM) ? highend_pfn : max_low_pfn;
|
2008-06-22 17:45:39 +08:00
|
|
|
#endif
|
2009-03-05 08:10:44 +08:00
|
|
|
__vmalloc_start_set = true;
|
|
|
|
|
2008-06-22 17:45:39 +08:00
|
|
|
printk(KERN_NOTICE "%ldMB LOWMEM available.\n",
|
|
|
|
pages_to_mb(max_low_pfn));
|
|
|
|
|
|
|
|
setup_bootmem_allocator();
|
|
|
|
}
|
2008-07-02 15:31:02 +08:00
|
|
|
#endif /* !CONFIG_NEED_MULTIPLE_NODES */
|
2008-06-22 17:45:39 +08:00
|
|
|
|
|
|
|
void __init setup_bootmem_allocator(void)
|
|
|
|
{
|
|
|
|
printk(KERN_INFO " mapped low ram: 0 - %08lx\n",
|
|
|
|
max_pfn_mapped<<PAGE_SHIFT);
|
2009-03-05 04:21:24 +08:00
|
|
|
printk(KERN_INFO " low ram: 0 - %08lx\n", max_low_pfn<<PAGE_SHIFT);
|
2008-06-25 03:18:14 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* paging_init() sets up the page tables - note that the first 8MB are
|
|
|
|
* already mapped by head.S.
|
|
|
|
*
|
|
|
|
* This routines also unmaps the page at virtual kernel address 0, so
|
|
|
|
* that we can trap those pesky NULL-reference errors in the kernel.
|
|
|
|
*/
|
|
|
|
void __init paging_init(void)
|
|
|
|
{
|
|
|
|
pagetable_init();
|
|
|
|
|
|
|
|
__flush_tlb_all();
|
|
|
|
|
|
|
|
kmap_init();
|
2008-06-24 10:51:10 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* NOTE: at this point the bootmem allocator is fully available.
|
|
|
|
*/
|
2010-11-17 14:09:52 +08:00
|
|
|
olpc_dt_build_devicetree();
|
2011-05-02 20:18:52 +08:00
|
|
|
sparse_memory_present_with_active_regions(MAX_NUMNODES);
|
2008-06-24 10:51:10 +08:00
|
|
|
sparse_init();
|
|
|
|
zone_sizes_init();
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Test if the WP bit works in supervisor mode. It isn't supported on 386's
|
2008-04-21 04:47:55 +08:00
|
|
|
* and also on some strange 486's. All 586+'s are OK. This used to involve
|
|
|
|
* black magic jumps to work around some nasty CPU bugs, but fortunately the
|
|
|
|
* switch to using exceptions got rid of all that.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
static void __init test_wp_bit(void)
|
|
|
|
{
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_INFO
|
|
|
|
"Checking if this processor honours the WP bit even in supervisor mode...");
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Any page-aligned address will do, the test is non-destructive */
|
2012-09-22 03:43:06 +08:00
|
|
|
__set_fixmap(FIX_WP_TEST, __pa(&swapper_pg_dir), PAGE_KERNEL_RO);
|
2005-04-17 06:20:36 +08:00
|
|
|
boot_cpu_data.wp_works_ok = do_test_wp_bit();
|
|
|
|
clear_fixmap(FIX_WP_TEST);
|
|
|
|
|
|
|
|
if (!boot_cpu_data.wp_works_ok) {
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_CONT "No.\n");
|
2012-11-29 03:50:28 +08:00
|
|
|
panic("Linux doesn't support CPUs with broken WP.");
|
2005-04-17 06:20:36 +08:00
|
|
|
} else {
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_CONT "Ok.\n");
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void __init mem_init(void)
|
|
|
|
{
|
2008-12-17 04:17:36 +08:00
|
|
|
pci_iommu_alloc();
|
|
|
|
|
2005-06-23 15:07:57 +08:00
|
|
|
#ifdef CONFIG_FLATMEM
|
2006-10-04 05:34:58 +08:00
|
|
|
BUG_ON(!mem_map);
|
2005-04-17 06:20:36 +08:00
|
|
|
#endif
|
2011-12-06 16:08:34 +08:00
|
|
|
/*
|
|
|
|
* With CONFIG_DEBUG_PAGEALLOC initialization of highmem pages has to
|
|
|
|
* be done before free_all_bootmem(). Memblock use free low memory for
|
|
|
|
* temporary data (see find_range_array()) and for this purpose can use
|
|
|
|
* pages that was already passed to the buddy allocator, hence marked as
|
|
|
|
* not accessible in the page tables when compiled with
|
|
|
|
* CONFIG_DEBUG_PAGEALLOC. Otherwise order of initialization is not
|
|
|
|
* important here.
|
|
|
|
*/
|
|
|
|
set_highmem_pages_init();
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* this will put all low memory onto the freelists */
|
2013-07-04 06:03:24 +08:00
|
|
|
free_all_bootmem();
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-11-17 11:39:16 +08:00
|
|
|
after_bootmem = 1;
|
|
|
|
|
2013-07-04 06:04:19 +08:00
|
|
|
mem_init_print_info(NULL);
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_INFO "virtual kernel memory layout:\n"
|
2008-01-30 20:34:10 +08:00
|
|
|
" fixmap : 0x%08lx - 0x%08lx (%4ld kB)\n"
|
2006-09-26 14:32:25 +08:00
|
|
|
#ifdef CONFIG_HIGHMEM
|
2008-01-30 20:34:10 +08:00
|
|
|
" pkmap : 0x%08lx - 0x%08lx (%4ld kB)\n"
|
2006-09-26 14:32:25 +08:00
|
|
|
#endif
|
2008-01-30 20:34:10 +08:00
|
|
|
" vmalloc : 0x%08lx - 0x%08lx (%4ld MB)\n"
|
|
|
|
" lowmem : 0x%08lx - 0x%08lx (%4ld MB)\n"
|
|
|
|
" .init : 0x%08lx - 0x%08lx (%4ld kB)\n"
|
|
|
|
" .data : 0x%08lx - 0x%08lx (%4ld kB)\n"
|
|
|
|
" .text : 0x%08lx - 0x%08lx (%4ld kB)\n",
|
|
|
|
FIXADDR_START, FIXADDR_TOP,
|
|
|
|
(FIXADDR_TOP - FIXADDR_START) >> 10,
|
2006-09-26 14:32:25 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
2008-01-30 20:34:10 +08:00
|
|
|
PKMAP_BASE, PKMAP_BASE+LAST_PKMAP*PAGE_SIZE,
|
|
|
|
(LAST_PKMAP*PAGE_SIZE) >> 10,
|
2006-09-26 14:32:25 +08:00
|
|
|
#endif
|
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
VMALLOC_START, VMALLOC_END,
|
|
|
|
(VMALLOC_END - VMALLOC_START) >> 20,
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
(unsigned long)__va(0), (unsigned long)high_memory,
|
|
|
|
((unsigned long)high_memory - (unsigned long)__va(0)) >> 20,
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
(unsigned long)&__init_begin, (unsigned long)&__init_end,
|
|
|
|
((unsigned long)&__init_end -
|
|
|
|
(unsigned long)&__init_begin) >> 10,
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
(unsigned long)&_etext, (unsigned long)&_edata,
|
|
|
|
((unsigned long)&_edata - (unsigned long)&_etext) >> 10,
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
(unsigned long)&_text, (unsigned long)&_etext,
|
|
|
|
((unsigned long)&_etext - (unsigned long)&_text) >> 10);
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2008-12-16 19:45:56 +08:00
|
|
|
/*
|
|
|
|
* Check boundaries twice: Some fundamental inconsistencies can
|
|
|
|
* be detected at build time already.
|
|
|
|
*/
|
|
|
|
#define __FIXADDR_TOP (-PAGE_SIZE)
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
BUILD_BUG_ON(PKMAP_BASE + LAST_PKMAP*PAGE_SIZE > FIXADDR_START);
|
|
|
|
BUILD_BUG_ON(VMALLOC_END > PKMAP_BASE);
|
|
|
|
#endif
|
|
|
|
#define high_memory (-128UL << 20)
|
|
|
|
BUILD_BUG_ON(VMALLOC_START >= VMALLOC_END);
|
|
|
|
#undef high_memory
|
|
|
|
#undef __FIXADDR_TOP
|
|
|
|
|
2006-09-26 14:32:25 +08:00
|
|
|
#ifdef CONFIG_HIGHMEM
|
2008-01-30 20:34:10 +08:00
|
|
|
BUG_ON(PKMAP_BASE + LAST_PKMAP*PAGE_SIZE > FIXADDR_START);
|
|
|
|
BUG_ON(VMALLOC_END > PKMAP_BASE);
|
2006-09-26 14:32:25 +08:00
|
|
|
#endif
|
2008-12-16 19:45:56 +08:00
|
|
|
BUG_ON(VMALLOC_START >= VMALLOC_END);
|
2008-01-30 20:34:10 +08:00
|
|
|
BUG_ON((unsigned long)high_memory > VMALLOC_START);
|
2006-09-26 14:32:25 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (boot_cpu_data.wp_works_ok < 0)
|
|
|
|
test_wp_bit();
|
|
|
|
}
|
|
|
|
|
2006-05-21 06:00:03 +08:00
|
|
|
#ifdef CONFIG_MEMORY_HOTPLUG
|
2015-08-10 03:29:06 +08:00
|
|
|
int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
|
2005-10-30 09:16:57 +08:00
|
|
|
{
|
2006-12-22 17:11:13 +08:00
|
|
|
struct pglist_data *pgdata = NODE_DATA(nid);
|
2014-08-07 07:07:40 +08:00
|
|
|
struct zone *zone = pgdata->node_zones +
|
2015-08-10 03:29:06 +08:00
|
|
|
zone_for_memory(nid, start, size, ZONE_HIGHMEM, for_device);
|
2005-10-30 09:16:57 +08:00
|
|
|
unsigned long start_pfn = start >> PAGE_SHIFT;
|
|
|
|
unsigned long nr_pages = size >> PAGE_SHIFT;
|
|
|
|
|
2009-01-07 06:39:14 +08:00
|
|
|
return __add_pages(nid, zone, start_pfn, nr_pages);
|
2005-10-30 09:16:57 +08:00
|
|
|
}
|
2013-02-23 08:32:58 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_MEMORY_HOTREMOVE
|
|
|
|
int arch_remove_memory(u64 start, u64 size)
|
|
|
|
{
|
|
|
|
unsigned long start_pfn = start >> PAGE_SHIFT;
|
|
|
|
unsigned long nr_pages = size >> PAGE_SHIFT;
|
|
|
|
struct zone *zone;
|
|
|
|
|
|
|
|
zone = page_zone(pfn_to_page(start_pfn));
|
|
|
|
return __remove_pages(zone, start_pfn, nr_pages);
|
|
|
|
}
|
|
|
|
#endif
|
2006-04-08 01:49:15 +08:00
|
|
|
#endif
|
2005-10-30 09:16:57 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* This function cannot be __init, since exceptions don't work in that
|
|
|
|
* section. Put this after the callers, so that it cannot be inlined.
|
|
|
|
*/
|
2008-01-30 20:34:10 +08:00
|
|
|
static noinline int do_test_wp_bit(void)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
char tmp_reg;
|
|
|
|
int flag;
|
|
|
|
|
|
|
|
__asm__ __volatile__(
|
2008-01-30 20:34:10 +08:00
|
|
|
" movb %0, %1 \n"
|
|
|
|
"1: movb %1, %0 \n"
|
|
|
|
" xorl %2, %2 \n"
|
2005-04-17 06:20:36 +08:00
|
|
|
"2: \n"
|
2008-02-04 23:47:58 +08:00
|
|
|
_ASM_EXTABLE(1b,2b)
|
2005-04-17 06:20:36 +08:00
|
|
|
:"=m" (*(char *)fix_to_virt(FIX_WP_TEST)),
|
|
|
|
"=q" (tmp_reg),
|
|
|
|
"=r" (flag)
|
|
|
|
:"2" (1)
|
|
|
|
:"memory");
|
2008-01-30 20:34:10 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
return flag;
|
|
|
|
}
|
|
|
|
|
2008-01-30 20:34:08 +08:00
|
|
|
const int rodata_test_data = 0xC3;
|
|
|
|
EXPORT_SYMBOL_GPL(rodata_test_data);
|
2006-01-06 16:12:02 +08:00
|
|
|
|
2009-10-29 10:46:56 +08:00
|
|
|
int kernel_set_to_readonly __read_mostly;
|
2009-02-18 06:57:30 +08:00
|
|
|
|
|
|
|
void set_kernel_text_rw(void)
|
|
|
|
{
|
|
|
|
unsigned long start = PFN_ALIGN(_text);
|
|
|
|
unsigned long size = PFN_ALIGN(_etext) - start;
|
|
|
|
|
|
|
|
if (!kernel_set_to_readonly)
|
|
|
|
return;
|
|
|
|
|
|
|
|
pr_debug("Set kernel text: %lx - %lx for read write\n",
|
|
|
|
start, start+size);
|
|
|
|
|
|
|
|
set_pages_rw(virt_to_page(start), size >> PAGE_SHIFT);
|
|
|
|
}
|
|
|
|
|
|
|
|
void set_kernel_text_ro(void)
|
|
|
|
{
|
|
|
|
unsigned long start = PFN_ALIGN(_text);
|
|
|
|
unsigned long size = PFN_ALIGN(_etext) - start;
|
|
|
|
|
|
|
|
if (!kernel_set_to_readonly)
|
|
|
|
return;
|
|
|
|
|
|
|
|
pr_debug("Set kernel text: %lx - %lx for read only\n",
|
|
|
|
start, start+size);
|
|
|
|
|
|
|
|
set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT);
|
|
|
|
}
|
|
|
|
|
2010-11-17 05:31:26 +08:00
|
|
|
static void mark_nxdata_nx(void)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* When this called, init has already been executed and released,
|
2011-03-18 03:24:16 +08:00
|
|
|
* so everything past _etext should be NX.
|
2010-11-17 05:31:26 +08:00
|
|
|
*/
|
|
|
|
unsigned long start = PFN_ALIGN(_etext);
|
|
|
|
/*
|
|
|
|
* This comes from is_kernel_text upper limit. Also HPAGE where used:
|
|
|
|
*/
|
|
|
|
unsigned long size = (((unsigned long)__init_end + HPAGE_SIZE) & HPAGE_MASK) - start;
|
|
|
|
|
|
|
|
if (__supported_pte_mask & _PAGE_NX)
|
|
|
|
printk(KERN_INFO "NX-protecting the kernel data: %luk\n", size >> 10);
|
|
|
|
set_pages_nx(virt_to_page(start), size >> PAGE_SHIFT);
|
|
|
|
}
|
|
|
|
|
2006-01-06 16:12:02 +08:00
|
|
|
void mark_rodata_ro(void)
|
|
|
|
{
|
[PATCH] x86: tighten kernel image page access rights
On x86-64, kernel memory freed after init can be entirely unmapped instead
of just getting 'poisoned' by overwriting with a debug pattern.
On i386 and x86-64 (under CONFIG_DEBUG_RODATA), kernel text and bug table
can also be write-protected.
Compared to the first version, this one prevents re-creating deleted
mappings in the kernel image range on x86-64, if those got removed
previously. This, together with the original changes, prevents temporarily
having inconsistent mappings when cacheability attributes are being
changed on such pages (e.g. from AGP code). While on i386 such duplicate
mappings don't exist, the same change is done there, too, both for
consistency and because checking pte_present() before using various other
pte_XXX functions is a requirement anyway. At once, i386 code gets
adjusted to use pte_huge() instead of open coding this.
AK: split out cpa() changes
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Signed-off-by: Andi Kleen <ak@suse.de>
2007-05-03 01:27:10 +08:00
|
|
|
unsigned long start = PFN_ALIGN(_text);
|
|
|
|
unsigned long size = PFN_ALIGN(_etext) - start;
|
2006-01-06 16:12:02 +08:00
|
|
|
|
2008-02-03 04:42:20 +08:00
|
|
|
set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT);
|
|
|
|
printk(KERN_INFO "Write protecting the kernel text: %luk\n",
|
|
|
|
size >> 10);
|
2008-01-30 20:33:42 +08:00
|
|
|
|
2009-02-18 06:57:30 +08:00
|
|
|
kernel_set_to_readonly = 1;
|
|
|
|
|
2008-01-30 20:33:42 +08:00
|
|
|
#ifdef CONFIG_CPA_DEBUG
|
2008-02-03 04:42:20 +08:00
|
|
|
printk(KERN_INFO "Testing CPA: Reverting %lx-%lx\n",
|
|
|
|
start, start+size);
|
|
|
|
set_pages_rw(virt_to_page(start), size>>PAGE_SHIFT);
|
2008-01-30 20:33:42 +08:00
|
|
|
|
2008-02-03 04:42:20 +08:00
|
|
|
printk(KERN_INFO "Testing CPA: write protecting again\n");
|
|
|
|
set_pages_ro(virt_to_page(start), size>>PAGE_SHIFT);
|
2007-07-27 03:07:21 +08:00
|
|
|
#endif
|
2008-05-13 03:20:56 +08:00
|
|
|
|
[PATCH] x86: tighten kernel image page access rights
On x86-64, kernel memory freed after init can be entirely unmapped instead
of just getting 'poisoned' by overwriting with a debug pattern.
On i386 and x86-64 (under CONFIG_DEBUG_RODATA), kernel text and bug table
can also be write-protected.
Compared to the first version, this one prevents re-creating deleted
mappings in the kernel image range on x86-64, if those got removed
previously. This, together with the original changes, prevents temporarily
having inconsistent mappings when cacheability attributes are being
changed on such pages (e.g. from AGP code). While on i386 such duplicate
mappings don't exist, the same change is done there, too, both for
consistency and because checking pte_present() before using various other
pte_XXX functions is a requirement anyway. At once, i386 code gets
adjusted to use pte_huge() instead of open coding this.
AK: split out cpa() changes
Signed-off-by: Jan Beulich <jbeulich@novell.com>
Signed-off-by: Andi Kleen <ak@suse.de>
2007-05-03 01:27:10 +08:00
|
|
|
start += size;
|
|
|
|
size = (unsigned long)__end_rodata - start;
|
2008-01-30 20:34:06 +08:00
|
|
|
set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT);
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_INFO "Write protecting the kernel read-only data: %luk\n",
|
|
|
|
size >> 10);
|
2008-01-30 20:34:08 +08:00
|
|
|
rodata_test();
|
2006-01-06 16:12:02 +08:00
|
|
|
|
2008-01-30 20:33:42 +08:00
|
|
|
#ifdef CONFIG_CPA_DEBUG
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_INFO "Testing CPA: undo %lx-%lx\n", start, start + size);
|
2008-01-30 20:34:06 +08:00
|
|
|
set_pages_rw(virt_to_page(start), size >> PAGE_SHIFT);
|
2008-01-30 20:33:42 +08:00
|
|
|
|
2008-01-30 20:34:10 +08:00
|
|
|
printk(KERN_INFO "Testing CPA: write protecting again\n");
|
2008-01-30 20:34:06 +08:00
|
|
|
set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT);
|
2008-01-30 20:33:42 +08:00
|
|
|
#endif
|
2010-11-17 05:31:26 +08:00
|
|
|
mark_nxdata_nx();
|
2015-10-06 00:55:20 +08:00
|
|
|
if (__supported_pte_mask & _PAGE_NX)
|
|
|
|
debug_checkwx();
|
2006-01-06 16:12:02 +08:00
|
|
|
}
|