2010-04-21 21:08:20 +08:00
|
|
|
The x86 kvm shadow mmu
|
|
|
|
======================
|
|
|
|
|
|
|
|
The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
|
|
|
|
for presenting a standard x86 mmu to the guest, while translating guest
|
|
|
|
physical addresses to host physical addresses.
|
|
|
|
|
|
|
|
The mmu code attempts to satisfy the following requirements:
|
|
|
|
|
|
|
|
- correctness: the guest should not be able to determine that it is running
|
|
|
|
on an emulated mmu except for timing (we attempt to comply
|
|
|
|
with the specification, not emulate the characteristics of
|
|
|
|
a particular implementation such as tlb size)
|
|
|
|
- security: the guest must not be able to touch host memory not assigned
|
|
|
|
to it
|
|
|
|
- performance: minimize the performance penalty imposed by the mmu
|
|
|
|
- scaling: need to scale to large memory and large vcpu guests
|
|
|
|
- hardware: support the full range of x86 virtualization hardware
|
|
|
|
- integration: Linux memory management code must be in control of guest memory
|
|
|
|
so that swapping, page migration, page merging, transparent
|
|
|
|
hugepages, and similar features work without change
|
|
|
|
- dirty tracking: report writes to guest memory to enable live migration
|
|
|
|
and framebuffer-based displays
|
|
|
|
- footprint: keep the amount of pinned kernel memory low (most memory
|
|
|
|
should be shrinkable)
|
2011-03-31 09:57:33 +08:00
|
|
|
- reliability: avoid multipage or GFP_ATOMIC allocations
|
2010-04-21 21:08:20 +08:00
|
|
|
|
|
|
|
Acronyms
|
|
|
|
========
|
|
|
|
|
|
|
|
pfn host page frame number
|
|
|
|
hpa host physical address
|
|
|
|
hva host virtual address
|
|
|
|
gfn guest frame number
|
|
|
|
gpa guest physical address
|
|
|
|
gva guest virtual address
|
|
|
|
ngpa nested guest physical address
|
|
|
|
ngva nested guest virtual address
|
|
|
|
pte page table entry (used also to refer generically to paging structure
|
|
|
|
entries)
|
|
|
|
gpte guest pte (referring to gfns)
|
|
|
|
spte shadow pte (referring to pfns)
|
|
|
|
tdp two dimensional paging (vendor neutral term for NPT and EPT)
|
|
|
|
|
|
|
|
Virtual and real hardware supported
|
|
|
|
===================================
|
|
|
|
|
|
|
|
The mmu supports first-generation mmu hardware, which allows an atomic switch
|
|
|
|
of the current paging mode and cr3 during guest entry, as well as
|
|
|
|
two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware
|
|
|
|
it exposes is the traditional 2/3/4 level x86 mmu, with support for global
|
2018-05-27 02:19:12 +08:00
|
|
|
pages, pae, pse, pse36, cr0.wp, and 1GB pages. Emulated hardware also
|
|
|
|
able to expose NPT capable hardware on NPT capable hosts.
|
2010-04-21 21:08:20 +08:00
|
|
|
|
|
|
|
Translation
|
|
|
|
===========
|
|
|
|
|
|
|
|
The primary job of the mmu is to program the processor's mmu to translate
|
|
|
|
addresses for the guest. Different translations are required at different
|
|
|
|
times:
|
|
|
|
|
|
|
|
- when guest paging is disabled, we translate guest physical addresses to
|
|
|
|
host physical addresses (gpa->hpa)
|
|
|
|
- when guest paging is enabled, we translate guest virtual addresses, to
|
|
|
|
guest physical addresses, to host physical addresses (gva->gpa->hpa)
|
|
|
|
- when the guest launches a guest of its own, we translate nested guest
|
|
|
|
virtual addresses, to nested guest physical addresses, to guest physical
|
|
|
|
addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
|
|
|
|
|
|
|
|
The primary challenge is to encode between 1 and 3 translations into hardware
|
|
|
|
that support only 1 (traditional) and 2 (tdp) translations. When the
|
|
|
|
number of required translations matches the hardware, the mmu operates in
|
|
|
|
direct mode; otherwise it operates in shadow mode (see below).
|
|
|
|
|
|
|
|
Memory
|
|
|
|
======
|
|
|
|
|
2010-04-26 16:59:21 +08:00
|
|
|
Guest memory (gpa) is part of the user address space of the process that is
|
|
|
|
using kvm. Userspace defines the translation between guest addresses and user
|
2010-06-17 16:49:22 +08:00
|
|
|
addresses (gpa->hva); note that two gpas may alias to the same hva, but not
|
2010-04-21 21:08:20 +08:00
|
|
|
vice versa.
|
|
|
|
|
2010-06-17 16:49:22 +08:00
|
|
|
These hvas may be backed using any method available to the host: anonymous
|
2010-04-21 21:08:20 +08:00
|
|
|
memory, file backed memory, and device memory. Memory might be paged by the
|
|
|
|
host at any time.
|
|
|
|
|
|
|
|
Events
|
|
|
|
======
|
|
|
|
|
|
|
|
The mmu is driven by events, some from the guest, some from the host.
|
|
|
|
|
|
|
|
Guest generated events:
|
|
|
|
- writes to control registers (especially cr3)
|
|
|
|
- invlpg/invlpga instruction execution
|
|
|
|
- access to missing or protected translations
|
|
|
|
|
|
|
|
Host generated events:
|
|
|
|
- changes in the gpa->hpa translation (either through gpa->hva changes or
|
|
|
|
through hva->hpa changes)
|
|
|
|
- memory pressure (the shrinker)
|
|
|
|
|
|
|
|
Shadow pages
|
|
|
|
============
|
|
|
|
|
|
|
|
The principal data structure is the shadow page, 'struct kvm_mmu_page'. A
|
|
|
|
shadow page contains 512 sptes, which can be either leaf or nonleaf sptes. A
|
|
|
|
shadow page may contain a mix of leaf and nonleaf sptes.
|
|
|
|
|
|
|
|
A nonleaf spte allows the hardware mmu to reach the leaf pages and
|
|
|
|
is not related to a translation directly. It points to other shadow pages.
|
|
|
|
|
|
|
|
A leaf spte corresponds to either one or two translations encoded into
|
|
|
|
one paging structure entry. These are always the lowest level of the
|
2010-04-26 16:59:21 +08:00
|
|
|
translation stack, with optional higher level translations left to NPT/EPT.
|
2010-04-21 21:08:20 +08:00
|
|
|
Leaf ptes point at guest pages.
|
|
|
|
|
|
|
|
The following table shows translations encoded by leaf ptes, with higher-level
|
|
|
|
translations in parentheses:
|
|
|
|
|
|
|
|
Non-nested guests:
|
|
|
|
nonpaging: gpa->hpa
|
|
|
|
paging: gva->gpa->hpa
|
|
|
|
paging, tdp: (gva->)gpa->hpa
|
|
|
|
Nested guests:
|
|
|
|
non-tdp: ngva->gpa->hpa (*)
|
|
|
|
tdp: (ngva->)ngpa->gpa->hpa
|
|
|
|
|
|
|
|
(*) the guest hypervisor will encode the ngva->gpa translation into its page
|
|
|
|
tables if npt is not present
|
|
|
|
|
|
|
|
Shadow pages contain the following information:
|
|
|
|
role.level:
|
|
|
|
The level in the shadow paging hierarchy that this shadow page belongs to.
|
|
|
|
1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
|
|
|
|
role.direct:
|
|
|
|
If set, leaf sptes reachable from this page are for a linear range.
|
|
|
|
Examples include real mode translation, large guest pages backed by small
|
|
|
|
host pages, and gpa->hpa translations when NPT or EPT is active.
|
|
|
|
The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
|
|
|
|
by role.level (2MB for first level, 1GB for second level, 0.5TB for third
|
|
|
|
level, 256TB for fourth level)
|
|
|
|
If clear, this page corresponds to a guest page table denoted by the gfn
|
|
|
|
field.
|
|
|
|
role.quadrant:
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
When role.gpte_is_8_bytes=0, the guest uses 32-bit gptes while the host uses 64-bit
|
2010-04-21 21:08:20 +08:00
|
|
|
sptes. That means a guest page table contains more ptes than the host,
|
|
|
|
so multiple shadow pages are needed to shadow one guest page.
|
|
|
|
For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
|
|
|
|
first or second 512-gpte block in the guest page table. For second-level
|
|
|
|
page tables, each 32-bit gpte is converted to two 64-bit sptes
|
|
|
|
(since each first-level guest page is shadowed by two first-level
|
|
|
|
shadow pages) so role.quadrant takes values in the range 0..3. Each
|
|
|
|
quadrant maps 1GB virtual address space.
|
|
|
|
role.access:
|
|
|
|
Inherited guest access permissions in the form uwx. Note execute
|
|
|
|
permission is positive, not negative.
|
|
|
|
role.invalid:
|
|
|
|
The page is invalid and should not be used. It is a root page that is
|
|
|
|
currently pinned (by a cpu hardware register pointing to it); once it is
|
|
|
|
unpinned it will be destroyed.
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.gpte_is_8_bytes:
|
|
|
|
Reflects the size of the guest PTE for which the page is valid, i.e. '1'
|
|
|
|
if 64-bit gptes are in use, '0' if 32-bit gptes are in use.
|
2010-05-11 14:36:58 +08:00
|
|
|
role.nxe:
|
2010-04-21 21:08:20 +08:00
|
|
|
Contains the value of efer.nxe for which the page is valid.
|
2010-05-12 16:48:18 +08:00
|
|
|
role.cr0_wp:
|
|
|
|
Contains the value of cr0.wp for which the page is valid.
|
2011-06-06 21:11:54 +08:00
|
|
|
role.smep_andnot_wp:
|
|
|
|
Contains the value of cr4.smep && !cr0.wp for which the page is valid
|
|
|
|
(pages for which this is true are different from other pages; see the
|
|
|
|
treatment of cr0.wp=0 below).
|
2015-05-11 22:55:21 +08:00
|
|
|
role.smap_andnot_wp:
|
|
|
|
Contains the value of cr4.smap && !cr0.wp for which the page is valid
|
|
|
|
(pages for which this is true are different from other pages; see the
|
|
|
|
treatment of cr0.wp=0 below).
|
KVM: x86: fix handling of role.cr4_pae and rename it to 'gpte_size'
The cr4_pae flag is a bit of a misnomer, its purpose is really to track
whether the guest PTE that is being shadowed is a 4-byte entry or an
8-byte entry. Prior to supporting nested EPT, the size of the gpte was
reflected purely by CR4.PAE. KVM fudged things a bit for direct sptes,
but it was mostly harmless since the size of the gpte never mattered.
Now that a spte may be tracking an indirect EPT entry, relying on
CR4.PAE is wrong and ill-named.
For direct shadow pages, force the gpte_size to '1' as they are always
8-byte entries; EPT entries can only be 8-bytes and KVM always uses
8-byte entries for NPT and its identity map (when running with EPT but
not unrestricted guest).
Likewise, nested EPT entries are always 8-bytes. Nested EPT presents a
unique scenario as the size of the entries are not dictated by CR4.PAE,
but neither is the shadow page a direct map. To handle this scenario,
set cr0_wp=1 and smap_andnot_wp=1, an otherwise impossible combination,
to denote a nested EPT shadow page. Use the information to avoid
incorrectly zapping an unsync'd indirect page in __kvm_sync_page().
Providing a consistent and accurate gpte_size fixes a bug reported by
Vitaly where fast_cr3_switch() always fails when switching from L2 to
L1 as kvm_mmu_get_page() would force role.cr4_pae=0 for direct pages,
whereas kvm_calc_mmu_role_common() would set it according to CR4.PAE.
Fixes: 7dcd575520082 ("x86/kvm/mmu: check if tdp/shadow MMU reconfiguration is needed")
Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-03-08 07:27:44 +08:00
|
|
|
role.ept_sp:
|
|
|
|
This is a virtual flag to denote a shadowed nested EPT page. ept_sp
|
|
|
|
is true if "cr0_wp && smap_andnot_wp", an otherwise invalid combination.
|
2015-05-18 21:03:39 +08:00
|
|
|
role.smm:
|
|
|
|
Is 1 if the page is valid in system management mode. This field
|
|
|
|
determines which of the kvm_memslots array was used to build this
|
|
|
|
shadow page; it is also used to go back from a struct kvm_mmu_page
|
|
|
|
to a memslot, through the kvm_memslots_for_spte_role macro and
|
|
|
|
__gfn_to_memslot.
|
2017-07-01 08:26:31 +08:00
|
|
|
role.ad_disabled:
|
|
|
|
Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D
|
|
|
|
bits before Haswell; shadow EPT page tables also cannot use A/D bits
|
|
|
|
if the L1 hypervisor does not enable them.
|
2010-04-21 21:08:20 +08:00
|
|
|
gfn:
|
|
|
|
Either the guest page table containing the translations shadowed by this
|
|
|
|
page, or the base page frame for linear translations. See role.direct.
|
|
|
|
spt:
|
2010-04-26 16:59:21 +08:00
|
|
|
A pageful of 64-bit sptes containing the translations for this page.
|
2010-04-21 21:08:20 +08:00
|
|
|
Accessed by both kvm and hardware.
|
|
|
|
The page pointed to by spt will have its page->private pointing back
|
|
|
|
at the shadow page structure.
|
|
|
|
sptes in spt point either at guest pages, or at lower-level shadow pages.
|
|
|
|
Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point
|
|
|
|
at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
|
|
|
|
The spt array forms a DAG structure with the shadow page as a node, and
|
|
|
|
guest pages as leaves.
|
|
|
|
gfns:
|
|
|
|
An array of 512 guest frame numbers, one for each present pte. Used to
|
2010-05-26 16:49:59 +08:00
|
|
|
perform a reverse map from a pte to a gfn. When role.direct is set, any
|
|
|
|
element of this array can be calculated from the gfn field when used, in
|
|
|
|
this case, the array of gfns is not allocated. See role.direct and gfn.
|
2010-04-21 21:08:20 +08:00
|
|
|
root_count:
|
|
|
|
A counter keeping track of how many hardware registers (guest cr3 or
|
|
|
|
pdptrs) are now pointing at the page. While this counter is nonzero, the
|
|
|
|
page cannot be destroyed. See role.invalid.
|
2013-06-19 17:09:19 +08:00
|
|
|
parent_ptes:
|
|
|
|
The reverse mapping for the pte/ptes pointing at this page's spt. If
|
2015-11-20 16:45:44 +08:00
|
|
|
parent_ptes bit 0 is zero, only one spte points at this page and
|
2013-06-19 17:09:19 +08:00
|
|
|
parent_ptes points at this single spte, otherwise, there exists multiple
|
|
|
|
sptes pointing at this page and (parent_ptes & ~0x1) points at a data
|
2015-11-20 16:45:44 +08:00
|
|
|
structure with a list of parent sptes.
|
2010-04-21 21:08:20 +08:00
|
|
|
unsync:
|
|
|
|
If true, then the translations in this page may not match the guest's
|
|
|
|
translation. This is equivalent to the state of the tlb when a pte is
|
|
|
|
changed but before the tlb entry is flushed. Accordingly, unsync ptes
|
|
|
|
are synchronized when the guest executes invlpg or flushes its tlb by
|
|
|
|
other means. Valid for leaf pages.
|
|
|
|
unsync_children:
|
|
|
|
How many sptes in the page point at pages that are unsync (or have
|
|
|
|
unsynchronized children).
|
|
|
|
unsync_child_bitmap:
|
|
|
|
A bitmap indicating which sptes in spt point (directly or indirectly) at
|
|
|
|
pages that may be unsynchronized. Used to quickly locate all unsychronized
|
|
|
|
pages reachable from a given page.
|
2013-06-19 17:09:20 +08:00
|
|
|
clear_spte_count:
|
|
|
|
Only present on 32-bit hosts, where a 64-bit spte cannot be written
|
|
|
|
atomically. The reader uses this while running out of the MMU lock
|
|
|
|
to detect in-progress updates and retry them until the writer has
|
|
|
|
finished the write.
|
2013-06-19 17:09:21 +08:00
|
|
|
write_flooding_count:
|
|
|
|
A guest may write to a page table many times, causing a lot of
|
|
|
|
emulations if the page needs to be write-protected (see "Synchronized
|
|
|
|
and unsynchronized pages" below). Leaf pages can be unsynchronized
|
|
|
|
so that they do not trigger frequent emulation, but this is not
|
|
|
|
possible for non-leafs. This field counts the number of emulations
|
|
|
|
since the last time the page table was actually used; if emulation
|
|
|
|
is triggered too frequently on this page, KVM will unmap the page
|
|
|
|
to avoid emulation in the future.
|
2010-04-21 21:08:20 +08:00
|
|
|
|
|
|
|
Reverse map
|
|
|
|
===========
|
|
|
|
|
|
|
|
The mmu maintains a reverse mapping whereby all ptes mapping a page can be
|
|
|
|
reached given its gfn. This is used, for example, when swapping out a page.
|
|
|
|
|
|
|
|
Synchronized and unsynchronized pages
|
|
|
|
=====================================
|
|
|
|
|
|
|
|
The guest uses two events to synchronize its tlb and page tables: tlb flushes
|
|
|
|
and page invalidations (invlpg).
|
|
|
|
|
|
|
|
A tlb flush means that we need to synchronize all sptes reachable from the
|
|
|
|
guest's cr3. This is expensive, so we keep all guest page tables write
|
|
|
|
protected, and synchronize sptes to gptes when a gpte is written.
|
|
|
|
|
|
|
|
A special case is when a guest page table is reachable from the current
|
|
|
|
guest cr3. In this case, the guest is obliged to issue an invlpg instruction
|
|
|
|
before using the translation. We take advantage of that by removing write
|
|
|
|
protection from the guest page, and allowing the guest to modify it freely.
|
|
|
|
We synchronize modified gptes when the guest invokes invlpg. This reduces
|
|
|
|
the amount of emulation we have to do when the guest modifies multiple gptes,
|
|
|
|
or when the a guest page is no longer used as a page table and is used for
|
|
|
|
random guest data.
|
|
|
|
|
2010-04-26 16:59:21 +08:00
|
|
|
As a side effect we have to resynchronize all reachable unsynchronized shadow
|
2010-04-21 21:08:20 +08:00
|
|
|
pages on a tlb flush.
|
|
|
|
|
|
|
|
|
|
|
|
Reaction to events
|
|
|
|
==================
|
|
|
|
|
|
|
|
- guest page fault (or npt page fault, or ept violation)
|
|
|
|
|
|
|
|
This is the most complicated event. The cause of a page fault can be:
|
|
|
|
|
|
|
|
- a true guest fault (the guest translation won't allow the access) (*)
|
|
|
|
- access to a missing translation
|
|
|
|
- access to a protected translation
|
|
|
|
- when logging dirty pages, memory is write protected
|
|
|
|
- synchronized shadow pages are write protected (*)
|
|
|
|
- access to untranslatable memory (mmio)
|
|
|
|
|
|
|
|
(*) not applicable in direct mode
|
|
|
|
|
|
|
|
Handling a page fault is performed as follows:
|
|
|
|
|
2013-06-19 17:09:22 +08:00
|
|
|
- if the RSV bit of the error code is set, the page fault is caused by guest
|
|
|
|
accessing MMIO and cached MMIO information is available.
|
|
|
|
- walk shadow page table
|
2013-06-19 17:09:25 +08:00
|
|
|
- check for valid generation number in the spte (see "Fast invalidation of
|
|
|
|
MMIO sptes" below)
|
2019-08-02 04:35:21 +08:00
|
|
|
- cache the information to vcpu->arch.mmio_gva, vcpu->arch.mmio_access and
|
2013-06-19 17:09:22 +08:00
|
|
|
vcpu->arch.mmio_gfn, and call the emulator
|
2013-06-19 17:09:23 +08:00
|
|
|
- If both P bit and R/W bit of error code are set, this could possibly
|
|
|
|
be handled as a "fast page fault" (fixed without taking the MMU lock). See
|
2019-07-24 15:24:49 +08:00
|
|
|
the description in Documentation/virt/kvm/locking.txt.
|
2010-04-21 21:08:20 +08:00
|
|
|
- if needed, walk the guest page tables to determine the guest translation
|
|
|
|
(gva->gpa or ngpa->gpa)
|
|
|
|
- if permissions are insufficient, reflect the fault back to the guest
|
|
|
|
- determine the host page
|
2013-06-19 17:09:22 +08:00
|
|
|
- if this is an mmio request, there is no host page; cache the info to
|
2019-08-02 04:35:21 +08:00
|
|
|
vcpu->arch.mmio_gva, vcpu->arch.mmio_access and vcpu->arch.mmio_gfn
|
2010-04-21 21:08:20 +08:00
|
|
|
- walk the shadow page table to find the spte for the translation,
|
|
|
|
instantiating missing intermediate page tables as necessary
|
2013-06-19 17:09:22 +08:00
|
|
|
- If this is an mmio request, cache the mmio info to the spte and set some
|
|
|
|
reserved bit on the spte (see callers of kvm_mmu_set_mmio_spte_mask)
|
2010-04-21 21:08:20 +08:00
|
|
|
- try to unsynchronize the page
|
|
|
|
- if successful, we can let the guest continue and modify the gpte
|
|
|
|
- emulate the instruction
|
|
|
|
- if failed, unshadow the page and let the guest continue
|
|
|
|
- update any translations that were modified by the instruction
|
|
|
|
|
|
|
|
invlpg handling:
|
|
|
|
|
|
|
|
- walk the shadow page hierarchy and drop affected translations
|
|
|
|
- try to reinstantiate the indicated translation in the hope that the
|
|
|
|
guest will use it in the near future
|
|
|
|
|
|
|
|
Guest control register updates:
|
|
|
|
|
|
|
|
- mov to cr3
|
|
|
|
- look up new shadow roots
|
|
|
|
- synchronize newly reachable shadow pages
|
|
|
|
|
|
|
|
- mov to cr0/cr4/efer
|
|
|
|
- set up mmu context for new paging mode
|
|
|
|
- look up new shadow roots
|
|
|
|
- synchronize newly reachable shadow pages
|
|
|
|
|
|
|
|
Host translation updates:
|
|
|
|
|
|
|
|
- mmu notifier called with updated hva
|
|
|
|
- look up affected sptes through reverse map
|
|
|
|
- drop (or update) translations
|
|
|
|
|
2010-05-27 19:46:04 +08:00
|
|
|
Emulating cr0.wp
|
|
|
|
================
|
|
|
|
|
|
|
|
If tdp is not enabled, the host must keep cr0.wp=1 so page write protection
|
|
|
|
works for the guest kernel, not guest guest userspace. When the guest
|
|
|
|
cr0.wp=1, this does not present a problem. However when the guest cr0.wp=0,
|
|
|
|
we cannot map the permissions for gpte.u=1, gpte.w=0 to any spte (the
|
|
|
|
semantics require allowing any guest kernel access plus user read access).
|
|
|
|
|
|
|
|
We handle this by mapping the permissions to two possible sptes, depending
|
|
|
|
on fault type:
|
|
|
|
|
|
|
|
- kernel write fault: spte.u=0, spte.w=1 (allows full kernel access,
|
|
|
|
disallows user access)
|
|
|
|
- read fault: spte.u=1, spte.w=0 (allows full read access, disallows kernel
|
|
|
|
write access)
|
|
|
|
|
|
|
|
(user write faults generate a #PF)
|
|
|
|
|
2015-05-11 22:55:21 +08:00
|
|
|
In the first case there are two additional complications:
|
|
|
|
- if CR4.SMEP is enabled: since we've turned the page into a kernel page,
|
|
|
|
the kernel may now execute it. We handle this by also setting spte.nx.
|
|
|
|
If we get a user fetch or read fault, we'll change spte.u=1 and
|
KVM: MMU: fix ept=0/pte.u=1/pte.w=0/CR0.WP=0/CR4.SMEP=1/EFER.NX=0 combo
Yes, all of these are needed. :) This is admittedly a bit odd, but
kvm-unit-tests access.flat tests this if you run it with "-cpu host"
and of course ept=0.
KVM runs the guest with CR0.WP=1, so it must handle supervisor writes
specially when pte.u=1/pte.w=0/CR0.WP=0. Such writes cause a fault
when U=1 and W=0 in the SPTE, but they must succeed because CR0.WP=0.
When KVM gets the fault, it sets U=0 and W=1 in the shadow PTE and
restarts execution. This will still cause a user write to fault, while
supervisor writes will succeed. User reads will fault spuriously now,
and KVM will then flip U and W again in the SPTE (U=1, W=0). User reads
will be enabled and supervisor writes disabled, going back to the
originary situation where supervisor writes fault spuriously.
When SMEP is in effect, however, U=0 will enable kernel execution of
this page. To avoid this, KVM also sets NX=1 in the shadow PTE together
with U=0. If the guest has not enabled NX, the result is a continuous
stream of page faults due to the NX bit being reserved.
The fix is to force EFER.NX=1 even if the CPU is taking care of the EFER
switch. (All machines with SMEP have the CPU_LOAD_IA32_EFER vm-entry
control, so they do not use user-return notifiers for EFER---if they did,
EFER.NX would be forced to the same value as the host).
There is another bug in the reserved bit check, which I've split to a
separate patch for easier application to stable kernels.
Cc: stable@vger.kernel.org
Cc: Andy Lutomirski <luto@amacapital.net>
Reviewed-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Fixes: f6577a5fa15d82217ca73c74cd2dcbc0f6c781dd
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2016-03-08 19:13:39 +08:00
|
|
|
spte.nx=gpte.nx back. For this to work, KVM forces EFER.NX to 1 when
|
|
|
|
shadow paging is in use.
|
2015-05-11 22:55:21 +08:00
|
|
|
- if CR4.SMAP is disabled: since the page has been changed to a kernel
|
|
|
|
page, it can not be reused when CR4.SMAP is enabled. We set
|
|
|
|
CR4.SMAP && !CR0.WP into shadow page's role to avoid this case. Note,
|
|
|
|
here we do not care the case that CR4.SMAP is enabled since KVM will
|
|
|
|
directly inject #PF to guest due to failed permission check.
|
2011-06-06 21:11:54 +08:00
|
|
|
|
|
|
|
To prevent an spte that was converted into a kernel page with cr0.wp=0
|
|
|
|
from being written by the kernel after cr0.wp has changed to 1, we make
|
|
|
|
the value of cr0.wp part of the page role. This means that an spte created
|
|
|
|
with one value of cr0.wp cannot be used when cr0.wp has a different value -
|
|
|
|
it will simply be missed by the shadow page lookup code. A similar issue
|
|
|
|
exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after
|
|
|
|
changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep
|
|
|
|
is also made a part of the page role.
|
|
|
|
|
2010-05-27 21:44:12 +08:00
|
|
|
Large pages
|
|
|
|
===========
|
|
|
|
|
|
|
|
The mmu supports all combinations of large and small guest and host pages.
|
|
|
|
Supported page sizes include 4k, 2M, 4M, and 1G. 4M pages are treated as
|
|
|
|
two separate 2M pages, on both guest and host, since the mmu always uses PAE
|
|
|
|
paging.
|
|
|
|
|
|
|
|
To instantiate a large spte, four constraints must be satisfied:
|
|
|
|
|
|
|
|
- the spte must point to a large host page
|
|
|
|
- the guest pte must be a large pte of at least equivalent size (if tdp is
|
2012-03-04 22:16:11 +08:00
|
|
|
enabled, there is no guest pte and this condition is satisfied)
|
2010-05-27 21:44:12 +08:00
|
|
|
- if the spte will be writeable, the large page frame may not overlap any
|
|
|
|
write-protected pages
|
|
|
|
- the guest page must be wholly contained by a single memory slot
|
|
|
|
|
2016-02-24 17:51:06 +08:00
|
|
|
To check the last two conditions, the mmu maintains a ->disallow_lpage set of
|
2010-05-27 21:44:12 +08:00
|
|
|
arrays for each memory slot and large page size. Every write protected page
|
2016-02-24 17:51:06 +08:00
|
|
|
causes its disallow_lpage to be incremented, thus preventing instantiation of
|
2010-05-27 21:44:12 +08:00
|
|
|
a large spte. The frames at the end of an unaligned memory slot have
|
2016-02-24 17:51:06 +08:00
|
|
|
artificially inflated ->disallow_lpages so they can never be instantiated.
|
2010-05-27 21:44:12 +08:00
|
|
|
|
2013-06-19 17:09:25 +08:00
|
|
|
Fast invalidation of MMIO sptes
|
|
|
|
===============================
|
|
|
|
|
|
|
|
As mentioned in "Reaction to events" above, kvm will cache MMIO
|
|
|
|
information in leaf sptes. When a new memslot is added or an existing
|
|
|
|
memslot is changed, this information may become stale and needs to be
|
|
|
|
invalidated. This also needs to hold the MMU lock while walking all
|
|
|
|
shadow pages, and is made more scalable with a similar technique.
|
|
|
|
|
|
|
|
MMIO sptes have a few spare bits, which are used to store a
|
|
|
|
generation number. The global generation number is stored in
|
|
|
|
kvm_memslots(kvm)->generation, and increased whenever guest memory info
|
Revert "KVM: MMU: document fast invalidate all pages"
Remove x86 KVM's fast invalidate mechanism, i.e. revert all patches
from the original series[1].
Though not explicitly stated, for all intents and purposes the fast
invalidate mechanism was added to speed up the scenario where removing
a memslot, e.g. as part of accessing reading PCI ROM, caused KVM to
flush all shadow entries[1]. Now that the memslot case flushes only
shadow entries belonging to the memslot, i.e. doesn't use the fast
invalidate mechanism, the only remaining usage of the mechanism are
when the VM is being destroyed and when the MMIO generation rolls
over.
When a VM is being destroyed, either there are no active vcpus, i.e.
there's no lock contention, or the VM has ungracefully terminated, in
which case we want to reclaim its pages as quickly as possible, i.e.
not release the MMU lock if there are still CPUs executing in the VM.
The MMIO generation scenario is almost literally a one-in-a-million
occurrence, i.e. is not a performance sensitive scenario.
Given that lock-breaking is not desirable (VM teardown) or irrelevant
(MMIO generation overflow), remove the fast invalidate mechanism to
simplify the code (a small amount) and to discourage future code from
zapping all pages as using such a big hammer should be a last restort.
This reverts commit f6f8adeef542a18b1cb26a0b772c9781a10bb477.
[1] https://lkml.kernel.org/r/1369960590-14138-1-git-send-email-xiaoguangrong@linux.vnet.ibm.com
Cc: Xiao Guangrong <guangrong.xiao@gmail.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2019-02-06 05:01:22 +08:00
|
|
|
changes.
|
2013-06-19 17:09:25 +08:00
|
|
|
|
|
|
|
When KVM finds an MMIO spte, it checks the generation number of the spte.
|
|
|
|
If the generation number of the spte does not equal the global generation
|
|
|
|
number, it will ignore the cached MMIO information and handle the page
|
|
|
|
fault through the slow path.
|
|
|
|
|
|
|
|
Since only 19 bits are used to store generation-number on mmio spte, all
|
|
|
|
pages are zapped when there is an overflow.
|
|
|
|
|
2014-08-19 06:46:06 +08:00
|
|
|
Unfortunately, a single memory access might access kvm_memslots(kvm) multiple
|
|
|
|
times, the last one happening when the generation number is retrieved and
|
|
|
|
stored into the MMIO spte. Thus, the MMIO spte might be created based on
|
|
|
|
out-of-date information, but with an up-to-date generation number.
|
|
|
|
|
|
|
|
To avoid this, the generation number is incremented again after synchronize_srcu
|
2019-02-06 05:01:18 +08:00
|
|
|
returns; thus, bit 63 of kvm_memslots(kvm)->generation set to 1 only during a
|
2014-08-19 06:46:06 +08:00
|
|
|
memslot update, while some SRCU readers might be using the old copy. We do not
|
|
|
|
want to use an MMIO sptes created with an odd generation number, and we can do
|
2019-02-06 05:01:18 +08:00
|
|
|
this without losing a bit in the MMIO spte. The "update in-progress" bit of the
|
|
|
|
generation is not stored in MMIO spte, and is so is implicitly zero when the
|
|
|
|
generation is extracted out of the spte. If KVM is unlucky and creates an MMIO
|
|
|
|
spte while an update is in-progress, the next access to the spte will always be
|
|
|
|
a cache miss. For example, a subsequent access during the update window will
|
|
|
|
miss due to the in-progress flag diverging, while an access after the update
|
|
|
|
window closes will have a higher generation number (as compared to the spte).
|
2014-08-19 06:46:06 +08:00
|
|
|
|
2013-06-19 17:09:25 +08:00
|
|
|
|
2010-04-21 21:08:20 +08:00
|
|
|
Further reading
|
|
|
|
===============
|
|
|
|
|
|
|
|
- NPT presentation from KVM Forum 2008
|
2018-05-26 07:46:54 +08:00
|
|
|
http://www.linux-kvm.org/images/c/c8/KvmForum2008%24kdf2008_21.pdf
|
2010-04-21 21:08:20 +08:00
|
|
|
|