Prepare of a next patch which will call tlbmiss_handler_setup_pgd on
microMIPS. MicroMIPS complains if the called code s not in the .text
section. To fix this we generate code into space reserved in
arch/mips/mm/tlb-funcs.S
While there, move the rest of the generated functions (handle_tlbl,
handle_tlbs, handle_tlbm) to the same file.
Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5542/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
When having enabled MIPS_PGD_C0_CONTEXT, trap_init() might call the
generated tlbmiss_handler_setup_pgd before it was committed to memory,
causing boot failures:
trap_init()
|- per_cpu_trap_init()
| |- TLBMISS_HANDLER_SETUP()
| |- tlbmiss_handler_setup_pgd()
|- flush_tlb_handlers()
To avoid this, move flush_tlb_handlers() into build_tlb_refill_handler()
right after they were generated. We can do this as the cache handling is
initialized just before creating the tlb handlers.
This issue was introduced in 3d8bfdd030
("MIPS: Use C0_KScratch (if present) to hold PGD pointer.").
Signed-off-by: Jonas Gorski <jogo@openwrt.org>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Jayachandran C <jchandra@broadcom.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5539/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
MIPS R2 documents state that an execution hazard barrier is needed
after a TLBR before reading EntryLo.
Original patch by Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Patchwork: https://patchwork.linux-mips.org/patch/5526/
The invalid value for scratch register is -1, so update the checks of
the form (scratch_reg > 0) to be (scratch_reg >= 0). This will fix
the case in Netlogic XLP where the scratch_reg can be 0.
Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5444/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
XLR/XLP COP0 scratch is register 22, sel 0-7. Add a function
c0_kscratch() which returns the scratch register for the platform,
and use the return value while generating TLB handlers.
Setup kscratch_mask to 0xf for XLR/XLP since the config4 register
does not exist. This allows the kernel to allocate scratch registers
0-3 if needed.
Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5445/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This reverts commit d532f3d267.
The original commit has several problems:
1) Doesn't work with 64-bit kernels.
2) Calls TLBMISS_HANDLER_SETUP() before the code is generated.
3) Calls TLBMISS_HANDLER_SETUP() twice in per_cpu_trap_init() when
only one call is needed.
[ralf@linux-mips.org: Also revert the bits of the ASID patch which were
hidden in the KVM merge.]
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: "Steven J. Hill" <Steven.Hill@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5242/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
All exceptions must be taken in microMIPS mode, never in classic
MIPS mode or the kernel falls apart. A few NOP instructions are
used to maintain the correct alignment of microMIPS versions of
the exception vectors.
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Changes for pure microMIPS cores to dynamically determine the ASID
size at boot time.
Includes bug fix https://patchwork.linux-mips.org/patch/5230/
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Jonas Gorski <jogo@openwrt.org>
Original patch by Ralf Baechle and removed by Harold Koerfgen
with commit f67e4ffc79905482c3b9b8c8dd65197bac7eb508. This
allows for more generic kernels since the size of the ASID
and corresponding masks can be determined at run-time. This
patch is also required for the new Aptiv cores and has been
tested on Malta and Malta Aptiv platforms.
[ralf@linux-mips.org: Added relevant part of fix
https://patchwork.linux-mips.org/patch/5213/]
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This and the next patch resolve memory corruption problems while CPU
hotplug. Without these patches, memory corruption can triggered easily
as below:
On a quad-core MIPS platform, use "spawn" of UnixBench-5.1.3 (http://
code.google.com/p/byte-unixbench/) and a CPU hotplug script like this
(hotplug.sh):
while true; do
echo 0 >/sys/devices/system/cpu/cpu1/online
echo 0 >/sys/devices/system/cpu/cpu2/online
echo 0 >/sys/devices/system/cpu/cpu3/online
sleep 1
echo 1 >/sys/devices/system/cpu/cpu1/online
echo 1 >/sys/devices/system/cpu/cpu2/online
echo 1 >/sys/devices/system/cpu/cpu3/online
sleep 1
done
Run "hotplug.sh" and then run "spawn 10000", spawn will get segfault
after a few minutes.
This patch:
Currently, clear_page()/copy_page() are generated by Micro-assembler
dynamically. But they are unavailable until uasm_resolve_relocs() has
finished because jump labels are illegal before that. Since these
functions are shared by every CPU, we only call build_clear_page()/
build_copy_page() only once at boot time. Without this patch, programs
will get random memory corruption (segmentation fault, bus error, etc.)
while CPU Hotplug (e.g. one CPU is using clear_page() while another is
generating it in cpu_cache_init()).
For similar reasons we modify build_tlb_refill_handler()'s invocation.
V2:
1, Rework the code to make CPU#0 can be online/offline.
2, Introduce cpu_has_local_ebase feature since some types of MIPS CPU
need a per-CPU tlb_refill_handler().
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Hongbing Hu <huhb@lemote.com>
Acked-by: David Daney <david.daney@cavium.com>
Patchwork: http://patchwork.linux-mips.org/patch/4994/
Acked-by: John Crispin <blogic@openwrt.org>
tlbmiss_handler_setup_pgd is run-time generated code and it was convenient
to pretend the symbol was an array in the generator but a function for
the users. LTO gcc won't tolerate this kind of lie anymore so solve the
problem through a cast and function pointer instead.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We need Huge TLBs for HUGETLB_PAGE, or the soon to follow
TRANSPARENT_HUGEPAGE. collect this information under a single Kconfig
symbol.
Signed-off-by: David Daney <david.daney@cavium.com>
Pgtable bits are assigned dynamically depending on processor feature and
statically based on kernel configuration. To make sense out of the
disassembled TLB exception handlers a list of the actual assignments
used for a particular configuration and hardware setup can be very useful.
Output the actual TLB exception handlers in a format that simplifies their
post processsing from dmesg output.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
From a software perspective R5000 and R5000A are the same thing which is
why the symbol CPU_R5000A never got used, so finally delete it.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The new functions introduced in commit 02a5417751 (MIPS: tlbex: Deal with
re-definition of label) should be marked __cpuinit, to eliminate a
warning that can pop up when CONFIG_EXPORT_UASM is disabled:
LD arch/mips/mm/built-in.o
WARNING: arch/mips/mm/built-in.o(.text+0x2a4c): Section mismatch in reference from the function uasm_bgezl_hazard() to the function .cpuinit.text:uasm_il_bgezl()
The function uasm_bgezl_hazard() references
the function __cpuinit uasm_il_bgezl().
This is often because uasm_bgezl_hazard lacks a __cpuinit
annotation or the annotation of uasm_il_bgezl is wrong.
WARNING: arch/mips/mm/built-in.o(.text+0x2a68): Section mismatch in reference from the function uasm_bgezl_label() to the function .cpuinit.text:uasm_build_label()
The function uasm_bgezl_label() references
the function __cpuinit uasm_build_label().
This is often because uasm_bgezl_label lacks a __cpuinit
annotation or the annotation of uasm_build_label is wrong.
(This warning might not occur if the function was inlined.)
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Patchwork: http://patchwork.linux-mips.org/patch/4517
Signed-off-by: John Crispin <blogic@openwrt.org>
R5000 and the Nevada CPUs (RM5230, RM5231, RM5260, RM5261, RM5270 and
RM5271) are basically the same CPU core and all are documented to require
two instructions separating a write to c0_pagemask, c0_entryhi, c0_entrylo0,
c0_entrylo1 or c0_index.
So far we were only providing on cycle before / after a TLBR/TLBWI
for R5000 but 3 cycles before and 1 cycles after for the Nevadas.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The microassembler used in tlbex.c does not notice if a label is redefined
resulting in relocations against such labels silently missrelocated.
The issues exists since commit add6eb04776db4189ea89f596cbcde31b899be9d
[Synthesize TLB exception handlers at runtime.] in 2.6.10 and went unnoticed
for so long because the relocations for the affected branches got computed
to do something *almost* sensible.
The issue affects R4000, R4400, QED/IDT RM5230, RM5231, RM5260, RM5261,
RM5270 and RM5271 processors.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We don't have to do a separate shift to eliminate the software bits,
just rotate them into the fill and they will be ignored.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/4294/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Remove usage of the 'kernel_uses_smartmips_rixi' macro from all files
and use new 'cpu_has_rixi' instead.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
Acked-by: David Daney <david.daney@cavium.com>
The EXT and INS instructions can be used to decrease code size and
thus speed up TLB handlers on MIPS32R2 and MIPS64R2 cores.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
The architecture specification says that an EHB instruction is
needed to avoid a hazard when writing TLB entries. However, some
cores do not have this hazard, and thus the EHB instruction causes
a costly pipeline stall. Detect these cores and do not use the EHB
instruction.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
For the case PM_DEFAULT_MASK == 0, we were placing a branch in the
delay slot of another branch. This leads to undefined behavior.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2775/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Only some GCC versions such as gcc 4.2 notice that the variable wr in
build_r3000_tlb_modify_handler is used uninitialized. When using one
of those GCCs the build will fail due to -Werror. GCC 4.6 does not
warn about the uninitialized use of wr.
This issue was introduced by 7211f4d7a3dcbe57c5d396c334dca525315dceb2
[MIPS: Close races in TLB modify handlers.]
Reported-by: Ganesan Ramalingam <ganesan18@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Page table entries are made invalid by writing a zero into the the PTE
slot in a page table. This creates a race condition with the TLB
modify handlers when they are updating the PTE.
CPU0 CPU1
Test for _PAGE_PRESENT
. set to not _PAGE_PRESENT (zero)
Set to _PAGE_VALID
So now the page not present value (zero) is suddenly valid and user
space programs have access to physical page zero.
We close the race by putting the test for _PAGE_PRESENT and setting of
_PAGE_VALID into an atomic LL/SC section. This requires more registers
than just K0 and K1 in the handlers, so we need to save some registers
to a save area and then restore them when we are done.
The save area is an array of cacheline aligned structures that should
not suffer cache line bouncing as they are CPU private.
[ralf@linux-mips.org: Fix !defined(CONFIG_MIPS_PGD_C0_CONTEXT) build error.]
Signed-off-by: David Daney <david.daney@cavium.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2577/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
CPU_XLR case added to mm/tlbex.c
CPU_XLR case added to mm/c-r4k.c for PINDEX attribute
Feature overrides for XLR cpu.
Signed-off-by: Jayachandran C <jayachandranc@netlogicmicro.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2333/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
CC arch/mips/mm/tlbex.o
arch/mips/mm/tlbex.c: In function 'build_r4000_tlb_refill_handler':
arch/mips/mm/tlbex.c:1155:22: error: variable 'vmalloc_mode' set but not used [-Werror=unused-but-set-variable]
arch/mips/mm/tlbex.c:1154:28: error: variable 'htlb_info' set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
It was reported that GCC-4.3.3 (with CodeSourcery extensions) fails
without this.
Reported-by: Jonas Gorski <jonas.gorski@gmail.com>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2010/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Octeon can use scratch registers in the TLB handlers. Octeon II can
use LDX instructions.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1904/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
If the CPU supports BBIT0 and BBIT1, use them in TLB handlers as they
are more efficient than an AND followed by an branch and then
restoring the clobbered register.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1873/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
BMIPS processor cores are used in 50+ different chipsets spread across
5+ product lines. In many cases the chipsets do not share the same
peripheral register layouts, the same register blocks, the same
interrupt controllers, the same memory maps, or much of anything else.
But, across radically different SoCs that share nothing more than the
same BMIPS CPU, a few things are still mostly constant:
SMP operations
Access to performance counters
DMA cache coherency quirks
Cache and memory bus configuration
So, it makes sense to treat each BMIPS processor type as a generic
"building block," rather than tying it to a specific SoC. This makes it
easier to support a large number of BMIPS-based chipsets without
unnecessary duplication of code, and provides the infrastructure needed
to support BMIPS-proprietary features.
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Cc: mbizon@freebox.fr
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Tested-by: Florian Fainelli <ffainelli@freebox.fr>
Patchwork: https://patchwork.linux-mips.org/patch/1706/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org
Ralf Baechle