openkylin
/
linux
mirror of https://gitee.com/openkylin/linux.git
9
0
Fork 0
Code Issues Projects Releases Wiki Activity
linux/arch/powerpc/kernel/entry_32.S

1350 lines
33 KiB
ArmAsm
Raw Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license as published by the free software foundation either version 2 of the license or at your option any later version extracted by the scancode license scanner the SPDX license identifier GPL-2.0-or-later has been chosen to replace the boilerplate/reference in 3029 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190527070032.746973796@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-27 14:55:01 +08:00
/* SPDX-License-Identifier: GPL-2.0-or-later */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* PowerPC version
* Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org)
* Rewritten by Cort Dougan (cort@fsmlabs.com) for PReP
* Copyright (C) 1996 Cort Dougan <cort@fsmlabs.com>
* Adapted for Power Macintosh by Paul Mackerras.
* Low-level exception handlers and MMU support
* rewritten by Paul Mackerras.
* Copyright (C) 1996 Paul Mackerras.
* MPC8xx modifications Copyright (C) 1997 Dan Malek (dmalek@jlc.net).
*
* This file contains the system call entry code, context switch
* code, and exception/interrupt return code for PowerPC.
*/
#include <linux/errno.h>
powerpc/kernel: Switch to using MAX_ERRNO Currently on powerpc we have our own #define for the highest (negative) errno value, called _LAST_ERRNO. This is defined to be 516, for reasons which are not clear. The generic code, and x86, use MAX_ERRNO, which is defined to be 4095. In particular seccomp uses MAX_ERRNO to restrict the value that a seccomp filter can return. Currently with the mismatch between _LAST_ERRNO and MAX_ERRNO, a seccomp tracer wanting to return 600, expecting it to be seen as an error, would instead find on powerpc that userspace sees a successful syscall with a return value of 600. To avoid this inconsistency, switch powerpc to use MAX_ERRNO. We are somewhat confident that generic syscalls that can return a non-error value above negative MAX_ERRNO have already been updated to use force_successful_syscall_return(). I have also checked all the powerpc specific syscalls, and believe that none of them expect to return a non-error value between -MAX_ERRNO and -516. So this change should be safe ... Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Reviewed-by: Kees Cook <keescook@chromium.org>
2015-07-23 18:21:01 +08:00
#include <linux/err.h>
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#include <linux/sys.h>
#include <linux/threads.h>
#include <asm/reg.h>
#include <asm/page.h>
#include <asm/mmu.h>
#include <asm/cputable.h>
#include <asm/thread_info.h>
#include <asm/ppc_asm.h>
#include <asm/asm-offsets.h>
#include <asm/unistd.h>
powerpc: Remove second definition of STACK_FRAME_OVERHEAD Since STACK_FRAME_OVERHEAD is defined in asm/ptrace.h and that is ASSEMBER safe, we can just include that instead of going via asm-offsets.h. Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2010-11-18 23:06:17 +08:00
#include <asm/ptrace.h>
ppc: move exports to definitions Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2016-01-14 12:33:46 +08:00
#include <asm/export.h>
powerpc/405: move PPC405_ERR77 in asm-405.h Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-07-06 00:24:55 +08:00
#include <asm/asm-405.h>
powerpc: clean inclusions of asm/feature-fixups.h files not using feature fixup don't need asm/feature-fixups.h files using feature fixup need asm/feature-fixups.h Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-07-06 00:25:01 +08:00
#include <asm/feature-fixups.h>
powerpc/fsl: Sanitize the syscall table for NXP PowerPC 32 bit platforms Used barrier_nospec to sanitize the syscall table. Signed-off-by: Diana Craciun <diana.craciun@nxp.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-07-28 07:06:38 +08:00
#include <asm/barrier.h>
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
#include <asm/kup.h>
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
#include <asm/bug.h>
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
powerpc/32: move LOAD_MSR_KERNEL() into head_32.h and use it As preparation for using head_32.h for head_40x.S, move LOAD_MSR_KERNEL() there and use it to load r10 with MSR_KERNEL value. In the mean time, this patch modifies it so that it takes into account the size of the passed value to determine if 'li' can be used or if 'lis/ori' is needed instead of using the size of MSR_KERNEL. This is done by using gas macro. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:38:51 +08:00
#include "head_32.h"
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
powerpc/32: Avoid risk of unrecoverable TLBmiss inside entry_32.S By default, the 8xx pins an ITLB on the first 8M of memory in order to avoid any ITLB miss on kernel code. However, with some debug functions like DEBUG_PAGEALLOC and DEBUG_RODATA, pinning TLBs is contradictory. In order to avoid any ITLB miss in a critical section without pinning TLBs, we have to ensure that there is no page boundary crossed between the setup of a new value in SRR0/SRR1 and the associated RFI. The functions modifying srr0/srr1 are all located in setup_32.S. They are spread over almost 4kbytes. The patch forces a 12 bits (4kbytes) alignment for those functions. This garanties that the functions remain in a single 4k page. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-07-12 18:08:49 +08:00
/*
* Align to 4k in order to ensure that all functions modyfing srr0/srr1
* fit into one page in order to not encounter a TLB miss between the
* modification of srr0/srr1 and the associated rfi.
*/
.align 12
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#ifdef CONFIG_BOOKE
.globl mcheck_transfer_to_handler
mcheck_transfer_to_handler:
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
mfspr r0,SPRN_DSRR0
stw r0,_DSRR0(r11)
mfspr r0,SPRN_DSRR1
stw r0,_DSRR1(r11)
/* fall through */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl debug_transfer_to_handler
debug_transfer_to_handler:
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
mfspr r0,SPRN_CSRR0
stw r0,_CSRR0(r11)
mfspr r0,SPRN_CSRR1
stw r0,_CSRR1(r11)
/* fall through */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl crit_transfer_to_handler
crit_transfer_to_handler:
powerpc/book-3e: Introduce concept of Book-3e MMU The Power ISA 2.06 spec introduces a standard MMU programming model that is based on the Freescale Book-E MMU programing model. The Freescale version is pretty backwards compatiable with the ISA 2.06 definition so we are starting to refactor some of the Freescale code so it can be easily shared. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2009-02-13 06:12:40 +08:00
#ifdef CONFIG_PPC_BOOK3E_MMU
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
mfspr r0,SPRN_MAS0
stw r0,MAS0(r11)
mfspr r0,SPRN_MAS1
stw r0,MAS1(r11)
mfspr r0,SPRN_MAS2
stw r0,MAS2(r11)
mfspr r0,SPRN_MAS3
stw r0,MAS3(r11)
mfspr r0,SPRN_MAS6
stw r0,MAS6(r11)
#ifdef CONFIG_PHYS_64BIT
mfspr r0,SPRN_MAS7
stw r0,MAS7(r11)
#endif /* CONFIG_PHYS_64BIT */
powerpc/book-3e: Introduce concept of Book-3e MMU The Power ISA 2.06 spec introduces a standard MMU programming model that is based on the Freescale Book-E MMU programing model. The Freescale version is pretty backwards compatiable with the ISA 2.06 definition so we are starting to refactor some of the Freescale code so it can be easily shared. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2009-02-13 06:12:40 +08:00
#endif /* CONFIG_PPC_BOOK3E_MMU */
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
#ifdef CONFIG_44x
mfspr r0,SPRN_MMUCR
stw r0,MMUCR(r11)
#endif
mfspr r0,SPRN_SRR0
stw r0,_SRR0(r11)
mfspr r0,SPRN_SRR1
stw r0,_SRR1(r11)
powerpc: regain entire stack space thread_info is not anymore in the stack, so the entire stack can now be used. There is also no risk anymore of corrupting task_cpu(p) with a stack overflow so the patch removes the test. When doing this, an explicit test for NULL stack pointer is needed in validate_sp() as it is not anymore implicitely covered by the sizeof(thread_info) gap. In the meantime, with the previous patch all pointers to the stacks are not anymore pointers to thread_info so this patch changes them to void* Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:00 +08:00
/* set the stack limit to the current stack */
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r8,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lwz r0,KSP_LIMIT(r8)
stw r0,SAVED_KSP_LIMIT(r11)
powerpc: regain entire stack space thread_info is not anymore in the stack, so the entire stack can now be used. There is also no risk anymore of corrupting task_cpu(p) with a stack overflow so the patch removes the test. When doing this, an explicit test for NULL stack pointer is needed in validate_sp() as it is not anymore implicitely covered by the sizeof(thread_info) gap. In the meantime, with the previous patch all pointers to the stacks are not anymore pointers to thread_info so this patch changes them to void* Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:00 +08:00
rlwinm r0,r1,0,0,(31 - THREAD_SHIFT)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
stw r0,KSP_LIMIT(r8)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* fall through */
#endif
#ifdef CONFIG_40x
.globl crit_transfer_to_handler
crit_transfer_to_handler:
lwz r0,crit_r10@l(0)
stw r0,GPR10(r11)
lwz r0,crit_r11@l(0)
stw r0,GPR11(r11)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
mfspr r0,SPRN_SRR0
stw r0,crit_srr0@l(0)
mfspr r0,SPRN_SRR1
stw r0,crit_srr1@l(0)
powerpc: regain entire stack space thread_info is not anymore in the stack, so the entire stack can now be used. There is also no risk anymore of corrupting task_cpu(p) with a stack overflow so the patch removes the test. When doing this, an explicit test for NULL stack pointer is needed in validate_sp() as it is not anymore implicitely covered by the sizeof(thread_info) gap. In the meantime, with the previous patch all pointers to the stacks are not anymore pointers to thread_info so this patch changes them to void* Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:00 +08:00
/* set the stack limit to the current stack */
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r8,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lwz r0,KSP_LIMIT(r8)
stw r0,saved_ksp_limit@l(0)
powerpc: regain entire stack space thread_info is not anymore in the stack, so the entire stack can now be used. There is also no risk anymore of corrupting task_cpu(p) with a stack overflow so the patch removes the test. When doing this, an explicit test for NULL stack pointer is needed in validate_sp() as it is not anymore implicitely covered by the sizeof(thread_info) gap. In the meantime, with the previous patch all pointers to the stacks are not anymore pointers to thread_info so this patch changes them to void* Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:00 +08:00
rlwinm r0,r1,0,0,(31 - THREAD_SHIFT)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
stw r0,KSP_LIMIT(r8)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* fall through */
#endif
/*
* This code finishes saving the registers to the exception frame
* and jumps to the appropriate handler for the exception, turning
* on address translation.
* Note that we rely on the caller having set cr0.eq iff the exception
* occurred in kernel mode (i.e. MSR:PR = 0).
*/
.globl transfer_to_handler_full
transfer_to_handler_full:
SAVE_NVGPRS(r11)
/* fall through */
.globl transfer_to_handler
transfer_to_handler:
stw r2,GPR2(r11)
stw r12,_NIP(r11)
stw r9,_MSR(r11)
andi. r2,r9,MSR_PR
mfctr r12
mfspr r2,SPRN_XER
stw r12,_CTR(r11)
stw r2,_XER(r11)
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r12,SPRN_SPRG_THREAD
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
beq 2f /* if from user, fix up THREAD.regs */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
addi r2, r12, -THREAD
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
addi r11,r1,STACK_FRAME_OVERHEAD
stw r11,PT_REGS(r12)
#if defined(CONFIG_40x) || defined(CONFIG_BOOKE)
/* Check to see if the dbcr0 register is set up to debug. Use the
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
internal debug mode bit to do this. */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r12,THREAD_DBCR0(r12)
powerpc/booke: Clean up the hardware watchpoint support * CONFIG_BOOKE is selected by CONFIG_44x so we dont need both * Fixed a few comments * Go back to only using DBCR0_IDM to determine if we are using debug resources. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-07-26 03:27:33 +08:00
andis. r12,r12,DBCR0_IDM@h
powerpc/32: Fix CONFIG_VIRT_CPU_ACCOUNTING_NATIVE for 40x/booke 40x/booke have another path to reach 3f from transfer_to_handler, make sure it also calls ACCOUNT_CPU_USER_ENTRY() when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is selected. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:10:31 +08:00
#endif
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
ACCOUNT_CPU_USER_ENTRY(r2, r11, r12)
powerpc/32s: Implement Kernel Userspace Execution Prevention. To implement Kernel Userspace Execution Prevention, this patch sets NX bit on all user segments on kernel entry and clears NX bit on all user segments on kernel exit. Note that powerpc 601 doesn't have the NX bit, so KUEP will not work on it. A warning is displayed at startup. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:35 +08:00
#ifdef CONFIG_PPC_BOOK3S_32
kuep_lock r11, r12
#endif
powerpc/32: Fix CONFIG_VIRT_CPU_ACCOUNTING_NATIVE for 40x/booke 40x/booke have another path to reach 3f from transfer_to_handler, make sure it also calls ACCOUNT_CPU_USER_ENTRY() when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is selected. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:10:31 +08:00
#if defined(CONFIG_40x) || defined(CONFIG_BOOKE)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
beq+ 3f
/* From user and task is ptraced - load up global dbcr0 */
li r12,-1 /* clear all pending debug events */
mtspr SPRN_DBSR,r12
lis r11,global_dbcr0@ha
tophys(r11,r11)
addi r11,r11,global_dbcr0@l
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
#ifdef CONFIG_SMP
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r9,TASK_CPU(r2)
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
slwi r9,r9,3
add r11,r11,r9
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r12,0(r11)
mtspr SPRN_DBCR0,r12
lwz r12,4(r11)
addi r12,r12,-1
stw r12,4(r11)
#endif
powerpc32: provide VIRT_CPU_ACCOUNTING This patch provides VIRT_CPU_ACCOUTING to PPC32 architecture. PPC32 doesn't have the PACA structure, so we use the task_info structure to store the accounting data. In order to reuse on PPC32 the PPC64 functions, all u64 data has been replaced by 'unsigned long' so that it is u32 on PPC32 and u64 on PPC64 Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Scott Wood <oss@buserror.net>
2016-05-17 14:33:46 +08:00
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b 3f
powerpc: Use correct sequence for putting CPU into nap mode We weren't using the recommended sequence for putting the CPU into nap mode. When I changed the idle loop, for some reason 7447A cpus started hanging when we put them into nap mode. Changing to the recommended sequence fixes that. The complexity here is that the recommended sequence is a loop that keeps putting the cpu back into nap mode. Clearly we need some way to break out of the loop when an interrupt (external interrupt, decrementer, performance monitor) occurs. Here we use a bit in the thread_info struct to indicate that we need this, and the exception entry code notices this and arranges for the exception to return to the value in the link register, thus breaking out of the loop. We use a new `local_flags' field in the thread_info which we can alter without needing to use an atomic update sequence. The PPC970 has the same recommended sequence, so we do the same thing there too. This also fixes a bug in the kernel stack overflow handling code on 32-bit, since it was causing a value that we needed in a register to get trashed. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-04-18 19:49:11 +08:00
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
2: /* if from kernel, check interrupted DOZE/NAP mode and
* check for stack overflow
*/
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
kuap_save_and_lock r11, r12, r9, r2, r0
addi r2, r12, -THREAD
[POWERPC] Add IRQSTACKS support on ppc32 This makes it possible to use separate stacks for hard and soft IRQs on 32-bit powerpc as well as on 64-bit. The code for 32-bit is just the 32-bit analog of the 64-bit code. * Added allocation and initialization of the irq stacks. We limit the stacks to be in lowmem for ppc32. * Implemented ppc32 versions of call_do_softirq() and call_handle_irq() to switch the stack pointers * Reworked how we do stack overflow detection. We now keep around the limit of the stack in the thread_struct and compare against the limit to see if we've overflowed. We can now use this on ppc64 if desired. [ paulus@samba.org: Fixed bug on 6xx where we need to reload r9 with the thread_info pointer. ] Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-04-28 14:21:22 +08:00
lwz r9,KSP_LIMIT(r12)
cmplw r1,r9 /* if r1 <= ksp_limit */
powerpc: Use correct sequence for putting CPU into nap mode We weren't using the recommended sequence for putting the CPU into nap mode. When I changed the idle loop, for some reason 7447A cpus started hanging when we put them into nap mode. Changing to the recommended sequence fixes that. The complexity here is that the recommended sequence is a loop that keeps putting the cpu back into nap mode. Clearly we need some way to break out of the loop when an interrupt (external interrupt, decrementer, performance monitor) occurs. Here we use a bit in the thread_info struct to indicate that we need this, and the exception entry code notices this and arranges for the exception to return to the value in the link register, thus breaking out of the loop. We use a new `local_flags' field in the thread_info which we can alter without needing to use an atomic update sequence. The PPC970 has the same recommended sequence, so we do the same thing there too. This also fixes a bug in the kernel stack overflow handling code on 32-bit, since it was causing a value that we needed in a register to get trashed. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-04-18 19:49:11 +08:00
ble- stack_ovf /* then the kernel stack overflowed */
5:
powerpc: change CONFIG_6xx to CONFIG_PPC_BOOK3S_32 Today we have: config PPC_BOOK3S_32 bool "512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx" [depends on PPC32 within a choice] config PPC_BOOK3S def_bool y depends on PPC_BOOK3S_32 || PPC_BOOK3S_64 config 6xx def_bool y depends on PPC32 && PPC_BOOK3S 6xx is therefore redundant with PPC_BOOK3S_32. In order to make the code clearer, lets use preferably PPC_BOOK3S_32. This will allow to remove CONFIG_6xx in a later patch. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-11-17 18:24:56 +08:00
#if defined(CONFIG_PPC_BOOK3S_32) || defined(CONFIG_E500)
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r12,TI_LOCAL_FLAGS(r2)
powerpc: Use correct sequence for putting CPU into nap mode We weren't using the recommended sequence for putting the CPU into nap mode. When I changed the idle loop, for some reason 7447A cpus started hanging when we put them into nap mode. Changing to the recommended sequence fixes that. The complexity here is that the recommended sequence is a loop that keeps putting the cpu back into nap mode. Clearly we need some way to break out of the loop when an interrupt (external interrupt, decrementer, performance monitor) occurs. Here we use a bit in the thread_info struct to indicate that we need this, and the exception entry code notices this and arranges for the exception to return to the value in the link register, thus breaking out of the loop. We use a new `local_flags' field in the thread_info which we can alter without needing to use an atomic update sequence. The PPC970 has the same recommended sequence, so we do the same thing there too. This also fixes a bug in the kernel stack overflow handling code on 32-bit, since it was causing a value that we needed in a register to get trashed. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-04-18 19:49:11 +08:00
mtcrf 0x01,r12
bt- 31-TLF_NAPPING,4f
[POWERPC] Defer processing of interrupts when the CPU wakes from sleep mode This provides a way to defer processing of an interrupt that wakes the processor out of sleep mode. On 32-bit platforms that use an interrupt to wake the processor, we have to have interrupts enabled in hardware at the point where we go to sleep, otherwise the processor will never wake up. However, because interrupts are logically disabled at this point, we don't want to process the interrupt straight away. This is handled by setting the _TLF_SLEEPING flag. When we get an interrupt and _TLF_SLEEPING is set, we firstly clear the MSR_EE (external interrupt enable) bit in the saved MSR value, and secondly we then return to the address in the link register, like we do for _TLF_NAPPING, but without actually handling the interrupt. Note that this is handled somewhat differently on powerbooks, so this new code will only be used on non-Apple machines. Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-05-14 12:30:48 +08:00
bt- 31-TLF_SLEEPING,7f
powerpc: change CONFIG_6xx to CONFIG_PPC_BOOK3S_32 Today we have: config PPC_BOOK3S_32 bool "512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx" [depends on PPC32 within a choice] config PPC_BOOK3S def_bool y depends on PPC_BOOK3S_32 || PPC_BOOK3S_64 config 6xx def_bool y depends on PPC32 && PPC_BOOK3S 6xx is therefore redundant with PPC_BOOK3S_32. In order to make the code clearer, lets use preferably PPC_BOOK3S_32. This will allow to remove CONFIG_6xx in a later patch. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-11-17 18:24:56 +08:00
#endif /* CONFIG_PPC_BOOK3S_32 || CONFIG_E500 */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl transfer_to_handler_cont
transfer_to_handler_cont:
3:
mflr r9
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
tovirt(r2, r2) /* set r2 to current */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r11,0(r9) /* virtual address of handler */
lwz r9,4(r9) /* where to go when done */
powerpc/8xx: Only perform perf counting when perf is in use. In TLB miss handlers, updating the perf counter is only useful when performing a perf analysis. As it has a noticeable overhead, let's only do it when needed. In order to do so, the exit of the miss handlers will be patched when starting/stopping 'perf': the first register restore instruction of each exit point will be replaced by a jump to the counting code. Once this is done, CONFIG_PPC_8xx_PERF_EVENT becomes useless as this feature doesn't add any overhead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-01-12 20:45:23 +08:00
#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
powerpc/8xx: Perf events on PPC 8xx This patch has been reworked since RFC version. In the RFC, this patch was preceded by a patch clearing MSR RI for all PPC32 at all time at exception prologs. Now MSR RI clearing is done only when this 8xx perf events functionality is compiled in, it is therefore limited to 8xx and merged inside this patch. Other main changes have been to take into account detailed review from Peter Zijlstra. The instructions counter has been reworked to behave as a free running counter like the three other counters. The 8xx has no PMU, however some events can be emulated by other means. This patch implements the following events (as reported by 'perf list'): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] dTLB-load-misses [Hardware cache event] iTLB-load-misses [Hardware cache event] 'cycles' event is implemented using the timebase clock. Timebase clock corresponds to CPU clock divided by 16, so number of cycles is approximatly 16 times the number of TB ticks On the 8xx, TLB misses are handled by software. It is therefore easy to count all TLB misses each time the TLB miss exception is called. 'instructions' is calculated by using instruction watchpoint counter. This patch sets counter A to count instructions at address greater than 0, hence we count all instructions executed while MSR RI bit is set. The counter is set to the maximum which is 0xffff. Every 65535 instructions, debug instruction breakpoint exception fires. The exception handler increments a counter in memory which then represent the upper part of the instruction counter. We therefore end up with a 48 bits counter. In order to avoid unnecessary overhead while no perf event is active, this counter is started when the first event referring to this counter is added, and the counter is stopped when the last event referring to it is deleted. In order to properly support breakpoint exceptions, MSR RI bit has to be unset in exception epilogs in order to avoid breakpoint exceptions during critical sections during changes to SRR0 and SRR1 would be problematic. All counters are handled as free running counters. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Scott Wood <oss@buserror.net>
2016-12-15 20:42:18 +08:00
mtspr SPRN_NRI, r0
#endif
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
/*
* When tracing IRQ state (lockdep) we enable the MMU before we call
* the IRQ tracing functions as they might access vmalloc space or
* perform IOs for console output.
*
* To speed up the syscall path where interrupts stay on, let's check
* first if we are changing the MSR value at all.
*/
tophys(r12, r1)
lwz r12,_MSR(r12)
andi. r12,r12,MSR_EE
bne 1f
/* MSR isn't changing, just transition directly */
#endif
mtspr SPRN_SRR0,r11
mtspr SPRN_SRR1,r10
mtlr r9
SYNC
RFI /* jump to handler, enable MMU */
#ifdef CONFIG_TRACE_IRQFLAGS
1: /* MSR is changing, re-enable MMU so we can notify lockdep. We need to
* keep interrupts disabled at this point otherwise we might risk
* taking an interrupt before we tell lockdep they are enabled.
*/
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
lis r12,reenable_mmu@h
ori r12,r12,reenable_mmu@l
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
LOAD_MSR_KERNEL(r0, MSR_KERNEL)
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
mtspr SPRN_SRR0,r12
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
mtspr SPRN_SRR1,r0
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
SYNC
RFI
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
reenable_mmu:
powerpc/trace: Add a dummy stack frame for trace_hardirqs_off The trace_hardirqs_off will use CALLER_ADDR0 and CALLER_ADDR1. If an exception occurs in user mode, there is only one stack frame on the stack and accessing the CALLER_ADDR1 will causes the following call trace. So we create a dummy stack frame to make trace_hardirqs_off happy. WARNING: at kernel/smp.c:459 Modules linked in: NIP: c0093280 LR: c00930a0 CTR: c0010780 REGS: edb87ae0 TRAP: 0700 Not tainted (3.1.0) MSR: 00021002 <ME,CE> CR: 28002888 XER: 00000000 TASK = edce2ac0[17658] 'mthread-lock-on' THREAD: edb86000 CPU: 5 GPR00: 00000001 edb87b90 edce2ac0 00000005 c0019594 edb87bd8 00000001 00000fe3 GPR08: 00041000 c084138c 4e20120d edb87b90 48002888 1001aa7c 00000000 00000000 GPR16: 48830000 10012a8c 00000000 10000af4 00000001 c0810000 00000000 00000000 GPR24: ee9aa920 c0816a18 00000000 00000005 c0019594 edb87bd8 ee20178c edb87b90 NIP [c0093280] smp_call_function_many+0x214/0x2b4 LR [c00930a0] smp_call_function_many+0x34/0x2b4 Call Trace: [edb87b90] [c00930a0] smp_call_function_many+0x34/0x2b4 (unreliable) [edb87bd0] [c00194ec] __flush_tlb_page+0xac/0x100 [edb87c00] [c001957c] flush_tlb_page+0x3c/0x54 [edb87c10] [c00180ac] ptep_set_access_flags+0x74/0x12c [edb87c40] [c0128068] handle_pte_fault+0x2f0/0x9ac [edb87cb0] [c0128c3c] handle_mm_fault+0x104/0x1dc [edb87ce0] [c05f40f4] do_page_fault+0x2dc/0x630 [edb87e50] [c001078c] handle_page_fault+0xc/0x80 Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-11-11 00:04:17 +08:00
/*
powerpc/32: Don't add dummy frames when calling trace_hardirqs_on/off No need to add dummy frames when calling trace_hardirqs_on or trace_hardirqs_off. GCC properly handles empty stacks. In addition, powerpc doesn't set CONFIG_FRAME_POINTER, therefore __builtin_return_address(1..) returns NULL at all time. So the dummy frames are definitely unneeded here. In the meantime, avoid reading memory for loading r1 with a value we already know. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:05 +08:00
* We save a bunch of GPRs,
powerpc: Fix page fault with lockdep regression commit a546498f3bf9aac311c66f965186373aee2ca0b0 introduced a regression on 32-bit when irq tracing is enabled by exposing an old bug in our irq tracing code for exception entry. The code would save and restore some GPRs around the calls to the C lockdep code, however, it tries to be too smart for its own good and restores some of the GPRs from the exception frame (as saved there on exception entry). However, for page faults, we do replace those GPRs with arguments to do_page_fault before we call transfer_to_handler and so restoring from the exception frame is plain wrong in this case. This was fine as long as we didn't touch the interrupt state when taking page fault, but when I started doing it, it would trigger the lockdep calls and the bug. This fixes it by cleaning up that code a bit. It did create a small stack frame for the sake of backtraces, so let's make it a bit bigger and use it to save and restore the stuff we care about. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-04-10 15:21:35 +08:00
* r3 can be different from GPR3(r1) at this point, r9 and r11
* contains the old MSR and handler address respectively,
* r4 & r5 can contain page fault arguments that need to be passed
* along as well. r12, CCR, CTR, XER etc... are left clobbered as
* they aren't useful past this point (aren't syscall arguments),
* the rest is restored from the exception frame.
powerpc/trace: Add a dummy stack frame for trace_hardirqs_off The trace_hardirqs_off will use CALLER_ADDR0 and CALLER_ADDR1. If an exception occurs in user mode, there is only one stack frame on the stack and accessing the CALLER_ADDR1 will causes the following call trace. So we create a dummy stack frame to make trace_hardirqs_off happy. WARNING: at kernel/smp.c:459 Modules linked in: NIP: c0093280 LR: c00930a0 CTR: c0010780 REGS: edb87ae0 TRAP: 0700 Not tainted (3.1.0) MSR: 00021002 <ME,CE> CR: 28002888 XER: 00000000 TASK = edce2ac0[17658] 'mthread-lock-on' THREAD: edb86000 CPU: 5 GPR00: 00000001 edb87b90 edce2ac0 00000005 c0019594 edb87bd8 00000001 00000fe3 GPR08: 00041000 c084138c 4e20120d edb87b90 48002888 1001aa7c 00000000 00000000 GPR16: 48830000 10012a8c 00000000 10000af4 00000001 c0810000 00000000 00000000 GPR24: ee9aa920 c0816a18 00000000 00000005 c0019594 edb87bd8 ee20178c edb87b90 NIP [c0093280] smp_call_function_many+0x214/0x2b4 LR [c00930a0] smp_call_function_many+0x34/0x2b4 Call Trace: [edb87b90] [c00930a0] smp_call_function_many+0x34/0x2b4 (unreliable) [edb87bd0] [c00194ec] __flush_tlb_page+0xac/0x100 [edb87c00] [c001957c] flush_tlb_page+0x3c/0x54 [edb87c10] [c00180ac] ptep_set_access_flags+0x74/0x12c [edb87c40] [c0128068] handle_pte_fault+0x2f0/0x9ac [edb87cb0] [c0128c3c] handle_mm_fault+0x104/0x1dc [edb87ce0] [c05f40f4] do_page_fault+0x2dc/0x630 [edb87e50] [c001078c] handle_page_fault+0xc/0x80 Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-11-11 00:04:17 +08:00
*/
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
powerpc: Fix page fault with lockdep regression commit a546498f3bf9aac311c66f965186373aee2ca0b0 introduced a regression on 32-bit when irq tracing is enabled by exposing an old bug in our irq tracing code for exception entry. The code would save and restore some GPRs around the calls to the C lockdep code, however, it tries to be too smart for its own good and restores some of the GPRs from the exception frame (as saved there on exception entry). However, for page faults, we do replace those GPRs with arguments to do_page_fault before we call transfer_to_handler and so restoring from the exception frame is plain wrong in this case. This was fine as long as we didn't touch the interrupt state when taking page fault, but when I started doing it, it would trigger the lockdep calls and the bug. This fixes it by cleaning up that code a bit. It did create a small stack frame for the sake of backtraces, so let's make it a bit bigger and use it to save and restore the stuff we care about. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-04-10 15:21:35 +08:00
stwu r1,-32(r1)
stw r9,8(r1)
stw r11,12(r1)
stw r3,16(r1)
stw r4,20(r1)
stw r5,24(r1)
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
/* If we are disabling interrupts (normal case), simply log it with
* lockdep
*/
1: bl trace_hardirqs_off
2: lwz r5,24(r1)
powerpc: Fix page fault with lockdep regression commit a546498f3bf9aac311c66f965186373aee2ca0b0 introduced a regression on 32-bit when irq tracing is enabled by exposing an old bug in our irq tracing code for exception entry. The code would save and restore some GPRs around the calls to the C lockdep code, however, it tries to be too smart for its own good and restores some of the GPRs from the exception frame (as saved there on exception entry). However, for page faults, we do replace those GPRs with arguments to do_page_fault before we call transfer_to_handler and so restoring from the exception frame is plain wrong in this case. This was fine as long as we didn't touch the interrupt state when taking page fault, but when I started doing it, it would trigger the lockdep calls and the bug. This fixes it by cleaning up that code a bit. It did create a small stack frame for the sake of backtraces, so let's make it a bit bigger and use it to save and restore the stuff we care about. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-04-10 15:21:35 +08:00
lwz r4,20(r1)
lwz r3,16(r1)
lwz r11,12(r1)
lwz r9,8(r1)
addi r1,r1,32
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
lwz r0,GPR0(r1)
lwz r6,GPR6(r1)
lwz r7,GPR7(r1)
lwz r8,GPR8(r1)
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
mtctr r11
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
mtlr r9
bctr /* jump to handler */
#endif /* CONFIG_TRACE_IRQFLAGS */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
powerpc: change CONFIG_6xx to CONFIG_PPC_BOOK3S_32 Today we have: config PPC_BOOK3S_32 bool "512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx" [depends on PPC32 within a choice] config PPC_BOOK3S def_bool y depends on PPC_BOOK3S_32 || PPC_BOOK3S_64 config 6xx def_bool y depends on PPC32 && PPC_BOOK3S 6xx is therefore redundant with PPC_BOOK3S_32. In order to make the code clearer, lets use preferably PPC_BOOK3S_32. This will allow to remove CONFIG_6xx in a later patch. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-11-17 18:24:56 +08:00
#if defined (CONFIG_PPC_BOOK3S_32) || defined(CONFIG_E500)
powerpc: Use correct sequence for putting CPU into nap mode We weren't using the recommended sequence for putting the CPU into nap mode. When I changed the idle loop, for some reason 7447A cpus started hanging when we put them into nap mode. Changing to the recommended sequence fixes that. The complexity here is that the recommended sequence is a loop that keeps putting the cpu back into nap mode. Clearly we need some way to break out of the loop when an interrupt (external interrupt, decrementer, performance monitor) occurs. Here we use a bit in the thread_info struct to indicate that we need this, and the exception entry code notices this and arranges for the exception to return to the value in the link register, thus breaking out of the loop. We use a new `local_flags' field in the thread_info which we can alter without needing to use an atomic update sequence. The PPC970 has the same recommended sequence, so we do the same thing there too. This also fixes a bug in the kernel stack overflow handling code on 32-bit, since it was causing a value that we needed in a register to get trashed. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-04-18 19:49:11 +08:00
4: rlwinm r12,r12,0,~_TLF_NAPPING
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
stw r12,TI_LOCAL_FLAGS(r2)
powerpc/85xx: add DOZE/NAP support for e500 core The e500 core enter DOZE/NAP power-saving modes when the core go to cpu_idle routine. The power management default running mode is DOZE, If the user echo 1 > /proc/sys/kernel/powersave-nap the system will change to NAP running mode. Signed-off-by: Dave Liu <daveliu@freescale.com> Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-06-19 05:26:52 +08:00
b power_save_ppc32_restore
[POWERPC] Defer processing of interrupts when the CPU wakes from sleep mode This provides a way to defer processing of an interrupt that wakes the processor out of sleep mode. On 32-bit platforms that use an interrupt to wake the processor, we have to have interrupts enabled in hardware at the point where we go to sleep, otherwise the processor will never wake up. However, because interrupts are logically disabled at this point, we don't want to process the interrupt straight away. This is handled by setting the _TLF_SLEEPING flag. When we get an interrupt and _TLF_SLEEPING is set, we firstly clear the MSR_EE (external interrupt enable) bit in the saved MSR value, and secondly we then return to the address in the link register, like we do for _TLF_NAPPING, but without actually handling the interrupt. Note that this is handled somewhat differently on powerbooks, so this new code will only be used on non-Apple machines. Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-05-14 12:30:48 +08:00
7: rlwinm r12,r12,0,~_TLF_SLEEPING
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
stw r12,TI_LOCAL_FLAGS(r2)
[POWERPC] Defer processing of interrupts when the CPU wakes from sleep mode This provides a way to defer processing of an interrupt that wakes the processor out of sleep mode. On 32-bit platforms that use an interrupt to wake the processor, we have to have interrupts enabled in hardware at the point where we go to sleep, otherwise the processor will never wake up. However, because interrupts are logically disabled at this point, we don't want to process the interrupt straight away. This is handled by setting the _TLF_SLEEPING flag. When we get an interrupt and _TLF_SLEEPING is set, we firstly clear the MSR_EE (external interrupt enable) bit in the saved MSR value, and secondly we then return to the address in the link register, like we do for _TLF_NAPPING, but without actually handling the interrupt. Note that this is handled somewhat differently on powerbooks, so this new code will only be used on non-Apple machines. Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-05-14 12:30:48 +08:00
lwz r9,_MSR(r11) /* if sleeping, clear MSR.EE */
rlwinm r9,r9,0,~MSR_EE
lwz r12,_LINK(r11) /* and return to address in LR */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
kuap_restore r11, r2, r3, r4, r5
[POWERPC] Defer processing of interrupts when the CPU wakes from sleep mode This provides a way to defer processing of an interrupt that wakes the processor out of sleep mode. On 32-bit platforms that use an interrupt to wake the processor, we have to have interrupts enabled in hardware at the point where we go to sleep, otherwise the processor will never wake up. However, because interrupts are logically disabled at this point, we don't want to process the interrupt straight away. This is handled by setting the _TLF_SLEEPING flag. When we get an interrupt and _TLF_SLEEPING is set, we firstly clear the MSR_EE (external interrupt enable) bit in the saved MSR value, and secondly we then return to the address in the link register, like we do for _TLF_NAPPING, but without actually handling the interrupt. Note that this is handled somewhat differently on powerbooks, so this new code will only be used on non-Apple machines. Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-05-14 12:30:48 +08:00
b fast_exception_return
powerpc: Unify the 32 and 64 bit idle loops This unifies the 32-bit (ARCH=ppc and ARCH=powerpc) and 64-bit idle loops. It brings over the concept of having a ppc_md.power_save function from 32-bit to ARCH=powerpc, which lets us get rid of native_idle(). With this we will also be able to simplify the idle handling for pSeries and cell. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-27 12:03:03 +08:00
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* On kernel stack overflow, load up an initial stack pointer
* and call StackOverflow(regs), which should not return.
*/
stack_ovf:
/* sometimes we use a statically-allocated stack, which is OK. */
powerpc: Use correct sequence for putting CPU into nap mode We weren't using the recommended sequence for putting the CPU into nap mode. When I changed the idle loop, for some reason 7447A cpus started hanging when we put them into nap mode. Changing to the recommended sequence fixes that. The complexity here is that the recommended sequence is a loop that keeps putting the cpu back into nap mode. Clearly we need some way to break out of the loop when an interrupt (external interrupt, decrementer, performance monitor) occurs. Here we use a bit in the thread_info struct to indicate that we need this, and the exception entry code notices this and arranges for the exception to return to the value in the link register, thus breaking out of the loop. We use a new `local_flags' field in the thread_info which we can alter without needing to use an atomic update sequence. The PPC970 has the same recommended sequence, so we do the same thing there too. This also fixes a bug in the kernel stack overflow handling code on 32-bit, since it was causing a value that we needed in a register to get trashed. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-04-18 19:49:11 +08:00
lis r12,_end@h
ori r12,r12,_end@l
cmplw r1,r12
ble 5b /* r1 <= &_end is OK */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
SAVE_NVGPRS(r11)
addi r3,r1,STACK_FRAME_OVERHEAD
lis r1,init_thread_union@ha
addi r1,r1,init_thread_union@l
addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
lis r9,StackOverflow@ha
addi r9,r9,StackOverflow@l
LOAD_MSR_KERNEL(r10,MSR_KERNEL)
powerpc/8xx: Only perform perf counting when perf is in use. In TLB miss handlers, updating the perf counter is only useful when performing a perf analysis. As it has a noticeable overhead, let's only do it when needed. In order to do so, the exit of the miss handlers will be patched when starting/stopping 'perf': the first register restore instruction of each exit point will be replaced by a jump to the counting code. Once this is done, CONFIG_PPC_8xx_PERF_EVENT becomes useless as this feature doesn't add any overhead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-01-12 20:45:23 +08:00
#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
powerpc/8xx: Perf events on PPC 8xx This patch has been reworked since RFC version. In the RFC, this patch was preceded by a patch clearing MSR RI for all PPC32 at all time at exception prologs. Now MSR RI clearing is done only when this 8xx perf events functionality is compiled in, it is therefore limited to 8xx and merged inside this patch. Other main changes have been to take into account detailed review from Peter Zijlstra. The instructions counter has been reworked to behave as a free running counter like the three other counters. The 8xx has no PMU, however some events can be emulated by other means. This patch implements the following events (as reported by 'perf list'): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] dTLB-load-misses [Hardware cache event] iTLB-load-misses [Hardware cache event] 'cycles' event is implemented using the timebase clock. Timebase clock corresponds to CPU clock divided by 16, so number of cycles is approximatly 16 times the number of TB ticks On the 8xx, TLB misses are handled by software. It is therefore easy to count all TLB misses each time the TLB miss exception is called. 'instructions' is calculated by using instruction watchpoint counter. This patch sets counter A to count instructions at address greater than 0, hence we count all instructions executed while MSR RI bit is set. The counter is set to the maximum which is 0xffff. Every 65535 instructions, debug instruction breakpoint exception fires. The exception handler increments a counter in memory which then represent the upper part of the instruction counter. We therefore end up with a 48 bits counter. In order to avoid unnecessary overhead while no perf event is active, this counter is started when the first event referring to this counter is added, and the counter is stopped when the last event referring to it is deleted. In order to properly support breakpoint exceptions, MSR RI bit has to be unset in exception epilogs in order to avoid breakpoint exceptions during critical sections during changes to SRR0 and SRR1 would be problematic. All counters are handled as free running counters. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Scott Wood <oss@buserror.net>
2016-12-15 20:42:18 +08:00
mtspr SPRN_NRI, r0
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mtspr SPRN_SRR0,r9
mtspr SPRN_SRR1,r10
SYNC
RFI
powerpc/32: implement fast entry for syscalls on non BOOKE This patch implements a fast entry for syscalls. Syscalls don't have to preserve non volatile registers except LR. This patch then implement a fast entry for syscalls, where volatile registers get clobbered. As this entry is dedicated to syscall it always sets MSR_EE and warns in case MSR_EE was previously off It also assumes that the call is always from user, system calls are unexpected from kernel. The overall series improves null_syscall selftest by 12,5% on an 83xx and by 17% on a 8xx. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:02 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
trace_syscall_entry_irq_off:
/*
* Syscall shouldn't happen while interrupts are disabled,
* so let's do a warning here.
*/
0: trap
EMIT_BUG_ENTRY 0b,__FILE__,__LINE__, BUGFLAG_WARNING
bl trace_hardirqs_on
/* Now enable for real */
LOAD_MSR_KERNEL(r10, MSR_KERNEL | MSR_EE)
mtmsr r10
REST_GPR(0, r1)
REST_4GPRS(3, r1)
REST_2GPRS(7, r1)
b DoSyscall
#endif /* CONFIG_TRACE_IRQFLAGS */
.globl transfer_to_syscall
transfer_to_syscall:
#ifdef CONFIG_TRACE_IRQFLAGS
andi. r12,r9,MSR_EE
beq- trace_syscall_entry_irq_off
#endif /* CONFIG_TRACE_IRQFLAGS */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* Handle a system call.
*/
.stabs "arch/powerpc/kernel/",N_SO,0,0,0f
.stabs "entry_32.S",N_SO,0,0,0f
0:
_GLOBAL(DoSyscall)
stw r3,ORIG_GPR3(r1)
li r12,0
stw r12,RESULT(r1)
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
/* Make sure interrupts are enabled */
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
mfmsr r11
andi. r12,r11,MSR_EE
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
/* We came in with interrupts disabled, we WARN and mark them enabled
* for lockdep now */
0: tweqi r12, 0
EMIT_BUG_ENTRY 0b,__FILE__,__LINE__, BUGFLAG_WARNING
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
#endif /* CONFIG_TRACE_IRQFLAGS */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r11,TI_FLAGS(r2)
powerpc: Rename _TIF_SYSCALL_T_OR_A to _TIF_SYSCALL_DOTRACE Once upon a time, at least 9 years ago (< 2.6.12), _TIF_SYSCALL_T_OR_A meant "TRACE or AUDIT". But these days it means TRACE or AUDIT or SECCOMP or TRACEPOINT or NOHZ. All of those are implemented via syscall_dotrace() so rename the flag to that to try and clarify things. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-01-15 09:01:42 +08:00
andi. r11,r11,_TIF_SYSCALL_DOTRACE
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
bne- syscall_dotrace
syscall_dotrace_cont:
cmplwi 0,r0,NR_syscalls
lis r10,sys_call_table@h
ori r10,r10,sys_call_table@l
slwi r0,r0,2
bge- 66f
powerpc/fsl: Sanitize the syscall table for NXP PowerPC 32 bit platforms Used barrier_nospec to sanitize the syscall table. Signed-off-by: Diana Craciun <diana.craciun@nxp.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-07-28 07:06:38 +08:00
barrier_nospec_asm
/*
* Prevent the load of the handler below (based on the user-passed
* system call number) being speculatively executed until the test
* against NR_syscalls and branch to .66f above has
* committed.
*/
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwzx r10,r10,r0 /* Fetch system call handler [ptr] */
mtlr r10
addi r9,r1,STACK_FRAME_OVERHEAD
PPC440EP_ERR42
blrl /* Call handler */
.globl ret_from_syscall
ret_from_syscall:
powerpc: Add syscall detection for restartable sequences Syscalls are not allowed inside restartable sequences, so add a call to rseq_syscall() at the very beginning of system call exiting path for CONFIG_DEBUG_RSEQ=y kernel. This could help us to detect whether there is a syscall issued inside restartable sequences. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Joel Fernandes <joelaf@google.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Dave Watson <davejwatson@fb.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Andi Kleen <andi@firstfloor.org> Cc: Paul Mackerras <paulus@samba.org> Cc: "H . Peter Anvin" <hpa@zytor.com> Cc: Chris Lameter <cl@linux.com> Cc: Russell King <linux@arm.linux.org.uk> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Andrew Hunter <ahh@google.com> Cc: Michael Kerrisk <mtk.manpages@gmail.com> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: "Paul E . McKenney" <paulmck@linux.vnet.ibm.com> Cc: Paul Turner <pjt@google.com> Cc: Josh Triplett <josh@joshtriplett.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Ben Maurer <bmaurer@fb.com> Cc: linux-api@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: Andy Lutomirski <luto@amacapital.net> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Link: https://lkml.kernel.org/r/20180602124408.8430-10-mathieu.desnoyers@efficios.com
2018-06-02 20:44:01 +08:00
#ifdef CONFIG_DEBUG_RSEQ
/* Check whether the syscall is issued inside a restartable sequence */
stw r3,GPR3(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
bl rseq_syscall
lwz r3,GPR3(r1)
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mr r6,r3
/* disable interrupts so current_thread_info()->flags can't change */
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
LOAD_MSR_KERNEL(r10,MSR_KERNEL) /* doesn't include MSR_EE */
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
/* Note: We don't bother telling lockdep about it */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
SYNC
MTMSRD(r10)
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r9,TI_FLAGS(r2)
powerpc/kernel: Switch to using MAX_ERRNO Currently on powerpc we have our own #define for the highest (negative) errno value, called _LAST_ERRNO. This is defined to be 516, for reasons which are not clear. The generic code, and x86, use MAX_ERRNO, which is defined to be 4095. In particular seccomp uses MAX_ERRNO to restrict the value that a seccomp filter can return. Currently with the mismatch between _LAST_ERRNO and MAX_ERRNO, a seccomp tracer wanting to return 600, expecting it to be seen as an error, would instead find on powerpc that userspace sees a successful syscall with a return value of 600. To avoid this inconsistency, switch powerpc to use MAX_ERRNO. We are somewhat confident that generic syscalls that can return a non-error value above negative MAX_ERRNO have already been updated to use force_successful_syscall_return(). I have also checked all the powerpc specific syscalls, and believe that none of them expect to return a non-error value between -MAX_ERRNO and -516. So this change should be safe ... Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Reviewed-by: Kees Cook <keescook@chromium.org>
2015-07-23 18:21:01 +08:00
li r8,-MAX_ERRNO
powerpc: Rename _TIF_SYSCALL_T_OR_A to _TIF_SYSCALL_DOTRACE Once upon a time, at least 9 years ago (< 2.6.12), _TIF_SYSCALL_T_OR_A meant "TRACE or AUDIT". But these days it means TRACE or AUDIT or SECCOMP or TRACEPOINT or NOHZ. All of those are implemented via syscall_dotrace() so rename the flag to that to try and clarify things. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-01-15 09:01:42 +08:00
andi. r0,r9,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP|_TIF_USER_WORK_MASK|_TIF_PERSYSCALL_MASK)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
bne- syscall_exit_work
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
cmplw 0,r3,r8
blt+ syscall_exit_cont
lwz r11,_CCR(r1) /* Load CR */
neg r3,r3
oris r11,r11,0x1000 /* Set SO bit in CR */
stw r11,_CCR(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
syscall_exit_cont:
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
lwz r8,_MSR(r1)
#ifdef CONFIG_TRACE_IRQFLAGS
/* If we are going to return from the syscall with interrupts
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
* off, we trace that here. It shouldn't normally happen.
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
*/
andi. r10,r8,MSR_EE
bne+ 1f
stw r3,GPR3(r1)
bl trace_hardirqs_off
lwz r3,GPR3(r1)
1:
#endif /* CONFIG_TRACE_IRQFLAGS */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
/* If the process has its own DBCR0 value, load it up. The internal
debug mode bit tells us that dbcr0 should be loaded. */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r0,THREAD+THREAD_DBCR0(r2)
powerpc/booke: Clean up the hardware watchpoint support * CONFIG_BOOKE is selected by CONFIG_44x so we dont need both * Fixed a few comments * Go back to only using DBCR0_IDM to determine if we are using debug resources. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-07-26 03:27:33 +08:00
andis. r10,r0,DBCR0_IDM@h
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
bnel- load_dbcr0
#endif
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
#ifdef CONFIG_44x
powerpc/47x: Base ppc476 support This patch adds the base support for the 476 processor. The code was primarily written by Ben Herrenschmidt and Torez Smith, but I've been maintaining it for a while. The goal is to have a single binary that will run on 44x and 47x, but we still have some details to work out. The biggest is that the L1 cache line size differs on the two platforms, but it's currently a compile-time option. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Torez Smith <lnxtorez@linux.vnet.ibm.com> Signed-off-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2010-03-05 18:43:12 +08:00
BEGIN_MMU_FTR_SECTION
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
lis r4,icache_44x_need_flush@ha
lwz r5,icache_44x_need_flush@l(r4)
cmplwi cr0,r5,0
bne- 2f
1:
powerpc/47x: Base ppc476 support This patch adds the base support for the 476 processor. The code was primarily written by Ben Herrenschmidt and Torez Smith, but I've been maintaining it for a while. The goal is to have a single binary that will run on 44x and 47x, but we still have some details to work out. The biggest is that the L1 cache line size differs on the two platforms, but it's currently a compile-time option. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Torez Smith <lnxtorez@linux.vnet.ibm.com> Signed-off-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2010-03-05 18:43:12 +08:00
END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_47x)
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
#endif /* CONFIG_44x */
[POWERPC] Avoid unpaired stwcx. on some processors The context switch code in the kernel issues a dummy stwcx. to clear the reservation, as recommended by the architecture. However, some processors can have issues if this stwcx to address A occurs while the reservation is already held to a different address B. To avoid this problem, the dummy stwcx. needs to be paired with a dummy lwarx to the same address. This adds the dummy lwarx, and creates a cpu feature bit to indicate which cpus are affected. Tested on mpc8641_hpcn_defconfig in arch/powerpc; build tested in arch/ppc. Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-11-10 06:17:49 +08:00
BEGIN_FTR_SECTION
lwarx r7,0,r1
END_FTR_SECTION_IFSET(CPU_FTR_NEED_PAIRED_STWCX)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
stwcx. r0,0,r1 /* to clear the reservation */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
ACCOUNT_CPU_USER_EXIT(r2, r5, r7)
powerpc/32s: Implement Kernel Userspace Execution Prevention. To implement Kernel Userspace Execution Prevention, this patch sets NX bit on all user segments on kernel entry and clears NX bit on all user segments on kernel exit. Note that powerpc 601 doesn't have the NX bit, so KUEP will not work on it. A warning is displayed at startup. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:35 +08:00
#ifdef CONFIG_PPC_BOOK3S_32
kuep_unlock r5, r7
#endif
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
kuap_check r2, r4
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r4,_LINK(r1)
lwz r5,_CCR(r1)
mtlr r4
mtcr r5
lwz r7,_NIP(r1)
lwz r2,GPR2(r1)
lwz r1,GPR1(r1)
powerpc/8xx: Only perform perf counting when perf is in use. In TLB miss handlers, updating the perf counter is only useful when performing a perf analysis. As it has a noticeable overhead, let's only do it when needed. In order to do so, the exit of the miss handlers will be patched when starting/stopping 'perf': the first register restore instruction of each exit point will be replaced by a jump to the counting code. Once this is done, CONFIG_PPC_8xx_PERF_EVENT becomes useless as this feature doesn't add any overhead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-01-12 20:45:23 +08:00
#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
powerpc/8xx: Perf events on PPC 8xx This patch has been reworked since RFC version. In the RFC, this patch was preceded by a patch clearing MSR RI for all PPC32 at all time at exception prologs. Now MSR RI clearing is done only when this 8xx perf events functionality is compiled in, it is therefore limited to 8xx and merged inside this patch. Other main changes have been to take into account detailed review from Peter Zijlstra. The instructions counter has been reworked to behave as a free running counter like the three other counters. The 8xx has no PMU, however some events can be emulated by other means. This patch implements the following events (as reported by 'perf list'): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] dTLB-load-misses [Hardware cache event] iTLB-load-misses [Hardware cache event] 'cycles' event is implemented using the timebase clock. Timebase clock corresponds to CPU clock divided by 16, so number of cycles is approximatly 16 times the number of TB ticks On the 8xx, TLB misses are handled by software. It is therefore easy to count all TLB misses each time the TLB miss exception is called. 'instructions' is calculated by using instruction watchpoint counter. This patch sets counter A to count instructions at address greater than 0, hence we count all instructions executed while MSR RI bit is set. The counter is set to the maximum which is 0xffff. Every 65535 instructions, debug instruction breakpoint exception fires. The exception handler increments a counter in memory which then represent the upper part of the instruction counter. We therefore end up with a 48 bits counter. In order to avoid unnecessary overhead while no perf event is active, this counter is started when the first event referring to this counter is added, and the counter is stopped when the last event referring to it is deleted. In order to properly support breakpoint exceptions, MSR RI bit has to be unset in exception epilogs in order to avoid breakpoint exceptions during critical sections during changes to SRR0 and SRR1 would be problematic. All counters are handled as free running counters. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Scott Wood <oss@buserror.net>
2016-12-15 20:42:18 +08:00
mtspr SPRN_NRI, r0
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mtspr SPRN_SRR0,r7
mtspr SPRN_SRR1,r8
SYNC
RFI
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
#ifdef CONFIG_44x
2: li r7,0
iccci r0,r0
stw r7,icache_44x_need_flush@l(r4)
b 1b
#endif /* CONFIG_44x */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
66: li r3,-ENOSYS
b ret_from_syscall
.globl ret_from_fork
ret_from_fork:
REST_NVGPRS(r1)
bl schedule_tail
li r3,0
b ret_from_syscall
powerpc: split ret_from_fork ... and get rid of in-kernel syscalls in kernel_thread() Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-09-13 06:32:42 +08:00
.globl ret_from_kernel_thread
ret_from_kernel_thread:
REST_NVGPRS(r1)
bl schedule_tail
mtlr r14
mr r3,r15
PPC440EP_ERR42
blrl
li r3,0
powerpc: switch to generic sys_execve()/kernel_execve() the only non-obvious part is that current_pt_regs() is really needed here - task_pt_regs() is NULL for kernel threads; it's OK for ptrace uses (the thing task_pt_regs() is intended for), but not for us. Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-09-01 03:48:05 +08:00
b ret_from_syscall
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* Traced system call support */
syscall_dotrace:
SAVE_NVGPRS(r1)
li r0,0xc00
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
addi r3,r1,STACK_FRAME_OVERHEAD
bl do_syscall_trace_enter
powerpc: Make syscall tracing use tracehook.h helpers This changes powerpc syscall tracing to use the new tracehook.h entry points. There is no change, only cleanup. In addition, the assembly changes allow do_syscall_trace_enter() to abort the syscall without losing the information about the original r0 value. Signed-off-by: Roland McGrath <roland@redhat.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-07-27 14:51:03 +08:00
/*
* Restore argument registers possibly just changed.
* We use the return value of do_syscall_trace_enter
* for call number to look up in the table (r0).
*/
mr r0,r3
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r3,GPR3(r1)
lwz r4,GPR4(r1)
lwz r5,GPR5(r1)
lwz r6,GPR6(r1)
lwz r7,GPR7(r1)
lwz r8,GPR8(r1)
REST_NVGPRS(r1)
powerpc/kernel: Change the do_syscall_trace_enter() API The API for calling do_syscall_trace_enter() is currently sensible enough, it just returns the (modified) syscall number. However once we enable seccomp filter it will get more complicated. When seccomp filter runs, the seccomp kernel code (via SECCOMP_RET_ERRNO), or a ptracer (via SECCOMP_RET_TRACE), may reject the syscall and *may* or may *not* set a return value in r3. That means the assembler that calls do_syscall_trace_enter() can not blindly return ENOSYS, it needs to only return ENOSYS if a return value has not already been set. There is no way to implement that logic with the current API. So change the do_syscall_trace_enter() API to make it deal with the return code juggling, and the assembler can then just return whatever return code it is given. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Reviewed-by: Kees Cook <keescook@chromium.org>
2015-07-23 18:21:02 +08:00
cmplwi r0,NR_syscalls
/* Return code is already in r3 thanks to do_syscall_trace_enter() */
bge- ret_from_syscall
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b syscall_dotrace_cont
syscall_exit_work:
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
andi. r0,r9,_TIF_RESTOREALL
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
beq+ 0f
REST_NVGPRS(r1)
b 2f
0: cmplw 0,r3,r8
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
blt+ 1f
andi. r0,r9,_TIF_NOERROR
bne- 1f
lwz r11,_CCR(r1) /* Load CR */
neg r3,r3
oris r11,r11,0x1000 /* Set SO bit in CR */
stw r11,_CCR(r1)
1: stw r6,RESULT(r1) /* Save result */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
stw r3,GPR3(r1) /* Update return value */
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
2: andi. r0,r9,(_TIF_PERSYSCALL_MASK)
beq 4f
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
/* Clear per-syscall TIF flags if any are set. */
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
li r11,_TIF_PERSYSCALL_MASK
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
addi r12,r2,TI_FLAGS
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
3: lwarx r8,0,r12
andc r8,r8,r11
#ifdef CONFIG_IBM405_ERR77
dcbt 0,r12
#endif
stwcx. r8,0,r12
bne- 3b
4: /* Anything which requires enabling interrupts? */
powerpc: Rename _TIF_SYSCALL_T_OR_A to _TIF_SYSCALL_DOTRACE Once upon a time, at least 9 years ago (< 2.6.12), _TIF_SYSCALL_T_OR_A meant "TRACE or AUDIT". But these days it means TRACE or AUDIT or SECCOMP or TRACEPOINT or NOHZ. All of those are implemented via syscall_dotrace() so rename the flag to that to try and clarify things. Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-01-15 09:01:42 +08:00
andi. r0,r9,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP)
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
beq ret_from_except
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
/* Re-enable interrupts. There is no need to trace that with
* lockdep as we are supposed to have IRQs on at this point
*/
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
ori r10,r10,MSR_EE
SYNC
MTMSRD(r10)
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
/* Save NVGPRS if they're not saved already */
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r4,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
andi. r4,r4,1
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
beq 5f
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
SAVE_NVGPRS(r1)
li r4,0xc00
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r4,_TRAP(r1)
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
5:
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
addi r3,r1,STACK_FRAME_OVERHEAD
bl do_syscall_trace_leave
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
b ret_from_except_full
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
[PATCH] syscall entry/exit revamp This cleanup patch speeds up the null syscall path on ppc64 by about 3%, and brings the ppc32 and ppc64 code slightly closer together. The ppc64 code was checking current_thread_info()->flags twice in the syscall exit path; once for TIF_SYSCALL_T_OR_A before disabling interrupts, and then again for TIF_SIGPENDING|TIF_NEED_RESCHED etc after disabling interrupts. Now we do the same as ppc32 -- check the flags only once in the fast path, and re-enable interrupts if necessary in the ptrace case. The patch abolishes the 'syscall_noerror' member of struct thread_info and replaces it with a TIF_NOERROR bit in the flags, which is handled in the slow path. This shortens the syscall entry code, which no longer needs to clear syscall_noerror. The patch adds a TIF_SAVE_NVGPRS flag which causes the syscall exit slow path to save the non-volatile GPRs into a signal frame. This removes the need for the assembly wrappers around sys_sigsuspend(), sys_rt_sigsuspend(), et al which existed solely to save those registers in advance. It also means I don't have to add new wrappers for ppoll() and pselect(), which is what I was supposed to be doing when I got distracted into this... Finally, it unifies the ppc64 and ppc32 methods of handling syscall exit directly into a signal handler (as required by sigsuspend et al) by introducing a TIF_RESTOREALL flag which causes _all_ the registers to be reloaded from the pt_regs by taking the ret_from_exception path, instead of the normal syscall exit path which stomps on the callee-saved GPRs. It appears to pass an LTP test run on ppc64, and passes basic testing on ppc32 too. Brief tests of ptrace functionality with strace and gdb also appear OK. I wouldn't send it to Linus for 2.6.15 just yet though :) Signed-off-by: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16 02:52:18 +08:00
* The fork/clone functions need to copy the full register set into
* the child process. Therefore we need to save all the nonvolatile
* registers (r13 - r31) before calling the C code.
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
*/
.globl ppc_fork
ppc_fork:
SAVE_NVGPRS(r1)
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r0,_TRAP(r1) /* register set saved */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b sys_fork
.globl ppc_vfork
ppc_vfork:
SAVE_NVGPRS(r1)
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r0,_TRAP(r1) /* register set saved */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b sys_vfork
.globl ppc_clone
ppc_clone:
SAVE_NVGPRS(r1)
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r0,_TRAP(r1) /* register set saved */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b sys_clone
powerpc: Fix various syscall/signal/swapcontext bugs A careful reading of the recent changes to the system call entry/exit paths revealed several problems, plus some things that could be simplified and improved: * 32-bit wasn't testing the _TIF_NOERROR bit in the syscall fast exit path, so it was only doing anything with it once it saw some other bit being set. In other words, the noerror behaviour would apply to the next system call where we had to reschedule or deliver a signal, which is not necessarily the current system call. * 32-bit wasn't doing the call to ptrace_notify in the syscall exit path when the _TIF_SINGLESTEP bit was set. * _TIF_RESTOREALL was in both _TIF_USER_WORK_MASK and _TIF_PERSYSCALL_MASK, which is odd since _TIF_RESTOREALL is only set by system calls. I took it out of _TIF_USER_WORK_MASK. * On 64-bit, _TIF_RESTOREALL wasn't causing the non-volatile registers to be restored (unless perhaps a signal was delivered or the syscall was traced or single-stepped). Thus the non-volatile registers weren't restored on exit from a signal handler. We probably got away with it mostly because signal handlers written in C wouldn't alter the non-volatile registers. * On 32-bit I simplified the code and made it more like 64-bit by making the syscall exit path jump to ret_from_except to handle preemption and signal delivery. * 32-bit was calling do_signal unnecessarily when _TIF_RESTOREALL was set - but I think because of that 32-bit was actually restoring the non-volatile registers on exit from a signal handler. * I changed the order of enabling interrupts and saving the non-volatile registers before calling do_syscall_trace_leave; now we enable interrupts first. Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-03-08 10:24:22 +08:00
.globl ppc_swapcontext
ppc_swapcontext:
SAVE_NVGPRS(r1)
lwz r0,_TRAP(r1)
rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
stw r0,_TRAP(r1) /* register set saved */
b sys_swapcontext
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* Top-level page fault handling.
* This is in assembler because if do_page_fault tells us that
* it is a bad kernel page fault, we want to save the non-volatile
* registers before calling bad_page_fault.
*/
.globl handle_page_fault
handle_page_fault:
stw r4,_DAR(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
powerpc: change CONFIG_6xx to CONFIG_PPC_BOOK3S_32 Today we have: config PPC_BOOK3S_32 bool "512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx" [depends on PPC32 within a choice] config PPC_BOOK3S def_bool y depends on PPC_BOOK3S_32 || PPC_BOOK3S_64 config 6xx def_bool y depends on PPC32 && PPC_BOOK3S 6xx is therefore redundant with PPC_BOOK3S_32. In order to make the code clearer, lets use preferably PPC_BOOK3S_32. This will allow to remove CONFIG_6xx in a later patch. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-11-17 18:24:56 +08:00
#ifdef CONFIG_PPC_BOOK3S_32
powerpc/32: Fix boot failure on non 6xx platforms Commit d300627c6a536 ("powerpc/6xx: Handle DABR match before calling do_page_fault") breaks non 6xx platforms. Failed to execute /init (error -14) Starting init: /bin/sh exists but couldn't execute it (error -14) Kernel panic - not syncing: No working init found. Try passing init= ... CPU: 0 PID: 1 Comm: init Not tainted 4.13.0-rc3-s3k-dev-00143-g7aa62e972a56 #56 Call Trace: panic+0x108/0x250 (unreliable) rootfs_mount+0x0/0x58 ret_from_kernel_thread+0x5c/0x64 Rebooting in 180 seconds.. This is because in handle_page_fault(), the call to do_page_fault() has been mistakenly enclosed inside an #ifdef CONFIG_6xx Fixes: d300627c6a536 ("powerpc/6xx: Handle DABR match before calling do_page_fault") Brown-paper-bag-to-be-worn-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-08-08 14:37:24 +08:00
andis. r0,r5,DSISR_DABRMATCH@h
powerpc/6xx: Handle DABR match before calling do_page_fault On legacy 6xx 32-bit procesors, we checked for the DABR match bit in DSISR from do_page_fault(), in the middle of a pile of ifdef's because all other CPU types do it in assembly prior to calling do_page_fault. Fix that. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> [mpe: Add #ifdef CONFIG_6xx] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-07-19 12:49:25 +08:00
bne- handle_dabr_fault
#endif
powerpc/32: Fix boot failure on non 6xx platforms Commit d300627c6a536 ("powerpc/6xx: Handle DABR match before calling do_page_fault") breaks non 6xx platforms. Failed to execute /init (error -14) Starting init: /bin/sh exists but couldn't execute it (error -14) Kernel panic - not syncing: No working init found. Try passing init= ... CPU: 0 PID: 1 Comm: init Not tainted 4.13.0-rc3-s3k-dev-00143-g7aa62e972a56 #56 Call Trace: panic+0x108/0x250 (unreliable) rootfs_mount+0x0/0x58 ret_from_kernel_thread+0x5c/0x64 Rebooting in 180 seconds.. This is because in handle_page_fault(), the call to do_page_fault() has been mistakenly enclosed inside an #ifdef CONFIG_6xx Fixes: d300627c6a536 ("powerpc/6xx: Handle DABR match before calling do_page_fault") Brown-paper-bag-to-be-worn-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-08-08 14:37:24 +08:00
bl do_page_fault
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
cmpwi r3,0
beq+ ret_from_except
SAVE_NVGPRS(r1)
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
clrrwi r0,r0,1
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r0,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mr r5,r3
addi r3,r1,STACK_FRAME_OVERHEAD
lwz r4,_DAR(r1)
bl bad_page_fault
b ret_from_except_full
powerpc: change CONFIG_6xx to CONFIG_PPC_BOOK3S_32 Today we have: config PPC_BOOK3S_32 bool "512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx" [depends on PPC32 within a choice] config PPC_BOOK3S def_bool y depends on PPC_BOOK3S_32 || PPC_BOOK3S_64 config 6xx def_bool y depends on PPC32 && PPC_BOOK3S 6xx is therefore redundant with PPC_BOOK3S_32. In order to make the code clearer, lets use preferably PPC_BOOK3S_32. This will allow to remove CONFIG_6xx in a later patch. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-11-17 18:24:56 +08:00
#ifdef CONFIG_PPC_BOOK3S_32
powerpc/6xx: Handle DABR match before calling do_page_fault On legacy 6xx 32-bit procesors, we checked for the DABR match bit in DSISR from do_page_fault(), in the middle of a pile of ifdef's because all other CPU types do it in assembly prior to calling do_page_fault. Fix that. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> [mpe: Add #ifdef CONFIG_6xx] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-07-19 12:49:25 +08:00
/* We have a data breakpoint exception - handle it */
handle_dabr_fault:
SAVE_NVGPRS(r1)
lwz r0,_TRAP(r1)
clrrwi r0,r0,1
stw r0,_TRAP(r1)
bl do_break
b ret_from_except_full
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* This routine switches between two different tasks. The process
* state of one is saved on its kernel stack. Then the state
* of the other is restored from its kernel stack. The memory
* management hardware is updated to the second process's state.
* Finally, we can return to the second process.
* On entry, r3 points to the THREAD for the current task, r4
* points to the THREAD for the new task.
*
* This routine is always called with interrupts disabled.
*
* Note: there are two ways to get to the "going out" portion
* of this code; either by coming in via the entry (_switch)
* or via "fork" which must set up an environment equivalent
* to the "_switch" path. If you change this , you'll have to
* change the fork code also.
*
* The code which creates the new task context is in 'copy_thread'
* in arch/ppc/kernel/process.c
*/
_GLOBAL(_switch)
stwu r1,-INT_FRAME_SIZE(r1)
mflr r0
stw r0,INT_FRAME_SIZE+4(r1)
/* r3-r12 are caller saved -- Cort */
SAVE_NVGPRS(r1)
stw r0,_NIP(r1) /* Return to switch caller */
mfmsr r11
li r0,MSR_FP /* Disable floating-point */
#ifdef CONFIG_ALTIVEC
BEGIN_FTR_SECTION
oris r0,r0,MSR_VEC@h /* Disable altivec */
mfspr r12,SPRN_VRSAVE /* save vrsave register value */
stw r12,THREAD+THREAD_VRSAVE(r2)
END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
#endif /* CONFIG_ALTIVEC */
#ifdef CONFIG_SPE
[POWERPC] Add cpu feature for SPE handling Make it so that SPE support can be determined at runtime. This is similiar to how we handle AltiVec. This allows us to have SPE support built in and work on processors with and without SPE. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-09-13 14:44:20 +08:00
BEGIN_FTR_SECTION
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
oris r0,r0,MSR_SPE@h /* Disable SPE */
mfspr r12,SPRN_SPEFSCR /* save spefscr register value */
stw r12,THREAD+THREAD_SPEFSCR(r2)
[POWERPC] Add cpu feature for SPE handling Make it so that SPE support can be determined at runtime. This is similiar to how we handle AltiVec. This allows us to have SPE support built in and work on processors with and without SPE. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-09-13 14:44:20 +08:00
END_FTR_SECTION_IFSET(CPU_FTR_SPE)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#endif /* CONFIG_SPE */
and. r0,r0,r11 /* FP or altivec or SPE enabled? */
beq+ 1f
andc r11,r11,r0
MTMSRD(r11)
isync
1: stw r11,_MSR(r1)
mfcr r10
stw r10,_CCR(r1)
stw r1,KSP(r3) /* Set old stack pointer */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
kuap_check r2, r4
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#ifdef CONFIG_SMP
/* We need a sync somewhere here to make sure that if the
* previous task gets rescheduled on another CPU, it sees all
* stores it has performed on this one.
*/
sync
#endif /* CONFIG_SMP */
tophys(r0,r4)
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mtspr SPRN_SPRG_THREAD,r0 /* Update current THREAD phys addr */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r1,KSP(r4) /* Load new stack pointer */
/* save the old current 'last' for return value */
mr r3,r2
addi r2,r4,-THREAD /* Update current */
#ifdef CONFIG_ALTIVEC
BEGIN_FTR_SECTION
lwz r0,THREAD+THREAD_VRSAVE(r2)
mtspr SPRN_VRSAVE,r0 /* if G4, restore VRSAVE reg */
END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
#endif /* CONFIG_ALTIVEC */
#ifdef CONFIG_SPE
[POWERPC] Add cpu feature for SPE handling Make it so that SPE support can be determined at runtime. This is similiar to how we handle AltiVec. This allows us to have SPE support built in and work on processors with and without SPE. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-09-13 14:44:20 +08:00
BEGIN_FTR_SECTION
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r0,THREAD+THREAD_SPEFSCR(r2)
mtspr SPRN_SPEFSCR,r0 /* restore SPEFSCR reg */
[POWERPC] Add cpu feature for SPE handling Make it so that SPE support can be determined at runtime. This is similiar to how we handle AltiVec. This allows us to have SPE support built in and work on processors with and without SPE. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-09-13 14:44:20 +08:00
END_FTR_SECTION_IFSET(CPU_FTR_SPE)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#endif /* CONFIG_SPE */
powerpc: Revert the initial stack protector support Unfortunately the stack protector support we merged recently only works on some toolchains. If the toolchain is built without glibc support everything works fine, but if glibc is built then it leads to a panic at boot. The solution is not rc5 material, so revert the support for now. This reverts commits: 6533b7c16ee5 ("powerpc: Initial stack protector (-fstack-protector) support") 902e06eb86cd ("powerpc/32: Change the stack protector canary value per task") Fixes: 6533b7c16ee5 ("powerpc: Initial stack protector (-fstack-protector) support") Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2017-01-24 18:37:20 +08:00
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r0,_CCR(r1)
mtcrf 0xFF,r0
/* r3-r12 are destroyed -- Cort */
REST_NVGPRS(r1)
lwz r4,_NIP(r1) /* Return to _switch caller in new task */
mtlr r4
addi r1,r1,INT_FRAME_SIZE
blr
.globl fast_exception_return
fast_exception_return:
#if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE))
andi. r10,r9,MSR_RI /* check for recoverable interrupt */
beq 1f /* if not, we've got problems */
#endif
2: REST_4GPRS(3, r11)
lwz r10,_CCR(r11)
REST_GPR(1, r11)
mtcr r10
lwz r10,_LINK(r11)
mtlr r10
powerpc/32: Clear on-stack exception marker upon exception return Clear the on-stack STACK_FRAME_REGS_MARKER on exception exit in order to avoid confusing stacktrace like the one below. Call Trace: [c0e9dca0] [c01c42a0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c4684] kasan_report+0xfc/0x180 [c0e9dd10] [c0895130] memchr+0x24/0x74 [c0e9dd30] [c00a9e38] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab710] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc60] vprintk_emit+0x188/0x1c4 --- interrupt: c0e9df00 at 0x400f330 LR = init_stack+0x1f00/0x2000 [c0e9de80] [c00ae3c4] printk+0xa8/0xcc (unreliable) [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 With this patch the trace becomes: Call Trace: [c0e9dca0] [c01c42c0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c46a4] kasan_report+0xfc/0x180 [c0e9dd10] [c0895150] memchr+0x24/0x74 [c0e9dd30] [c00a9e58] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab730] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc80] vprintk_emit+0x188/0x1c4 [c0e9de80] [c00ae3e4] printk+0xa8/0xcc [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 Cc: stable@vger.kernel.org Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-02-27 19:45:30 +08:00
/* Clear the exception_marker on the stack to avoid confusing stacktrace */
li r10, 0
stw r10, 8(r11)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
REST_GPR(10, r11)
powerpc/8xx: Only perform perf counting when perf is in use. In TLB miss handlers, updating the perf counter is only useful when performing a perf analysis. As it has a noticeable overhead, let's only do it when needed. In order to do so, the exit of the miss handlers will be patched when starting/stopping 'perf': the first register restore instruction of each exit point will be replaced by a jump to the counting code. Once this is done, CONFIG_PPC_8xx_PERF_EVENT becomes useless as this feature doesn't add any overhead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-01-12 20:45:23 +08:00
#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
powerpc/8xx: Perf events on PPC 8xx This patch has been reworked since RFC version. In the RFC, this patch was preceded by a patch clearing MSR RI for all PPC32 at all time at exception prologs. Now MSR RI clearing is done only when this 8xx perf events functionality is compiled in, it is therefore limited to 8xx and merged inside this patch. Other main changes have been to take into account detailed review from Peter Zijlstra. The instructions counter has been reworked to behave as a free running counter like the three other counters. The 8xx has no PMU, however some events can be emulated by other means. This patch implements the following events (as reported by 'perf list'): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] dTLB-load-misses [Hardware cache event] iTLB-load-misses [Hardware cache event] 'cycles' event is implemented using the timebase clock. Timebase clock corresponds to CPU clock divided by 16, so number of cycles is approximatly 16 times the number of TB ticks On the 8xx, TLB misses are handled by software. It is therefore easy to count all TLB misses each time the TLB miss exception is called. 'instructions' is calculated by using instruction watchpoint counter. This patch sets counter A to count instructions at address greater than 0, hence we count all instructions executed while MSR RI bit is set. The counter is set to the maximum which is 0xffff. Every 65535 instructions, debug instruction breakpoint exception fires. The exception handler increments a counter in memory which then represent the upper part of the instruction counter. We therefore end up with a 48 bits counter. In order to avoid unnecessary overhead while no perf event is active, this counter is started when the first event referring to this counter is added, and the counter is stopped when the last event referring to it is deleted. In order to properly support breakpoint exceptions, MSR RI bit has to be unset in exception epilogs in order to avoid breakpoint exceptions during critical sections during changes to SRR0 and SRR1 would be problematic. All counters are handled as free running counters. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Scott Wood <oss@buserror.net>
2016-12-15 20:42:18 +08:00
mtspr SPRN_NRI, r0
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mtspr SPRN_SRR1,r9
mtspr SPRN_SRR0,r12
REST_GPR(9, r11)
REST_GPR(12, r11)
lwz r11,GPR11(r11)
SYNC
RFI
#if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE))
/* check if the exception happened in a restartable section */
1: lis r3,exc_exit_restart_end@ha
addi r3,r3,exc_exit_restart_end@l
cmplw r12,r3
bge 3f
lis r4,exc_exit_restart@ha
addi r4,r4,exc_exit_restart@l
cmplw r12,r4
blt 3f
lis r3,fee_restarts@ha
tophys(r3,r3)
lwz r5,fee_restarts@l(r3)
addi r5,r5,1
stw r5,fee_restarts@l(r3)
mr r12,r4 /* restart at exc_exit_restart */
b 2b
[POWERPC] Fix COMMON symbol warnings We get the following warnings in various ARCH=powerpc builds: WARNING: "ee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "fee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "htab_hash_searches" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "next_slot" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "mmu_hash_lock" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "primary_pteg_full" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "global_dbcr0" [arch/powerpc/kernel/built-in] is COMMON symbol Switch to moving local symbols (except mmu_hash_lock which is global) and space directive instead. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-05-15 06:11:58 +08:00
.section .bss
.align 2
fee_restarts:
.space 4
.previous
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* aargh, a nonrecoverable interrupt, panic */
/* aargh, we don't know which trap this is */
/* but the 601 doesn't implement the RI bit, so assume it's OK */
3:
BEGIN_FTR_SECTION
b 2b
END_FTR_SECTION_IFSET(CPU_FTR_601)
li r10,-1
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r10,_TRAP(r11)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
addi r3,r1,STACK_FRAME_OVERHEAD
lis r10,MSR_KERNEL@h
ori r10,r10,MSR_KERNEL@l
bl transfer_to_handler_full
powerpc/traps: merge unrecoverable_exception() and nonrecoverable_exception() PPC32 uses nonrecoverable_exception() while PPC64 uses unrecoverable_exception(). Both functions are doing almost the same thing. This patch removes nonrecoverable_exception() Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-09-25 22:10:04 +08:00
.long unrecoverable_exception
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.long ret_from_except
#endif
.globl ret_from_except_full
ret_from_except_full:
REST_NVGPRS(r1)
/* fall through */
.globl ret_from_except
ret_from_except:
/* Hard-disable interrupts so that current_thread_info()->flags
* can't change between when we test it and when we return
* from the interrupt. */
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
/* Note: We don't bother telling lockdep about it */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
LOAD_MSR_KERNEL(r10,MSR_KERNEL)
SYNC /* Some chip revs have problems here... */
MTMSRD(r10) /* disable interrupts */
lwz r3,_MSR(r1) /* Returning to user mode? */
andi. r0,r3,MSR_PR
beq resume_kernel
user_exc_return: /* r10 contains MSR_KERNEL here */
/* Check current_thread_info()->flags */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r9,TI_FLAGS(r2)
[POWERPC] Define and use TLF_RESTORE_SIGMASK Replace TIF_RESTORE_SIGMASK with TLF_RESTORE_SIGMASK and define our own set_restore_sigmask() function. This saves the costly SMP-safe set_bit operation, which we do not need for the sigmask flag since TIF_SIGPENDING always has to be set too. Signed-off-by: Roland McGrath <roland@redhat.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-04-28 15:30:37 +08:00
andi. r0,r9,_TIF_USER_WORK_MASK
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
bne do_work
restore_user:
#if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
/* Check whether this process has its own DBCR0 value. The internal
debug mode bit tells us that dbcr0 should be loaded. */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r0,THREAD+THREAD_DBCR0(r2)
powerpc/booke: Clean up the hardware watchpoint support * CONFIG_BOOKE is selected by CONFIG_44x so we dont need both * Fixed a few comments * Go back to only using DBCR0_IDM to determine if we are using debug resources. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-07-26 03:27:33 +08:00
andis. r10,r0,DBCR0_IDM@h
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
bnel- load_dbcr0
#endif
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
ACCOUNT_CPU_USER_EXIT(r2, r10, r11)
powerpc/32s: Implement Kernel Userspace Execution Prevention. To implement Kernel Userspace Execution Prevention, this patch sets NX bit on all user segments on kernel entry and clears NX bit on all user segments on kernel exit. Note that powerpc 601 doesn't have the NX bit, so KUEP will not work on it. A warning is displayed at startup. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:35 +08:00
#ifdef CONFIG_PPC_BOOK3S_32
kuep_unlock r10, r11
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
b restore
/* N.B. the only way to get here is from the beq following ret_from_except. */
resume_kernel:
powerpc/kprobe: Complete kprobe and migrate exception frame We can't emulate stwu since that may corrupt current exception stack. So we will have to do real store operation in the exception return code. Firstly we'll allocate a trampoline exception frame below the kprobed function stack and copy the current exception frame to the trampoline. Then we can do this real store operation to implement 'stwu', and reroute the trampoline frame to r1 to complete this exception migration. Signed-off-by: Tiejun Chen <tiejun.chen@windriver.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-09-17 07:54:30 +08:00
/* check current_thread_info, _TIF_EMULATE_STACK_STORE */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r8,TI_FLAGS(r2)
powerpc/32bit:Store temporary result in r0 instead of r8 Commit a9c4e541ea9b22944da356f2a9258b4eddcc953b "powerpc/kprobe: Complete kprobe and migrate exception frame" introduced a regression: While returning from exception handling in case of PREEMPT enabled, _TIF_NEED_RESCHED bit is checked in TI_FLAGS (thread_info flag) of current task. Only if this bit is set, it should continue with the process of calling preempt_schedule_irq() to schedule highest priority task if available. Current code assumes that r8 contains TI_FLAGS and check this for _TIF_NEED_RESCHED, but as r8 is modified in the code which executes before this check, r8 no longer contains the expected TI_FLAGS information. As a result check for comparison with _TIF_NEED_RESCHED was failing even if NEED_RESCHED bit is set in the current thread_info flag. Due to this, preempt_schedule_irq() and in turn scheduler was not getting called even if highest priority task is ready for execution. So, store temporary results in r0 instead of r8 to prevent r8 from getting modified as subsequent code is dependent on its value. Signed-off-by: Priyanka Jain <Priyanka.Jain@freescale.com> CC: <stable@vger.kernel.org> [v3.7+] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-05-31 09:20:02 +08:00
andis. r0,r8,_TIF_EMULATE_STACK_STORE@h
powerpc/kprobe: Complete kprobe and migrate exception frame We can't emulate stwu since that may corrupt current exception stack. So we will have to do real store operation in the exception return code. Firstly we'll allocate a trampoline exception frame below the kprobed function stack and copy the current exception frame to the trampoline. Then we can do this real store operation to implement 'stwu', and reroute the trampoline frame to r1 to complete this exception migration. Signed-off-by: Tiejun Chen <tiejun.chen@windriver.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-09-17 07:54:30 +08:00
beq+ 1f
addi r8,r1,INT_FRAME_SIZE /* Get the kprobed function entry */
lwz r3,GPR1(r1)
subi r3,r3,INT_FRAME_SIZE /* dst: Allocate a trampoline exception frame */
mr r4,r1 /* src: current exception frame */
mr r1,r3 /* Reroute the trampoline frame to r1 */
/* Copy from the original to the trampoline. */
li r5,INT_FRAME_SIZE/4 /* size: INT_FRAME_SIZE */
li r6,0 /* start offset: 0 */
mtctr r5
2: lwzx r0,r6,r4
stwx r0,r6,r3
addi r6,r6,4
bdnz 2b
/* Do real store operation to complete stwu */
lwz r5,GPR1(r1)
stw r8,0(r5)
/* Clear _TIF_EMULATE_STACK_STORE flag */
lis r11,_TIF_EMULATE_STACK_STORE@h
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
addi r5,r2,TI_FLAGS
powerpc/kprobe: Complete kprobe and migrate exception frame We can't emulate stwu since that may corrupt current exception stack. So we will have to do real store operation in the exception return code. Firstly we'll allocate a trampoline exception frame below the kprobed function stack and copy the current exception frame to the trampoline. Then we can do this real store operation to implement 'stwu', and reroute the trampoline frame to r1 to complete this exception migration. Signed-off-by: Tiejun Chen <tiejun.chen@windriver.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-09-17 07:54:30 +08:00
0: lwarx r8,0,r5
andc r8,r8,r11
#ifdef CONFIG_IBM405_ERR77
dcbt 0,r5
#endif
stwcx. r8,0,r5
bne- 0b
1:
#ifdef CONFIG_PREEMPT
/* check current_thread_info->preempt_count */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r0,TI_PREEMPT(r2)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
cmpwi 0,r0,0 /* if non-zero, just restore regs and return */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
bne restore_kuap
powerpc/kprobe: Complete kprobe and migrate exception frame We can't emulate stwu since that may corrupt current exception stack. So we will have to do real store operation in the exception return code. Firstly we'll allocate a trampoline exception frame below the kprobed function stack and copy the current exception frame to the trampoline. Then we can do this real store operation to implement 'stwu', and reroute the trampoline frame to r1 to complete this exception migration. Signed-off-by: Tiejun Chen <tiejun.chen@windriver.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-09-17 07:54:30 +08:00
andi. r8,r8,_TIF_NEED_RESCHED
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
beq+ restore_kuap
powerpc/kprobe: Complete kprobe and migrate exception frame We can't emulate stwu since that may corrupt current exception stack. So we will have to do real store operation in the exception return code. Firstly we'll allocate a trampoline exception frame below the kprobed function stack and copy the current exception frame to the trampoline. Then we can do this real store operation to implement 'stwu', and reroute the trampoline frame to r1 to complete this exception migration. Signed-off-by: Tiejun Chen <tiejun.chen@windriver.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-09-17 07:54:30 +08:00
lwz r3,_MSR(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
andi. r0,r3,MSR_EE /* interrupts off? */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
beq restore_kuap /* don't schedule if so */
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
/* Lockdep thinks irqs are enabled, we need to call
* preempt_schedule_irq with IRQs off, so we inform lockdep
* now that we -did- turn them off already
*/
bl trace_hardirqs_off
#endif
powerpc/entry: Remove unneeded need_resched() loop Since the enabling and disabling of IRQs within preempt_schedule_irq() is contained in a need_resched() loop, we don't need the outer arch code loop. Signed-off-by: Valentin Schneider <valentin.schneider@arm.com> [mpe: Rebase since CURRENT_THREAD_INFO() removal] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-12 06:47:46 +08:00
bl preempt_schedule_irq
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
/* And now, to properly rebalance the above, we tell lockdep they
* are being turned back on, which will happen when we return
*/
bl trace_hardirqs_on
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#endif /* CONFIG_PREEMPT */
powerpc/32: Prepare for Kernel Userspace Access Protection This patch adds ASM macros for saving, restoring and checking the KUAP state, and modifies setup_32 to call them on exceptions from kernel. The macros are defined as empty by default for when CONFIG_PPC_KUAP is not selected and/or for platforms which don't handle (yet) KUAP. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-03-11 16:30:31 +08:00
restore_kuap:
kuap_restore r1, r2, r9, r10, r0
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* interrupts are hard-disabled at this point */
restore:
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
#ifdef CONFIG_44x
powerpc/47x: Base ppc476 support This patch adds the base support for the 476 processor. The code was primarily written by Ben Herrenschmidt and Torez Smith, but I've been maintaining it for a while. The goal is to have a single binary that will run on 44x and 47x, but we still have some details to work out. The biggest is that the L1 cache line size differs on the two platforms, but it's currently a compile-time option. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Torez Smith <lnxtorez@linux.vnet.ibm.com> Signed-off-by: Dave Kleikamp <shaggy@linux.vnet.ibm.com> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2010-03-05 18:43:12 +08:00
BEGIN_MMU_FTR_SECTION
b 1f
END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_47x)
[POWERPC] 4xx: Deal with 44x virtually tagged icache The 44x family has an interesting "feature" which is a virtually tagged instruction cache (yuck !). So far, we haven't dealt with it properly, which means we've been mostly lucky or people didn't report the problems, unless people have been running custom patches in their distro... This is an attempt at fixing it properly. I chose to do it by setting a global flag whenever we change a PTE that was previously marked executable, and flush the entire instruction cache upon return to user space when that happens. This is a bit heavy handed, but it's hard to do more fine grained flushes as the icbi instruction, on those processor, for some very strange reasons (since the cache is virtually mapped) still requires a valid TLB entry for reading in the target address space, which isn't something I want to deal with. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
2007-10-31 13:42:19 +08:00
lis r4,icache_44x_need_flush@ha
lwz r5,icache_44x_need_flush@l(r4)
cmplwi cr0,r5,0
beq+ 1f
li r6,0
iccci r0,r0
stw r6,icache_44x_need_flush@l(r4)
1:
#endif /* CONFIG_44x */
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
lwz r9,_MSR(r1)
#ifdef CONFIG_TRACE_IRQFLAGS
/* Lockdep doesn't know about the fact that IRQs are temporarily turned
* off in this assembly code while peeking at TI_FLAGS() and such. However
* we need to inform it if the exception turned interrupts off, and we
* are about to trun them back on.
*/
andi. r10,r9,MSR_EE
beq 1f
powerpc/ppc32/tracing: Add stack frame to calls of trace_hardirqs_on/off 32-bit variant of the previous patch for 64-bit: << When an interrupt occurs in userspace, we can call trace_hardirqs_on/off() With one level stack. But if we have irqsoff tracing enabled, it checks both CALLER_ADDR0 and CALLER_ADDR1. The second call goes two stack frames up. If this is from user space, then there may not exist a second stack.... >> Signed-off-by: Steven Rostedt <rostedt@goodmis.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2010-12-23 00:42:56 +08:00
stwu r1,-32(r1)
mflr r0
stw r0,4(r1)
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
bl trace_hardirqs_on
powerpc/32: Don't add dummy frames when calling trace_hardirqs_on/off No need to add dummy frames when calling trace_hardirqs_on or trace_hardirqs_off. GCC properly handles empty stacks. In addition, powerpc doesn't set CONFIG_FRAME_POINTER, therefore __builtin_return_address(1..) returns NULL at all time. So the dummy frames are definitely unneeded here. In the meantime, avoid reading memory for loading r1 with a value we already know. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:05 +08:00
addi r1, r1, 32
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
lwz r9,_MSR(r1)
1:
#endif /* CONFIG_TRACE_IRQFLAGS */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lwz r0,GPR0(r1)
lwz r2,GPR2(r1)
REST_4GPRS(3, r1)
REST_2GPRS(7, r1)
lwz r10,_XER(r1)
lwz r11,_CTR(r1)
mtspr SPRN_XER,r10
mtctr r11
PPC405_ERR77(0,r1)
[POWERPC] Avoid unpaired stwcx. on some processors The context switch code in the kernel issues a dummy stwcx. to clear the reservation, as recommended by the architecture. However, some processors can have issues if this stwcx to address A occurs while the reservation is already held to a different address B. To avoid this problem, the dummy stwcx. needs to be paired with a dummy lwarx to the same address. This adds the dummy lwarx, and creates a cpu feature bit to indicate which cpus are affected. Tested on mpc8641_hpcn_defconfig in arch/powerpc; build tested in arch/ppc. Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-11-10 06:17:49 +08:00
BEGIN_FTR_SECTION
lwarx r11,0,r1
END_FTR_SECTION_IFSET(CPU_FTR_NEED_PAIRED_STWCX)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
stwcx. r0,0,r1 /* to clear the reservation */
#if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE))
andi. r10,r9,MSR_RI /* check if this exception occurred */
beql nonrecoverable /* at a bad place (MSR:RI = 0) */
lwz r10,_CCR(r1)
lwz r11,_LINK(r1)
mtcrf 0xFF,r10
mtlr r11
powerpc/32: Clear on-stack exception marker upon exception return Clear the on-stack STACK_FRAME_REGS_MARKER on exception exit in order to avoid confusing stacktrace like the one below. Call Trace: [c0e9dca0] [c01c42a0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c4684] kasan_report+0xfc/0x180 [c0e9dd10] [c0895130] memchr+0x24/0x74 [c0e9dd30] [c00a9e38] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab710] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc60] vprintk_emit+0x188/0x1c4 --- interrupt: c0e9df00 at 0x400f330 LR = init_stack+0x1f00/0x2000 [c0e9de80] [c00ae3c4] printk+0xa8/0xcc (unreliable) [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 With this patch the trace becomes: Call Trace: [c0e9dca0] [c01c42c0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c46a4] kasan_report+0xfc/0x180 [c0e9dd10] [c0895150] memchr+0x24/0x74 [c0e9dd30] [c00a9e58] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab730] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc80] vprintk_emit+0x188/0x1c4 [c0e9de80] [c00ae3e4] printk+0xa8/0xcc [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 Cc: stable@vger.kernel.org Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-02-27 19:45:30 +08:00
/* Clear the exception_marker on the stack to avoid confusing stacktrace */
li r10, 0
stw r10, 8(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* Once we put values in SRR0 and SRR1, we are in a state
* where exceptions are not recoverable, since taking an
* exception will trash SRR0 and SRR1. Therefore we clear the
* MSR:RI bit to indicate this. If we do take an exception,
* we can't return to the point of the exception but we
* can restart the exception exit path at the label
* exc_exit_restart below. -- paulus
*/
LOAD_MSR_KERNEL(r10,MSR_KERNEL & ~MSR_RI)
SYNC
MTMSRD(r10) /* clear the RI bit */
.globl exc_exit_restart
exc_exit_restart:
lwz r12,_NIP(r1)
mtspr SPRN_SRR0,r12
mtspr SPRN_SRR1,r9
REST_4GPRS(9, r1)
lwz r1,GPR1(r1)
.globl exc_exit_restart_end
exc_exit_restart_end:
SYNC
RFI
#else /* !(CONFIG_4xx || CONFIG_BOOKE) */
/*
* This is a bit different on 4xx/Book-E because it doesn't have
* the RI bit in the MSR.
* The TLB miss handler checks if we have interrupted
* the exception exit path and restarts it if so
* (well maybe one day it will... :).
*/
lwz r11,_LINK(r1)
mtlr r11
lwz r10,_CCR(r1)
mtcrf 0xff,r10
powerpc/32: Clear on-stack exception marker upon exception return Clear the on-stack STACK_FRAME_REGS_MARKER on exception exit in order to avoid confusing stacktrace like the one below. Call Trace: [c0e9dca0] [c01c42a0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c4684] kasan_report+0xfc/0x180 [c0e9dd10] [c0895130] memchr+0x24/0x74 [c0e9dd30] [c00a9e38] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab710] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc60] vprintk_emit+0x188/0x1c4 --- interrupt: c0e9df00 at 0x400f330 LR = init_stack+0x1f00/0x2000 [c0e9de80] [c00ae3c4] printk+0xa8/0xcc (unreliable) [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 With this patch the trace becomes: Call Trace: [c0e9dca0] [c01c42c0] print_address_description+0x64/0x2bc (unreliable) [c0e9dcd0] [c01c46a4] kasan_report+0xfc/0x180 [c0e9dd10] [c0895150] memchr+0x24/0x74 [c0e9dd30] [c00a9e58] msg_print_text+0x124/0x574 [c0e9dde0] [c00ab730] console_unlock+0x114/0x4f8 [c0e9de40] [c00adc80] vprintk_emit+0x188/0x1c4 [c0e9de80] [c00ae3e4] printk+0xa8/0xcc [c0e9df20] [c0c27e44] early_irq_init+0x38/0x108 [c0e9df50] [c0c15434] start_kernel+0x310/0x488 [c0e9dff0] [00003484] 0x3484 Cc: stable@vger.kernel.org Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-02-27 19:45:30 +08:00
/* Clear the exception_marker on the stack to avoid confusing stacktrace */
li r10, 0
stw r10, 8(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
REST_2GPRS(9, r1)
.globl exc_exit_restart
exc_exit_restart:
lwz r11,_NIP(r1)
lwz r12,_MSR(r1)
exc_exit_start:
mtspr SPRN_SRR0,r11
mtspr SPRN_SRR1,r12
REST_2GPRS(11, r1)
lwz r1,GPR1(r1)
.globl exc_exit_restart_end
exc_exit_restart_end:
PPC405_ERR77_SYNC
rfi
b . /* prevent prefetch past rfi */
/*
* Returning from a critical interrupt in user mode doesn't need
* to be any different from a normal exception. For a critical
* interrupt in the kernel, we just return (without checking for
* preemption) since the interrupt may have happened at some crucial
* place (e.g. inside the TLB miss handler), and because we will be
* running with r1 pointing into critical_stack, not the current
* process's kernel stack (and therefore current_thread_info() will
* give the wrong answer).
* We have to restore various SPRs that may have been in use at the
* time of the critical interrupt.
*
*/
#ifdef CONFIG_40x
#define PPC_40x_TURN_OFF_MSR_DR \
/* avoid any possible TLB misses here by turning off MSR.DR, we \
* assume the instructions here are mapped by a pinned TLB entry */ \
li r10,MSR_IR; \
mtmsr r10; \
isync; \
tophys(r1, r1);
#else
#define PPC_40x_TURN_OFF_MSR_DR
#endif
#define RET_FROM_EXC_LEVEL(exc_lvl_srr0, exc_lvl_srr1, exc_lvl_rfi) \
REST_NVGPRS(r1); \
lwz r3,_MSR(r1); \
andi. r3,r3,MSR_PR; \
LOAD_MSR_KERNEL(r10,MSR_KERNEL); \
bne user_exc_return; \
lwz r0,GPR0(r1); \
lwz r2,GPR2(r1); \
REST_4GPRS(3, r1); \
REST_2GPRS(7, r1); \
lwz r10,_XER(r1); \
lwz r11,_CTR(r1); \
mtspr SPRN_XER,r10; \
mtctr r11; \
PPC405_ERR77(0,r1); \
stwcx. r0,0,r1; /* to clear the reservation */ \
lwz r11,_LINK(r1); \
mtlr r11; \
lwz r10,_CCR(r1); \
mtcrf 0xff,r10; \
PPC_40x_TURN_OFF_MSR_DR; \
lwz r9,_DEAR(r1); \
lwz r10,_ESR(r1); \
mtspr SPRN_DEAR,r9; \
mtspr SPRN_ESR,r10; \
lwz r11,_NIP(r1); \
lwz r12,_MSR(r1); \
mtspr exc_lvl_srr0,r11; \
mtspr exc_lvl_srr1,r12; \
lwz r9,GPR9(r1); \
lwz r12,GPR12(r1); \
lwz r10,GPR10(r1); \
lwz r11,GPR11(r1); \
lwz r1,GPR1(r1); \
PPC405_ERR77_SYNC; \
exc_lvl_rfi; \
b .; /* prevent prefetch past exc_lvl_rfi */
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
#define RESTORE_xSRR(exc_lvl_srr0, exc_lvl_srr1) \
lwz r9,_##exc_lvl_srr0(r1); \
lwz r10,_##exc_lvl_srr1(r1); \
mtspr SPRN_##exc_lvl_srr0,r9; \
mtspr SPRN_##exc_lvl_srr1,r10;
powerpc/book-3e: Introduce concept of Book-3e MMU The Power ISA 2.06 spec introduces a standard MMU programming model that is based on the Freescale Book-E MMU programing model. The Freescale version is pretty backwards compatiable with the ISA 2.06 definition so we are starting to refactor some of the Freescale code so it can be easily shared. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2009-02-13 06:12:40 +08:00
#if defined(CONFIG_PPC_BOOK3E_MMU)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
#ifdef CONFIG_PHYS_64BIT
#define RESTORE_MAS7 \
lwz r11,MAS7(r1); \
mtspr SPRN_MAS7,r11;
#else
#define RESTORE_MAS7
#endif /* CONFIG_PHYS_64BIT */
#define RESTORE_MMU_REGS \
lwz r9,MAS0(r1); \
lwz r10,MAS1(r1); \
lwz r11,MAS2(r1); \
mtspr SPRN_MAS0,r9; \
lwz r9,MAS3(r1); \
mtspr SPRN_MAS1,r10; \
lwz r10,MAS6(r1); \
mtspr SPRN_MAS2,r11; \
mtspr SPRN_MAS3,r9; \
mtspr SPRN_MAS6,r10; \
RESTORE_MAS7;
#elif defined(CONFIG_44x)
#define RESTORE_MMU_REGS \
lwz r9,MMUCR(r1); \
mtspr SPRN_MMUCR,r9;
#else
#define RESTORE_MMU_REGS
#endif
#ifdef CONFIG_40x
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl ret_from_crit_exc
ret_from_crit_exc:
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r9,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lis r10,saved_ksp_limit@ha;
lwz r10,saved_ksp_limit@l(r10);
tovirt(r9,r9);
stw r10,KSP_LIMIT(r9)
lis r9,crit_srr0@ha;
lwz r9,crit_srr0@l(r9);
lis r10,crit_srr1@ha;
lwz r10,crit_srr1@l(r10);
mtspr SPRN_SRR0,r9;
mtspr SPRN_SRR1,r10;
powerpc: Unify opcode definitions and support Create a new header that becomes a single location for defining PowerPC opcodes used by code that is either generationg instructions at runtime (fixups, debug, etc.), emulating instructions, or just compiling instructions old assemblers don't know about. We currently don't handle the floating point emulation or alignment decode as both are better handled by the specific decode support they already have. Added support for the new dcbzl, dcbal, msgsnd, tlbilx, & wait instructions since older assemblers don't know about them. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-02-11 04:10:44 +08:00
RET_FROM_EXC_LEVEL(SPRN_CSRR0, SPRN_CSRR1, PPC_RFCI)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
#endif /* CONFIG_40x */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#ifdef CONFIG_BOOKE
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
.globl ret_from_crit_exc
ret_from_crit_exc:
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r9,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lwz r10,SAVED_KSP_LIMIT(r1)
stw r10,KSP_LIMIT(r9)
RESTORE_xSRR(SRR0,SRR1);
RESTORE_MMU_REGS;
powerpc: Unify opcode definitions and support Create a new header that becomes a single location for defining PowerPC opcodes used by code that is either generationg instructions at runtime (fixups, debug, etc.), emulating instructions, or just compiling instructions old assemblers don't know about. We currently don't handle the floating point emulation or alignment decode as both are better handled by the specific decode support they already have. Added support for the new dcbzl, dcbal, msgsnd, tlbilx, & wait instructions since older assemblers don't know about them. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-02-11 04:10:44 +08:00
RET_FROM_EXC_LEVEL(SPRN_CSRR0, SPRN_CSRR1, PPC_RFCI)
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl ret_from_debug_exc
ret_from_debug_exc:
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r9,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lwz r10,SAVED_KSP_LIMIT(r1)
stw r10,KSP_LIMIT(r9)
RESTORE_xSRR(SRR0,SRR1);
RESTORE_xSRR(CSRR0,CSRR1);
RESTORE_MMU_REGS;
powerpc: Unify opcode definitions and support Create a new header that becomes a single location for defining PowerPC opcodes used by code that is either generationg instructions at runtime (fixups, debug, etc.), emulating instructions, or just compiling instructions old assemblers don't know about. We currently don't handle the floating point emulation or alignment decode as both are better handled by the specific decode support they already have. Added support for the new dcbzl, dcbal, msgsnd, tlbilx, & wait instructions since older assemblers don't know about them. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-02-11 04:10:44 +08:00
RET_FROM_EXC_LEVEL(SPRN_DSRR0, SPRN_DSRR1, PPC_RFDI)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
.globl ret_from_mcheck_exc
ret_from_mcheck_exc:
powerpc: Use names rather than numbers for SPRGs (v2) The kernel uses SPRG registers for various purposes, typically in low level assembly code as scratch registers or to hold per-cpu global infos such as the PACA or the current thread_info pointer. We want to be able to easily shuffle the usage of those registers as some implementations have specific constraints realted to some of them, for example, some have userspace readable aliases, etc.. and the current choice isn't always the best. This patch should not change any code generation, and replaces the usage of SPRN_SPRGn everywhere in the kernel with a named replacement and adds documentation next to the definition of the names as to what those are used for on each processor family. The only parts that still use the original numbers are bits of KVM or suspend/resume code that just blindly needs to save/restore all the SPRGs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-15 04:52:54 +08:00
mfspr r9,SPRN_SPRG_THREAD
[POWERPC] 40x/Book-E: Save/restore volatile exception registers On machines with more than one exception level any system register that might be modified by the "normal" exception level needs to be saved and restored on taking a higher level exception. We already are saving and restoring ESR and DEAR. For critical level add SRR0/1. For debug level add CSRR0/1 and SRR0/1. For machine check level add DSRR0/1, CSRR0/1, and SRR0/1. On FSL Book-E parts we always save/restore the MAS registers for critical, debug, and machine check level exceptions. On 44x we always save/restore the MMUCR. Additionally, we save and restore the ksp_limit since we have to adjust it for each exception level. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Acked-by: Paul Mackerras <paulus@samba.org>
2008-04-30 18:23:21 +08:00
lwz r10,SAVED_KSP_LIMIT(r1)
stw r10,KSP_LIMIT(r9)
RESTORE_xSRR(SRR0,SRR1);
RESTORE_xSRR(CSRR0,CSRR1);
RESTORE_xSRR(DSRR0,DSRR1);
RESTORE_MMU_REGS;
powerpc: Unify opcode definitions and support Create a new header that becomes a single location for defining PowerPC opcodes used by code that is either generationg instructions at runtime (fixups, debug, etc.), emulating instructions, or just compiling instructions old assemblers don't know about. We currently don't handle the floating point emulation or alignment decode as both are better handled by the specific decode support they already have. Added support for the new dcbzl, dcbal, msgsnd, tlbilx, & wait instructions since older assemblers don't know about them. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-02-11 04:10:44 +08:00
RET_FROM_EXC_LEVEL(SPRN_MCSRR0, SPRN_MCSRR1, PPC_RFMCI)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#endif /* CONFIG_BOOKE */
/*
* Load the DBCR0 value for a task that is being ptraced,
* having first saved away the global DBCR0. Note that r0
* has the dbcr0 value to set upon entry to this.
*/
load_dbcr0:
mfmsr r10 /* first disable debug exceptions */
rlwinm r10,r10,0,~MSR_DE
mtmsr r10
isync
mfspr r10,SPRN_DBCR0
lis r11,global_dbcr0@ha
addi r11,r11,global_dbcr0@l
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
#ifdef CONFIG_SMP
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r9,TASK_CPU(r2)
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
slwi r9,r9,3
add r11,r11,r9
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
stw r10,0(r11)
mtspr SPRN_DBCR0,r0
lwz r10,4(r11)
addi r10,r10,1
stw r10,4(r11)
li r11,-1
mtspr SPRN_DBSR,r11 /* clear all pending debug events */
blr
[POWERPC] Fix COMMON symbol warnings We get the following warnings in various ARCH=powerpc builds: WARNING: "ee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "fee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "htab_hash_searches" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "next_slot" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "mmu_hash_lock" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "primary_pteg_full" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "global_dbcr0" [arch/powerpc/kernel/built-in] is COMMON symbol Switch to moving local symbols (except mmu_hash_lock which is global) and space directive instead. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-05-15 06:11:58 +08:00
.section .bss
.align 4
powerpc/32: implement fast entry for syscalls on non BOOKE This patch implements a fast entry for syscalls. Syscalls don't have to preserve non volatile registers except LR. This patch then implement a fast entry for syscalls, where volatile registers get clobbered. As this entry is dedicated to syscall it always sets MSR_EE and warns in case MSR_EE was previously off It also assumes that the call is always from user, system calls are unexpected from kernel. The overall series improves null_syscall selftest by 12,5% on an 83xx and by 17% on a 8xx. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:02 +08:00
.global global_dbcr0
[POWERPC] Fix COMMON symbol warnings We get the following warnings in various ARCH=powerpc builds: WARNING: "ee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "fee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "htab_hash_searches" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "next_slot" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "mmu_hash_lock" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "primary_pteg_full" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "global_dbcr0" [arch/powerpc/kernel/built-in] is COMMON symbol Switch to moving local symbols (except mmu_hash_lock which is global) and space directive instead. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-05-15 06:11:58 +08:00
global_dbcr0:
[POWERPC] Make Book-E debug handling SMP safe global_dbcr0 needs to be a per cpu set of save areas instead of a single global on all processors. Also, we switch to using DBCR0_IDM to determine if the user space app is being debugged as its a more consistent way. In the future we should support features like hardware breakpoint and watchpoints which will have DBCR0_IDM set but not necessarily DBCR0_IC (single step). Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2008-04-10 05:15:40 +08:00
.space 8*NR_CPUS
[POWERPC] Fix COMMON symbol warnings We get the following warnings in various ARCH=powerpc builds: WARNING: "ee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "fee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "htab_hash_searches" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "next_slot" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "mmu_hash_lock" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "primary_pteg_full" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "global_dbcr0" [arch/powerpc/kernel/built-in] is COMMON symbol Switch to moving local symbols (except mmu_hash_lock which is global) and space directive instead. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-05-15 06:11:58 +08:00
.previous
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
#endif /* !(CONFIG_4xx || CONFIG_BOOKE) */
do_work: /* r10 contains MSR_KERNEL here */
andi. r0,r9,_TIF_NEED_RESCHED
beq do_user_signal
do_resched: /* r10 contains MSR_KERNEL here */
powerpc: Fix 32-bit handling of MSR_EE on exceptions [text mostly copied from benh's RFC/WIP] ppc32 are still doing something rather gothic and wrong on 32-bit which we stopped doing on 64-bit a while ago. We have that thing where some handlers "copy" the EE value from the original stack frame into the new MSR before transferring to the handler. Thus for a number of exceptions, we enter the handlers with interrupts enabled. This is rather fishy, some of the stuff that handlers might do early on such as irq_enter/exit or user_exit, context tracking, etc... should be run with interrupts off afaik. Generally our handlers know when to re-enable interrupts if needed. The problem we were having is that we assumed these interrupts would return with interrupts enabled. However that isn't the case. Instead, this patch changes things so that we always enter exception handlers with interrupts *off* with the notable exception of syscalls which are special (and get a fast path). Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-04-30 20:39:01 +08:00
#ifdef CONFIG_TRACE_IRQFLAGS
bl trace_hardirqs_on
mfmsr r10
#endif
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
ori r10,r10,MSR_EE
SYNC
MTMSRD(r10) /* hard-enable interrupts */
bl schedule
recheck:
powerpc: Add irqtrace support for 32-bit powerpc Based on initial work from: Dale Farnsworth <dale@farnsworth.org> Add the low level irq tracing hooks for 32-bit powerpc needed to enable full lockdep functionality. The approach taken to deal with the code in entry_32.S is that we don't trace all the transitions of MSR:EE when we just turn it off to peek at TI_FLAGS without races. Only when we are calling into C code or returning from exceptions with a state that have changed from what lockdep thinks. There's a little bugger though: If we take an exception that keeps interrupts enabled (such as an alignment exception) while interrupts are enabled, we will call trace_hardirqs_on() on the way back spurriously. Not a big deal, but to get rid of it would require remembering in pt_regs that the exception was one of the type that kept interrupts enabled which we don't know at this stage. (Well, we could test all cases for regs->trap but that sucks too much). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Kumar Gala <galak@kernel.crashing.org>
2009-06-18 01:43:59 +08:00
/* Note: And we don't tell it we are disabling them again
* neither. Those disable/enable cycles used to peek at
* TI_FLAGS aren't advertised.
*/
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
LOAD_MSR_KERNEL(r10,MSR_KERNEL)
SYNC
MTMSRD(r10) /* disable interrupts */
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU Now that thread_info is similar to task_struct, its address is in r2 so CURRENT_THREAD_INFO() macro is useless. This patch removes it. This patch also moves the 'tovirt(r2, r2)' down just before the reactivation of MMU translation, so that we keep the physical address of 'current' in r2 until then. It avoids a few calls to tophys(). At the same time, as the 'cpu' field is not anymore in thread_info, TI_CPU is renamed TASK_CPU by this patch. It also allows to get rid of a couple of '#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE' as ACCOUNT_CPU_USER_ENTRY() and ACCOUNT_CPU_USER_EXIT() are empty when CONFIG_VIRT_CPU_ACCOUNTING_NATIVE is not defined. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> [mpe: Fix a missed conversion of TI_CPU idle_6xx.S] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:09:04 +08:00
lwz r9,TI_FLAGS(r2)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
andi. r0,r9,_TIF_NEED_RESCHED
bne- do_resched
[POWERPC] Define and use TLF_RESTORE_SIGMASK Replace TIF_RESTORE_SIGMASK with TLF_RESTORE_SIGMASK and define our own set_restore_sigmask() function. This saves the costly SMP-safe set_bit operation, which we do not need for the sigmask flag since TIF_SIGPENDING always has to be set too. Signed-off-by: Roland McGrath <roland@redhat.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-04-28 15:30:37 +08:00
andi. r0,r9,_TIF_USER_WORK_MASK
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
beq restore_user
do_user_signal: /* r10 contains MSR_KERNEL here */
ori r10,r10,MSR_EE
SYNC
MTMSRD(r10) /* hard-enable interrupts */
/* save r13-r31 in the exception frame, if not already done */
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r3,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
andi. r0,r3,1
beq 2f
SAVE_NVGPRS(r1)
rlwinm r3,r3,0,0,30
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r3,_TRAP(r1)
powerpc: Add TIF_NOTIFY_RESUME support for tracehook This adds TIF_NOTIFY_RESUME support for powerpc. When set, we call tracehook_notify_resume() on the way to user mode. This overloads do_signal() to do the work, but changes its arguments to it has the TIF_* bits handy in a register and drops the useless first argument that was always zero. Signed-off-by: Roland McGrath <roland@redhat.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-07-27 14:52:52 +08:00
2: addi r3,r1,STACK_FRAME_OVERHEAD
mr r4,r9
powerpc: Fix various issues with return to userspace We have a few problems when returning to userspace. This is a quick set of fixes for 3.3, I'll look into a more comprehensive rework for 3.4. This fixes: - We kept interrupts soft-disabled when schedule'ing or calling do_signal when returning to userspace as a result of a hardware interrupt. - Rename do_signal to do_notify_resume like all other archs (and do_signal_pending back to do_signal, which it was before Roland changed it). - Add the missing call to key_replace_session_keyring() to do_notify_resume(). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> ---
2012-02-22 13:48:32 +08:00
bl do_notify_resume
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
REST_NVGPRS(r1)
b recheck
/*
* We come here when we are at the end of handling an exception
* that occurred at a place where taking an exception will lose
* state information, such as the contents of SRR0 and SRR1.
*/
nonrecoverable:
lis r10,exc_exit_restart_end@ha
addi r10,r10,exc_exit_restart_end@l
cmplw r12,r10
bge 3f
lis r11,exc_exit_restart@ha
addi r11,r11,exc_exit_restart@l
cmplw r12,r11
blt 3f
lis r10,ee_restarts@ha
lwz r12,ee_restarts@l(r10)
addi r12,r12,1
stw r12,ee_restarts@l(r10)
mr r12,r11 /* restart at exc_exit_restart */
blr
3: /* OK, we can't recover, kill this process */
/* but the 601 doesn't implement the RI bit, so assume it's OK */
BEGIN_FTR_SECTION
blr
END_FTR_SECTION_IFSET(CPU_FTR_601)
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
lwz r3,_TRAP(r1)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
andi. r0,r3,1
powerpc: Activate CONFIG_THREAD_INFO_IN_TASK This patch activates CONFIG_THREAD_INFO_IN_TASK which moves the thread_info into task_struct. Moving thread_info into task_struct has the following advantages: - It protects thread_info from corruption in the case of stack overflows. - Its address is harder to determine if stack addresses are leaked, making a number of attacks more difficult. This has the following consequences: - thread_info is now located at the beginning of task_struct. - The 'cpu' field is now in task_struct, and only exists when CONFIG_SMP is active. - thread_info doesn't have anymore the 'task' field. This patch: - Removes all recopy of thread_info struct when the stack changes. - Changes the CURRENT_THREAD_INFO() macro to point to current. - Selects CONFIG_THREAD_INFO_IN_TASK. - Modifies raw_smp_processor_id() to get ->cpu from current without including linux/sched.h to avoid circular inclusion and without including asm/asm-offsets.h to avoid symbol names duplication between ASM constants and C constants. - Modifies klp_init_thread_info() to take a task_struct pointer argument. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Reviewed-by: Nicholas Piggin <npiggin@gmail.com> [mpe: Add task_stack.h to livepatch.h to fix build fails] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:08:58 +08:00
beq 5f
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
SAVE_NVGPRS(r1)
rlwinm r3,r3,0,0,30
powerpc: Rename asm offset TRAP to _TRAP for 32-bit ... for consistency with 64-bit. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-28 20:45:25 +08:00
stw r3,_TRAP(r1)
powerpc: Activate CONFIG_THREAD_INFO_IN_TASK This patch activates CONFIG_THREAD_INFO_IN_TASK which moves the thread_info into task_struct. Moving thread_info into task_struct has the following advantages: - It protects thread_info from corruption in the case of stack overflows. - Its address is harder to determine if stack addresses are leaked, making a number of attacks more difficult. This has the following consequences: - thread_info is now located at the beginning of task_struct. - The 'cpu' field is now in task_struct, and only exists when CONFIG_SMP is active. - thread_info doesn't have anymore the 'task' field. This patch: - Removes all recopy of thread_info struct when the stack changes. - Changes the CURRENT_THREAD_INFO() macro to point to current. - Selects CONFIG_THREAD_INFO_IN_TASK. - Modifies raw_smp_processor_id() to get ->cpu from current without including linux/sched.h to avoid circular inclusion and without including asm/asm-offsets.h to avoid symbol names duplication between ASM constants and C constants. - Modifies klp_init_thread_info() to take a task_struct pointer argument. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Reviewed-by: Nicholas Piggin <npiggin@gmail.com> [mpe: Add task_stack.h to livepatch.h to fix build fails] Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-01-31 18:08:58 +08:00
5: mfspr r2,SPRN_SPRG_THREAD
addi r2,r2,-THREAD
tovirt(r2,r2) /* set back r2 to current */
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
4: addi r3,r1,STACK_FRAME_OVERHEAD
powerpc/traps: merge unrecoverable_exception() and nonrecoverable_exception() PPC32 uses nonrecoverable_exception() while PPC64 uses unrecoverable_exception(). Both functions are doing almost the same thing. This patch removes nonrecoverable_exception() Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-09-25 22:10:04 +08:00
bl unrecoverable_exception
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/* shouldn't return */
b 4b
[POWERPC] Fix COMMON symbol warnings We get the following warnings in various ARCH=powerpc builds: WARNING: "ee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "fee_restarts" [arch/powerpc/kernel/built-in] is COMMON symbol WARNING: "htab_hash_searches" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "next_slot" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "mmu_hash_lock" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "primary_pteg_full" [arch/powerpc/mm/built-in] is COMMON symbol WARNING: "global_dbcr0" [arch/powerpc/kernel/built-in] is COMMON symbol Switch to moving local symbols (except mmu_hash_lock which is global) and space directive instead. Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-05-15 06:11:58 +08:00
.section .bss
.align 2
ee_restarts:
.space 4
.previous
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* PROM code for specific machines follows. Put it
* here so it's easy to add arch-specific sections later.
* -- Cort
*/
powerpc: Merge rtas.c into arch/powerpc/kernel This splits arch/ppc64/kernel/rtas.c into arch/powerpc/kernel/rtas.c, which contains generic RTAS functions useful on any CHRP platform, and arch/powerpc/platforms/pseries/rtas-fw.[ch], which contain some pSeries-specific firmware flashing bits. The parts of rtas.c that are to do with pSeries-specific error logging are protected by a new CONFIG_RTAS_ERROR_LOGGING symbol. The inclusion of rtas.o is controlled by the CONFIG_PPC_RTAS symbol, and the relevant platforms select that. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-26 15:05:24 +08:00
#ifdef CONFIG_PPC_RTAS
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
/*
* On CHRP, the Run-Time Abstraction Services (RTAS) have to be
* called with the MMU off.
*/
_GLOBAL(enter_rtas)
stwu r1,-INT_FRAME_SIZE(r1)
mflr r0
stw r0,INT_FRAME_SIZE+4(r1)
[PATCH] powerpc: Cleanup LOADADDR etc. asm macros This patch consolidates the variety of macros used for loading 32 or 64-bit constants in assembler (LOADADDR, LOADBASE, SET_REG_TO_*). The idea is to make the set of macros consistent across 32 and 64 bit and to make it more obvious which is the appropriate one to use in a given situation. The new macros and their semantics are described in the comments in ppc_asm.h. In the process, we change several places that were unnecessarily using immediate loads on ppc64 to use the GOT/TOC. Likewise we cleanup a couple of places where we were clumsily subtracting PAGE_OFFSET with asm instructions to use assemble-time arithmetic or the toreal() macro instead. Signed-off-by: David Gibson <dwg@au1.ibm.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-01-13 11:56:25 +08:00
LOAD_REG_ADDR(r4, rtas)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
lis r6,1f@ha /* physical return address for rtas */
addi r6,r6,1f@l
tophys(r6,r6)
tophys(r7,r1)
powerpc: Merge rtas.c into arch/powerpc/kernel This splits arch/ppc64/kernel/rtas.c into arch/powerpc/kernel/rtas.c, which contains generic RTAS functions useful on any CHRP platform, and arch/powerpc/platforms/pseries/rtas-fw.[ch], which contain some pSeries-specific firmware flashing bits. The parts of rtas.c that are to do with pSeries-specific error logging are protected by a new CONFIG_RTAS_ERROR_LOGGING symbol. The inclusion of rtas.o is controlled by the CONFIG_PPC_RTAS symbol, and the relevant platforms select that. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-26 15:05:24 +08:00
lwz r8,RTASENTRY(r4)
lwz r4,RTASBASE(r4)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mfmsr r9
stw r9,8(r1)
LOAD_MSR_KERNEL(r0,MSR_KERNEL)
SYNC /* disable interrupts so SRR0/1 */
MTMSRD(r0) /* don't get trashed */
li r9,MSR_KERNEL & ~(MSR_IR|MSR_DR)
mtlr r6
powerpc/6xx: Don't use SPRN_SPRG2 for storing stack pointer while in RTAS When calling RTAS, the stack pointer is stored in SPRN_SPRG2 in order to be able to restore it in case of machine check in RTAS. As machine check is not a perfomance critical path, this patch frees SPRN_SPRG2 by using a field in thread struct instead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-02-21 18:37:54 +08:00
stw r7, THREAD + RTAS_SP(r2)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mtspr SPRN_SRR0,r8
mtspr SPRN_SRR1,r9
RFI
1: tophys(r9,r1)
lwz r8,INT_FRAME_SIZE+4(r9) /* get return address */
lwz r9,8(r9) /* original msr value */
addi r1,r1,INT_FRAME_SIZE
li r0,0
powerpc/6xx: Don't use SPRN_SPRG2 for storing stack pointer while in RTAS When calling RTAS, the stack pointer is stored in SPRN_SPRG2 in order to be able to restore it in case of machine check in RTAS. As machine check is not a perfomance critical path, this patch frees SPRN_SPRG2 by using a field in thread struct instead. Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2019-02-21 18:37:54 +08:00
tophys(r7, r2)
stw r0, THREAD + RTAS_SP(r7)
powerpc: Introduce entry_{32,64}.S, misc_{32,64}.S, systbl.S The system call table has been consolidated into systbl.S. We have separate 32-bit and 64-bit versions of entry.S and misc.S since the code is mostly sufficiently different to be not worth merging. There are some common bits that will be extracted in future. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-10 20:36:14 +08:00
mtspr SPRN_SRR0,r8
mtspr SPRN_SRR1,r9
RFI /* return to caller */
.globl machine_check_in_rtas
machine_check_in_rtas:
twi 31,0,0
/* XXX load up BATs and panic */
powerpc: Merge rtas.c into arch/powerpc/kernel This splits arch/ppc64/kernel/rtas.c into arch/powerpc/kernel/rtas.c, which contains generic RTAS functions useful on any CHRP platform, and arch/powerpc/platforms/pseries/rtas-fw.[ch], which contain some pSeries-specific firmware flashing bits. The parts of rtas.c that are to do with pSeries-specific error logging are protected by a new CONFIG_RTAS_ERROR_LOGGING symbol. The inclusion of rtas.o is controlled by the CONFIG_PPC_RTAS symbol, and the relevant platforms select that. Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-10-26 15:05:24 +08:00
#endif /* CONFIG_PPC_RTAS */