openkylin
/
linux
mirror of https://gitee.com/openkylin/linux.git
9
0
Fork 0
Code Issues Projects Releases Wiki Activity
linux/arch/powerpc/kernel/tm.S

482 lines
12 KiB
ArmAsm
Raw Normal View History

powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/*
* Transactional memory support routines to reclaim and recheckpoint
* transactional process state.
*
* Copyright 2012 Matt Evans & Michael Neuling, IBM Corporation.
*/
#include <asm/asm-offsets.h>
#include <asm/ppc_asm.h>
#include <asm/ppc-opcode.h>
#include <asm/ptrace.h>
#include <asm/reg.h>
powerpc/tm: Add checking to treclaim/trechkpt If we do a treclaim and we are not in TM suspend mode, it results in a TM bad thing (ie. a 0x700 program check). Similarly if we do a trechkpt and we have an active transaction or TEXASR Failure Summary (FS) is not set, we also take a TM bad thing. This should never happen, but if it does (ie. a kernel bug), the cause is almost impossible to debug as the GPR state is mostly userspace and hence we don't get a call chain. This adds some checks in these cases case a BUG_ON() (in asm) in case we ever hit these cases. It moves the register saving around to preserve r1 till later also. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-03-28 13:40:34 +08:00
#include <asm/bug.h>
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
#ifdef CONFIG_VSX
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
/* See fpu.S, this is borrowed from there */
#define __SAVE_32FPRS_VSRS(n,c,base) \
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
BEGIN_FTR_SECTION \
b 2f; \
END_FTR_SECTION_IFSET(CPU_FTR_VSX); \
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
SAVE_32FPRS(n,base); \
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
b 3f; \
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
2: SAVE_32VSRS(n,c,base); \
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
3:
#define __REST_32FPRS_VSRS(n,c,base) \
BEGIN_FTR_SECTION \
b 2f; \
END_FTR_SECTION_IFSET(CPU_FTR_VSX); \
REST_32FPRS(n,base); \
b 3f; \
2: REST_32VSRS(n,c,base); \
3:
#else
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
#define __SAVE_32FPRS_VSRS(n,c,base) SAVE_32FPRS(n, base)
#define __REST_32FPRS_VSRS(n,c,base) REST_32FPRS(n, base)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
#endif
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
#define SAVE_32FPRS_VSRS(n,c,base) \
__SAVE_32FPRS_VSRS(n,__REG_##c,__REG_##base)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
#define REST_32FPRS_VSRS(n,c,base) \
__REST_32FPRS_VSRS(n,__REG_##c,__REG_##base)
/* Stack frame offsets for local variables. */
#define TM_FRAME_L0 TM_FRAME_SIZE-16
#define TM_FRAME_L1 TM_FRAME_SIZE-8
/* In order to access the TM SPRs, TM must be enabled. So, do so: */
_GLOBAL(tm_enable)
mfmsr r4
li r3, MSR_TM >> 32
sldi r3, r3, 32
and. r0, r4, r3
bne 1f
or r4, r4, r3
mtmsrd r4
1: blr
_GLOBAL(tm_save_sprs)
mfspr r0, SPRN_TFHAR
std r0, THREAD_TM_TFHAR(r3)
mfspr r0, SPRN_TEXASR
std r0, THREAD_TM_TEXASR(r3)
mfspr r0, SPRN_TFIAR
std r0, THREAD_TM_TFIAR(r3)
blr
_GLOBAL(tm_restore_sprs)
ld r0, THREAD_TM_TFHAR(r3)
mtspr SPRN_TFHAR, r0
ld r0, THREAD_TM_TEXASR(r3)
mtspr SPRN_TEXASR, r0
ld r0, THREAD_TM_TFIAR(r3)
mtspr SPRN_TFIAR, r0
blr
/* Passed an 8-bit failure cause as first argument. */
_GLOBAL(tm_abort)
TABORT(R3)
blr
/* void tm_reclaim(struct thread_struct *thread,
* unsigned long orig_msr,
* uint8_t cause)
*
* - Performs a full reclaim. This destroys outstanding
* transactions and updates thread->regs.tm_ckpt_* with the
* original checkpointed state. Note that thread->regs is
* unchanged.
* - FP regs are written back to thread->transact_fpr before
* reclaiming. These are the transactional (current) versions.
*
* Purpose is to both abort transactions of, and preserve the state of,
* a transactions at a context switch. We preserve/restore both sets of process
* state to restore them when the thread's scheduled again. We continue in
* userland as though nothing happened, but when the transaction is resumed
* they will abort back to the checkpointed state we save out here.
*
* Call with IRQs off, stacks get all out of sync for some periods in here!
*/
_GLOBAL(tm_reclaim)
mfcr r6
mflr r0
powerpc: Use 32 bit loads and stores when operating on condition register values The condition register (CR) is a 32 bit quantity so we should use 32 bit loads and stores. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-10-15 11:36:31 +08:00
stw r6, 8(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
std r0, 16(r1)
powerpc/tm: Fix GOT save offset for ABIv2 The r2 TOC/GOT save offset is 40 on ABIv1 and 24 on ABIv2. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:52:17 +08:00
std r2, STK_GOT(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
stdu r1, -TM_FRAME_SIZE(r1)
/* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD]. */
powerpc/tm: Use STK_PARAM Get rid of the tm specific STACK_PARAM and use STK_PARAM Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:48:44 +08:00
std r3, STK_PARAM(R3)(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
SAVE_NVGPRS(r1)
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
/* We need to setup MSR for VSX register save instructions. Here we
* also clear the MSR RI since when we do the treclaim, we won't have a
* valid kernel pointer for a while. We clear RI here as it avoids
* adding another mtmsr closer to the treclaim. This makes the region
* maked as non-recoverable wider than it needs to be but it saves on
* inserting another mtmsrd later.
*/
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mfmsr r14
mr r15, r14
ori r15, r15, MSR_FP
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
li r16, MSR_RI
powerpc/tm: Turn interrupts hard off in tm_reclaim() We can't take IRQs in tm_reclaim as we might have a bogus r13 and r1. This turns IRQs hard off in this function. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-10-02 15:15:15 +08:00
ori r16, r16, MSR_EE /* IRQs hard off */
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
andc r15, r15, r16
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
oris r15, r15, MSR_VEC@h
#ifdef CONFIG_VSX
BEGIN_FTR_SECTION
oris r15,r15, MSR_VSX@h
END_FTR_SECTION_IFSET(CPU_FTR_VSX)
#endif
mtmsrd r15
std r14, TM_FRAME_L0(r1)
/* Stash the stack pointer away for use after reclaim */
std r1, PACAR1(r13)
/* ******************** FPR/VR/VSRs ************
* Before reclaiming, capture the current/transactional FPR/VR
* versions /if used/.
*
* (If VSX used, FP and VMX are implied. Or, we don't need to look
* at MSR.VSX as copying FP regs if .FP, vector regs if .VMX covers it.)
*
* We're passed the thread's MSR as parameter 2.
*
* We enabled VEC/FP/VSX in the msr above, so we can execute these
* instructions!
*/
andis. r0, r4, MSR_VEC@h
beq dont_backup_vec
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
addi r7, r3, THREAD_TRANSACT_VRSTATE
SAVE_32VRS(0, r6, r7) /* r6 scratch, r7 transact vr state */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mfvscr vr0
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
li r6, VRSTATE_VSCR
stvx vr0, r7, r6
powerpc: Fix VRSAVE handling Since 2002, the kernel has not saved VRSAVE on exception entry and restored it on exit; rather, VRSAVE gets context-switched in _switch. This means that when executing in process context in the kernel, the userspace VRSAVE value is live in the VRSAVE register. However, the signal code assumes that current->thread.vrsave holds the current VRSAVE value, which is incorrect. Therefore, this commit changes it to use the actual VRSAVE register instead. (It still uses current->thread.vrsave as a temporary location to store it in, as __get_user and __put_user can only transfer to/from a variable, not an SPR.) This also modifies the transactional memory code to save and restore VRSAVE regardless of whether VMX is enabled in the MSR. This is because accesses to VRSAVE are not controlled by the MSR.VEC bit, but can happen at any time. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-05 12:13:16 +08:00
dont_backup_vec:
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mfspr r0, SPRN_VRSAVE
std r0, THREAD_TRANSACT_VRSAVE(r3)
andi. r0, r4, MSR_FP
beq dont_backup_fp
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
addi r7, r3, THREAD_TRANSACT_FPSTATE
SAVE_32FPRS_VSRS(0, R6, R7) /* r6 scratch, r7 transact fp state */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mffs fr0
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
stfd fr0,FPSTATE_FPSCR(r7)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
dont_backup_fp:
powerpc/tm: Add checking to treclaim/trechkpt If we do a treclaim and we are not in TM suspend mode, it results in a TM bad thing (ie. a 0x700 program check). Similarly if we do a trechkpt and we have an active transaction or TEXASR Failure Summary (FS) is not set, we also take a TM bad thing. This should never happen, but if it does (ie. a kernel bug), the cause is almost impossible to debug as the GPR state is mostly userspace and hence we don't get a call chain. This adds some checks in these cases case a BUG_ON() (in asm) in case we ever hit these cases. It moves the register saving around to preserve r1 till later also. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-03-28 13:40:34 +08:00
/* Do sanity check on MSR to make sure we are suspended */
li r7, (MSR_TS_S)@higher
srdi r6, r14, 32
and r6, r6, r7
1: tdeqi r6, 0
EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* The moment we treclaim, ALL of our GPRs will switch
* to user register state. (FPRs, CCR etc. also!)
* Use an sprg and a tm_scratch in the PACA to shuffle.
*/
TRECLAIM(R5) /* Cause in r5 */
/* ******************** GPRs ******************** */
/* Stash the checkpointed r13 away in the scratch SPR and get the real
* paca
*/
SET_SCRATCH0(r13)
GET_PACA(r13)
/* Stash the checkpointed r1 away in paca tm_scratch and get the real
* stack pointer back
*/
std r1, PACATMSCRATCH(r13)
ld r1, PACAR1(r13)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* Store the PPR in r11 and reset to decent value */
std r11, GPR11(r1) /* Temporary stash */
mfspr r11, SPRN_PPR
HMT_MEDIUM
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* Now get some more GPRS free */
std r7, GPR7(r1) /* Temporary stash */
std r12, GPR12(r1) /* '' '' '' */
powerpc/tm: Use STK_PARAM Get rid of the tm specific STACK_PARAM and use STK_PARAM Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:48:44 +08:00
ld r12, STK_PARAM(R3)(r1) /* Param 0, thread_struct * */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
std r11, THREAD_TM_PPR(r12) /* Store PPR and free r11 */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
addi r7, r12, PT_CKPT_REGS /* Thread's ckpt_regs */
/* Make r7 look like an exception frame so that we
* can use the neat GPRx(n) macros. r7 is NOT a pt_regs ptr!
*/
subi r7, r7, STACK_FRAME_OVERHEAD
/* Sync the userland GPRs 2-12, 14-31 to thread->regs: */
SAVE_GPR(0, r7) /* user r0 */
SAVE_GPR(2, r7) /* user r2 */
SAVE_4GPRS(3, r7) /* user r3-r6 */
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
SAVE_GPR(8, r7) /* user r8 */
SAVE_GPR(9, r7) /* user r9 */
SAVE_GPR(10, r7) /* user r10 */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
ld r3, PACATMSCRATCH(r13) /* user r1 */
ld r4, GPR7(r1) /* user r7 */
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
ld r5, GPR11(r1) /* user r11 */
ld r6, GPR12(r1) /* user r12 */
GET_SCRATCH0(8) /* user r13 */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
std r3, GPR1(r7)
std r4, GPR7(r7)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
std r5, GPR11(r7)
std r6, GPR12(r7)
std r8, GPR13(r7)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
SAVE_NVGPRS(r7) /* user r14-r31 */
/* ******************** NIP ******************** */
mfspr r3, SPRN_TFHAR
std r3, _NIP(r7) /* Returns to failhandler */
/* The checkpointed NIP is ignored when rescheduling/rechkpting,
* but is used in signal return to 'wind back' to the abort handler.
*/
/* ******************** CR,LR,CCR,MSR ********** */
mfctr r3
mflr r4
mfcr r5
mfxer r6
std r3, _CTR(r7)
std r4, _LINK(r7)
std r5, _CCR(r7)
std r6, _XER(r7)
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* ******************** TAR, DSCR ********** */
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
mfspr r3, SPRN_TAR
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
mfspr r4, SPRN_DSCR
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
std r3, THREAD_TM_TAR(r12)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
std r4, THREAD_TM_DSCR(r12)
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* MSR and flags: We don't change CRs, and we don't need to alter
* MSR.
*/
/* TM regs, incl TEXASR -- these live in thread_struct. Note they've
* been updated by the treclaim, to explain to userland the failure
* cause (aborted).
*/
mfspr r0, SPRN_TEXASR
mfspr r3, SPRN_TFHAR
mfspr r4, SPRN_TFIAR
std r0, THREAD_TM_TEXASR(r12)
std r3, THREAD_TM_TFHAR(r12)
std r4, THREAD_TM_TFIAR(r12)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* AMR is checkpointed too, but is unsupported by Linux. */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* Restore original MSR/IRQ state & clear TM mode */
ld r14, TM_FRAME_L0(r1) /* Orig MSR */
li r15, 0
rldimi r14, r15, MSR_TS_LG, (63-MSR_TS_LG)-1
mtmsrd r14
REST_NVGPRS(r1)
addi r1, r1, TM_FRAME_SIZE
powerpc: Use 32 bit loads and stores when operating on condition register values The condition register (CR) is a 32 bit quantity so we should use 32 bit loads and stores. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-10-15 11:36:31 +08:00
lwz r4, 8(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
ld r0, 16(r1)
mtcr r4
mtlr r0
powerpc/tm: Fix GOT save offset for ABIv2 The r2 TOC/GOT save offset is 40 on ABIv1 and 24 on ABIv2. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:52:17 +08:00
ld r2, STK_GOT(r1)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
powerpc: Fix regression of per-CPU DSCR setting Since commit "efcac65 powerpc: Per process DSCR + some fixes (try#4)" it is no longer possible to set the DSCR on a per-CPU basis. The old behaviour was to minipulate the DSCR SPR directly but this is no longer sufficient: the value is quickly overwritten by context switching. This patch stores the per-CPU DSCR value in a kernel variable rather than directly in the SPR and it is used whenever a process has not set the DSCR itself. The sysfs interface (/sys/devices/system/cpu/cpuN/dscr) is unchanged. Writes to the old global default (/sys/devices/system/cpu/dscr_default) now set all of the per-CPU values and reads return the last written value. The new per-CPU default is added to the paca_struct and is used everywhere outside of sysfs.c instead of the old global default. Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-05-21 14:32:38 +08:00
/* Load CPU's default DSCR */
ld r0, PACA_DSCR(r13)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
mtspr SPRN_DSCR, r0
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
blr
/* void tm_recheckpoint(struct thread_struct *thread,
* unsigned long orig_msr)
* - Restore the checkpointed register state saved by tm_reclaim
* when we switch_to a process.
*
* Call with IRQs off, stacks get all out of sync for
* some periods in here!
*/
powerpc/tm: Disable IRQ in tm_recheckpoint We can't take an IRQ when we're about to do a trechkpt as our GPR state is set to user GPR values. We've hit this when running some IBM Java stress tests in the lab resulting in the following dump: cpu 0x3f: Vector: 700 (Program Check) at [c000000007eb3d40] pc: c000000000050074: restore_gprs+0xc0/0x148 lr: 00000000b52a8184 sp: ac57d360 msr: 8000000100201030 current = 0xc00000002c500000 paca = 0xc000000007dbfc00 softe: 0 irq_happened: 0x00 pid = 34535, comm = Pooled Thread # R00 = 00000000b52a8184 R16 = 00000000b3e48fda R01 = 00000000ac57d360 R17 = 00000000ade79bd8 R02 = 00000000ac586930 R18 = 000000000fac9bcc R03 = 00000000ade60000 R19 = 00000000ac57f930 R04 = 00000000f6624918 R20 = 00000000ade79be8 R05 = 00000000f663f238 R21 = 00000000ac218a54 R06 = 0000000000000002 R22 = 000000000f956280 R07 = 0000000000000008 R23 = 000000000000007e R08 = 000000000000000a R24 = 000000000000000c R09 = 00000000b6e69160 R25 = 00000000b424cf00 R10 = 0000000000000181 R26 = 00000000f66256d4 R11 = 000000000f365ec0 R27 = 00000000b6fdcdd0 R12 = 00000000f66400f0 R28 = 0000000000000001 R13 = 00000000ada71900 R29 = 00000000ade5a300 R14 = 00000000ac2185a8 R30 = 00000000f663f238 R15 = 0000000000000004 R31 = 00000000f6624918 pc = c000000000050074 restore_gprs+0xc0/0x148 cfar= c00000000004fe28 dont_restore_vec+0x1c/0x1a4 lr = 00000000b52a8184 msr = 8000000100201030 cr = 24804888 ctr = 0000000000000000 xer = 0000000000000000 trap = 700 This moves tm_recheckpoint to a C function and moves the tm_restore_sprs into that function. It then adds IRQ disabling over the trechkpt critical section. It also sets the TEXASR FS in the signals code to ensure this is never set now that we explictly write the TM sprs in tm_recheckpoint. Signed-off-by: Michael Neuling <mikey@neuling.org> cc: stable@vger.kernel.org Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-04-04 17:19:48 +08:00
_GLOBAL(__tm_recheckpoint)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mfcr r5
mflr r0
powerpc: Use 32 bit loads and stores when operating on condition register values The condition register (CR) is a 32 bit quantity so we should use 32 bit loads and stores. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-10-15 11:36:31 +08:00
stw r5, 8(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
std r0, 16(r1)
powerpc/tm: Fix GOT save offset for ABIv2 The r2 TOC/GOT save offset is 40 on ABIv1 and 24 on ABIv2. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:52:17 +08:00
std r2, STK_GOT(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
stdu r1, -TM_FRAME_SIZE(r1)
/* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD].
* This is used for backing up the NVGPRs:
*/
SAVE_NVGPRS(r1)
/* Load complete register state from ts_ckpt* registers */
addi r7, r3, PT_CKPT_REGS /* Thread's ckpt_regs */
/* Make r7 look like an exception frame so that we
* can use the neat GPRx(n) macros. r7 is now NOT a pt_regs ptr!
*/
subi r7, r7, STACK_FRAME_OVERHEAD
SET_SCRATCH0(r1)
mfmsr r6
/* R4 = original MSR to indicate whether thread used FP/Vector etc. */
/* Enable FP/vec in MSR if necessary! */
lis r5, MSR_VEC@h
ori r5, r5, MSR_FP
and. r5, r4, r5
beq restore_gprs /* if neither, skip both */
#ifdef CONFIG_VSX
BEGIN_FTR_SECTION
oris r5, r5, MSR_VSX@h
END_FTR_SECTION_IFSET(CPU_FTR_VSX)
#endif
or r5, r6, r5 /* Set MSR.FP+.VSX/.VEC */
mtmsr r5
powerpc: fix compiling CONFIG_PPC_TRANSACTIONAL_MEM when CONFIG_ALTIVEC=n We can't compile a kernel with CONFIG_ALTIVEC=n when CONFIG_PPC_TRANSACTIONAL_MEM=y. We currently get: arch/powerpc/kernel/tm.S:320: Error: unsupported relocation against THREAD_VSCR arch/powerpc/kernel/tm.S:323: Error: unsupported relocation against THREAD_VR0 arch/powerpc/kernel/tm.S:323: Error: unsupported relocation against THREAD_VR0 etc. The below fixes this with a sprinkling of #ifdefs. This was found by mpe with kisskb: http://kisskb.ellerman.id.au/kisskb/buildresult/8539442/ Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
2013-04-09 14:18:55 +08:00
#ifdef CONFIG_ALTIVEC
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* FP and VEC registers: These are recheckpointed from thread.fpr[]
* and thread.vr[] respectively. The thread.transact_fpr[] version
* is more modern, and will be loaded subsequently by any FPUnavailable
* trap.
*/
andis. r0, r4, MSR_VEC@h
beq dont_restore_vec
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
addi r8, r3, THREAD_VRSTATE
li r5, VRSTATE_VSCR
lvx vr0, r8, r5
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
mtvscr vr0
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
REST_32VRS(0, r5, r8) /* r5 scratch, r8 ptr */
powerpc: Fix VRSAVE handling Since 2002, the kernel has not saved VRSAVE on exception entry and restored it on exit; rather, VRSAVE gets context-switched in _switch. This means that when executing in process context in the kernel, the userspace VRSAVE value is live in the VRSAVE register. However, the signal code assumes that current->thread.vrsave holds the current VRSAVE value, which is incorrect. Therefore, this commit changes it to use the actual VRSAVE register instead. (It still uses current->thread.vrsave as a temporary location to store it in, as __get_user and __put_user can only transfer to/from a variable, not an SPR.) This also modifies the transactional memory code to save and restore VRSAVE regardless of whether VMX is enabled in the MSR. This is because accesses to VRSAVE are not controlled by the MSR.VEC bit, but can happen at any time. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-05 12:13:16 +08:00
dont_restore_vec:
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
ld r5, THREAD_VRSAVE(r3)
mtspr SPRN_VRSAVE, r5
powerpc: fix compiling CONFIG_PPC_TRANSACTIONAL_MEM when CONFIG_ALTIVEC=n We can't compile a kernel with CONFIG_ALTIVEC=n when CONFIG_PPC_TRANSACTIONAL_MEM=y. We currently get: arch/powerpc/kernel/tm.S:320: Error: unsupported relocation against THREAD_VSCR arch/powerpc/kernel/tm.S:323: Error: unsupported relocation against THREAD_VR0 arch/powerpc/kernel/tm.S:323: Error: unsupported relocation against THREAD_VR0 etc. The below fixes this with a sprinkling of #ifdefs. This was found by mpe with kisskb: http://kisskb.ellerman.id.au/kisskb/buildresult/8539442/ Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
2013-04-09 14:18:55 +08:00
#endif
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
andi. r0, r4, MSR_FP
beq dont_restore_fp
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
addi r8, r3, THREAD_FPSTATE
lfd fr0, FPSTATE_FPSCR(r8)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
MTFSF_L(fr0)
powerpc: Put FP/VSX and VR state into structures This creates new 'thread_fp_state' and 'thread_vr_state' structures to store FP/VSX state (including FPSCR) and Altivec/VSX state (including VSCR), and uses them in the thread_struct. In the thread_fp_state, the FPRs and VSRs are represented as u64 rather than double, since we rarely perform floating-point computations on the values, and this will enable the structures to be used in KVM code as well. Similarly FPSCR is now a u64 rather than a structure of two 32-bit values. This takes the offsets out of the macros such as SAVE_32FPRS, REST_32FPRS, etc. This enables the same macros to be used for normal and transactional state, enabling us to delete the transactional versions of the macros. This also removes the unused do_load_up_fpu and do_load_up_altivec, which were in fact buggy since they didn't create large enough stack frames to account for the fact that load_up_fpu and load_up_altivec are not designed to be called from C and assume that their caller's stack frame is an interrupt frame. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-10 18:20:42 +08:00
REST_32FPRS_VSRS(0, R4, R8)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
dont_restore_fp:
mtmsr r6 /* FP/Vec off again! */
restore_gprs:
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* ******************** CR,LR,CCR,MSR ********** */
ld r4, _CTR(r7)
ld r5, _LINK(r7)
ld r8, _XER(r7)
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
mtctr r4
mtlr r5
mtxer r8
powerpc/tm: Fix context switching TAR, PPR and DSCR SPRs If a transaction is rolled back, the Target Address Register (TAR), Processor Priority Register (PPR) and Data Stream Control Register (DSCR) should be restored to the checkpointed values before the transaction began. Any changes to these SPRs inside the transaction should not be visible in the abort handler. Currently Linux doesn't save or restore the checkpointed TAR, PPR or DSCR. If we preempt a processes inside a transaction which has modified any of these, on process restore, that same transaction may be aborted we but we won't see the checkpointed versions of these SPRs. This adds checkpointed versions of these SPRs to the thread_struct and adds the save/restore of these three SPRs to the treclaim/trechkpt code. Without this if any of these SPRs are modified during a transaction, users may incorrectly see a speculated SPR value even if the transaction is aborted. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> [v3.10] Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-08-09 15:29:31 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* ******************** TAR ******************** */
ld r4, THREAD_TM_TAR(r3)
mtspr SPRN_TAR, r4
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* Load up the PPR and DSCR in GPRs only at this stage */
ld r5, THREAD_TM_DSCR(r3)
ld r6, THREAD_TM_PPR(r3)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
/* Clear the MSR RI since we are about to change R1. EE is already off
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
*/
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
li r4, 0
mtmsrd r4, 1
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
powerpc/tm: Add checking to treclaim/trechkpt If we do a treclaim and we are not in TM suspend mode, it results in a TM bad thing (ie. a 0x700 program check). Similarly if we do a trechkpt and we have an active transaction or TEXASR Failure Summary (FS) is not set, we also take a TM bad thing. This should never happen, but if it does (ie. a kernel bug), the cause is almost impossible to debug as the GPR state is mostly userspace and hence we don't get a call chain. This adds some checks in these cases case a BUG_ON() (in asm) in case we ever hit these cases. It moves the register saving around to preserve r1 till later also. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-03-28 13:40:34 +08:00
REST_GPR(0, r7) /* GPR0 */
REST_2GPRS(2, r7) /* GPR2-3 */
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
REST_GPR(4, r7) /* GPR4 */
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
REST_4GPRS(8, r7) /* GPR8-11 */
REST_2GPRS(12, r7) /* GPR12-13 */
REST_NVGPRS(r7) /* GPR14-31 */
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
/* Load up PPR and DSCR here so we don't run with user values for long
*/
mtspr SPRN_DSCR, r5
mtspr SPRN_PPR, r6
powerpc/tm: Add checking to treclaim/trechkpt If we do a treclaim and we are not in TM suspend mode, it results in a TM bad thing (ie. a 0x700 program check). Similarly if we do a trechkpt and we have an active transaction or TEXASR Failure Summary (FS) is not set, we also take a TM bad thing. This should never happen, but if it does (ie. a kernel bug), the cause is almost impossible to debug as the GPR state is mostly userspace and hence we don't get a call chain. This adds some checks in these cases case a BUG_ON() (in asm) in case we ever hit these cases. It moves the register saving around to preserve r1 till later also. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-03-28 13:40:34 +08:00
/* Do final sanity check on TEXASR to make sure FS is set. Do this
* here before we load up the userspace r1 so any bugs we hit will get
* a call chain */
mfspr r5, SPRN_TEXASR
srdi r5, r5, 16
li r6, (TEXASR_FS)@h
and r6, r6, r5
1: tdeqi r6, 0
EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0
/* Do final sanity check on MSR to make sure we are not transactional
* or suspended
*/
mfmsr r6
li r5, (MSR_TS_MASK)@higher
srdi r6, r6, 32
and r6, r6, r5
1: tdnei r6, 0
EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0
/* Restore CR */
ld r6, _CCR(r7)
mtcr r6
REST_GPR(1, r7) /* GPR1 */
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
REST_GPR(5, r7) /* GPR5-7 */
REST_GPR(6, r7)
ld r7, GPR7(r7)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* Commit register state as checkpointed state: */
TRECHKPT
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
HMT_MEDIUM
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
/* Our transactional state has now changed.
*
* Now just get out of here. Transactional (current) state will be
* updated once restore is called on the return path in the _switch-ed
* -to process.
*/
GET_PACA(r13)
GET_SCRATCH0(r1)
powerpc/tm: Clear MSR RI in non-recoverable TM code When we treclaim and trecheckpoint there's an unavoidable period when r1 will not be a valid kernel stack pointer. This patch clears the MSR recoverable interrupt (RI) bit over these regions to indicate we have an invalid kernel stack pointer. For treclaim, the region over which we clear MSR RI is larger than required to avoid the need for an extra costly mtmsrd. Thanks to Paulus for suggesting this change. Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-06-28 16:17:09 +08:00
/* R1 is restored, so we are recoverable again. EE is still off */
li r4, MSR_RI
mtmsrd r4, 1
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
REST_NVGPRS(r1)
addi r1, r1, TM_FRAME_SIZE
powerpc: Use 32 bit loads and stores when operating on condition register values The condition register (CR) is a 32 bit quantity so we should use 32 bit loads and stores. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-10-15 11:36:31 +08:00
lwz r4, 8(r1)
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
ld r0, 16(r1)
mtcr r4
mtlr r0
powerpc/tm: Fix GOT save offset for ABIv2 The r2 TOC/GOT save offset is 40 on ABIv1 and 24 on ABIv2. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-03-10 07:52:17 +08:00
ld r2, STK_GOT(r1)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
powerpc: Fix regression of per-CPU DSCR setting Since commit "efcac65 powerpc: Per process DSCR + some fixes (try#4)" it is no longer possible to set the DSCR on a per-CPU basis. The old behaviour was to minipulate the DSCR SPR directly but this is no longer sufficient: the value is quickly overwritten by context switching. This patch stores the per-CPU DSCR value in a kernel variable rather than directly in the SPR and it is used whenever a process has not set the DSCR itself. The sysfs interface (/sys/devices/system/cpu/cpuN/dscr) is unchanged. Writes to the old global default (/sys/devices/system/cpu/dscr_default) now set all of the per-CPU values and reads return the last written value. The new per-CPU default is added to the paca_struct and is used everywhere outside of sysfs.c instead of the old global default. Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2014-05-21 14:32:38 +08:00
/* Load CPU's default DSCR */
ld r0, PACA_DSCR(r13)
powerpc/tm: Switch out userspace PPR and DSCR sooner When we do a treclaim or trecheckpoint we end up running with userspace PPR and DSCR values. Currently we don't do anything special to avoid running with user values which could cause a severe performance degradation. This patch moves the PPR and DSCR save and restore around treclaim and trecheckpoint so that we run with user values for a much shorter period. More care is taken with the PPR as it's impact is greater than the DSCR. This is similar to user exceptions, where we run HTM_MEDIUM early to ensure that we don't run with a userspace PPR values in the kernel. Signed-off-by: Michael Neuling <mikey@neuling.org> Cc: <stable@vger.kernel.org> # 3.9+ Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-09-26 11:29:09 +08:00
mtspr SPRN_DSCR, r0
powerpc: Add helper functions for transactional memory context switching Here we add the helper functions to be used when context switching. These allow us to fully reclaim and recheckpoint a transaction. We introduce a new paca field called tm_scratch to help us store away register values when doing the low level tm reclaim register save. Signed-off-by: Matt Evans <matt@ozlabs.org> Signed-off-by: Michael Neuling <mikey@neuling.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2013-02-14 00:21:35 +08:00
blr
/* ****************************************************************** */