openkylin
/
linux
mirror of https://gitee.com/openkylin/linux.git
9
0
Fork 0
Code Issues Projects Releases Wiki Activity
linux/arch/mips/lib/memset.S

329 lines
7.6 KiB
ArmAsm
Raw Normal View History

Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
/*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
*
* Copyright (C) 1998, 1999, 2000 by Ralf Baechle
* Copyright (C) 1999, 2000 Silicon Graphics, Inc.
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
* Copyright (C) 2007 by Maciej W. Rozycki
* Copyright (C) 2011, 2012 MIPS Technologies, Inc.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
*/
#include <asm/asm.h>
kbuild: mips use generic asm-offsets.h support Removed obsolete stuff from arch makefile. mips had a special rule for generating asm-offsets.h so preserved it using an architecture specific hook in top-level Kbuild file. Renamed .h file to asm-offsets.h Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
2005-09-10 04:32:31 +08:00
#include <asm/asm-offsets.h>
MIPS: Export memcpy & memset functions alongside their definitions Now that EXPORT_SYMBOL can be used from assembly source, move the EXPORT_SYMBOL invocations for the memcpy & memset functions & variants thereof to be alongside their definitions. Signed-off-by: Paul Burton <paul.burton@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/14514/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-11-07 19:14:15 +08:00
#include <asm/export.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#include <asm/regdef.h>
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
#if LONGSIZE == 4
#define LONG_S_L swl
#define LONG_S_R swr
#else
#define LONG_S_L sdl
#define LONG_S_R sdr
#endif
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
#ifdef CONFIG_CPU_MICROMIPS
#define STORSIZE (LONGSIZE * 2)
#define STORMASK (STORSIZE - 1)
#define FILL64RG t8
#define FILLPTRG t7
#undef LONG_S
#define LONG_S LONG_SP
#else
#define STORSIZE LONGSIZE
#define STORMASK LONGMASK
#define FILL64RG a1
#define FILLPTRG t0
#endif
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
#define LEGACY_MODE 1
#define EVA_MODE 2
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
/*
* No need to protect it with EVA #ifdefery. The generated block of code
* will never be assembled if EVA is not enabled.
*/
#define __EVAFY(insn, reg, addr) __BUILD_EVA_INSN(insn##e, reg, addr)
#define ___BUILD_EVA_INSN(insn, reg, addr) __EVAFY(insn, reg, addr)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#define EX(insn,reg,addr,handler) \
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
.if \mode == LEGACY_MODE; \
9: insn reg, addr; \
.else; \
9: ___BUILD_EVA_INSN(insn, reg, addr); \
.endif; \
MIPS: Whitespace cleanup. Having received another series of whitespace patches I decided to do this once and for all rather than dealing with this kind of patches trickling in forever. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2013-01-22 19:59:30 +08:00
.section __ex_table,"a"; \
PTR 9b, handler; \
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
.previous
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
.macro f_fill64 dst, offset, val, fixup, mode
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
EX(LONG_S, \val, (\offset + 0 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 1 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 2 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 3 * STORSIZE)(\dst), \fixup)
#if ((defined(CONFIG_CPU_MICROMIPS) && (LONGSIZE == 4)) || !defined(CONFIG_CPU_MICROMIPS))
EX(LONG_S, \val, (\offset + 4 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 5 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 6 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 7 * STORSIZE)(\dst), \fixup)
#endif
#if (!defined(CONFIG_CPU_MICROMIPS) && (LONGSIZE == 4))
EX(LONG_S, \val, (\offset + 8 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 9 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 10 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 11 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 12 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 13 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 14 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 15 * STORSIZE)(\dst), \fixup)
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
.endm
.align 5
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
/*
* Macro to generate the __bzero{,_user} symbol
* Arguments:
* mode: LEGACY_MODE or EVA_MODE
*/
.macro __BUILD_BZERO mode
/* Initialize __memset if this is the first time we call this macro */
.ifnotdef __memset
.set __memset, 1
.hidden __memset /* Make sure it does not leak */
.endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
sltiu t0, a2, STORSIZE /* very small region? */
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set noreorder
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
bnez t0, .Lsmall_memset\@
MIPS: memset.S: Reinstate delay slot indentation Assembly language within the MIPS kernel conventionally indents instructions which are in a branch delay slot to make them easier to see. Commit 8483b14aaa81 ("MIPS: lib: memset: Whitespace fixes") rather inexplicably removed all of these indentations from memset.S. Reinstate the convention for all instructions in a branch delay slot. This effectively reverts the above commit, plus other locations introduced with MIPSR6 support. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/19111/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 23:40:02 +08:00
andi t0, a0, STORMASK /* aligned? */
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set reorder
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
#ifdef CONFIG_CPU_MICROMIPS
move t8, a1 /* used by 'swp' instruction */
move t9, a1
#endif
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set noreorder
[MIPS] R4000/R4400 daddiu erratum workaround This complements the generic R4000/R4400 errata workaround code and adds bits for the daddiu problem. In most places it just modifies handwritten assembly code so that the assembler is allowed to use a temporary register as daddiu may now be treated as a macro that expands to a sequence of li and daddu. It is the AT register or, where AT is unavailable or used explicitly for another purpose, an explicitly-named register is selected, using the .set at=<reg> feature added recently to gas. This feature is only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the workaround remains disabled, the required version of binutils stays unchanged. Similarly, daddiu instructions put in branch delay slots in noreorder fragments are now taken out of them and the assembler is allowed to reorder them itself as possible (which it does making the whole idea of scheduling them into delay slots manually questionable). Also in the very few places where such a simple conversion was not possible, a handcoded longer sequence is implemented. Other than that there are changes to code responsible for building the TLB fault and page clear/copy handlers to avoid daddiu as appropriate. These are only effective if the erratum is verified to be present at the run time. Finally there is a trivial update to __delay(), because it uses daddiu in a branch delay slot. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-10-23 19:43:25 +08:00
#ifndef CONFIG_CPU_DADDI_WORKAROUNDS
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
beqz t0, 1f
MIPS: memset.S: Reinstate delay slot indentation Assembly language within the MIPS kernel conventionally indents instructions which are in a branch delay slot to make them easier to see. Commit 8483b14aaa81 ("MIPS: lib: memset: Whitespace fixes") rather inexplicably removed all of these indentations from memset.S. Reinstate the convention for all instructions in a branch delay slot. This effectively reverts the above commit, plus other locations introduced with MIPSR6 support. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/19111/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 23:40:02 +08:00
PTR_SUBU t0, STORSIZE /* alignment in bytes */
[MIPS] R4000/R4400 daddiu erratum workaround This complements the generic R4000/R4400 errata workaround code and adds bits for the daddiu problem. In most places it just modifies handwritten assembly code so that the assembler is allowed to use a temporary register as daddiu may now be treated as a macro that expands to a sequence of li and daddu. It is the AT register or, where AT is unavailable or used explicitly for another purpose, an explicitly-named register is selected, using the .set at=<reg> feature added recently to gas. This feature is only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the workaround remains disabled, the required version of binutils stays unchanged. Similarly, daddiu instructions put in branch delay slots in noreorder fragments are now taken out of them and the assembler is allowed to reorder them itself as possible (which it does making the whole idea of scheduling them into delay slots manually questionable). Also in the very few places where such a simple conversion was not possible, a handcoded longer sequence is implemented. Other than that there are changes to code responsible for building the TLB fault and page clear/copy handlers to avoid daddiu as appropriate. These are only effective if the erratum is verified to be present at the run time. Finally there is a trivial update to __delay(), because it uses daddiu in a branch delay slot. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-10-23 19:43:25 +08:00
#else
.set noat
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
li AT, STORSIZE
[MIPS] R4000/R4400 daddiu erratum workaround This complements the generic R4000/R4400 errata workaround code and adds bits for the daddiu problem. In most places it just modifies handwritten assembly code so that the assembler is allowed to use a temporary register as daddiu may now be treated as a macro that expands to a sequence of li and daddu. It is the AT register or, where AT is unavailable or used explicitly for another purpose, an explicitly-named register is selected, using the .set at=<reg> feature added recently to gas. This feature is only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the workaround remains disabled, the required version of binutils stays unchanged. Similarly, daddiu instructions put in branch delay slots in noreorder fragments are now taken out of them and the assembler is allowed to reorder them itself as possible (which it does making the whole idea of scheduling them into delay slots manually questionable). Also in the very few places where such a simple conversion was not possible, a handcoded longer sequence is implemented. Other than that there are changes to code responsible for building the TLB fault and page clear/copy handlers to avoid daddiu as appropriate. These are only effective if the erratum is verified to be present at the run time. Finally there is a trivial update to __delay(), because it uses daddiu in a branch delay slot. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-10-23 19:43:25 +08:00
beqz t0, 1f
MIPS: memset.S: Reinstate delay slot indentation Assembly language within the MIPS kernel conventionally indents instructions which are in a branch delay slot to make them easier to see. Commit 8483b14aaa81 ("MIPS: lib: memset: Whitespace fixes") rather inexplicably removed all of these indentations from memset.S. Reinstate the convention for all instructions in a branch delay slot. This effectively reverts the above commit, plus other locations introduced with MIPSR6 support. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/19111/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 23:40:02 +08:00
PTR_SUBU t0, AT /* alignment in bytes */
[MIPS] R4000/R4400 daddiu erratum workaround This complements the generic R4000/R4400 errata workaround code and adds bits for the daddiu problem. In most places it just modifies handwritten assembly code so that the assembler is allowed to use a temporary register as daddiu may now be treated as a macro that expands to a sequence of li and daddu. It is the AT register or, where AT is unavailable or used explicitly for another purpose, an explicitly-named register is selected, using the .set at=<reg> feature added recently to gas. This feature is only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the workaround remains disabled, the required version of binutils stays unchanged. Similarly, daddiu instructions put in branch delay slots in noreorder fragments are now taken out of them and the assembler is allowed to reorder them itself as possible (which it does making the whole idea of scheduling them into delay slots manually questionable). Also in the very few places where such a simple conversion was not possible, a handcoded longer sequence is implemented. Other than that there are changes to code responsible for building the TLB fault and page clear/copy handlers to avoid daddiu as appropriate. These are only effective if the erratum is verified to be present at the run time. Finally there is a trivial update to __delay(), because it uses daddiu in a branch delay slot. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-10-23 19:43:25 +08:00
.set at
#endif
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set reorder
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#ifndef CONFIG_CPU_NO_LOAD_STORE_LR
[MIPS] IP28: added cache barrier to assembly routines IP28 needs special treatment to avoid speculative accesses. gcc takes care for .c code, but for assembly code we need to do it manually. This is taken from Peter Fuersts IP28 patches. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-11-25 18:47:56 +08:00
R10KCBARRIER(0(ra))
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#ifdef __MIPSEB__
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
EX(LONG_S_L, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
MIPS: lib: memset: Clean up some MIPS{EL,EB} ifdefery The toolchain defines exactly one of __MIPSEB__ and __MIPSEL__. As a result, simplify the ifdefery a little bit. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/8522/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2014-11-19 16:58:10 +08:00
#else
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
EX(LONG_S_R, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#endif
PTR_SUBU a0, t0 /* long align ptr */
PTR_ADDU a2, t0 /* correct size */
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#else /* CONFIG_CPU_NO_LOAD_STORE_LR */
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
#define STORE_BYTE(N) \
EX(sb, a1, N(a0), .Lbyte_fixup\@); \
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set noreorder; \
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
beqz t0, 0f; \
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
PTR_ADDU t0, 1; \
.set reorder;
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
PTR_ADDU a2, t0 /* correct size */
PTR_ADDU t0, 1
STORE_BYTE(0)
STORE_BYTE(1)
#if LONGSIZE == 4
EX(sb, a1, 2(a0), .Lbyte_fixup\@)
#else
STORE_BYTE(2)
STORE_BYTE(3)
STORE_BYTE(4)
STORE_BYTE(5)
EX(sb, a1, 6(a0), .Lbyte_fixup\@)
#endif
0:
ori a0, STORMASK
xori a0, STORMASK
PTR_ADDIU a0, STORSIZE
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#endif /* CONFIG_CPU_NO_LOAD_STORE_LR */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
1: ori t1, a2, 0x3f /* # of full blocks */
xori t1, 0x3f
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
andi t0, a2, 0x40-STORSIZE
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
beqz t1, .Lmemset_partial\@ /* no block to fill */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
PTR_ADDU t1, a0 /* end address */
1: PTR_ADDIU a0, 64
[MIPS] IP28: added cache barrier to assembly routines IP28 needs special treatment to avoid speculative accesses. gcc takes care for .c code, but for assembly code we need to do it manually. This is taken from Peter Fuersts IP28 patches. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-11-25 18:47:56 +08:00
R10KCBARRIER(0(ra))
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@, \mode
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
bne t1, a0, 1b
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Lmemset_partial\@:
[MIPS] IP28: added cache barrier to assembly routines IP28 needs special treatment to avoid speculative accesses. gcc takes care for .c code, but for assembly code we need to do it manually. This is taken from Peter Fuersts IP28 patches. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-11-25 18:47:56 +08:00
R10KCBARRIER(0(ra))
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
PTR_LA t1, 2f /* where to start */
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
#ifdef CONFIG_CPU_MICROMIPS
LONG_SRL t7, t0, 1
#endif
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
#if LONGSIZE == 4
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
PTR_SUBU t1, FILLPTRG
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
#else
.set noat
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
LONG_SRL AT, FILLPTRG, 1
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
PTR_SUBU t1, AT
[MIPS] R4000/R4400 daddiu erratum workaround This complements the generic R4000/R4400 errata workaround code and adds bits for the daddiu problem. In most places it just modifies handwritten assembly code so that the assembler is allowed to use a temporary register as daddiu may now be treated as a macro that expands to a sequence of li and daddu. It is the AT register or, where AT is unavailable or used explicitly for another purpose, an explicitly-named register is selected, using the .set at=<reg> feature added recently to gas. This feature is only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the workaround remains disabled, the required version of binutils stays unchanged. Similarly, daddiu instructions put in branch delay slots in noreorder fragments are now taken out of them and the assembler is allowed to reorder them itself as possible (which it does making the whole idea of scheduling them into delay slots manually questionable). Also in the very few places where such a simple conversion was not possible, a handcoded longer sequence is implemented. Other than that there are changes to code responsible for building the TLB fault and page clear/copy handlers to avoid daddiu as appropriate. These are only effective if the erratum is verified to be present at the run time. Finally there is a trivial update to __delay(), because it uses daddiu in a branch delay slot. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-10-23 19:43:25 +08:00
.set at
[MIPS] Unify memset.S The 32-bit version and 64-bit version are almost equal. Unify them. This makes further improvements (for example, supporting CDEX, etc.) easier. Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-12-17 23:07:40 +08:00
#endif
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
PTR_ADDU a0, t0 /* dest ptr */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
jr t1
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
/* ... but first do longs ... */
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@, \mode
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
2: andi a2, STORMASK /* At most one long to go */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set noreorder
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
beqz a2, 1f
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#ifndef CONFIG_CPU_NO_LOAD_STORE_LR
MIPS: memset.S: Reinstate delay slot indentation Assembly language within the MIPS kernel conventionally indents instructions which are in a branch delay slot to make them easier to see. Commit 8483b14aaa81 ("MIPS: lib: memset: Whitespace fixes") rather inexplicably removed all of these indentations from memset.S. Reinstate the convention for all instructions in a branch delay slot. This effectively reverts the above commit, plus other locations introduced with MIPSR6 support. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/19111/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 23:40:02 +08:00
PTR_ADDU a0, a2 /* What's left */
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set reorder
[MIPS] IP28: added cache barrier to assembly routines IP28 needs special treatment to avoid speculative accesses. gcc takes care for .c code, but for assembly code we need to do it manually. This is taken from Peter Fuersts IP28 patches. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-11-25 18:47:56 +08:00
R10KCBARRIER(0(ra))
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#ifdef __MIPSEB__
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@)
MIPS: lib: memset: Clean up some MIPS{EL,EB} ifdefery The toolchain defines exactly one of __MIPSEB__ and __MIPSEL__. As a result, simplify the ifdefery a little bit. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/8522/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2014-11-19 16:58:10 +08:00
#else
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#endif
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#else /* CONFIG_CPU_NO_LOAD_STORE_LR */
MIPS: memset.S: Reinstate delay slot indentation Assembly language within the MIPS kernel conventionally indents instructions which are in a branch delay slot to make them easier to see. Commit 8483b14aaa81 ("MIPS: lib: memset: Whitespace fixes") rather inexplicably removed all of these indentations from memset.S. Reinstate the convention for all instructions in a branch delay slot. This effectively reverts the above commit, plus other locations introduced with MIPSR6 support. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/19111/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 23:40:02 +08:00
PTR_SUBU t0, $0, a2
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set reorder
MIPS: memset.S: Fix byte_fixup for MIPSr6 The __clear_user function is defined to return the number of bytes that could not be cleared. From the underlying memset / bzero implementation this means setting register a2 to that number on return. Currently if a page fault is triggered within the MIPSr6 version of setting of initial unaligned bytes, the value loaded into a2 on return is meaningless. During the MIPSr6 version of the initial unaligned bytes block, register a2 contains the number of bytes to be set beyond the initial unaligned bytes. The t0 register is initally set to the number of unaligned bytes - STORSIZE, effectively a negative version of the number of unaligned bytes. This is then incremented before each byte is saved. The label .Lbyte_fixup\@ is jumped to on page fault. Currently the value in a2 is incorrectly replaced by 0 - t0 + 1, effectively the number of unaligned bytes remaining. This leads to the failures being reported by the following test code: static int __init test_clear_user(void) { int j, k; pr_info("\n\n\nTesting clear_user\n"); for (j = 0; j < 512; j++) { if ((k = clear_user(NULL+3, j)) != j) { pr_err("clear_user (NULL %d) returned %d\n", j, k); } } return 0; } late_initcall(test_clear_user); Which reports: [ 3.965439] Testing clear_user [ 3.973169] clear_user (NULL 8) returned 6 [ 3.976782] clear_user (NULL 9) returned 6 [ 3.980390] clear_user (NULL 10) returned 6 [ 3.984052] clear_user (NULL 11) returned 6 [ 3.987524] clear_user (NULL 12) returned 6 Fix this by subtracting t0 from a2 (rather than $0), effectivey giving: unset_bytes = (#bytes - (#unaligned bytes)) - (-#unaligned bytes remaining + 1) + 1 a2 = a2 - t0 + 1 This fixes the value returned from __clear user when the number of bytes to set is > LONGSIZE and the address is invalid and unaligned. Unfortunately, this breaks the fixup handling for unaligned bytes after the final long, where register a2 still contains the number of bytes remaining to be set and the t0 register is to 0 - the number of unaligned bytes remaining. Because t0 is now is now subtracted from a2 rather than 0, the number of bytes unset is reported incorrectly: static int __init test_clear_user(void) { char *test; int j, k; pr_info("\n\n\nTesting clear_user\n"); test = vmalloc(PAGE_SIZE); for (j = 256; j < 512; j++) { if ((k = clear_user(test + PAGE_SIZE - 254, j)) != j - 254) { pr_err("clear_user (%px %d) returned %d\n", test + PAGE_SIZE - 254, j, k); } } return 0; } late_initcall(test_clear_user); [ 3.976775] clear_user (c00000000000df02 256) returned 4 [ 3.981957] clear_user (c00000000000df02 257) returned 6 [ 3.986425] clear_user (c00000000000df02 258) returned 8 [ 3.990850] clear_user (c00000000000df02 259) returned 10 [ 3.995332] clear_user (c00000000000df02 260) returned 12 [ 3.999815] clear_user (c00000000000df02 261) returned 14 Fix this by ensuring that a2 is set to 0 during the set of final unaligned bytes. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Fixes: 8c56208aff77 ("MIPS: lib: memset: Add MIPS R6 support") Patchwork: https://patchwork.linux-mips.org/patch/19338/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org # v4.0+
2018-05-23 21:39:58 +08:00
move a2, zero /* No remaining longs */
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
PTR_ADDIU t0, 1
STORE_BYTE(0)
STORE_BYTE(1)
#if LONGSIZE == 4
EX(sb, a1, 2(a0), .Lbyte_fixup\@)
#else
STORE_BYTE(2)
STORE_BYTE(3)
STORE_BYTE(4)
STORE_BYTE(5)
EX(sb, a1, 6(a0), .Lbyte_fixup\@)
#endif
0:
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#endif /* CONFIG_CPU_NO_LOAD_STORE_LR */
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
1: move a2, zero
jr ra
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Lsmall_memset\@:
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
PTR_ADDU t1, a0, a2
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
beqz a2, 2f
1: PTR_ADDIU a0, 1 /* fill bytewise */
[MIPS] IP28: added cache barrier to assembly routines IP28 needs special treatment to avoid speculative accesses. gcc takes care for .c code, but for assembly code we need to do it manually. This is taken from Peter Fuersts IP28 patches. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-11-25 18:47:56 +08:00
R10KCBARRIER(0(ra))
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set noreorder
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
bne t1, a0, 1b
MIPS: memset.S: EVA & fault support for small_memset The MIPS kernel memset / bzero implementation includes a small_memset branch which is used when the region to be set is smaller than a long (4 bytes on 32bit, 8 bytes on 64bit). The current small_memset implementation uses a simple store byte loop to write the destination. There are 2 issues with this implementation: 1. When EVA mode is active, user and kernel address spaces may overlap. Currently the use of the sb instruction means kernel mode addressing is always used and an intended write to userspace may actually overwrite some critical kernel data. 2. If the write triggers a page fault, for example by calling __clear_user(NULL, 2), instead of gracefully handling the fault, an OOPS is triggered. Fix these issues by replacing the sb instruction with the EX() macro, which will emit EVA compatible instuctions as required. Additionally implement a fault fixup for small_memset which sets a2 to the number of bytes that could not be cleared (as defined by __clear_user). Reported-by: Chuanhua Lei <chuanhua.lei@intel.com> Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: stable@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/18975/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-03-29 17:28:23 +08:00
EX(sb, a1, -1(a0), .Lsmall_fixup\@)
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
.set reorder
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
2: move a2, zero
jr ra /* done */
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.if __memset == 1
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
END(memset)
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.set __memset, 0
.hidden __memset
.endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#ifdef CONFIG_CPU_NO_LOAD_STORE_LR
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
.Lbyte_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/*
* unset_bytes = (#bytes - (#unaligned bytes)) - (-#unaligned bytes remaining + 1) + 1
* a2 = a2 - t0 + 1
*/
MIPS: memset.S: Fix byte_fixup for MIPSr6 The __clear_user function is defined to return the number of bytes that could not be cleared. From the underlying memset / bzero implementation this means setting register a2 to that number on return. Currently if a page fault is triggered within the MIPSr6 version of setting of initial unaligned bytes, the value loaded into a2 on return is meaningless. During the MIPSr6 version of the initial unaligned bytes block, register a2 contains the number of bytes to be set beyond the initial unaligned bytes. The t0 register is initally set to the number of unaligned bytes - STORSIZE, effectively a negative version of the number of unaligned bytes. This is then incremented before each byte is saved. The label .Lbyte_fixup\@ is jumped to on page fault. Currently the value in a2 is incorrectly replaced by 0 - t0 + 1, effectively the number of unaligned bytes remaining. This leads to the failures being reported by the following test code: static int __init test_clear_user(void) { int j, k; pr_info("\n\n\nTesting clear_user\n"); for (j = 0; j < 512; j++) { if ((k = clear_user(NULL+3, j)) != j) { pr_err("clear_user (NULL %d) returned %d\n", j, k); } } return 0; } late_initcall(test_clear_user); Which reports: [ 3.965439] Testing clear_user [ 3.973169] clear_user (NULL 8) returned 6 [ 3.976782] clear_user (NULL 9) returned 6 [ 3.980390] clear_user (NULL 10) returned 6 [ 3.984052] clear_user (NULL 11) returned 6 [ 3.987524] clear_user (NULL 12) returned 6 Fix this by subtracting t0 from a2 (rather than $0), effectivey giving: unset_bytes = (#bytes - (#unaligned bytes)) - (-#unaligned bytes remaining + 1) + 1 a2 = a2 - t0 + 1 This fixes the value returned from __clear user when the number of bytes to set is > LONGSIZE and the address is invalid and unaligned. Unfortunately, this breaks the fixup handling for unaligned bytes after the final long, where register a2 still contains the number of bytes remaining to be set and the t0 register is to 0 - the number of unaligned bytes remaining. Because t0 is now is now subtracted from a2 rather than 0, the number of bytes unset is reported incorrectly: static int __init test_clear_user(void) { char *test; int j, k; pr_info("\n\n\nTesting clear_user\n"); test = vmalloc(PAGE_SIZE); for (j = 256; j < 512; j++) { if ((k = clear_user(test + PAGE_SIZE - 254, j)) != j - 254) { pr_err("clear_user (%px %d) returned %d\n", test + PAGE_SIZE - 254, j, k); } } return 0; } late_initcall(test_clear_user); [ 3.976775] clear_user (c00000000000df02 256) returned 4 [ 3.981957] clear_user (c00000000000df02 257) returned 6 [ 3.986425] clear_user (c00000000000df02 258) returned 8 [ 3.990850] clear_user (c00000000000df02 259) returned 10 [ 3.995332] clear_user (c00000000000df02 260) returned 12 [ 3.999815] clear_user (c00000000000df02 261) returned 14 Fix this by ensuring that a2 is set to 0 during the set of final unaligned bytes. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Fixes: 8c56208aff77 ("MIPS: lib: memset: Add MIPS R6 support") Patchwork: https://patchwork.linux-mips.org/patch/19338/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org # v4.0+
2018-05-23 21:39:58 +08:00
PTR_SUBU a2, t0
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
PTR_ADDIU a2, 1
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
jr ra
MIPS: make CPU_HAS_LOAD_STORE_LR opt-out CPU_HAS_LOAD_STORE_LR was introduced in 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions") to make code in kernel/unaligned.c and lib/mem{cpy,set}.S more intuitive and give a possibility to easily add new CPUs without these instruction sets in future. Hovewer, this variant is not optimal for mainly two reasons: * For now, we have 20+ CPUs with such instructions and only two (MIPS R6) without. It will obviously be more effective and straightforward to have an option for these two rather than for the rest. * You can easily miss the fact that you need to select this option when adding a new CPU, while all processors lacking these sets are well-known, so the probability of missing something is way much lower. We can address both points by turning CPU_HAS_LOAD_STORE_LR into opt-out CPU_NO_LOAD_STORE_LR. This also makes MIPS root Kconfig more clear and understandable. Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Alexandre Belloni <alexandre.belloni@bootlin.com> Cc: Microchip Linux Driver Support <UNGLinuxDriver@microchip.com> Cc: Will Deacon <will@kernel.org> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Michal Simek <michal.simek@xilinx.com> Cc: Allison Randal <allison@lohutok.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-22 18:58:50 +08:00
#endif /* CONFIG_CPU_NO_LOAD_STORE_LR */
MIPS: lib: memset: Add MIPS R6 support MIPS R6 dropped the unaligned load and store instructions so we need to re-write this part of the code for R6 to store one byte at a time. Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-11-18 17:04:34 +08:00
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Lfirst_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/* unset_bytes already in a2 */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
jr ra
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Lfwd_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/*
* unset_bytes = partial_start_addr + #bytes - fault_addr
* a2 = t1 + (a2 & 3f) - $28->task->BUADDR
*/
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
PTR_L t0, TI_TASK($28)
andi a2, 0x3f
MIPS: Separate two consecutive loads in memset.S partial_fixup is used in noreorder block. Separating two consecutive loads can save one cycle on processors with GPR intrelock and can fix load-use on processors that need a load delay slot. Also do so for fwd_fixup. [Ralf: Only R2000/R3000 class processors are lacking the the load-user interlock and even some of those got it retrofitted. With R2000/R3000 being fairly uncommon these days the impact of this bug should be minor.] Signed-off-by: Tony Wu <tung7970@gmail.com> To: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/1768/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-11-10 21:48:15 +08:00
LONG_L t0, THREAD_BUADDR(t0)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
LONG_ADDU a2, t1
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
LONG_SUBU a2, t0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
jr ra
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Lpartial_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/*
* unset_bytes = partial_end_addr + #bytes - fault_addr
* a2 = a0 + (a2 & STORMASK) - $28->task->BUADDR
*/
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
PTR_L t0, TI_TASK($28)
MIPS: microMIPS: Optimise 'memset' core library function. Optimise 'memset' to use microMIPS instructions and/or optimisations for binary size reduction. When the microMIPS ISA is not being used, the library function compiles to the original binary code. Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
2013-03-26 02:40:49 +08:00
andi a2, STORMASK
MIPS: Separate two consecutive loads in memset.S partial_fixup is used in noreorder block. Separating two consecutive loads can save one cycle on processors with GPR intrelock and can fix load-use on processors that need a load delay slot. Also do so for fwd_fixup. [Ralf: Only R2000/R3000 class processors are lacking the the load-user interlock and even some of those got it retrofitted. With R2000/R3000 being fairly uncommon these days the impact of this bug should be minor.] Signed-off-by: Tony Wu <tung7970@gmail.com> To: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/1768/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-11-10 21:48:15 +08:00
LONG_L t0, THREAD_BUADDR(t0)
MIPS: memset.S: Fix return of __clear_user from Lpartial_fixup The __clear_user function is defined to return the number of bytes that could not be cleared. From the underlying memset / bzero implementation this means setting register a2 to that number on return. Currently if a page fault is triggered within the memset_partial block, the value loaded into a2 on return is meaningless. The label .Lpartial_fixup\@ is jumped to on page fault. In order to work out how many bytes failed to copy, the exception handler should find how many bytes left in the partial block (andi a2, STORMASK), add that to the partial block end address (a2), and subtract the faulting address to get the remainder. Currently it incorrectly subtracts the partial block start address (t1), which has additionally been clobbered to generate a jump target in memset_partial. Fix this by adding the block end address instead. This issue was found with the following test code: int j, k; for (j = 0; j < 512; j++) { if ((k = clear_user(NULL, j)) != j) { pr_err("clear_user (NULL %d) returned %d\n", j, k); } } Which now passes on Creator Ci40 (MIPS32) and Cavium Octeon II (MIPS64). Suggested-by: James Hogan <jhogan@kernel.org> Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: stable@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/19108/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-04-17 22:52:21 +08:00
LONG_ADDU a2, a0
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
LONG_SUBU a2, t0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
jr ra
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.Llast_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/* unset_bytes already in a2 */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
jr ra
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
MIPS: memset.S: EVA & fault support for small_memset The MIPS kernel memset / bzero implementation includes a small_memset branch which is used when the region to be set is smaller than a long (4 bytes on 32bit, 8 bytes on 64bit). The current small_memset implementation uses a simple store byte loop to write the destination. There are 2 issues with this implementation: 1. When EVA mode is active, user and kernel address spaces may overlap. Currently the use of the sb instruction means kernel mode addressing is always used and an intended write to userspace may actually overwrite some critical kernel data. 2. If the write triggers a page fault, for example by calling __clear_user(NULL, 2), instead of gracefully handling the fault, an OOPS is triggered. Fix these issues by replacing the sb instruction with the EX() macro, which will emit EVA compatible instuctions as required. Additionally implement a fault fixup for small_memset which sets a2 to the number of bytes that could not be cleared (as defined by __clear_user). Reported-by: Chuanhua Lei <chuanhua.lei@intel.com> Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: stable@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/18975/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-03-29 17:28:23 +08:00
.Lsmall_fixup\@:
MIPS: memset.S: Add comments to fault fixup handlers It is not immediately obvious what the expected inputs to these fault handlers is and how they calculate the number of unset bytes. Having stared deeply at this in order to fix some corner cases, add some comments to assist those who follow. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/19339/ Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: <linux-mips@linux-mips.org> Cc: <linux-kernel@vger.kernel.org>
2018-05-23 21:39:59 +08:00
/*
* unset_bytes = end_addr - current_addr + 1
* a2 = t1 - a0 + 1
*/
MIPS: memset.S: EVA & fault support for small_memset The MIPS kernel memset / bzero implementation includes a small_memset branch which is used when the region to be set is smaller than a long (4 bytes on 32bit, 8 bytes on 64bit). The current small_memset implementation uses a simple store byte loop to write the destination. There are 2 issues with this implementation: 1. When EVA mode is active, user and kernel address spaces may overlap. Currently the use of the sb instruction means kernel mode addressing is always used and an intended write to userspace may actually overwrite some critical kernel data. 2. If the write triggers a page fault, for example by calling __clear_user(NULL, 2), instead of gracefully handling the fault, an OOPS is triggered. Fix these issues by replacing the sb instruction with the EX() macro, which will emit EVA compatible instuctions as required. Additionally implement a fault fixup for small_memset which sets a2 to the number of bytes that could not be cleared (as defined by __clear_user). Reported-by: Chuanhua Lei <chuanhua.lei@intel.com> Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: stable@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/18975/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-03-29 17:28:23 +08:00
PTR_SUBU a2, t1, a0
MIPS: memset: Fix CPU_DADDI_WORKAROUNDS `small_fixup' regression Fix a commit 8a8158c85e1e ("MIPS: memset.S: EVA & fault support for small_memset") regression and remove assembly warnings: arch/mips/lib/memset.S: Assembler messages: arch/mips/lib/memset.S:243: Warning: Macro instruction expanded into multiple instructions in a branch delay slot triggering with the CPU_DADDI_WORKAROUNDS option set and this code: PTR_SUBU a2, t1, a0 jr ra PTR_ADDIU a2, 1 This is because with that option in place the DADDIU instruction, which the PTR_ADDIU CPP macro expands to, becomes a GAS macro, which in turn expands to an LI/DADDU (or actually ADDIU/DADDU) sequence: 13c: 01a4302f dsubu a2,t1,a0 140: 03e00008 jr ra 144: 24010001 li at,1 148: 00c1302d daddu a2,a2,at ... Correct this by switching off the `noreorder' assembly mode and letting GAS schedule this jump's delay slot, as there is nothing special about it that would require manual scheduling. With this change in place correct code is produced: 13c: 01a4302f dsubu a2,t1,a0 140: 24010001 li at,1 144: 03e00008 jr ra 148: 00c1302d daddu a2,a2,at ... Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> Signed-off-by: Paul Burton <paul.burton@mips.com> Fixes: 8a8158c85e1e ("MIPS: memset.S: EVA & fault support for small_memset") Patchwork: https://patchwork.linux-mips.org/patch/20833/ Cc: Ralf Baechle <ralf@linux-mips.org> Cc: stable@vger.kernel.org # 4.17+
2018-10-02 19:50:11 +08:00
PTR_ADDIU a2, 1
MIPS: memset.S: EVA & fault support for small_memset The MIPS kernel memset / bzero implementation includes a small_memset branch which is used when the region to be set is smaller than a long (4 bytes on 32bit, 8 bytes on 64bit). The current small_memset implementation uses a simple store byte loop to write the destination. There are 2 issues with this implementation: 1. When EVA mode is active, user and kernel address spaces may overlap. Currently the use of the sb instruction means kernel mode addressing is always used and an intended write to userspace may actually overwrite some critical kernel data. 2. If the write triggers a page fault, for example by calling __clear_user(NULL, 2), instead of gracefully handling the fault, an OOPS is triggered. Fix these issues by replacing the sb instruction with the EX() macro, which will emit EVA compatible instuctions as required. Additionally implement a fault fixup for small_memset which sets a2 to the number of bytes that could not be cleared (as defined by __clear_user). Reported-by: Chuanhua Lei <chuanhua.lei@intel.com> Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: linux-mips@linux-mips.org Cc: stable@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/18975/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-03-29 17:28:23 +08:00
jr ra
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
.endm
/*
* memset(void *s, int c, size_t n)
*
* a0: start of area to clear
* a1: char to fill with
* a2: size of area to clear
*/
LEAF(memset)
MIPS: Export memcpy & memset functions alongside their definitions Now that EXPORT_SYMBOL can be used from assembly source, move the EXPORT_SYMBOL invocations for the memcpy & memset functions & variants thereof to be alongside their definitions. Signed-off-by: Paul Burton <paul.burton@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/14514/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-11-07 19:14:15 +08:00
EXPORT_SYMBOL(memset)
MIPS: memset: Limit excessive `noreorder' assembly mode use Rewrite to use the `reorder' assembly mode and remove manually scheduled delay slots except where GAS cannot schedule a delay-slot instruction due to a data dependency or a section switch (as is the case with the EX macro). No change in machine code produced. Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org> [paul.burton@mips.com: Fix conflict with commit 932afdeec18b ("MIPS: Add Kconfig variable for CPUs with unaligned load/store instructions")] Signed-off-by: Paul Burton <paul.burton@mips.com> Patchwork: https://patchwork.linux-mips.org/patch/20834/ Cc: Ralf Baechle <ralf@linux-mips.org>
2018-10-02 19:50:16 +08:00
move v0, a0 /* result */
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
beqz a1, 1f
andi a1, 0xff /* spread fillword */
LONG_SLL t1, a1, 8
or a1, t1
LONG_SLL t1, a1, 16
#if LONGSIZE == 8
or a1, t1
LONG_SLL t1, a1, 32
#endif
or a1, t1
1:
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
#ifndef CONFIG_EVA
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
FEXPORT(__bzero)
MIPS: Export memcpy & memset functions alongside their definitions Now that EXPORT_SYMBOL can be used from assembly source, move the EXPORT_SYMBOL invocations for the memcpy & memset functions & variants thereof to be alongside their definitions. Signed-off-by: Paul Burton <paul.burton@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/14514/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-11-07 19:14:15 +08:00
EXPORT_SYMBOL(__bzero)
MIPS: uaccess: Take EVA into account in [__]clear_user __clear_user() (and clear_user() which uses it), always access the user mode address space, which results in EVA store instructions when EVA is enabled even if the current user address limit is KERNEL_DS. Fix this by adding a new symbol __bzero_kernel for the normal kernel address space bzero in EVA mode, and call that from __clear_user() if eva_kernel_access(). Signed-off-by: James Hogan <james.hogan@imgtec.com> Cc: Markos Chandras <markos.chandras@imgtec.com> Cc: Paul Burton <paul.burton@imgtec.com> Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/10844/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-08-05 23:41:39 +08:00
#else
FEXPORT(__bzero_kernel)
MIPS: Export memcpy & memset functions alongside their definitions Now that EXPORT_SYMBOL can be used from assembly source, move the EXPORT_SYMBOL invocations for the memcpy & memset functions & variants thereof to be alongside their definitions. Signed-off-by: Paul Burton <paul.burton@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/14514/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-11-07 19:14:15 +08:00
EXPORT_SYMBOL(__bzero_kernel)
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
#endif
MIPS: lib: memset: Use macro to build the __bzero symbol Build the __bzero symbol using a macor. In EVA mode we will need to use similar code to do the userspace load operations so it is better if we use a macro to avoid code duplications. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 17:23:16 +08:00
__BUILD_BZERO LEGACY_MODE
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
#ifdef CONFIG_EVA
LEAF(__bzero)
MIPS: Export memcpy & memset functions alongside their definitions Now that EXPORT_SYMBOL can be used from assembly source, move the EXPORT_SYMBOL invocations for the memcpy & memset functions & variants thereof to be alongside their definitions. Signed-off-by: Paul Burton <paul.burton@imgtec.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/14514/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-11-07 19:14:15 +08:00
EXPORT_SYMBOL(__bzero)
MIPS: lib: memset: Add EVA support for the __bzero function. Build the __bzero function using the EVA load/store instructions when operating in the EVA mode. This function is only used when accessing user code so there is no need to build two distinct symbols for user and kernel operations respectively. Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
2014-01-03 18:11:45 +08:00
__BUILD_BZERO EVA_MODE
END(__bzero)
#endif