__clear_user() (and clear_user() which uses it), always access the user
mode address space, which results in EVA store instructions when EVA is
enabled even if the current user address limit is KERNEL_DS.
Fix this by adding a new symbol __bzero_kernel for the normal kernel
address space bzero in EVA mode, and call that from __clear_user() if
eva_kernel_access().
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10844/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
When EVA is in use, __copy_from_user() was unconditionally using the EVA
instructions to read the user address space, however this can also be
used for kernel access. If the address isn't a valid user address it
will cause an address error or TLB exception, and if it is then user
memory may be read instead of kernel memory.
For example in the following stack trace from Linux v3.10 (changes since
then will prevent this particular one still happening) kernel_sendmsg()
set the user address limit to KERNEL_DS, and tcp_sendmsg() goes on to
use __copy_from_user() with a kernel address in KSeg0.
[<8002d434>] __copy_fromuser_common+0x10c/0x254
[<805710e0>] tcp_sendmsg+0x5f4/0xf00
[<804e8e3c>] sock_sendmsg+0x78/0xa0
[<804e8f28>] kernel_sendmsg+0x24/0x38
[<804ee0f8>] sock_no_sendpage+0x70/0x7c
[<8017c820>] pipe_to_sendpage+0x80/0x98
[<8017c6b0>] splice_from_pipe_feed+0xa8/0x198
[<8017cc54>] __splice_from_pipe+0x4c/0x8c
[<8017e844>] splice_from_pipe+0x58/0x78
[<8017e884>] generic_splice_sendpage+0x20/0x2c
[<8017d690>] do_splice_from+0xb4/0x110
[<8017d710>] direct_splice_actor+0x24/0x30
[<8017d394>] splice_direct_to_actor+0xd8/0x208
[<8017d51c>] do_splice_direct+0x58/0x7c
[<8014eaf4>] do_sendfile+0x1dc/0x39c
[<8014f82c>] SyS_sendfile+0x90/0xf8
Add the eva_kernel_access() check in __copy_from_user() like the one in
copy_from_user().
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10843/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The strlen_user() function calls __strlen_kernel_asm in both branches of
the eva_kernel_access() conditional. For EVA it should be calling
__strlen_user_eva for user accesses, otherwise it will load from the
kernel address space instead of the user address space, and the access
checking will likely be ineffective at preventing it due to EVA's
overlapping user and kernel address spaces.
This was found after extending the test_user_copy module to cover user
string access functions, which gave the following error with EVA:
test_user_copy: illegal strlen_user passed
Fortunately the use of strlen_user() has been all but eradicated from
the mainline kernel, so only out of tree modules could be affected.
Fixes: e3a9b07a9c ("MIPS: asm: uaccess: Add EVA support for str*_user operations")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: <stable@vger.kernel.org> # 3.15.x-
Patchwork: https://patchwork.linux-mips.org/patch/10842/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Pull MIPS updates from Ralf Baechle:
- Improvements to the tlb_dump code
- KVM fixes
- Add support for appended DTB
- Minor improvements to the R12000 support
- Minor improvements to the R12000 support
- Various platform improvments for BCM47xx
- The usual pile of minor cleanups
- A number of BPF fixes and improvments
- Some improvments to the support for R3000 and DECstations
- Some improvments to the ATH79 platform support
- A major patchset for the JZ4740 SOC adding support for the CI20 platform
- Add support for the Pistachio SOC
- Minor BMIPS/BCM63xx platform support improvments.
- Avoid "SYNC 0" as memory barrier when unlocking spinlocks
- Add support for the XWR-1750 board.
- Paul's __cpuinit/__cpuinitdata cleanups.
- New Malta CPU board support large memory so enable ZONE_DMA32.
* 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus: (131 commits)
MIPS: spinlock: Adjust arch_spin_lock back-off time
MIPS: asmmacro: Ensure 64-bit FP registers are used with MSA
MIPS: BCM47xx: Simplify handling SPROM revisions
MIPS: Cobalt Don't use module_init in non-modular MTD registration.
MIPS: BCM47xx: Move NVRAM driver to the drivers/firmware/
MIPS: use for_each_sg()
MIPS: BCM47xx: Don't select BCMA_HOST_PCI
MIPS: BCM47xx: Add helper variable for storing NVRAM length
MIPS: IRQ/IP27: Move IRQ allocation API to platform code.
MIPS: Replace smp_mb with release barrier function in unlocks.
MIPS: i8259: DT support
MIPS: Malta: Basic DT plumbing
MIPS: include errno.h for ENODEV in mips-cm.h
MIPS: Define GCR_GIC_STATUS register fields
MIPS: BPF: Introduce BPF ASM helpers
MIPS: BPF: Use BPF register names to describe the ABI
MIPS: BPF: Move register definition to the BPF header
MIPS: net: BPF: Replace RSIZE with SZREG
MIPS: BPF: Free up some callee-saved registers
MIPS: Xtalk: Update xwidget.h with known Xtalk device numbers
...
Commits ac1d8590d3 (MIPS: asm: uaccess: Use EVA instructions
wrappers), 05c6516005 (MIPS: asm: uaccess: Add EVA support to
copy_{in, to,from}_user) & e3a9b07a9c (MIPS: asm: uaccess: Add EVA
support for str*_user operations) added checks to various user memory
access functions & macros in order to determine whether to perform
standard memory accesses or their EVA userspace equivalents. In kernels
built without support for EVA these checks are entirely redundant. Avoid
emitting them & allow the compiler to optimise out the EVA userspace
code in such kernels by checking config_enabled(CONFIG_EVA).
This reduces the size of a malta_defconfig kernel built using GCC 4.9.2
by approximately 33KB (from 5995072 to 5962304 bytes).
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/10165/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
When EVA is turned on and prefetching is being used in memcpy.S,
the v1 register is being used as a helper register to the PREFE
instruction. However, v1 ($3) was not in the clobber list, which
means that the compiler did not preserve it across function calls,
and that could corrupt the value of the register leading to all
sorts of userland crashes. We fix this problem by using the
DADDI_SCRATCH macro to define the clobbered register when
CONFIG_EVA && CONFIG_CPU_HAS_PREFETCH are enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: <stable@vger.kernel.org> # v3.15+
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8510/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The str*_user functions are used to securely access NULL terminated
strings from userland. Therefore, it's necessary to use the appropriate
EVA function. However, if the string is in kernel space, then the normal
instructions are being used to access it. The __str*_kernel_asm and
__str*_user_asm symbols are the same for non-EVA mode so there is no
functional change for the non-EVA kernels.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Use the EVA specific functions from memcpy.S to perform
userspace operations. When get_fs() == get_ds() the usual load/store
instructions are used because the destination address is located in
the kernel address space region. Otherwise, the EVA specifc load/store
instructions are used which will go through th TLB to perform the virtual
to physical translation for the userspace address.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
The {get,put}_user_asm functions can be used to load data from
kernel or the user address space so rename them to avoid
confusion.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Use the EVA instruction wrappers from asm.h to perform
read/write operations from userland.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
ulb, ulh, ulw are macros which emulate unaligned access for MIPS.
However, no such macros exist for EVA mode, so the only way to do
EVA unaligned accesses is in the ADE exception handler. As a result
of which, disable these macros for EVA.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Similar to __get_user_* functions, move common code to
__put_user_*_common so it can be shared among similar users.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
In preparation for EVA support, an instruction argument is needed
for the __get_user_asm{,_ll32} functions to allow instruction overrides in
EVA mode. Even though EVA only works for MIPS 32-bit, both codepaths are
changed (32-bit and 64-bit) for consistency reasons.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Partially revert commit e0c14a260d66ba35935600d6435940a566fe806b
and turn off LL/SC when building a pure microMIPS kernel. This is
a temporary fix until the cmpxchg assembly macro functions are
re-written to not use the HI/LO registers in address calculations.
Also add .insn in selected user access functions which would
otherwise produce ISA mode jump incompatibilities. This is also a
temporary fix.
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We can save the 451 lines of code that comprise memcpy-inatomic.S at the
expense of a single instruction in the memcpy prolog. We also use an
additional register (t6), so this may cause increased register pressure in
some places as well. But I think the reduced maintenance burden, of not
having two nearly identical implementations, makes it worth it.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The TASK_SIZE macro should reflect the size of a user process virtual
address space. Previously for 64-bit kernels, this was not the case.
The immediate cause of pain was in
hugetlbfs/inode.c:hugetlb_get_unmapped_area() where 32-bit processes
trying to mmap a huge page would be served a page with an address
outside of the 32-bit address range. But there are other uses of
TASK_SIZE in the kernel as well that would like an accurate value.
The new definition is nice because it now makes TASK_SIZE and
TASK_SIZE_OF() yield the same value for any given process.
For 32-bit kernels there should be no change, although I did factor
out some code in asm/processor.h that became identical for the 32-bit and
64-bit cases.
__UA_LIMIT is now set to ~((1 << SEGBITS) - 1) for 64-bit kernels.
This should eliminate the possibility of getting a
AddressErrorException in the kernel for addresses that pass the
access_ok() test.
With the patch applied, I can still run o32, n32 and n64 processes,
and have an o32 shell fork/exec both n32 and n64 processes.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1701/
The type of the second argument of access_ok should be (void __user *).
The unnecessary conversion of the clear_user address argument was causing
sparse to emit warnings on the __chk_user_ptr check.
Signed-off-by: Wu Zhangjin <wuzhangjin@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Al Viro