linux/arch
Ma Ling 59daa706fb x86, mem: Optimize memcpy by avoiding memory false dependece
All read operations after allocation stage can run speculatively,
all write operation will run in program order, and if addresses are
different read may run before older write operation, otherwise wait
until write commit. However CPU don't check each address bit,
so read could fail to recognize different address even they
are in different page.For example if rsi is 0xf004, rdi is 0xe008,
in following operation there will generate big performance latency.
1. movq (%rsi),	%rax
2. movq %rax,	(%rdi)
3. movq 8(%rsi), %rax
4. movq %rax,	8(%rdi)

If %rsi and rdi were in really the same meory page, there are TRUE
read-after-write dependence because instruction 2 write 0x008 and
instruction 3 read 0x00c, the two address are overlap partially.
Actually there are in different page and no any issues,
but without checking each address bit CPU could think they are
in the same page, and instruction 3 have to wait for instruction 2
to write data into cache from write buffer, then load data from cache,
the cost time read spent is equal to mfence instruction. We may avoid it by
tuning operation sequence as follow.

1. movq 8(%rsi), %rax
2. movq %rax,	8(%rdi)
3. movq (%rsi),	%rax
4. movq %rax,	(%rdi)

Instruction 3 read 0x004, instruction 2 write address 0x010, no any
dependence.  At last on Core2 we gain 1.83x speedup compared with
original instruction sequence.  In this patch we first handle small
size(less 20bytes), then jump to different copy mode. Based on our
micro-benchmark small bytes from 1 to 127 bytes, we got up to 2X
improvement, and up to 1.5X improvement for 1024 bytes on Corei7.  (We
use our micro-benchmark, and will do further test according to your
requirment)

Signed-off-by: Ma Ling <ling.ma@intel.com>
LKML-Reference: <1277753065-18610-1-git-send-email-ling.ma@intel.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2010-08-23 14:56:41 -07:00
..
alpha Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
arm s5pc110: SDHCI-s3c can override host capabilities 2010-08-20 09:34:55 -07:00
avr32 Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
blackfin Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
cris Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
frv Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
h8300 Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
ia64 [IA64] Fix build error: conflicting types for ‘sys_execve’ 2010-08-18 10:17:44 -07:00
m32r Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
m68k Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu 2010-08-18 09:27:10 -07:00
m68knommu Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu 2010-08-18 09:27:10 -07:00
microblaze Merge branch 'merge-devicetree' of git://git.secretlab.ca/git/linux-2.6 2010-08-18 09:26:17 -07:00
mips Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
mn10300 Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
parisc Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
powerpc Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
s390 Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
score Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
sh Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
sparc Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc-2.6 2010-08-17 18:14:08 -07:00
tile Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
um uml: fix compile error in dma_get_cache_alignment() 2010-08-20 09:34:55 -07:00
x86 x86, mem: Optimize memcpy by avoiding memory false dependece 2010-08-23 14:56:41 -07:00
xtensa Make do_execve() take a const filename pointer 2010-08-17 18:07:43 -07:00
.gitignore
Kconfig Merge branch 'perf/nmi' into perf/core 2010-08-05 08:45:05 +02:00