redis/utils
Yuan Wang 70a079db5e
Improve multithreaded performance with memory prefetching (#14017)
This PR is based on: https://github.com/valkey-io/valkey/pull/861

> ### Memory Access Amortization
> (Designed and implemented by [dan
touitou](https://github.com/touitou-dan))
> 
> Memory Access Amortization (MAA) is a technique designed to optimize
the performance of dynamic data structures by reducing the impact of
memory access latency. It is applicable when multiple operations need to
be executed concurrently. The principle behind it is that for certain
dynamic data structures, executing operations in a batch is more
efficient than executing each one separately.
> 
> Rather than executing operations sequentially, this approach
interleaves the execution of all operations. This is done in such a way
that whenever a memory access is required during an operation, the
program prefetches the necessary memory and transitions to another
operation. This ensures that when one operation is blocked awaiting
memory access, other memory accesses are executed in parallel, thereby
reducing the average access latency.
> 
> We applied this method in the development of dictPrefetch, which takes
as parameters a vector of keys and dictionaries. It ensures that all
memory addresses required to execute dictionary operations for these
keys are loaded into the L1-L3 caches when executing commands.
Essentially, dictPrefetch is an interleaved execution of dictFind for
all the keys.

### Implementation of Redis
When the main thread processes clients with ready-to-execute commands
(i.e., clients for which the IO thread has parsed the commands), a batch
of up to 16 commands is created. Initially, the command's argv, which
were allocated by the IO thread, is prefetched to the main thread's L1
cache. Subsequently, all the dict entries and values required for the
commands are prefetched from the dictionary before the command
execution.

#### Memory prefetching for main hash table
As shown in the picture, after https://github.com/redis/redis/pull/13806
, we unify key value and the dict uses no_value optimization, so the
memory prefetching has 4 steps:

1. prefetch the bucket of the hash table
2. prefetch the entry associated with the given key's hash
3. prefetch the kv object of the entry
4. prefetch the value data of the kv object

we also need to handle the case that the dict entry is the pointer of kv
object, just skip step 3.

MAA can improves single-threaded memory access efficiency by
interleaving the execution of multiple independent operations, allowing
memory-level parallelism and better CPU utilization. Its key point is
batch-wise interleaved execution. Split a batch of independent
operations (such as multiple key lookups) into multiple state machines,
and interleave their progress within a single thread to hide the memory
access latency of individual requests.

The difference between serial execution and interleaved execution:
**naive serial execution**
```
key1: step1 → wait → step2 → wait → done
key2: step1 → wait → step2 → wait → done
```
**interleaved execution**
```
key1: step1   → step2   → done
key2:   step1 → step2   → done
key3:     step1 → step2 → done
         ↑ While waiting for key1’s memory, progress key2/key3
```

#### New configuration
This PR involves a new configuration `prefetch-batch-max-size`, but we
think it is a low level optimization, so we hide this config:
When multiple commands are parsed by the I/O threads and ready for
execution, we take advantage of knowing the next set of commands and
prefetch their required dictionary entries in a batch. This reduces
memory access costs. The optimal batch size depends on the specific
workflow of the user. The default batch size is 16, which can be
modified using the 'prefetch-batch-max-size' config.
When the config is set to 0, prefetching is disabled.

---------

Co-authored-by: Uri Yagelnik <uriy@amazon.com>
Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
2025-06-05 08:57:43 +08:00
..
create-cluster Add restart option to create-cluster script (#12885) 2023-12-25 18:36:44 -08:00
graphs/commits-over-time Added Tcl program to show commits graphicaly. 2015-11-20 15:45:25 +01:00
hyperloglog Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
lru Outdated comments, replace COUNTER_INIT_VAL with LFU_INIT_VAL, fix typo (#10888) 2022-06-21 08:14:31 +03:00
releasetools Fixes to release scripts (#7547) 2020-07-21 14:07:06 +03:00
req-res-validator Add reply_schema to command json files (internal for now) (#10273) 2023-03-11 10:14:16 +02:00
srandmember Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
build-static-symbols.tcl Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
cluster_fail_time.tcl Cluster: Tcl script to check avg pfail->fail time. 2015-01-30 12:03:17 +01:00
corrupt_rdb.c Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
gen-test-certs.sh Typo fix in gen-test-certs.sh (#8841) 2021-04-25 10:11:16 +03:00
generate-command-code.py Temporarily hide the new SFLUSH command by marking it as experimental (#13600) 2024-10-15 11:02:51 +03:00
generate-commands-json.py Fix some commands key spec in json files (#10779) 2022-05-27 12:58:00 +03:00
generate-fmtargs.py Improve multithreaded performance with memory prefetching (#14017) 2025-06-05 08:57:43 +08:00
generate-module-api-doc.rb DOC-3932: remove frontmatter attributes from module API spec generator (#13377) 2024-07-02 08:25:42 +03:00
install_server.sh Do not install SysV init-scripts on systemd-enabled hosts 2019-11-19 18:55:44 +02:00
redis-copy.rb Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
redis-sha1.rb Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
redis_init_script Add INIT INFO to the provided init script. 2018-03-26 11:29:16 +02:00
redis_init_script.tpl Update redis_init_script.tpl 2014-10-26 11:09:45 -07:00
reply_schema_linter.js Add reply_schema to command json files (internal for now) (#10273) 2023-03-11 10:14:16 +02:00
req-res-log-validator.py Vector Sets: VISMEMBER and other fixes (#13941) 2025-04-15 22:58:57 +03:00
speed-regression.tcl Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
systemd-redis_multiple_servers@.service Provide example systemd service unit files for redis-server 2019-11-19 18:55:44 +02:00
systemd-redis_server.service Start redis after network is online (#7639) 2020-08-11 12:30:32 +03:00
tracking_collisions.c Adding AGPLv3 as a license option to Redis! (#13997) 2025-05-01 14:04:22 +01:00
whatisdoing.sh Update old links for modules-api-ref.md (#13479) 2024-11-04 18:18:22 +02:00