mirrors
/
redis
mirror of https://mirror.osredm.com/root/redis.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
redis/src
History
Eduardo Semprebon 91d0c758e5
Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 
For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:

- Avoid `LOADING` response during failed and successful synchronization for cases where the
  replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
  time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
  to spend the extra memory usage.

General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
  repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline. 
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
  and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
  during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
  here anyways to avoid complications.

Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
  be cases where async_loading should be checked as well and cases where it shouldn't (would require
  very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
  handle commands coming from the AOF client differently than ones coming from real clients, changed
  to check CLIENT_ID_AOF instead.

**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
  contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
  Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
  ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
  to allow modules to declare they support the diskless replication with async loading (when absent, we fall
  back to disk-based loading).

Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
 		2021-11-04 10:46:50 +02:00
..
modules Replace deprecated REDISMODULE_POSTPONED_ARRAY_LEN in module tests and examples (#9677) 2021-10-25 12:00:43 +03:00
.gitignore
…
Makefile Allow to override OPENSSL_PREFIX (#9567) 2021-09-30 15:51:19 +03:00
acl.c Fix sentinel commands, ACL dictIter leak (#9661) 2021-10-21 12:50:58 +03:00
adlist.c Free value if dup succeed but listAddNodeTail failed. (#8901) 2021-05-03 13:39:07 +03:00
adlist.h Speedup: unblock clients on keys in O(1). 2020-04-08 12:55:57 +02:00
ae.c Improve doc comment about AE_DONT_WAIT (#9165) 2021-06-29 14:37:02 +03:00
ae.h Use H/W Monotonic clock and updates to AE (#7644) 2020-08-28 11:54:10 +03:00
ae_epoll.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
ae_evport.c Typo: Change port_get to port_getn in aeApiPoll panic message. (#8920) 2021-05-07 15:34:35 +03:00
ae_kqueue.c Fix the timing of read and write events under kqueue (#9416) 2021-09-02 11:07:51 +03:00
ae_select.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
anet.c Implement anetPipe() to combine creating pipe and setting flags (#9511) 2021-10-06 16:08:13 +03:00
anet.h Implement anetPipe() to combine creating pipe and setting flags (#9511) 2021-10-06 16:08:13 +03:00
aof.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
asciilogo.h Changes http to https in texts (#8495) 2021-03-10 19:11:16 +02:00
atomicvar.h Several (mostly Solaris-related) cleanups (#8171) 2020-12-13 17:09:54 +02:00
bio.c Delete unused 'time' fields from struct bio_job (#9622) 2021-10-10 08:17:54 +03:00
bio.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
bitops.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
blocked.c Make tracking invalidation messages always after command's reply (#9422) 2021-10-07 15:13:42 +03:00
call_reply.c Fix callReplyParseCollection memleak when use AutoMemory (#9446) 2021-09-09 11:03:05 +03:00
call_reply.h Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
childinfo.c Implement anetPipe() to combine creating pipe and setting flags (#9511) 2021-10-06 16:08:13 +03:00
cli_common.c Added URI support to redis-benchmark (cli and benchmark share the same uri-parsing methods) (#9314) 2021-09-14 19:45:06 +03:00
cli_common.h Added URI support to redis-benchmark (cli and benchmark share the same uri-parsing methods) (#9314) 2021-09-14 19:45:06 +03:00
cluster.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
cluster.h Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
config.c Add timestamp annotations in AOF (#9326) 2021-10-25 13:08:34 +03:00
config.h Use fcntl(fd,F_FULLFSYNC) instead of fsync on OSX, improve power failure safety (#9545) 2021-10-15 08:44:25 +03:00
connection.c fix: call CLIENT INFO from redis module will crash the server (#8560) 2021-03-01 08:18:14 +02:00
connection.h Retry when a blocked connection system call is interrupted by a signal (#9629) 2021-11-04 09:09:28 +02:00
connhelpers.h Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crc16.c
…
crc16_slottable.h Added basic support for clusters to redis-benchmark. 2019-03-01 17:53:14 +01:00
crc64.c Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
crc64.h Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
crcspeed.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crcspeed.h Added crcspeed library 2020-04-24 17:11:21 -07:00
db.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
debug.c Add support for list type to store elements larger than 4GB (#9357) 2021-11-03 20:47:18 +02:00
debugmacro.h Fix some redundancy use of semicolon in do-while macros (#8221) 2020-12-21 22:57:45 -08:00
defrag.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
dict.c Slot-to-keys using dict entry metadata (#9356) 2021-08-30 23:25:36 -07:00
dict.h Slot-to-keys using dict entry metadata (#9356) 2021-08-30 23:25:36 -07:00
endianconv.c Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
endianconv.h Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
evict.c Replication backlog and replicas use one global shared replication buffer (#9166) 2021-10-25 09:24:31 +03:00
expire.c Add latency monitor sample when key is deleted via lazy expire (#9317) 2021-08-05 11:09:24 +03:00
fmacros.h Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
geo.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
geo.h
…
geohash.c FIX: truncate max/min longitude,latitude related geo_point (ex: {180, 85.05112878} ) 2020-04-09 17:48:29 +08:00
geohash.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
geohash_helper.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
geohash_helper.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
help.h Remove useless - from help.h on commands with no arguments (#8939) 2021-05-18 17:13:10 +03:00
hyperloglog.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
intset.c Fix Integer overflow issue with intsets (CVE-2021-32687) (#9586) 2021-10-04 12:09:25 +03:00
intset.h Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
latency.c dict struct memory optimizations (#9228) 2021-08-05 08:25:58 +03:00
latency.h Disable THP if enabled (#7381) 2020-10-27 15:04:18 +02:00
lazyfree.c Replication backlog and replicas use one global shared replication buffer (#9166) 2021-10-25 09:24:31 +03:00
listpack.c Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) 2021-10-04 12:11:02 +03:00
listpack.h Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) 2021-10-04 12:11:02 +03:00
listpack_malloc.h Optimize listpack for stream usage to avoid repeated reallocs (#6281) 2021-02-16 16:17:38 +02:00
localtime.c update leap year comment when div by 400 2018-09-18 17:04:00 +08:00
lolwut.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut.h add include guard for lolwut.h 2020-05-05 23:35:08 -04:00
lolwut5.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut6.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lzf.h
…
lzfP.h Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
lzf_c.c
…
lzf_d.c Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed 2020-12-06 14:54:34 +02:00
memtest.c Resolve nonsense static analysis warnings 2021-05-03 18:59:47 +03:00
mkreleasehdr.sh Update mkreleasehdr.sh 2019-11-19 17:23:47 +08:00
module.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
monotonic.c Use H/W Monotonic clock and updates to AE (#7644) 2020-08-28 11:54:10 +03:00
monotonic.h Use H/W Monotonic clock and updates to AE (#7644) 2020-08-28 11:54:10 +03:00
mt19937-64.c Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
mt19937-64.h Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
multi.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
networking.c Replication backlog and replicas use one global shared replication buffer (#9166) 2021-10-25 09:24:31 +03:00
notify.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
object.c Add support for list type to store elements larger than 4GB (#9357) 2021-11-03 20:47:18 +02:00
pqsort.c
…
pqsort.h
…
pubsub.c Treat subcommands as commands (#9504) 2021-10-20 11:52:57 +03:00
quicklist.c Add support for list type to store elements larger than 4GB (#9357) 2021-11-03 20:47:18 +02:00
quicklist.h Add support for list type to store elements larger than 4GB (#9357) 2021-11-03 20:47:18 +02:00
rand.c
…
rand.h
…
rax.c Simplify logic in raxSeek, eliminate it->key reassembly on gt and lt (#9115) 2021-06-30 08:21:19 +03:00
rax.h Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
rax_malloc.h Cluster: hash slots tracking using a radix tree. 2017-03-27 16:37:22 +02:00
rdb.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
rdb.h Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
redis-benchmark.c Cleanup typos, incorrect comments, and fixed small memory leak in redis-cli (#9153) 2021-10-02 22:19:33 -07:00
redis-check-aof.c Add checks for overflow in redis-check-aof and loadAppendOnlyFile (#9669) 2021-11-02 17:03:07 +02:00
redis-check-rdb.c Add support for list type to store elements larger than 4GB (#9357) 2021-11-03 20:47:18 +02:00
redis-cli.c Add sleep interval to redis-cli --scan option (#3751) 2021-10-13 16:54:35 +03:00
redis-trib.rb Redis-trib deprecated: it no longer works and it 2018-07-13 10:51:58 +02:00
redisassert.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
redisassert.h Sanitize dump payload: performance optimizations and tuning 2020-12-06 14:54:34 +02:00
redismodule.h Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
release.c Speedup INFO server section. 2019-10-02 11:30:20 +02:00
replication.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
resp_parser.c Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
resp_parser.h Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
rio.c Retry when a blocked connection system call is interrupted by a signal (#9629) 2021-11-04 09:09:28 +02:00
rio.h diskless replication rdb transfer uses pipe, and writes to sockets form the parent process. 2019-10-07 21:06:30 +03:00
scripting.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
sds.c sdstrim remove excessive check (#4045) 2021-10-17 20:37:52 +03:00
sds.h Query buffer shrinking improvements (#5013) 2021-07-05 09:30:16 +03:00
sdsalloc.h Sanitize dump payload: fail RESTORE if memory allocation fails 2020-12-06 14:54:34 +02:00
sentinel.c Sentinel: don't log auth-pass value for better security (#9652) 2021-10-26 13:13:12 +03:00
server.c Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
server.h Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) 2021-11-04 10:46:50 +02:00
setcpuaffinity.c cpu affinity: DragonFlyBSD support (#7956) 2020-10-25 14:14:05 +02:00
setproctitle.c Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
sha1.c Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
sha1.h Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
sha256.c ACL: add slightly modified version of sha256.c for password hashing. 2019-09-12 12:21:37 +02:00
sha256.h fix explanation of sha256 (#9220) 2021-07-10 10:04:54 -05:00
siphash.c Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
slowlog.c slowlog get command supports passing in -1 to get all logs. (#9018) 2021-06-14 16:46:45 +03:00
slowlog.h Added the missed macro definition in slowlog.h 2019-12-02 00:28:41 -05:00
solarisfixes.h
…
sort.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
sparkline.c Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
sparkline.h
…
stream.h Fix if consumer is created as a side effect without notify and dirty++ (#9263) 2021-08-02 08:31:33 +03:00
syncio.c
…
t_hash.c Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) 2021-10-04 12:11:02 +03:00
t_list.c Fixes LPOP/RPOP wrong replies when count is 0 (#9692) 2021-11-04 09:43:08 +02:00
t_set.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
t_stream.c XADD - skip rewrite the id arg if it was given and is valid. (#9599) 2021-10-11 13:09:18 +03:00
t_string.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
t_zset.c fix: lookupKey on SETNX and SETXX only once (#9640) 2021-11-03 14:12:33 +02:00
testhelp.h Fix some redundancy use of semicolon in do-while macros (#8221) 2020-12-21 22:57:45 -08:00
timeout.c Add missing calls to raxStop (#7532) 2020-07-21 08:13:05 +03:00
tls.c TLS build fix on OpenBSD when built with LibreSSL. (#9486) 2021-09-11 22:54:09 +03:00
tracking.c Make tracking invalidation messages always after command's reply (#9422) 2021-10-07 15:13:42 +03:00
util.c Client eviction (#8687) 2021-09-23 14:02:16 +03:00
util.h config memory limits: handle values larger than (signed) LLONG_MAX (#9313) 2021-08-23 21:00:40 +03:00
valgrind.sup Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed 2020-12-06 14:54:34 +02:00
version.h Add Module API for version and compatibility checks (#7865) 2020-10-11 17:21:58 +03:00
ziplist.c Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) 2021-10-04 12:11:02 +03:00
ziplist.h Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) 2021-10-04 12:11:02 +03:00
zipmap.c Sanitize dump payload: handle remaining empty key when RDB loading and restore command (#9349) 2021-08-09 17:13:46 +03:00
zipmap.h Add run all test support with define REDIS_TEST (#8570) 2021-03-10 09:13:11 +02:00
zmalloc.c fix a compilation error around madvise when make with jemalloc on MacOS (#9350) 2021-08-10 11:32:27 +03:00
zmalloc.h Use madvise(MADV_DONTNEED) to release memory to reduce COW (#8974) 2021-08-04 23:01:46 +03:00