Commit Graph

233 Commits

Author SHA1 Message Date
antirez 68681f2bcf Fix issue #4505, Lua RDB AUX field loading of existing scripts.
Unfortunately, as outlined by @soloestoy in #4505, "lua" AUX RDB field
loading in case of duplicated script was still broken. This commit fixes
this problem and also a memory leak introduced by the past commit.

Note that now we have a regression test able to duplicate the issue, so
this commit was actually tested against the regression. The original PR
also had a valid fix, but I prefer to hide the details of scripting.c
outside scripting.c, and later "SCRIPT LOAD" should also be able to use
the function luaCreateFunction() instead of redoing the work.
2017-12-04 10:33:04 +01:00
antirez 65a9740fa8 Fix loading of RDB files lua AUX fields when the script is defined.
In the case of slaves loading the RDB from master, or in other similar
cases, the script is already defined, and the function registering the
script should not fail in the assert() call.
2017-12-01 16:01:10 +01:00
antirez 045d65c3af PSYNC2: Fix off by one buffer size in luaCreateFunction(). 2017-11-30 18:38:29 +01:00
antirez 452ad2e928 PSYNC2: just store script bodies into RDB.
Related to #4483. As suggested by @soloestoy, we can retrieve the SHA1
from the body. Given that in the new implementation using AUX fields we
ended copying around a lot to create new objects and strings, extremize
such concept and trade CPU for space inside the RDB file.
2017-11-30 18:38:26 +01:00
antirez 28dfdca733 PSYNC2: luaCreateFunction() should handle NULL client parameter.
See #4483. This is needed because luaCreateFunction() is now called
from RDB loading code outside a client context.
2017-11-30 18:37:52 +01:00
Itamar Haber 59d52f7fab Standardizes the 'help' subcommand
This adds a new `addReplyHelp` helper that's used by commands
when returning a help text. The following commands have been
touched: DEBUG, OBJECT, COMMAND, PUBSUB, SCRIPT and SLOWLOG.

WIP

Fix entry command table entry for OBJECT for HELP option.

After #4472 the command may have just 2 arguments.

Improve OBJECT HELP descriptions.

See #4472.

WIP 2

WIP 3
2017-11-28 21:15:45 +02:00
Yossi Gottlieb 2c70d28295 Nested MULTI/EXEC may replicate in different cases.
For example:
1. A module command called within a MULTI section.
2. A Lua script with replicate_commands() called within a MULTI section.
3. A module command called from a Lua script in the above context.
2017-11-22 22:02:51 +02:00
liangsijian ffbbe5a720 Fix lua ldb command log 2017-07-24 19:24:06 +08:00
antirez 7bad78bd2f Fix abort typo in Lua debugger help screen. 2017-06-30 12:12:00 +02:00
antirez 4a850be4dc Set lua-time-limit default value at safe place.
Otherwise, as it was, it will overwrite whatever the user set.

Close #3703.
2017-04-11 16:56:00 +02:00
Salvatore Sanfilippo b44ad302d2 Merge pull request #732 from evilpacket/remove_dofile
Removes dofile() from Lua
2016-05-08 18:04:41 +02:00
antirez 2205c465ca Cluster: don't check scripts key slots during AOF loading. 2016-05-05 23:37:08 +02:00
Salvatore Sanfilippo f0fcc36ce1 Merge pull request #2956 from pkulchenko/global-protection-msg-typo
Update global protection error message
2016-05-05 17:26:35 +02:00
antirez e85d6f22cf Fix INFO commandstats reporting when argv is rewritten.
We want to report the original command in the stats, for example GEOADD,
even when what is actually executed is the ZADD implementation.
2016-03-02 08:56:50 +01:00
antirez f43c794b0b Scripting: handle trailing comments.
This fix, provided by Paul Kulchenko (@pkulchenko), allows the Lua
scripting engine to evaluate statements with a trailing comment like the
following one:

    EVAL "print() --comment" 0

Lua can't parse the above if the string does not end with a newline, so
now a final newline is always added automatically. This does not change
the SHA1 of scripts since the SHA1 is computed on the body we pass to
EVAL, without the other code we add to register the function.

Close #2951.
2016-01-08 15:44:21 +01:00
antirez a75aa4bf92 Lua debugger: fix crash printing nested or deep objects.
Example of offending code:

> script debug yes
OK
> eval "local a = {1} a[1] = a\nprint(a)" 0
1) * Stopped at 1, stop reason = step over
2) -> 1   local a = {1} a[1] = a
> next
1) * Stopped at 2, stop reason = step over
2) -> 2   print(a)
> print

... server crash ...

Close #2955.
2016-01-08 09:14:13 +01:00
Salvatore Sanfilippo 075ea1646f Merge pull request #2954 from pkulchenko/debug-table-pretty-printing
Update pretty printing during debugging to generate valid Lua code for tables
2015-12-22 09:00:36 +01:00
Salvatore Sanfilippo f054b4ac47 Merge pull request #2957 from pkulchenko/debug-userdata-pretty-printing
Update pretty printing in debugging to generate valid Lua code for userdata-like types.
2015-12-22 08:59:48 +01:00
antirez b9aeb98156 Suppress harmless warnings. 2015-12-16 12:36:32 +01:00
Paul Kulchenko b754c8e18b Update pretty printing in debugging to generate valid Lua code for userdata-like types. 2015-12-15 20:24:41 -08:00
Paul Kulchenko 3969e9d3df Update pretty printing in debugging to generate valid Lua code for tables. 2015-12-15 18:15:39 -08:00
Paul Kulchenko 2f3f3fd5e0 Update global protection error message to fix a typo. 2015-12-15 18:13:09 -08:00
Itamar Haber 36801f7e43 Revert Lua's `redis.LOG_<level>` to original
Fixes #2898
2015-11-27 15:55:38 +02:00
antirez 4b0b28b469 Lua debugger: infinite loop detection. 2015-11-18 10:23:49 +01:00
antirez 1f35f2dd5a Lua debugger: fix trace command infinite loop.
Thanks to Itamar Haber for bug report and test case to reproduce.
2015-11-17 16:24:27 +01:00
antirez 0cc19174f9 Lua debugger: maxlen command implemented.
Let the user control the replies truncation.
2015-11-17 15:43:24 +01:00
antirez c560c645e9 Lua debugger: trace command implemented. 2015-11-17 15:43:24 +01:00
antirez 70a51694de Lua debugger: print without args show all local vars. 2015-11-17 15:43:23 +01:00
antirez fb53459ce8 Lua debugger: default behavior of "list" command changed.
Now it lists code around the current position by default. Can list any
other part using other arguments, but a new "whole" command was added in
order to show the whole source code easily.
2015-11-17 15:43:23 +01:00
antirez 34aadf79c3 Lua debugging: fix error message for SCRIPT DEBUG.
"async" -> "sync".

Thanks to Itamar Haber for reporting.
2015-11-17 15:43:23 +01:00
antirez 3d24cd6bf8 Lua debugger: reply +OK to SCRIPT DEBUG no.
Thanks to Itamar Haber for reporting.
2015-11-17 15:43:23 +01:00
antirez 333547dab6 Lua debugger: call wait3() if there are pending forked debugging sessions. 2015-11-17 15:43:23 +01:00
antirez 01636435fa Lua debugger: abort implemented. 2015-11-17 15:43:23 +01:00
antirez d99ce09343 Lua debugger: ldbSendLogs() memory leak fixed. 2015-11-17 15:43:23 +01:00
antirez 87672adee2 Lua debugger: better support for synchronous mode. 2015-11-17 15:43:22 +01:00
antirez 7be9170585 Lua debugger: handle forked sessions children during shutdown. 2015-11-17 15:43:22 +01:00
antirez 56d9bb8c55 Lua debugger: fix help typo, beark -> break. 2015-11-17 15:43:22 +01:00
antirez e386cd8ccf Lua debugger: clear end of session protocol.
When the debugger exits now it produces an <endsession> tag that informs
redis-cli (or other debugging clients) that the session terminated.
This way the client knows there is yet another reply to read (the one of
the EVAL script itself), and can switch to non-debugging mode ASAP.
2015-11-17 15:43:22 +01:00
antirez 7492237c3c Lua debugger: redis.debug() implemented. 2015-11-17 15:43:22 +01:00
antirez cd112db0ae Lua debugger: removing breakpoints now works. 2015-11-17 15:43:22 +01:00
antirez a076e421e9 Lua debugger: redis command implemented. 2015-11-17 15:43:22 +01:00
antirez e6eb6eadec Lua debugger: try to eval as expression first.
It's handly to just eval "5+5" without the return and see it printed on
the screen as result. However prepending "return" does not always result
into valid Lua code. So what we do is to exploit a common Lua community
trick of trying to compile with return prepended, and if compilation
fails then it's not an expression that can be returned, so we try again
without prepending "return". Works great apparently.
2015-11-17 15:43:21 +01:00
antirez 1f8fdafe65 Lua debugger: much better Lua values pretty printer. 2015-11-17 15:43:21 +01:00
antirez f4805800dc Lua debugger: print now handles ARGV and KEYS. 2015-11-17 15:43:21 +01:00
antirez 36392dd867 Lua debugger: added comment about helper functions. 2015-11-17 15:43:21 +01:00
antirez 3a04cb05ee Lua debugger: redis.breakpoint() implemented. 2015-11-17 15:43:21 +01:00
antirez cf4700bda4 Lua debugger: output improvements, eval command. 2015-11-17 15:43:21 +01:00
antirez 1f8d614423 Lua debugger: breakpoints. 2015-11-17 15:43:21 +01:00
antirez 5c4f492844 Lua debugger: ability to show local vars content. 2015-11-17 15:43:21 +01:00
antirez 5417217c87 Lua debugger: log Redis commands. List command. 2015-11-17 15:43:21 +01:00
antirez d3d1fa9437 Lua debugger: initial REPL. 2015-11-17 15:43:20 +01:00
antirez c494db89b5 Lua debugger: foundations implemented. 2015-11-17 15:43:20 +01:00
Salvatore Sanfilippo c950facf43 Merge pull request #2848 from badboy/removed-printf
Remove printf
2015-11-09 18:06:41 +01:00
antirez 1b5d24eb2b Scripting: fix redis.call() error reporting.
Arguments arity and arguments type error of redis.call() were not
reported correctly to Lua, so the command acted in this regard like
redis.pcall(), but just for two commands. Redis.call() should always
raise errors instead.
2015-11-09 11:43:51 +01:00
antirez cd8f19e9ca Initialize all Lua scripting related things into scripting.c 2015-11-05 11:37:39 +01:00
antirez 9aa1f94449 scripting.c source code better organized into sections. 2015-11-05 10:37:10 +01:00
Jan-Erik Rediger 35afefc7cb Remove printf 2015-11-04 11:48:14 +01:00
antirez ff6d296000 Scripting: ability to turn on Lua commands style replication globally.
Currently this feature is only accessible via DEBUG for testing, since
otherwise depending on the instance configuration a given script works
or is broken, which is against the Redis philosophy.
2015-10-30 12:06:09 +01:00
antirez ebaa9226ee Scripting: fix error reporting of many Redis provided functions. 2015-10-30 12:06:09 +01:00
antirez 2dabf82d5f Fix call() FORCE_REPL/AOF flags setting.
This commit also inverts two stanzas of the code just becuase they are
more logical like that, not because currently it makes any difference.
2015-10-30 12:06:08 +01:00
antirez 514a234722 Lua script selective replication fixes. 2015-10-30 12:06:08 +01:00
antirez a3e8de0430 Lua script selective replication WIP. 2015-10-30 12:06:08 +01:00
antirez fc38235664 Scripting: single commands replication mode implemented.
By calling redis.replicate_commands(), the scripting engine of Redis
switches to commands replication instead of replicating whole scripts.
This is useful when the script execution is costly but only results in a
few writes performed to the dataset.

Morover, in this mode, it is possible to call functions with side
effects freely, since the script execution does not need to be
deterministic: anyway we'll capture the outcome from the point of view
of changes to the dataset.

In this mode math.random() returns different sequences at every call.

If redis.replicate_commnads() is not called before any other write, the
command returns false and sticks to whole scripts replication instead.
2015-10-30 12:06:08 +01:00
antirez 4ff3c17a20 Lazyfree: client output buffers no longer use Redis Objects. 2015-10-01 13:02:24 +02:00
antirez 32f80e2f1b RDMF: More consistent define names. 2015-07-27 14:37:58 +02:00
antirez 40eb548a80 RDMF: REDIS_OK REDIS_ERR -> C_OK C_ERR. 2015-07-26 23:17:55 +02:00
antirez 2d9e3eb107 RDMF: redisAssert -> serverAssert. 2015-07-26 15:29:53 +02:00
antirez 14ff572482 RDMF: OBJ_ macros for object related stuff. 2015-07-26 15:28:00 +02:00
antirez 554bd0e7bd RDMF: use client instead of redisClient, like Disque. 2015-07-26 15:20:52 +02:00
antirez 424fe9afd9 RDMF: redisLog -> serverLog. 2015-07-26 15:17:43 +02:00
antirez cef054e868 RDMF (Redis/Disque merge friendlyness) refactoring WIP 1. 2015-07-26 15:17:18 +02:00
antirez 935251259f Merge branch 'sds' into unstable 2015-07-24 08:49:23 +02:00
Jiahao Huang 92c146dfd3 config tcp-keepalive should be numerical field not bool 2015-07-16 15:53:44 +08:00
Oran Agra f15df8ba5d sds size classes - memory optimization 2015-07-14 17:17:06 +02:00
Ben Murphy ffd6637e90 hide access to debug table 2015-06-03 13:33:28 +02:00
antirez 2f4240b9d9 Cluster: fix Lua scripts replication to slave nodes. 2015-03-22 22:24:08 +01:00
antirez e467cf5db3 luaRedisGenericCommand(): log error at WARNING level when re-entered.
Rationale is that when re-entering, it is likely due to Lua debugging
hooks. Returning an error will be ignored in most cases, going totally
unnoticed. With the log at least we leave a trace.

Related to issue #2302.
2015-01-20 23:21:21 +01:00
antirez bc8675612f luaRedisGenericCommand() recursion: just return an error.
Instead of calling redisPanic() to abort the server.

Related to issue #2302.
2015-01-20 23:16:19 +01:00
antirez 839767ad0b Panic on recursive calls to luaRedisGenericCommand().
Related to issue #2302.
2015-01-20 18:02:26 +01:00
antirez da95d22ad2 Prevent Lua scripts from violating Redis Cluster keyspace access rules.
Before this commit scripts were able to access / create keys outside the
set of hash slots served by the local node.
2015-01-09 11:23:22 +01:00
Matt Stancliff 3fecb96122 Lua: Add bitop
A few people have written custom C commands because bit
manipulation isn't exposed through Lua.  Let's give
them Mike Pall's bitop.

This adds bitop 1.0.2 (2012-05-08) from http://bitop.luajit.org/

bitop is imported as "bit" into the global namespace.

New Lua commands: bit.tobit, bit.tohex, bit.bnot, bit.band, bit.bor, bit.bxor,
bit.lshift, bit.rshift, bit.arshift, bit.rol, bit.ror, bit.bswap

Verification of working (the asserts would abort on error, so (nil) is correct):
127.0.0.1:6379> eval "assert(bit.tobit(1) == 1); assert(bit.band(1) == 1); assert(bit.bxor(1,2) == 3); assert(bit.bor(1,2,4,8,16,32,64,128) == 255)" 0
(nil)
127.0.0.1:6379> eval 'assert(0x7fffffff == 2147483647, "broken hex literals"); assert(0xffffffff == -1 or 0xffffffff == 2^32-1, "broken hex literals"); assert(tostring(-1) == "-1", "broken tostring()"); assert(tostring(0xffffffff) == "-1" or tostring(0xffffffff) == "4294967295", "broken tostring()")' 0
(nil)

Tests also integrated into the scripting tests and can be run with:
./runtest --single unit/scripting

Tests are excerpted from `bittest.lua` included in the bitop distribution.
2014-10-09 11:51:30 -04:00
Juarez Bochi a63b9c24f0 Fix typo in scripting.c comment
Closes #1960
2014-09-29 06:49:08 -04:00
antirez 12b56a969f luaRedisGenericCommand() cached argv handling simplified.
As discussed in issue #1945.
2014-09-10 15:45:59 +02:00
Paddy Byers 86cde329d1 Store the length of the static argv when first allocated. 2014-09-01 10:42:27 +02:00
antirez edca2b14d2 Remove warnings and improve integer sign correctness. 2014-08-13 11:44:38 +02:00
Matt Stancliff f17f8521f0 scripting: no eval with negative key count
Negative key count causes segfault in Lua functions.

Fixes #1842
Closes #1843
2014-08-07 12:38:36 +02:00
antirez 96e0fe6232 Fix semantics of Lua calls to SELECT.
Lua scripts are executed in the context of the currently selected
database (as selected by the caller of the script).

However Lua scripts are also free to use the SELECT command in order to
affect other DBs. When SELECT is called frm Lua, the old behavior, before
this commit, was to automatically set the Lua caller selected DB to the
last DB selected by Lua. See for example the following sequence of
commands:

    SELECT 0
    SET x 10
    EVAL "redis.call('select','1')" 0
    SET x 20

Before this commit after the execution of this sequence of commands,
we'll have x=10 in DB 0, and x=20 in DB 1.

Because of the problem above, there was a bug affecting replication of
Lua scripts, because of the actual implementation of replication. It was
possible to fix the implementation of Lua scripts in order to fix the
issue, but looking closely, the bug is the consequence of the behavior
of Lua ability to set the caller's DB.

Under the old semantics, a script selecting a different DB, has no simple
ways to restore the state and select back the previously selected DB.
Moreover the script auhtor must remember that the restore is needed,
otherwise the new commands executed by the caller, will be executed in
the context of a different DB.

So this commit fixes both the replication issue, and this hard-to-use
semantics, by removing the ability of Lua, after the script execution,
to force the caller to switch to the DB selected by the Lua script.

The new behavior of the previous sequence of commadns is to just set
X=20 in DB 0. However Lua scripts are still capable of writing / reading
from different DBs if needed.

WARNING: This is a semantical change that will break programs that are
conceived to select the client selected DB via Lua scripts.

This fixes issue #1811.
2014-06-12 16:05:52 +02:00
antirez 73fefd0bc0 Scripting: Fix for a #1118 regression simplified.
It is more straightforward to just test for a numerical type avoiding
Lua's automatic conversion. The code is technically more correct now,
however Lua should automatically convert to number only if the original
type is a string that "looks like a number", and not from other types,
so practically speaking the fix is identical AFAIK.
2014-06-11 10:10:58 +02:00
Matt Stancliff 76efe1225f Scripting: Fix regression from #1118
The new check-for-number behavior of Lua arguments broke
users who use large strings of just integers.

The Lua number check would convert the string to a number, but
that breaks user data because
Lua numbers have limited precision compared to an arbitrarily
precise number wrapped in a string.

Regression fixed and new test added.

Fixes #1118 again.
2014-06-10 14:26:13 -04:00
antirez 3758f27bc1 Fixed dbuf variable scope in luaRedisGenericCommand().
I'm not sure if while the visibility is the inner block, the fact we
point to 'dbuf' is a problem or not, probably the stack var isx
guaranteed to live until the function returns. However obvious code is
better anyway.
2014-06-04 18:57:12 +02:00
antirez 072982d83c Scripting: better Lua number -> string conversion in luaRedisGenericCommand().
The lua_to*string() family of functions use a non optimal format
specifier when converting integers to strings. This has both the problem
of the number being converted in exponential notation, which we don't
use as a Redis return value when floating point numbers are involed,
and, moreover, there is a loss of precision since the default format
specifier is not able to represent numbers that must be represented
exactly in the IEEE 754 number mantissa.

The new code handles it as a special case using a saner conversion.

This fixes issue #1118.
2014-06-04 18:33:24 +02:00
antirez ce2b2f22d9 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2014-05-20 16:15:13 +02:00
antirez 4ddc77041f Remove trailing spaces from scripting.c 2014-05-20 16:11:22 +02:00
michael-grunder ea0e2524aa Fix LUA_OBJCACHE segfault.
When scanning the argument list inside of a redis.call() invocation
for pre-cached values, there was no check being done that the
argument we were on was in fact within the bounds of the cache size.

So if a redis.call() command was ever executed with more than 32
arguments (current cache size #define setting) redis-server could
segfault.
2014-05-19 13:18:13 -07:00
antirez 4f686555ce Scripting: objects caching for Lua c->argv creation.
Reusing small objects when possible is a major speedup under certain
conditions, since it is able to avoid the malloc/free pattern that
otherwise is performed for every argument in the client command vector.
2014-05-07 16:12:32 +02:00
antirez 1e4ba6e7e6 Scripting: Use faster API for Lua client c->argv creation.
Replace the three calls to Lua API lua_tostring, lua_lua_strlen,
and lua_isstring, with a single call to lua_tolstring.

~ 5% consistent speed gain measured.
2014-05-07 16:12:32 +02:00
antirez 76fda9f8e1 Scripting: don't call lua_gc() after Lua script run.
Calling lua_gc() after every script execution is too expensive, and
apparently does not make the execution smoother: the same peak latency
was measured before and after the commit.

This change accounts for scripts execution speedup in the order of 10%.
2014-05-07 16:12:32 +02:00
antirez 48c49c4851 Scripting: cache argv in luaRedisGenericCommand().
~ 4% consistently measured speed improvement.
2014-05-07 16:12:32 +02:00
antirez 3318b74705 Fixed missing c->bufpos reset in luaRedisGenericCommand().
Bug introduced when adding a fast path to avoid copying the reply buffer
for small replies that fit into the client static buffer.
2014-05-07 16:12:32 +02:00
antirez c49955fd77 Scripting: replace tolower() with faster code in evalGenericCommand().
The function showed up consuming a non trivial amount of time in the
profiler output. After this change benchmarking gives a 6% speed
improvement that can be consistently measured.
2014-05-07 16:12:32 +02:00