Commit Graph

4171 Commits

Author SHA1 Message Date
antirez 0bbdaca6a0 PFDEBUG TODENSE added.
Converts HyperLogLogs from sparse to dense. Used for testing.
2014-04-16 09:05:42 +02:00
antirez 402110f9fd User-defined switch point between sparse-dense HLL encodings. 2014-04-15 17:46:51 +02:00
antirez d541f65d66 PFSELFTEST improved with sparse encoding checks. 2014-04-15 10:10:38 +02:00
antirez dde8dff73f PFDEBUG ENCODING added. 2014-04-14 19:35:00 +02:00
antirez 54f0156e8c Set HLL_SPARSE_MAX to 3000.
After running a few benchmarks, 3000 looks like a reasonable value to
keep HLLs with a few thousand elements small while the CPU cost is
still not huge.

This covers all the cases where the dense representation would use N
orders of magnitude more space, like in the case of many HLLs with
carinality of a few tens or hundreds.

It is not impossible that in the future this gets user configurable,
however it is easy to pick an unreasoable value just looking at savings
in the space dimension without checking what happens in the time
dimension.
2014-04-14 16:15:55 +02:00
antirez 848d0461f9 Error message for invalid HLL objects unified. 2014-04-14 16:11:54 +02:00
antirez 81ceef7d22 PFMERGE fixed to work with sparse encoding. 2014-04-14 16:09:32 +02:00
antirez 9df77fc0c4 Mark PFDEBUG as write command in the commands table.
It is safer since it is able to have side effects.
2014-04-14 15:57:50 +02:00
antirez 3bc35f9ce9 Correctly replicate PFDEBUG GETREG.
Even if it is a debugging command, make sure that when it forces a
change in encoding, the command is propagated.
2014-04-14 15:57:19 +02:00
antirez ba0afb4566 Added assertion in hllSparseAdd() when promotion to dense occurs.
If we converted to dense, a register must be updated in the dense
representation.
2014-04-14 15:55:21 +02:00
antirez e9cd51c7eb hllSparseAdd(): speed optimization.
Mostly by reordering opcodes check conditional by frequency of opcodes
in larger sparse-encoded HLLs.
2014-04-14 15:42:05 +02:00
antirez 681bf7468b Detect corrupted sparse HLLs in hllSparseSum(). 2014-04-14 15:20:26 +02:00
antirez db40da0a47 hllSparseAdd(): faster code removing conditional.
Bottleneck found profiling. Big run time improvement found when testing
after the change.
2014-04-14 12:58:46 +02:00
antirez 4e0a99ba51 Comment typo in hllSparseAdd(). first -> fits. 2014-04-14 12:12:53 +02:00
antirez 5532b5308a Merge adjacent VAL opcodes in hllSparseAdd().
As more values are added splitting ZERO or XZERO opcodes, try to merge
adjacent VAL opcodes if they have the same value.
2014-04-14 12:11:39 +02:00
antirez 837ca39081 More robust HLL_SPARSE macros protecting 'p' with parens.
Now the macros will work with arguments such as "ptr+1".
2014-04-14 11:49:53 +02:00
antirez 142d133c8a hllSparseAdd() opcode seek stop condition fixed. 2014-04-14 11:04:11 +02:00
antirez 1ee18db922 Fixed error message generation in PFDEBUG GETREG.
Bulk length for registers was emitted too early, so if there was a bug
the reply looked like a long array with just one element, blocking the
client as result.
2014-04-14 10:25:19 +02:00
antirez 82c31f750d Fixed memmove() count in hllSparseAdd(). 2014-04-14 09:40:07 +02:00
antirez 3b20003503 hllSparseAdd(): more correct dense conversion conditional.
We want to promote if the total string size exceeds the resulting size
after the upgrade.
2014-04-14 09:36:32 +02:00
antirez b7571b7453 hllSparseToDense(): sanity check added.
The function checks if all the HLL_REGISTERS were processed during the
convertion from sparse to dense encoding, returning REDIS_OK or
REDIS_ERR to signal a corruption problem.

A bug in PFDEBUG GETREG was fixed: when the object is converted to the
dense representation we need to reassign the new pointer to the header
structure pointer.
2014-04-14 09:27:01 +02:00
antirez f9dc3cb04d PFDEBUG DECODE added.
Provides a human readable description of the opcodes composing a
run-length encoded HLL (sparse encoding).
The command is only useful for debugging / development tasks.
2014-04-14 09:00:53 +02:00
antirez 261da523e8 PFDEBUG added, PFGETREG removed.
PFDEBUG will be the interface to do debugging tasks with a key
containing an HLL object.
2014-04-13 23:01:21 +02:00
antirez e8e717e145 hllSparseToDense API changed to take ref to object.
The new API takes directly the object doing everything needed to
turn it into a dense representation, including setting the new
representation as object->ptr.
2014-04-13 22:59:27 +02:00
antirez 2067644a8c hllSparseAdd() sanity check for span != 0 added. 2014-04-13 10:19:12 +02:00
antirez 80140fa006 Fix hllSparseAdd() new sequence replacement when next is NULL.
sdsIncrLen() must be called anyway even if we are replacing the last
oppcode of the sparse representation.
2014-04-12 23:55:44 +02:00
antirez 3c3c16561a Fix seqlen computation in hllSparseAdd(). 2014-04-12 23:52:36 +02:00
antirez a9e057e095 Abstract hllSparseAdd() / hllDenseAdd() via hllAdd(). 2014-04-12 23:42:56 +02:00
antirez 0b7d08efb9 hllSparseSum(): multiply 1 * runlen for zero entries. 2014-04-12 16:47:50 +02:00
antirez d9314079ca Macro HLL_SPARSE_XZERO_LEN fixed. 2014-04-12 16:46:08 +02:00
antirez f5c03044a6 Fix HLL sparse object creation #2.
Two vars initialized to wrong values in createHLLObject().
2014-04-12 16:37:50 +02:00
antirez b5659cb0a6 Increment pointer while iterating sparse HLL object. 2014-04-12 11:02:14 +02:00
antirez 1ccb661569 Fix HLL sparse object creation.
The function didn't considered the fact that each XZERO opcode is
two bytes.
2014-04-12 10:59:12 +02:00
antirez a79386b1af Create HyperLogLog objects with sparse encoding. 2014-04-12 10:56:18 +02:00
antirez 1fc04a6221 HyperLogLog sparse to dense conversion function. 2014-04-12 10:55:42 +02:00
antirez c756936b1d HyperLogLog sparse representation initial implementation.
Code never tested, but the basic layout is shaped in this commit.
Also missing:

1) Sparse -> Dense conversion function.
2) New HLL object creation using the sparse representation.
3) Implementation of PFMERGE for the sparse representation.
2014-04-11 17:34:32 +02:00
antirez 8ea5b46d30 hllCount() refactored to support multiple representations. 2014-04-11 10:25:07 +02:00
antirez 1efc1e052d hllAdd() refactored into two functions.
Also dense representation access macro renamed accordingly.
2014-04-11 09:47:52 +02:00
antirez d55474e558 HyperLogLog refactoring to support different encodings.
Metadata are now placed at the start of the representation as an header.
There is a proper structure to access the representation.
Still work to do in order to truly abstract the implementation from the
representation, commands still work assuming dense representation.
2014-04-11 09:26:45 +02:00
antirez 9c037ba85f HyperLogLog sparse representation slightly modified.
After running a few simulations with different alternative encodings,
it was found that the VAL opcode performs better using 5 bits for the
value and 2 bits for the run length, at least for cardinalities in the
range of interest.
2014-04-10 16:36:31 +02:00
antirez da2fbcf93d HyperLogLog sparse representation description and macros. 2014-04-09 18:56:00 +02:00
antirez 67bb2c46b2 Add casting to match printf format.
adjustOpenFilesLimit() and clusterUpdateSlotsWithConfig() that were
assuming uint64_t is the same as unsigned long long, which is true
probably for all the systems out there that we target, but still GCC
emitted a warning since technically they are two different types.
2014-04-07 08:58:06 +02:00
antirez 3a6a1e42f1 ZRANGEBYLEX and ZREVRANGEBYLEX implementation. 2014-04-05 11:41:43 +02:00
antirez d5be696db8 PFCOUNT: always unshare/decode the object.
This will be a non-op most of the times since the object will be
unshared / decoded, however it is more technically correct to start this
way since the object may be decoded even in the read-only code path.
2014-04-04 17:25:55 +02:00
antirez 1c12bcbcfb tryObjectEncoding() refactoring.
We also avoid to re-create an object that is already in EMBSTR encoding.
2014-04-04 17:25:35 +02:00
antirez 433ce7f85c Changed HyperLogLog hash seed to a non-zero value.
Using a seed of zero has the side effect of having the empty string
hashing to what is a very special case in the context of HyperLogLog: a
very long run of zeroes.

This did not influenced the correctness of the result with 16k registers
because of the harmonic mean, but still it is inconvenient that a so
obvious value maps to a so special hash.

The seed 0xadc83b19 is used instead, which is the first 64 bits of the
SHA1 of the empty string.

Reference: issue #1657.
2014-04-04 09:36:32 +02:00
antirez 352208ff68 Initial HyperLogLog tests. 2014-04-03 22:16:05 +02:00
antirez d2ca4bb62d Return "WRONGTYPE" error on PF* type mismatch. 2014-04-03 22:10:20 +02:00
antirez 349c978189 Fix PFADD infinite loop.
We need to guarantee that the last bit is 1, otherwise an element may
hash to just zeroes with probability 1/(2^64) and trigger an infinite
loop.

See issue #1657.
2014-04-03 19:31:26 +02:00
antirez b612affba3 Make hll-gnuplot-graph.rb callable from cli. 2014-04-03 16:38:11 +02:00