FROM sonarsource/local-travis

RUN apt-get -y update && apt-get install -y build-essential cmake python python-pip

RUN pip install redis

This is what ReJSON (https://github.com/redislabsmodules/rejson) currently has ready for the MVP:

* Code is under src/

* Building with CMake

- Need to verify on OSX

- Currently does not have an `install` option - needed?

* Documentation

- docs/commands.md

- docs/design.md ~ 30% done

- README.md ~ 85% done

- Missing about/what is ReJSON

- Some notes about performance

- Perhaps a Node.js example

- Source code is about 90% documented

* AGPLv3 license

* Copyright

1. Peer review of project CTO->RnD/QA?

1. Number overflows in number operations

1. Something is printing "inf"

1. Review and standardize use of int/size_t/uint32...

1. Improve/remove error reporting and logging in case of module internal errors

1. Need to include a simple standalone "benchmark", either w/ redis-benchmark or not ~ 30% done, need to complete some suites and generate graphs from output

TBD

1. A session token that also has a list of last seen times, what stack though

1. Node.js example perhaps

References:

* [Parsing JSON Is A Minefield](http://seriot.ch/parsing_json.php)

1. Memory usage: implemented JSON.MEMORY, need to compile an automated reporting tool

1. Performance with callgrind and/or gperftools

1. Run https://github.com/nst/JSONTestSuite and report like http://seriot.ch/parsing_json.php

1. Need a dependable cycle to check for memory leaks

1. Once we have a way to check baseline performance, add regression

1. Fuzz all module commands with a mix of keys paths and values

1. Memory leaks suite to run

`valgrind --tool=memcheck --suppressions=../redis/src/valgrind.sup ../redis/src/redis-server --loadmodule ./lib/rejson.so`

1. Verify each command's syntax - need a YAML

1. Add CI to repo?

1. Add array slice

Encode as trie over a certain size threshold to save memory and increase lookup performance. Alternatively, use a hash dictionary.

Integrate with @dvirsky's `secondary` library.

Support [JSON Schema](http://json-schema.org/).

JSON.SETSCHEMA <key> <json>

Notes:

1. Could be replaced by a JSON.SET modifier

2. Indexing will be specified in the schema

3. Cluster needs to be taken into account as well

JSON.VALIDATE <schema_key> <json>

JSON.EXPIRE <key> <path> <ttl>

Manage a cache inside the module for frequently accessed object in order to avoid repeatative

serialization.

Add a node type that references a Redis key that is either a JSON data type or a regular Redis key.

The module's API can transparently support resolving referenced keys and querying the data in them.

KeyRefs in cluster mode will only be allowed if in the same slot.

Redis core data types can be mapped to flat (i.e. non-nested) JSON structure as follows:

* A Redis String is a JSON String (albeit some escaping may be needed)

* A List: a JSON Array of JSON Strings, where the indices are the identical

* A Hash: a JSON Object where the Hash fields are the Object's keys and the values are JSON Strings

* A Set: a JSON Object where the Set members are the keys and their values are always a JSON Null

* A Sorted Set: a JSON Array that is made from two elements:

* A JSON Object where each key is a member and value is the score

* A JSON Array of all members ordered by score in ascending order

Compress (string only? entire objects?) values over a (configureable?) size threshold with zstd.

JSON.STATS

Print statistics about encountered values, parsing performance and such

JSON.OBJSET <key> <path> <value>

An alias for 'JSON.SET'

JSON.COUNT <key> <path> <json-scalar>

P: count JS: ? R: N/A

Counts the number of occurances for scalar in the array

JSON.REMOVE <key> <path> <json-scalar> [count]

P: builtin del JS: ? R: LREM (but also has a count and direction)

Removes the first `count` occurances (default 1) of value from array. If index is negative,

traversal is reversed.

JSON.EXISTS <key> <path>

P: in JS: ? R: HEXISTS/LINDEX

Checks if path key or array index exists. Syntactic sugar for JSON.TYPE.

JSON.REVERSE <key> <path>

P: reverse JS: ? R: N/A

Reverses the array. Nice to have.

JSON.SORT <key> <path>

P: sort JS: ? R: SORT

Sorts the values in an array. Nice to have.

title,size,concurrency,rate,avgLatency,50.00%-tile,90.00%-tile,95.00%-tile,99.00%-tile,99.50%-tile,100.00%-tile

JSON.SET {key} .,380,16,52537.50,0.30,0.27,0.43,0.50,0.64,0.69,7.05

JSON.GET {key} .,380,16,60343.12,0.26,0.24,0.36,0.44,0.59,0.65,6.41

JSON.GET {key} sclr,380,16,98514.45,0.16,0.15,0.23,0.26,0.33,0.36,4.88

JSON.SET {key} sclr,380,16,84124.03,0.19,0.17,0.27,0.31,0.39,0.42,7.10

JSON.SET {key} .,1441,16,30105.82,0.53,0.47,0.75,0.90,1.12,1.20,6.80

JSON.GET {key} .,1441,16,17754.51,0.90,0.77,1.29,1.55,1.94,2.00,7.61

JSON.GET {key} [0],1441,16,84786.94,0.19,0.17,0.27,0.30,0.38,0.41,7.09

JSON.SET {key} [0],1441,16,82954.30,0.19,0.18,0.27,0.31,0.39,0.43,5.71

JSON.SET {key} .,3468,16,21529.24,0.74,0.70,0.81,0.96,1.52,1.68,5.88

JSON.GET {key} .,3468,16,10497.95,1.52,1.45,1.58,1.99,3.14,3.51,8.57

"JSON.GET {key} [""web-app""].servlet[0][""servlet-name""]",3468,16,83277.94,0.19,0.18,0.26,0.30,0.37,0.40,6.64

"JSON.SET {key} [""web-app""].servlet[0][""servlet-name""] ""bar""",3468,16,84622.61,0.19,0.17,0.27,0.31,0.39,0.42,5.79

JSON.SET {key} .,18446,16,5664.91,2.82,2.73,2.83,3.16,5.38,6.25,12.30

JSON.GET {key} .,18446,16,1062.04,15.05,14.82,15.05,15.93,22.90,28.02,48.96

JSON.GET {key} ResultSet.totalResultsAvailable,18446,16,92883.96,0.17,0.16,0.24,0.27,0.35,0.39,7.41

JSON.SET {key} ResultSet.totalResultsAvailable 1,18446,16,83471.43,0.19,0.17,0.27,0.31,0.38,0.41,8.69

JSON.SET {key} .,39491,16,2953.48,5.41,5.33,5.45,5.54,8.39,9.99,16.92

JSON.GET {key} .,39491,16,513.05,31.13,30.18,32.18,36.13,58.42,70.90,84.10

JSON.GET {key} message.code,39491,16,91236.38,0.17,0.16,0.24,0.27,0.34,0.38,7.73

JSON.SET {key} message.code 1,39491,16,81865.64,0.19,0.18,0.27,0.30,0.38,0.42,5.66

import matplotlib.pyplot as plt

import numpy as np

data = np.genfromtxt('benchmark.csv', delimiter=',', names=True, dtype=None)

NOP = 4

N = int(len(data) / NOP)

ind = np.arange(N)

width = (1 - .3) / NOP

colors = ['r','g','b','y']

fig, ax = plt.subplots()

for oidx in range(NOP):

d = data[oidx::NOP]

# ax1 = ax[oidx]

# plt.subplot(NOP,1,oidx+1)

# Plot the rate as bars

plt.bar(ind+oidx*width, d['rate'], width, align='center', color=colors[oidx])

# Plot the latency as line

# tax = plt.twinx()

# tax.set_yscale('log')

# tax.plot(ind, d['avgLatency'], color='r')

# plt.title('{} rate and average latency'.format(d['title'][0]))

# ax1.set_ylabel('Rate (op/s)', color='b')

# ax1.set_xlabel('Object size (bytes)')

# for t in ax1.get_yticklabels():

# t.set_color('b')

# ax2.set_ylabel('Average latency (msec)', color='r')

# for t in ax2.get_yticklabels():

# t.set_color('r')

# plt.xticks(ind, d['size'])

plt.grid(True)

plt.xticks(ind + width, d['size'])

plt.show()

title,concurrency,rate,average latency,50.00%-tile,90.00%-tile,95.00%-tile,99.00%-tile,99.50%-tile,100.00%-tile

JSON.SET {key} . {size: 2 B},16,79313.90,0.20,0.18,0.29,0.32,0.40,0.43,9.41

JSON.GET {key} .,16,96787.14,0.16,0.15,0.23,0.26,0.33,0.36,7.23

JSON.SET {key} . {size: 380 B},16,52537.50,0.30,0.27,0.43,0.50,0.64,0.69,7.05

JSON.GET {key} .,16,60343.12,0.26,0.24,0.36,0.44,0.59,0.65,6.41

JSON.GET {key} sclr,16,98514.45,0.16,0.15,0.23,0.26,0.33,0.36,4.88

JSON.SET {key} sclr {size: 1 B},16,84124.03,0.19,0.17,0.27,0.31,0.39,0.42,7.10

JSON.GET {key} sub_doc,16,79419.40,0.20,0.18,0.29,0.32,0.41,0.45,5.73

JSON.GET {key} sub_doc.sclr,16,89612.06,0.18,0.16,0.25,0.28,0.35,0.39,7.44

JSON.GET {key} array_of_docs,16,67806.08,0.23,0.21,0.34,0.40,0.51,0.57,6.45

JSON.GET {key} array_of_docs[1],16,76575.85,0.21,0.19,0.30,0.33,0.42,0.47,5.26

JSON.GET {key} array_of_docs[1].sclr,16,89853.27,0.18,0.16,0.25,0.28,0.35,0.38,6.96

JSON.SET {key} . {size: 1.4 kB},16,30105.82,0.53,0.47,0.75,0.90,1.12,1.20,6.80

JSON.GET {key} .,16,17754.51,0.90,0.77,1.29,1.55,1.94,2.00,7.61

JSON.GET {key} [0],16,84786.94,0.19,0.17,0.27,0.30,0.38,0.41,7.09

JSON.SET {key} [0] {size: 5 B},16,82954.30,0.19,0.18,0.27,0.31,0.39,0.43,5.71

JSON.GET {key} [7],16,89787.85,0.18,0.16,0.25,0.28,0.34,0.38,5.81

JSON.GET {key} [8].zero,16,89826.21,0.18,0.16,0.24,0.28,0.35,0.39,8.95

JSON.SET {key} . {size: 3.5 kB},16,21529.24,0.74,0.70,0.81,0.96,1.52,1.68,5.88

JSON.GET {key} .,16,10497.95,1.52,1.45,1.58,1.99,3.14,3.51,8.57

"JSON.GET {key} [""web-app""]",16,10726.16,1.49,1.44,1.51,1.70,2.82,3.22,7.11

"JSON.GET {key} [""web-app""].servlet[0]",16,17250.85,0.92,0.89,1.00,1.09,1.77,1.97,7.10

"JSON.SET {key} [""web-app""].servlet[0][""servlet-name""] {size: 5 B}",16,84622.61,0.19,0.17,0.27,0.31,0.39,0.42,5.79

"JSON.GET {key} [""web-app""].servlet[0][""servlet-name""]",16,83277.94,0.19,0.18,0.26,0.30,0.37,0.40,6.64

JSON.SET {key} . {size: 18 kB},16,5664.91,2.82,2.73,2.83,3.16,5.38,6.25,12.30

JSON.GET {key} .,16,1062.04,15.05,14.82,15.05,15.93,22.90,28.02,48.96

JSON.SET {key} ResultSet.totalResultsAvailable {size: 1 B},16,83471.43,0.19,0.17,0.27,0.31,0.38,0.41,8.69

JSON.GET {key} ResultSet.totalResultsAvailable,16,92883.96,0.17,0.16,0.24,0.27,0.35,0.39,7.41

JSON.SET {key} . {size: 40 kB},16,2953.48,5.41,5.33,5.45,5.54,8.39,9.99,16.92

JSON.GET {key} .,16,513.05,31.13,30.18,32.18,36.13,58.42,70.90,84.10

JSON.SET {key} message.code {size: 1 B},16,81865.64,0.19,0.18,0.27,0.30,0.38,0.42,5.66

JSON.GET {key} message.code,16,91236.38,0.17,0.16,0.24,0.27,0.34,0.38,7.73

The purpose of this module is to provide native support for JSON documents stored in Redis, allowing

users to:

1. Store a JSON blob

2. Manipulate just a part of the json object without retrieving it to the client

3. Retrieve just a portion of the object as JSON

Later on, we can use the inernal object implementation in this module to produce similar modules for

other serialization formats, namely XML and BSON.

* Documents are added as JSON but are stored in an internal representation and not as strings.

* Internal representation does not depend on any JSON parser or library, to allow connecting other formats to it later.

* The internal representation will initially be limited to the types supported by JSON, but can later be extended to types like timestamps, etc.

* Queries that include internal paths of objects will be expressed in JSON path expressionse (e.g. `foo.bar[3].baz`)

* We will not implement our own JSON parser and composer, but use existing libraries.

* The code apart from the implementation of the redis commands will not depend on redis and will be testable without being compiled as a module.

The internal representation of JSON objects will be stored in a redis data type called Object [TBD].

These will be optimized for memory efficiency and path search speed.

See [src/object.h](src/object.h) for the API specification.

When updating, reading and deleting parts of json objects, we'll use path specifiers.

These too will have internal representation disconnected from their JSON path representation.

We only support a limited subset of it. Furthermore, jsonsl's jpr implementation may be worth looking into.

| JSONPath | rejson | Description |

| ---------------- | ----------- | ----------------------------------------------------------------- |

| `$` | key name | the root element |

| `*` | N/A #1 | wildcard, can be used instead of name or index |

| `..` | N/A #2 | recursive descent a.k.a deep scan, can be used instead of name |

| `.` or `[]` | `.` or `[]` | child operator |

| `[]` | `[]` | subscript operator |

| `[,]` | N/A #3 | Union operator. Allows alternate names or array indices as a set. |

| `@` | N/A #4 | the current element being proccessed by a filter predicate |

| [start:end:step] | N/A #3 | array slice operator |

| ?() | N/A #4 | applies a filter (script) expression |

| () | N/A #4 | script expression, using the underlying script engine |

ref: http://goessner.net/articles/JsonPath/

1. Wildcard should be added, but mainly useful for filters

1. Deep scan should be added

1. Union and slice operators should be added to ARR*, GET, MGET, DEL...

1. Filtering and scripting (min,max,...) should wait until some indexing is supported

google_analytics:

- 'UA-89559992-1'

- 'auto'