Sync and async

Summary: Sync and async functions

Content

By default, each time a function is invoked, it is executed synchronously. This ensures the atomicity property, meaning that no other commands will be executed on Redis while the function is running. The atomicity property offers several advantages:

* Multiple keys can be updated simultaneously, guaranteeing that other clients see the complete update rather than partial updates.
* The data in Redis remains unchanged while it is being processed.

However, the major disadvantage of the atomicity property is that Redis is blocked throughout the entire invocation, preventing it from serving other clients.

Redis Stack's triggers and functions feature aims to provide greater flexibility to function writers by enabling the invocation of functions in the background. When a function is invoked in the background, it cannot directly access the Redis key space. To interact with the Redis key space from the background, the function must block Redis and enter an atomic section where the atomicity property is once again guaranteed.

To run triggers and functions in the background, functions can be implemented as JS coroutines using the `registerAsyncFunction` API. The coroutine is invoked on a background thread and does not block the Redis processes. Here's an example:

[code example]

The simple function shown above will return the value 'test' and will execute on a background thread without blocking Redis. This allows Redis to continue accepting commands from other clients while the function is running.

The coroutine also accepts an optional client argument, which differs from the client used in synchronous functions. This client argument does not allow direct invocation of Redis commands. Instead, it provides the capability to block Redis and enter an atomic section where the atomicity property is once again guaranteed. Here's an example that demonstrates invoking a ping command from within an async coroutine:

[code example]

Running this function will return a `pong` reply:

[code example]

Notice that this time, in order to invoke the function, we used `TFCALLASYNC`. **We can only invoke async functions using `TFCALLASYNC`**.

Now let's look at a more complex example. Assume we want to write a function that counts the number of hashes in Redis that have a `name` property with some value. As a first attempt, we'll write a synchronous function that uses the [`SCAN`]() command to scan the key space:

[code example]

While this function works, it has the potential to block Redis for a long time. So let's modify this function to run in the background as a coroutine:

[code example]

Both implementations return the same result, but the second function runs in the background and blocks Redis just to analyze the next batch of keys that are returned from the [`SCAN`]() command. Other commands will be processed in between [`SCAN`]() batches. Notice that the coroutine approach allows the key space to be changed while the scanning it. The function writer will need to decide if this is acceptable.

# Start sync and move to async

The previous example, although functional, has a drawback in terms of performance. Even though Redis is not blocked, it still takes time to return the reply to the user. However, if we modify the requirement slightly and agree to obtain an approximate value, we can achieve much better performance in most cases. This can be done by implementing result caching using a key named `<name>_count` and setting an expiration time on that key, which triggers recalculation of the value periodically. Here's the updated code:

[code example]

The above code works as expected. It first checks the cache and if the cache exists it's returned. Otherwise it will perform the calculation and update the cache. But the above example is not optimal. The callback is a coroutine, which means that it will always be calculated on a background thread. Intrinsically, moving to a background thread is costly. The best approach would be to check the cache synchronously and, only if its not there, move to the background. Triggers and functions provides for starting synchronously and then moving asynchronous execution using `executeAsync` function as required. The new code:

[code example]

`executeAsync` will return a `Promise` object. When triggers and functions sees that the function returns a Promise, it waits for the promise to be resolved and returns its result to the client. The above implementation will be much faster in the case of cache hit.

**Notice** that even though we registered a synchronous function (not a coroutine) we still used `registerAsyncFunction`. This is because our function has the potential of blocking the client, taking the execution to the background. If we had used `registerFunction`, Triggers and functions would not have allowed the function to block the client and it would have ignored the returned promise object.

**Also notice** it is not always possible to wait for a promise to be resolved. If the command is called inside a `multi/exec` it is not possible to block it and wait for the promise. In such cases the client will get an error. It is possible to check if blocking the client is allowed using the `client.isBlockAllowed()` function, which will return `true` if it is OK to wait for a promise to be resolved and `false` if it is not possible.

# Call blocking commands

Redis has a few commands that blocks the client and executed asynchronously when some condition holds (commands like [blpop]()). In general, such commands are not suppose to be called inside a script and calling them will result in running their none blocking logic. For example, [blpop]() will basically runs lpop and return empty result if the list it empty.

RedisGears allows running blocking commands using `client.callAsync` API. `client.callAsync` will execute the blocking command and return a promise object which will be resolved when the command invocation finished (notice that `client.callAsync` allow calling any command and not just blocking but it will always return a promise object that will be resolve later, so **using it for regular commands is less efficient**).

Example:

[code example]

The following function will continue popping elements from the requested list up until it will encounter the requested value. In case the list is empty it will wait until elements will be added to the list.

RedisGears also provided `client.callAsyncRaw` API, which is the same as `client.callAsync` but will not decode the replies as utf8.

**Notice**: There is no guarantee when the promise returned from `client.callAsyn` will be resolved. So the **function writer should not make any assumption about atomicity guarantees.**

# Block Redis failure

Blocking Redis might fail for a few reasons:

* Redis reached OOM state and the `redis.functionFlags.NO_WRITES` or `redis.functionFlags.ALLOW_OOM` flags are not set (see [functions flags]() for more information)
* `redis.functionFlags.NO_WRITES` flag is not set and the Redis instance changed roles and is now a replica.
* The ACL user that invoked the function was deleted.

The failure will result in an exception that the function writer can choose to handle or throw it to be caught by triggers and functions.

# Block Redis timeout

Blocking Redis for a long time is discouraged and is considered an unsafe operation. The triggers and functions feature attempts to protect the function writer and will time out the blocking function if it continues for too long. The timeout can be set as a [module configuration]() along side the fatal failure policy that indicates how to handle the timeout. Policies can be one of the following:

* Abort - Stop the function invocation even at the cost of losing the atomicity property.
* Kill - Keep the atomicity property and do not stop the function invocation. In this case there is a risk of an external process killing the Redis server, thinking that the shard is not responding.

#!js api_version=1.0 name=lib

redis.registerAsyncFunction('test', async function(){
    return 'test';
});

#!js api_version=1.0 name=lib

redis.registerFunction('test', async function(client){
    return client.block(function(redis_client){
        return redis_client.call('ping');
    });
});

127.0.0.1:6379> TFCALLASYNC lib.test 0
"PONG"

#!js api_version=1.0 name=lib

redis.registerFunction('test', function(client, expected_name){
    var count = 0;
    var cursor = '0';
    do{
        var res = client.call('scan', cursor);
        cursor = res[0];
        var keys = res[1];
        keys.forEach((key) => {
            if (client.call('hget', key, 'name') == expected_name) {
                count += 1;
            }
        });
    } while(cursor != '0');
    return count;
});

#!js api_version=1.0 name=lib

redis.registerAsyncFunction('test', async function(async_client, expected_name){
    var count = 0;
    var cursor = '0';
    do{
        async_client.block((client)=>{
            var res = client.call('scan', cursor);
            cursor = res[0];
            var keys = res[1];
            keys.forEach((key) => {
                if (client.call('hget', key, 'name') == expected_name) {
                    count += 1;
                }
            });
        });
    } while(cursor != '0');
    return count;
});

#!js api_version=1.0 name=lib

redis.registerAsyncFunction('test', async function(async_client, expected_name){
    // check the cache first
    var cached_value = async_client.block((client)=>{
        return client.call('get', expected_name + '_count');
    });

    if (cached_value != null) {
        return cached_value;
    }

    var count = 0;
    var cursor = '0';
    do{
        async_client.block((client)=>{
            var res = client.call('scan', cursor);
            cursor = res[0];
            var keys = res[1];
            keys.forEach((key) => {
                if (client.call('hget', key, 'name') == expected_name) {
                    count += 1;
                }
            });
        });
    } while(cursor != '0');

    // set count to the cache wil 5 seconds expiration
    async_client.block((client)=>{
        client.call('set', expected_name + '_count', count);
        client.call('expire', expected_name + '_count', 5);
    });

    return count;
});

#!js api_version=1.0 name=lib

redis.registerAsyncFunction('test', function(client, expected_name){
    // check the cache first
    var cached_value = client.call('get', expected_name + '_count');
    if (cached_value != null) {
        return cached_value;
    }

    // cache is not set, move to background
    return client.executeAsync(async function(async_client) {
        var count = 0;
        var cursor = '0';
        do{
            async_client.block((client)=>{
                var res = client.call('scan', cursor);
                cursor = res[0];
                var keys = res[1];
                keys.forEach((key) => {
                    if (client.call('hget', key, 'name') == expected_name) {
                        count += 1;
                    }
                });
            });
        } while(cursor != '0');

        // set count to the cache wil 5 seconds expiration
        async_client.block((client)=>{
            client.call('set', expected_name + '_count', count);
            client.call('expire', expected_name + '_count', 5);
        });

        return count;
    });
});

#!js api_version=1.0 name=lib

redis.registerAsyncFunction('my_blpop', async function(client, key, expected_val) {
    var res = null
    do {
        res = await client.block((c) => {
            return c.callAsync('blpop', key, '0');
        })
    } while (res[1] != expected_val);
    return res;
});


Triggers

Summary: Different ways of executing JavaScript functions

Summary: Execute JavaScript functions via `TFCALL` or `TFCALLASYNC`

User functions

Summary: Execute a JavaScript function based on a keyspace notification

Keyspace triggers

Summary: Execute a JavaScript function when an item is added to a stream

Stream triggers


Keyspace triggers

Overview

Keyspace triggers allow you to register a function that will be executed whenever an event occurs in the database. Most events are triggered by command invocations, but there are two special events that can occur independently of a command:

1. Expired: This event is fired when a key expires from the database.
2. Evicted: This event is fired when a key is evicted from the database.

For a complete list of supported events, please refer to the [Redis keyspace notifications page]().

To register a keyspace trigger, you need to use the `redis.registerKeySpaceTrigger` API when loading your library. The following example demonstrates how to register a database trigger that adds a "last updated" field whenever a hash key is modified:

[code example]

Argument Description:

* `consumer`: The consumer name.
* `prefix `: The key prefix on which the trigger should be fired.
* `callback`: The callback function to invoke, following the same rules of [Sync and Async invocation](). The callback will only be invoked on the primary shard.

Run the example:

[code example]

The `data` argument, which is passed to the consumer callback, are in the following format:

[code example]

Notice that the `key` field is given only if the key can be decoded as a `JS` `String`, otherwise the value will be `null`.

We can display trigger information using `TFUNCTION LIST` command:

[code example]

Trigger guarantees

If the callback function passed to the trigger is a `JS` function (not a Coroutine), it is guaranteed that the callback will be invoked atomically along side the operation that caused the trigger; meaning all clients will see the data only after the callback has completed. In addition, it is guaranteed that the effect of the callback will be replicated to the replica and the AOF in a `multi/exec` block together with the command that fired the trigger.

If the callback is a Coroutine, it will be executed in the background and there is no guarantee on where or if it will be executed. The guarantees are the same as described on [sync and async invocation]().

Upgrades

When upgrading existing trigger code using the `REPLACE` option of `TFUNCTION LOAD` command, all trigger parameters can be modified.

Advanced usage

For most use cases, the `registerKeySpaceTrigger` API is sufficient. But there are some use cases where you might need a better guarantee on when a trigger will be fired. Let's look at the following example:

[code example]

Whenever a hash key is changed, the example above will read the field `name` from the hash, for example `tom`, and increase the value of the key `name_tom`. (**Notice: This function will not work properly on cluster. We need to use `{}` on the key's name to make sure we are writing to a key located on the current shard. For the sake of simplicity we will ignore the cluster issues for the time being**). Running the function will give the following results:

[code example]

We can see that the key `name_tom` was incremented once, and the key `name_jerry` was incremented once. Will we get the same results if we wrap the `hset` call with a `multi`/`exec`?

[code example]

What just happened? `name_jerry` was incremented twice while `name_tom` was not incremented at all. This happened because, in case of a `multi`/`exec` or Lua function, the notifications are fired at the end of the transaction, so all the clients will receive notifications of the last value written, which is `jerry`.

To fix the code and still get the expected results even on `multi`/`exec`. Triggers and functions allows you to specify an optional callback that will run exactly when the notification happened (and not at the end of the transaction). The constraint on this callback is that it can only **read** data without performing any writes. The new code will be as follow:

[code example]

The above code gives an optional function argument, `onTriggerFired`, to our trigger. The function will be fired right after a key change and will allow us to read the content of the key. We are adding the content into the `data` argument, which will be given to the actual trigger function that can write the data. The above code works as expected:

[code example]

#!js api_version=1.0 name=myFirstLibrary

redis.registerKeySpaceTrigger("consumer", "", function(client, data){
    if (client.call("type", data.key) != "hash") {
        // key is not a has, do not touch it.
        return;
    }
    // get the current time in ms
    var curr_time = client.call("time")[0];
    // set '__last_updated__' with the current time value
    client.call('hset', data.key, '__last_updated__', curr_time);
});

<key name that the event was fired on as String>

<key name that the event was fired on as ArrayBuffer>

127.0.0.1:6379> TFUNCTION list vvv
1)  1) "engine"
    2) "js"
    3) "api_version"
    4) "1.0"
    5) "name"
    6) "foo"
    7) "pending_jobs"
    8) (integer) 0
    9) "user"
    10) "default"
    11) "functions"
   12) (empty array)
   13) "keyspace_triggers"
   14) (empty array)
   15) "stream_triggers"
   16) 1)  1) "name"
           2) "consumer"
           3) "num_triggered"
           4) (integer) 2
           5) "num_finished"
           6) (integer) 2
           7) "num_success"
           8) (integer) 1
           9) "num_failed"
          10) (integer) 1
          11) "last_error"
          12) "TypeError: redis.call is not a function"
          13) "last_exection_time"
          14) (integer) 0
          15) "total_exection_time"
          16) (integer) 0
          17) "avg_exection_time"
          18) "0"

#!js api_version=1.0 name=myFirstLibrary

redis.registerKeySpaceTrigger("consumer", "", function(client, data){
    if (client.call("type", data.key) != "hash") {
        // key is not a has, do not touch it.
        return;
    }
    var name = client.call('hget', data.key, 'name');
    client.call('incr', `name_${name}`);
});

#!js api_version=1.0 name=lib

redis.registerKeySpaceTrigger("consumer", "", function(client, data){
    if (data.name !== undefined) {
        client.call('incr', `name_${data.name}`);
    }
},{
    onTriggerFired: (client, data) => {
        if (client.call("type", data.key) != "hash") {
            // key is not a has, do not touch it.
            return;
        }
        data.name = client.call('hget', data.key, 'name');
    }
});


Stream triggers

Overview

Redis Stack's triggers and functions feature comes with a full stream API to process data from [Redis streams](). Unlike RedisGears v1 that provided a micro batching API, the new triggers and functions feature provides a **real streaming** API, which means that the data will be processed as soon as it enters the stream.

Register a stream consumer

Triggers and functions provide an API that allows to register a stream trigger. Do not get confused with [Redis streams consumer groups](), triggers and functions uses the Redis Module API to efficiently read the stream and manage its consumers. This approach gives a much better performance as there is no need to invoke any Redis commands in order to read from the stream. Lets see a simple example:

[code example]

Argument Description:

* consumer - the consumer name.
* stream - streams name prefix on which to trigger the callback.
* callback - the callback to invoke on each element in the stream. Following the same rules of [sync and async invocation](). The callback will be invoke only on primary shard.

If we register this library (see the [quick start]() section to learn how to Register a RedisGears function) and run the following command on our Redis:

[code example]

We will see the following line on the Redis log file:

[code example]

The `data` argument which pass to the stream consumer callback are in the following format:

[code example]

The reason why the record is a list of touples and not an object is because the Redis Stream specifications allows duplicate keys.

Notice that `stream_name` and `record` fields might contains `null`'s if the data can not be decoded as string. the `*_raw` fields will always be provided and will contains the data as `JS` `ArrayBuffer`.

We can observe the streams which are tracked by our registered consumer using `TFUNCTION LIST` command:

[code example]

Enable trimming and set window

We can enable stream trimming by adding `isStreamTrimmed` optional argument after the trigger callback, we can also set the `window` argument that controls how many elements can be processed simultaneously. example:

[code example]

The default values are:
* `isStreamTrimmed` - `false`
* `window` - 1

It is enough that a single consumer will enable trimming so that the stream will be trimmed. The stream will be trim according to the slowest consumer that consume the stream at a given time (even if this is not the consumer that enabled the trimming). Raising exception during the callback invocation will **not prevent the trimming**. The callback should decide how to handle failures by invoke a retry or write some error log. The error will be added to the `last_error` field on `TFUNCTION LIST` command.

Data processing guarantees

As long as the primary shard is up and running we guarantee exactly once property (the callback will be triggered exactly one time on each element in the stream). In case of failure such as shard crashing, we guarantee at least once property (the callback will be triggered at least one time on each element in the stream)

Upgrades

When upgrading the consumer code (using the `REPLACE` option of `TFUNCTION LOAD` command) the following consumer parameters can be updated:

* Window
* Trimming

Any attempt to update any other parameter will result in an error when loading the library.

#!js api_version=1.0 name=myFirstLibrary

redis.registerStreamTrigger(
    "consumer", // consumer name
    "stream", // streams prefix
    function(c, data) {
        // callback to run on each element added to the stream
        redis.log(JSON.stringify(data, (key, value) =>
            typeof value === 'bigint'
                ? value.toString()
                : value // return everything else unchanged
        ));
    }
);

XADD stream:1 * foo1 bar1
XADD stream:1 * foo2 bar2
XADD stream:2 * foo1 bar1
XADD stream:2 * foo2 bar2

{
    "id": ["<ms>", "<seq>"],
    "stream_name": "<stream name>",
    "stream_name_raw": "<stream name as ArrayBuffer>",
    "record":[
        ["<key>", "<value>"],
        .
        .
        ["<key>", "<value>"]
    ],
    "record_raw":[
        ["<key_raw>", "<value_raw>"],
        .
        .
        ["<key_raw>", "<value_raw>"]
    ],
}

127.0.0.1:6379> TFUNCTION LIST LIBRARY lib vvv
1)  1) "engine"
    1) "js"
    2) "api_version"
    3) "1.0"
    4) "name"
    5) "lib"
    6) "pending_jobs"
    7) (integer) 0
    8) "user"
    9)  "default"
    10) "functions"
   1)  (empty array)
   2)  "stream_triggers"
   3)  1)  1) "name"
           1) "consumer"
           2) "prefix"
           3) "stream"
           4) "window"
           5) (integer) 1
           6) "trim"
           7) "disabled"
           8) "num_streams"
          1)  (integer) 2
          2)  "streams"
          3)  1)  1) "name"
                  1) "stream:2"
                  2) "last_processed_time"
                  3) (integer) 0
                  4) "avg_processed_time"
                  5) "0"
                  6) "last_lag"
                  7) (integer) 0
                  8) "avg_lag"
                 1)  "0"
                 2)  "total_record_processed"
                 3)  (integer) 2
                 4)  "id_to_read_from"
                 5)  "1657030412715-0"
                 6)  "last_error"
                 7)  "None"
                 17) "pending_ids"
                 18) (empty array)
              1)  1) "name"
                  1) "stream:1"
                  2) "last_processed_time"
                  3) (integer) 1
                  4) "avg_processed_time"
                  5) "0.5"
                  6) "last_lag"
                  7) (integer) 1
                  8) "avg_lag"
                 1)  "0.5"
                 2)  "total_record_processed"
                 3)  (integer) 2
                 4)  "id_to_read_from"
                 5)  "1657030405323-0"
                 6)  "last_error"
                 7)  "None"
                 8)  "pending_ids"
                 9)  (empty array)
   4)  "keyspace_triggers"
   5)  (empty array)

#!js api_version=1.0 name=myFirstLibrary

redis.registerStreamTrigger(
    "consumer", // consumer name
    "stream", // streams prefix
    function(c, data) {
        // callback to run on each element added to the stream
        redis.log(JSON.stringify(data, (key, value) =>
            typeof value === 'bigint'
                ? value.toString()
                : value // return everything else unchanged
        ));
    },
    {
        isStreamTrimmed: true,
        window: 3
    }
);


User functions

Content

All `TFCALL` command arguments that follow the function name are passed to the function callback. The following example shows how to implement a simple function that returns the value of a key of type string or hash:

[code example]

Example of running the preceding function:

[code example]

It is also possible to get all the function arguments as a `JS` array. This is how we can extend the above example to accept multiple keys and return their values:

[code example]

Run example:

[code example]

#!js api_version=1.0 name=lib

redis.registerFunction('my_get', function(client, key_name){
    if (client.call('type', key_name) == 'string') {
        return client.call('get', key_name);
    }
    if (client.call('type', key_name) == 'hash') {
        return client.call('hgetall', key_name);
    }
    throw "Unsupported type";
});

#!js api_version=1.0 name=lib

redis.registerFunction('my_get', function(client, ...keys){
    var results = [];
    keys.forEach((key_name)=> {
            if (client.call('type', key_name) == 'string') {
                results.push(client.call('get', key_name));
                return;
            }
            if (client.call('type', key_name) == 'hash') {
                results.push(client.call('hgetall', key_name));
                return;
            }
            results.push("Unsupported type");
        }
    );
    return results;

});


Configuration

Summary: Configure the operation parameters

Overview

Redis Stack's triggers and functions feature provides configuration options to control its operation. These options can be set when the module is bootstrapped and, in some cases, at runtime.

The following sections describe the configuration options and how to set them.

Bootstrap configuration

You can set your configuration options when the module is loaded.
When the module is loaded at start time, the module configuration can be defined in the Redis configuration file. When loading the module at runtime the configuration can be given to the [`MODULE LOADEX`]() command. Each configuration must be prefixed with the module name, `redisgears_2.<configuration name>`.

Runtime configuration

You may set certain configuration options at runtime. Setting a configuration at runtime is done using [`CONFIG SET`]() command. Here each configuration must be prefixed with the module name, `redisgears_2.<configuration name>`.

Example:

[code example]

Configurations

execution-threads

The `execution-threads` configuration option controls the number of background threads that run JS code. **Note that libraries are considered single threaded**. This configuration allows Redis to parallelize the invocation of multiple libraries.

_Expected Value_

Integer

_Default_

1

_Minimum Value_

1

_Maximum Value_

32

_Runtime Configurability_

No

library-fatal-failure-policy

The `library-fatal-failure-policy` configuration option controls how to handle a fatal error. A fatal error is consider one of the following:

* Block timeout - The function blocks the Redis processes for too long (configurable using the [lock-redis-timeout](#lock-redis-timeout) configuration value)
* OOM - The function consumes too much memory (configurable using the [v8-maxmemory](#v8-maxmemory) configuration value).

This configuration basically allows choosing between two options:

* Do not break atomicity property, even at the cost of killing the Redis processes.
* Keep my Redis processes alive, even at the cost of losing atomicity.

_Expected Value_

* kill - Save the atomicity property. Risk of killing the Redis processes.
* abort - Abort the invocation of the function and keep the Redis processes alive. Risk of losing the atomicity property.

_Default_

abort

_Runtime Configurability_

Yes

v8-maxmemory

The `v8-maxmemory` configuration option controls the maximum amount of memory used by all V8 libraries. Exceeding this limit is considered a fatal error and will be handled base on the [library-fatal-failure-policy](#library-fatal-failure-policy) configuration value.

_Expected Value_

Integer

_Default_

200M

_Minimum Value_

50M

_Maximum Value_

1G

_Runtime Configurability_

No

v8-library-initial-memory-usage

The `v8-library-initial-memory-usage` configuration option controls the initial memory given to a single V8 library. This value can not be greater then [`v8-library-initial-memory-limit`](#v8-library-initial-memory-limit) or [v8-maxmemory](#v8-maxmemory).

_Expected Value_

Integer

_Default_

2M

_Minimum Value_

1M

_Maximum Value_

10M

_Runtime Configurability_

No

v8-library-initial-memory-limit

The `v8-library-initial-memory-limit` configuration option controls the initial memory limit on a single V8 library. This value can not be greater then [v8-maxmemory](#v8-maxmemory).

_Expected Value_

Integer

_Default_

3M

_Minimum Value_

2M

_Maximum Value_

20M

_Runtime Configurability_

No

v8-library-memory-usage-delta

The `v8-library-memory-usage-delta` configuration option controls the delta by which we will increase the V8 library memory limit once the limit reached. This value can not be greater then [v8-maxmemory](#v8-maxmemory).

_Expected Value_

Integer

_Default_

1M

_Minimum Value_

1M

_Maximum Value_

10M

_Runtime Configurability_

No

lock-redis-timeout

The `lock-redis-timeout` configuration option controls the maximum amount of time (in MS) a library can lock Redis. Exceeding this limit is considered a fatal error and will be handled based on the [library-fatal-failure-policy](#library-fatal-failure-policy) configuration value. This
configuration only affects library loading at runtime with `TFUNCTION LOAD`.
The timeout for loading a library from RDB is set separately via
[db-loading-lock-redis-timeout](#db-loading-lock-redis-timeout).

_Expected Value_

Integer

_Default_

500 MS

_Minimum Value_

100 MS

_Maximum Value_

Unlimited

_Runtime Configurability_

Yes

#### Side effects

When setting `lock-redis-timeout`, if the new value is higher than the
`db-loading-lock-redis-timeout`, the `db-loading-lock-redis-timeout` is also updated to
this value.

db-loading-lock-redis-timeout

This timeout configuration is used for setting the upper time limit
(in milliseconds) for the library loading from RDB.

_Expected Value_

Integer

_Default_

30000 MS

_Minimum Value_

100 MS

_Maximum Value_

Unlimited

_Runtime Configurability_

Yes

#### Notes

The value cannot be lower than the value of `lock-redis-timeout`.

remote-task-default-timeout

The `remote-task-default-timeout` configuration option controls the timeout when waiting for a remote task to finish. If the timeout is reached an error will result.

_Expected Value_

Integer

_Default_

500 MS

_Minimum Value_

1 MS

_Maximum Value_

Unlimited

_Runtime Configurability_

Yes

error-verbosity

The `error-verbosity` configuration option controls the verbosity of error messages that will be provided by triggers and functions. The higher the value the more verbose the error messages will be (for example, including stack traces and extra information for better analysis and debugging).

_Expected Value_

Integer

_Default_

1

_Minimum Value_

1

_Maximum Value_

2

_Runtime Configurability_

Yes

> config set redisgears_2.lock-redis-timeout 1000
OK


Debugging

Summary: Methods for debugging your Redis Stack functions

Overview

There are two methods you can use to debug your Redis Stack functions:

1. Make judicious use of the `redis.log` function, which writes to the Redis log file.
1. Use Redis [pub/sub]().

Use `redis.log`

If you have access to the Redis log files, `redis.log` is a good method to use when debugging. However, there is a drawback. Redis Cloud users do not have access to the Redis log files, and it's pretty common that only system administrators have access to them on self-hosted installations. Fortunately, you can also use Redis pub/sub, which will be discussed in the next section.

You don't have to do anything special to use `redis.log`, as it's always available. Here is an example:

[code example]

After loading the library and executing the function with `TFCALL`, you'll see something like the following in your Redis log file:

[code example]

Use Redis pub/sub

If you don't have access to your Redis database log files, you can use pub/sub. The following example demonstrates how to use the [client.call]() API to publish to a pub/sub channel.

[code example]
In a CLI session, subscribe to the `tfLogChannel` channel and watch for messages.

[code example]

In another CLI session, load the library, replacing what was already there, and then call the new function:

[code example]

You'll see the following in the previous CLI session:

[code example]

There is a downside to using pub/sub. Redis pub/sub provides at-most-once message delivery semantics, which means that once a message is sent, it won't be sent again. So, if a message isn't consumed, it's gone forever. This is likely okay for debugging, but for the longer term, `redis.log` is the better solution for log persistence.

#!js api_version=1.0 name=lib

redis.registerFunction('hello', ()=> {
  redis.log('Hello log')
  return 'Hello from an external file'
})

#!js api_version=1.0 name=lib

const logChannel = 'tfLogChannel'

function publog(client, message) {
  client.call('publish', logChannel, message)
}

redis.registerFunction('tflog', (client) => {
  publog(client, 'sample pub/sub log message')
  return 'sample'
})

$ redis-cli
127.0.0.1:6379> subscribe tfLogChannel
1) "subscribe"
2) "tfLogChannel"
3) (integer) 1
Reading messages... (press Ctrl-C to quit or any key to type command)

127.0.0.1:6379> TFCALL lib.tflog 0
"sample"

1) "message"
2) "tfLogChannel"
3) "sample pub/sub log message"


Development

Summary: How to develop for triggers and functions

Overview

To aid in the development of new libraries of triggers and functions, you can use the type declaration files for the [triggers and functions API](), which allows your preferred development environment to provide autocompletion and type checking. You can install this information using the following command:

[code example]

Or you can manually add it as a devDependency to your `package.json` file:

[code example]

Example project setup

Create an empty directory for your new triggers and functions project, `my_first_project`. Navigate to the folder and run the following command:

[code example]

Update the `package.json` to add the gears API to the `devDependencies` object.

[code example]

Install the dependencies using `npm install`.

Create a new file, `index.js` and import the gears-api module. While this step is not strictly necessary, doing so will enable code intelligent code completion when adding the `registerFunction`:

[code example]

To automate the deployment, update the `scripts` section in `package.json`:

[code example]

Now you can deploy your code to Redis Stack using the following command:

[code example]

The provided URL should follow the following format:

`<redis[s]://[[username][:password]@][host][:port][/db-number]>`

You can now run your Redis Stack function:

[code example]

Example function using a third-party `npm` package

If you want to use a third-party `npm` package in your Redis Stack function, the overall process is nearly the same as already discussed, with just a couple of differences.

First, you'll need to install your package using `npm install`. For this example, the `pi` symbol from the `math.js` library will be used in a simple function that calculates the area of a circle.

[code example]

Note: your `package.json` file will also be updated with the new dependency. For example:

[code example]

Next, you'll import the required symbol(s) into your function.

[code example]

Load and run your new function as described in the previous section.

npm install https://gitpkg.now.sh/RedisGears/RedisGears/js_api --save-dev

"devDependencies": {
  "@redis/gears-api": "https://gitpkg.now.sh/RedisGears/RedisGears/js_api"
}

$ npm init -y -f
npm WARN using --force Recommended protections disabled.
Wrote to /home/work/my_first_project/package.json:

{
  "name": "my_first_project",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

echo "Error: no test specified" && exit 1

#!js name=lib api_version=1.0

import { redis } from '@redis/gears-api';

redis.registerFunction("test", ()=>{
    return "test";
});

"scripts": {
  "deploy": "gears-api index.js"
}

> npm run deploy -- -r redis://localhost:6379

> deploy
> gears-api index.js

Deployed! :)

"dependencies": {
    "mathjs": "^12.0.0"
}

#!js api_version=1.0 name=lib

import { redis } from '@redis/gears-api';
import { pi } from 'mathjs';

function calculateCircleArea(client, radius) {
  return pi * radius * radius;
}

redis.registerFunction('calculateArea', calculateCircleArea);


Triggers and functions examples

Summary: How triggers and functions can be used

Overview

Redis Stack's triggers and functions feature enables the detection of changes to data as they happen and guarantee seamless execution of business logic at the data source. This ensures that new options to manipulate the data are possible and can be delivered for all the clients at the same time, while deployment and maintenance are simplified. Let's explore some industry-specific use cases where these capabilities shine:

- **Retail**: In the retail sector, a function can be developed to update the inventory immediately upon order receipt. This empowers businesses to accurately predict stock requirements for specific warehouses on any given day.
- **Travel**: For the travel industry, a trigger can be utilized to detect new flight bookings and efficiently load the relevant information into a queue for different consumers. Services can then leverage this data to provide recommendations for hotels, restaurants, car rental services, and more. Leveraging Redis geometries, powerful recommendation systems can offer localized and personalized suggestions.
- **Subscription services**: In the realm of subscription services, employing a keyspace trigger can automatically identify users whose subscriptions have been renewed, seamlessly changing their status to active. Further operations can be performed on these users, such as adding them to a queue for the delivery of notifications or executing additional actions.

These examples highlight the practical application of triggers and functions in different industries, showcasing their value in streamlining processes and delivering efficient solutions.

Using keyspace and Stream triggers to capture events and execute the desired JavaScript functions requires few lines of code to enrich applications with new behaviors. The following examples show how typical problems are solved with JavaScript.

Auditing and logging

Redis Triggers can be used to detect changes to specific data structures, and log an auditing trail in the Redis log. The timestamp of the latest change can be updated in the same function.

[code example]

Enrich and transform data

Data can be extracted, enriched or transformed, and loaded again. As an example, upon insertion of a document in a Hash data structure, a Trigger can launch the execution of a Function that computes the number of words in the text (in the example, a simple tokenization is presented but the logic can be as complex as required). The counter is finally stored in the same Hash together with the original document.

[code example]

Batch operations

JavaScript functions can be executed when required, for example as part of scheduled or periodic maintenance routines. An example could be deleting data identified by the desired pattern.

[code example]

The function `del_keys` performs an asynchronous batch scan of all the keys in the database matching the desired pattern, and enters a blocking section where it gets partial results. The asynchronous design of this function permits non-blocking execution, so other clients will not starve while the iteration is performed on a keyspace of arbitrary size. The function can be invoked as follows.

[code example]

Automatic expiration

Sometimes it is useful to enforce expiration time for data that by nature is short-lived. An example could be session data, or authentication tokens. A trigger fits the use case and can execute a function that sets the desired TTL.

[code example]

function alertUserChanged(client, data) {
    // detect the event and log an audit trail
    if (data.event == 'hset'){
        redis.log('User data changed: ' + data.key);
    }
    else if (data.event == 'del'){
        redis.log('User deleted: ' + data.key);
    }

    var curr_time = client.call("time")[0];

    // add the current timestamp to the Hash
    client.call('hset', data.key, 'last', curr_time);
}

redis.registerKeySpaceTrigger('alert_user_changed', 'user:',
    alertUserChanged, {description: 'Report user data change.'});

function wordsCounter(client, data){
    text = client.call('hget', data.key, 'content');
    words = text.split(' ').length;
    client.call('hset', data.key, 'cnt', words.toString());
    redis.log('Number of words: ' + words.toString()); //This log is for demo purposes, be aware of spamming the log file in production
}

redis.registerKeySpaceTrigger('words_counter', 'doc:',
    wordsCounter, {description: 'Count words in a document.'});

#!js api_version=1.0 name=utils

redis.registerAsyncFunction('del_keys', async function(async_client, pattern){
    var count = 0;
    var cursor = '0';
    do {
        async_client.block((client)=>{
            var res = client.call('scan', cursor, 'match', pattern);
            cursor = res[0];
            var keys = res[1];
            keys.forEach((key) => {
                client.call('del', key);
            });
        });
    } while(cursor != '0');
    return count;
});

function automaticSessionExpiry(client, data){
    client.call('expire', data.key, '3600');
    redis.log('Session ' + data.key + ' has been automatically expired'); //This log is for demo purposes, be aware of spamming the log file in production
}

redis.registerKeySpaceTrigger('automaticSessionExpiry', 'session:',
    automaticSessionExpiry, {description: 'Set the session TTL to one hour.'});


Triggers and functions

Summary: Trigger and execute JavaScript functions in the Redis process

Summary: Get started with triggers and functions using Redis Insight

Quick start using Redis Insight

Summary: Get started with triggers and functions using redis-cli

Quick start using redis-cli

Summary: Configure the operation parameters

Configuration

Development

Debugging

Summary: Details about triggers, the JavaScript API, and advanced concepts

Concepts

Summary: How triggers and functions can be used

Triggers and functions examples

Summary: Overview of the known limitations

Known limitations

Overview

[![discord](https://img.shields.io/discord/697882427875393627?style=flat-square)](https://discord.gg/xTbqgTB)
[![Github](https://img.shields.io/static/v1?label=&message=repository&color=5961FF&logo=github)](https://github.com/RedisGears/RedisGears/)

The triggers and functions feature of Redis Stack allows running JavaScript functions inside Redis. These functions can be executed on-demand, by an event-driven trigger, or by a stream processing trigger.

Quick links

* [Command documentation](https://github.com/RedisGears/RedisGears/tree/master/docs/commands)
* [Source code](https://github.com/RedisGears/RedisGears)
* [Latest release](https://github.com/RedisGears/RedisGears/releases)
* [Docker image](https://hub.docker.com/r/redis/redis-stack-server/)

Primary features

* JavaScript engine for functions
* On-demand functions
* Keyspace triggers
* Stream triggers
* Async handling of functions
* Read data from across the cluster

Cluster support

Triggers and functions support deployment and execution of functions across a cluster. Functions are executed on the correct shard based on the key that is changed or read functions can be executed on all to return a correct view of the data.

References

Blog posts

- [Expanding the Database Trigger Features in Redis](https://redis.com/blog/database-trigger-features/)

Overview


Known limitations

Summary: Overview of the known limitations

Limited write options

JavaScript remote functions are limited to **read operations** only. Any attempt to perform a write operation of the following functions on a shard different than the one executing the function will result in an error.

- `async_client.runOnShards` runs the remote function on all the shards
- `async_client.runOnKey` runs the remote function on the shard responsible for a given key

In addition, keyspace modification performed by JavaScript functions that are registered using any of the methods available should perform write operations locally:

- If the function is registered with `registerFunction` or `registerAsyncFunction`, it can insert, modify or delete keys that are in the same shard where the function is executed.
- If the function is registered with `registerKeySpaceTrigger` or `registerStreamTrigger`, keyspace modification must be local to the shard that originated the events.

It is also recommended to co-locate the keys to be modified in the same hash slot as the key or Stream that originated the event. As an example, if the user profile stored in the Hash `myserv:user:1234` is subject to changes and we'd like to count them in an external counter, we would name the counter using hash tags: `{myserv:user:1234}:cnt`.

Exclusive access to the keyspace

By design, asynchronous functions guarantee exclusive single-threaded access to the keyspace, the distinctive feature of Redis. In asynchronous programming with JavaScript functions, access to the keyspace in read or write mode must be blocking, while if not accessing the keyspace, the execution may be non-blocking. This implementation maintains the same level of data consistency as Redis standard commands or Lua scripts and functions but takes advantage of asynchronous execution, a feature of the JavaScript engine.

JavaScript variables

Not all the JavaScript global variables are made available by the JavaScript engine loaded by Redis (e.g. `console`, `document`). The `redis` global variable can be used to manage functions registration, logging etc.

Sandboxed

This feature is sandboxed, meaning, from within a function, it’s not possible to make calls to external services, including other Redis databases, or APIs.


Quick start using redis-cli

Overview

Make sure that you have [Redis Stack installed]() and running. Alternatively, you can create a [free Redis Cloud account](https://redis.com/try-free/?utm_source=redisio&utm_medium=referral&utm_campaign=2023-09-try_free&utm_content=cu-redis_cloud_users). The triggers and functions preview is available in the fixed subscription plan for the Google Cloud Asia Pacific (Tokyo) and AWS Asia Pacific (Singapore) regions.

Connect to Redis Stack

[code example]

Load a library

Use the `TFUNCION LOAD` command to create a new library in your Redis instance.

[code example]

When the library is created successfully, an `OK` response is returned. Run the `TFUNCTION LIST` command to confirm your library was added to Redis.

[code example]

The `TFCALL` command is used to execute the JavaScript Function. If the command fails, an error will be returned.

[code example]

To update the library run the `TFUNCTION LOAD` command with the additional parameter `REPLACE`.

[code example]

Uploading an external file

Use the `redis-cli` command to upload JavaScript from an external file. The file needs to contain the header, which contains the engine identifier, the API version, and the library name: `#!js api_version=1.0 name=myFirstLibrary`.

[code example]

Use the `redis-cli -x` option to send the file with the command and use the `TFUNCTION LOAD REPLACE` to replace the inline library with the one from the `main.js` file.

[code example]

Creating triggers

Functions within Redis can respond to events using keyspace triggers. While the majority of these events are initiated by command invocations, they also include events that occur when a key expires or is removed from the database.

For the full list of supported events, please refer to the [Redis keyspace notifications page]().

The following code creates a new keyspace trigger that adds a new field to a new or updated hash with the latest update time.

Load the code in your database:

[code example]

Add a new hash with the required prefix to trigger our function.

[code example]

Check if the last updated time is added to the example.

[code example]

127.0.0.1:6379> TFUNCTION LOAD "#!js api_version=1.0 name=myFirstLibrary\n redis.registerFunction('hello', ()=>{ return 'Hello World'})"
OK

> TFUNCTION LIST
1) 1) "api_version"
   2) "1.0"
   3) "cluster_functions"
   4) (empty list or set)
   5) "configuration"
   6) "null"
   7) "engine"
   8) "js"
   9) "functions"
   10) 1) "hello"
   11) "keyspace_triggers"
   12) (empty list or set)
   13) "name"
   14) "myFirstLibrary"
   15) "pending_async_calls"
   16) (empty list or set)
   17) "pending_jobs"
   18) "0"
   19) "stream_triggers"
   20) (empty list or set)
   21) "user"
   22) "default"

127.0.0.1:6379> TFCALL myFirstLibrary.hello 0
"Hello World"

127.0.0.1:6379> TFUNCTION LOAD REPLACE "#!js api_version=1.0 name=myFirstLibrary\n redis.registerFunction('hello', ()=>{ return 'Hello World updated'})"
OK

#!js api_version=1.0 name=lib

redis.registerFunction('hello', ()=> {
  return 'Hello from an external file'
})

redis-cli -x TFUNCTION LOAD REPLACE < ./main.js

TFUNCTION LOAD REPLACE "#!js name=myFirstLibrary api_version=1.0\n
    function addLastUpdatedField(client, data) {
        if(data.event == 'hset') {
            var currentDateTime = Date.now();
            client.call('hset', data.key, 'last_updated', currentDateTime.toString());
        }
    }
    redis.registerKeySpaceTrigger('addLastUpdated', 'fellowship:', addLastUpdatedField);" // Register the KeySpaceTrigger 'AddLastUpdated' for keys with the prefix 'fellowship' with a callback to the function 'addLastUpdatedField'

127.0.0.1:6379> HSET fellowship:1 name "Frodo Baggins" title "The One Ring Bearer"

127.0.0.1:6379> HGETALL fellowship:1
1) "name"
2) "Frodo Baggins"
3) "title"
4) "The One Ring Bearer"
5) "last_updated"
6) "1693238681822"


Quick start using Redis Insight

Overview

Make sure that you have [Redis Stack installed]() and running. Alternatively, you can create a [free Redis Cloud account](https://redis.com/try-free/?utm_source=redisio&utm_medium=referral&utm_campaign=2023-09-try_free&utm_content=cu-redis_cloud_users).

If you haven't already installed Redis Insight, you can download the latest version [here](https://redis.com/redis-enterprise/redis-insight/?_ga=2.232184223.127667221.1704724457-86137583.1685485233&_gl=1*1gygred*_ga*ODYxMzc1ODMuMTY4NTQ4NTIzMw..*_ga_8BKGRQKRPV*MTcwNDkyMzExMC40MDEuMS4xNzA0OTI3MjQ2LjUyLjAuMA..*_gcl_au*MTQzODY1OTU4OS4xNzAxMTg0MzY0). If this is your first time using Redis Insight, you may wish to read through the [Redis Insight guide]() before continuing with this guide.

Connect to Redis Stack

Open the Redis Insight application, and connect to your database by clicking on its database alias.

Load a library

Click on the triggers and functions icon and then on **+ Library** as shown below.

Add your code to the **Library Code** section of the right-hand panel and then click **Add Library**.

You'll see the following when the library was added:

The `TFCALL` command is used to execute the JavaScript Function. If the command fails, an error will be returned. Click on the **>_ CLI**  button in the lower left-hand corner to open a console window and then run the command shown below.

To update a library you can edit the library code directly in the interface by clicking on the edit (pencil) icon. When you save your changes, the libary will be reloaded.

Uploading an external file

Click on the **+ Add Library** button as before and, instead of adding the code directly into the editor, click on the **Upload** button, select the file from your file browser, and then click on **Add Library**. The file needs to contain the header, which contains the engine identifier, the API version, and the library name: `#!js api_version=1.0 name=myFirstLibrary`.

Creating triggers

Functions within Redis can respond to events using keyspace triggers. While the majority of these events are initiated by command invocations, they also include events that occur when a key expires or is removed from the database.

For the full list of supported events, please refer to the [Redis keyspace notifications page]().

The following code creates a new keyspace trigger that adds a new field to a new or updated hash with the latest update time.

[code example]

Update the existing library as before and then, using the Redis Insight console, add a new hash with the required prefix to trigger the function.

[code example]

Run the [`HGETALL`]() command to check if the last updated time is added to the example.

[code example]

#!js name=myFirstLibrary api_version=1.0

function addLastUpdatedField(client, data) {
    if(data.event == 'hset') {
        var currentDateTime = Date.now();
        client.call('hset', data.key, 'last_updated', currentDateTime.toString());
    }
}

// Register the KeySpaceTrigger 'AddLastUpdated' for keys with the prefix 'fellowship'
// with the callback function 'addLastUpdatedField'.
redis.registerKeySpaceTrigger('addLastUpdated', 'fellowship:', addLastUpdatedField);"

> HSET fellowship:1 name "Frodo Baggins" title "The One Ring Bearer"

> HGETALL fellowship:1
1) "name"
2) "Frodo Baggins"
3) "title"
4) "The One Ring Bearer"
5) "last_updated"
6) "1693238681822"


Redis Software and Redis Open Source feature compatibility

Summary: Describes the Redis Software features supported by each Redis Open Source feature.

Overview

This article describes compatibility between Redis Software features and Redis Open Source features. Version numbers indicate the minimum module version required for feature support.

Supported Redis Open Source features

The following table shows which Redis Open Source features are supported by Redis Software and Redis Cloud.

| Feature | Redis
Software | Redis
Cloud |
|:-------|:-------------------------|:-----------------------|
| [Search and query]() | ✅ Supported | ✅ Supported |
| [JSON]()   | ✅ Supported | ✅ Supported |
| [Time series]() | ✅ Supported | ✅ Supported |
| [Probabilistic]() | ✅ Supported | ✅ Supported |
| [Gears]() | ✅ Supported | ❌ Not supported |
| [Triggers and functions]() | ⚠ Deprecated | ⚠ Deprecated |
| [Graph]() | ⚠ Deprecated | ⚠ Deprecated |

Feature compatibility

The following tables show Redis Software feature support for each Redis Open Source feature.

Version numbers indicate when the feature was first supported.  If you're using an earlier version than what's shown in the table, the feature is not supported.

For details about individual features, see the corresponding documentation.

| Feature name/capability   | [Search and query]() | [JSON]()    |
|---------------------------|:--------------:|:------------:|
| Active-Active (CRDB)[^5]  | Yes (v2.0)     | Yes (v2.2)   |
| Backup/Restore            | Yes (v1.4)     | Yes (v1.0)   |
| Clustering                | Yes (v1.6)[^3] | Yes (v1.0)   |
| Custom hashing policy     | Yes (v2.0)     | Yes (v1.0)   |
| Eviction expiration       | Yes (v2.0)     | Yes (v1.0)   |
| Failover/migration        | Yes (v1.4)     | Yes (v1.0)   |
| Internode encryption      | Yes (v2.0.11)  | Yes (v1.0.8) |
| Module data types         | Yes            | Yes          |
| Persistence (AOF)         | Yes (v1.4)     | Yes (v1.0)   |
| Persistence (snapshot)    | Yes (v1.6)     | Yes (v1.0)   |
| Auto Tiering [^4]         | Yes (v2.0)     | Yes (v1.0)   |
| Redis Flex                | No             | Yes (v8.0)   |
| Replica Of                | Yes (v1.6)[^2] | Yes (v1.0)   |
| Reshard/rebalance         | Yes (v2.0)     | Yes (v1.0)   |

[^1]: Graphs are compatible with clustering; however, individual graphs contained in a key reside in a single shard, which can affect pricing.  To learn more, [contact support](https://redis.com/company/support/).

[^2]: RediSearch version 1.6 supported Replica Of only between databases with the same number of shards.  This limitation was fixed in v2.0.

[^3]: You cannot use search and query with the [OSS Cluster API](). This limitation was fixed in Redis Software version 8.0.

[^4]: You currently cannot combine Auto Tiering with Redis Open Source features in Redis Cloud.

[^5]: With the exception of JSON, you currently cannot combine Active-Active with Redis Open Source features in Redis Cloud.

[^6]: Although time series are compatible with Auto Tiering, the entire series either lives in RAM or on flash.

| Feature name/capability | [Time series]() | [Probabilistic]() | [Gears]() |
|--------------------------|:--------------:|:------------:|:----------:|
| Active-Active (CRDB)[^5] | No             | No           | Yes (v1.0) |
| Backup/Restore           | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Clustering               | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Custom hashing policy    | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Eviction expiration      | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Failover/migration       | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Internode encryption     | Yes (v1.4.9)   | Yes (v2.2.6) | Yes (v1.2) |
| Module data types        | Yes            | Yes          | Yes        |
| Persistence (AOF)        | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Persistence (snapshot)   | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |
| Auto Tiering [^4]        | Yes (v1.6)[^6] | Yes (vTBD)   | Yes (vTBD) |
| Redis Flex               | No               | Yes (v8.0)   | No |
| Replica Of               | Yes (v1.2)     | Yes (v2.0)   | No         |
| Reshard/rebalance        | Yes (v1.2)     | Yes (v2.0)   | Yes (v1.0) |

Feature descriptions

The following table briefly describes each feature shown in the earlier tables.

| Feature name/capability | Description |
|-------------------------|-------------|
| Active-Active (CRDB)    | Compatible with Active-Active (CRDB) databases  |
| Backup/Restore          | Supports import and export features |
| Clustering              | Compatible with sharded databases and shard migration |
| Custom hashing policy   | Compatible with databases using custom hashing policies |
| Eviction expiration     | Allows data to be evicted when the database reaches memory limits |
| Failover/migration      | Compatible with primary/replica failover and the migration of shards between nodes within the cluster |
| Internode encryption    | Compatible with encryption on the data plane |
| Persistence (AOF)       | Compatible with databases using AoF persistence |
| Persistence (snapshot)  | Compatible with databases using snapshot persistence |
| Auto Tiering    | Compatible with Auto Tiering |
| Redis Flex      | Compatible with Redis Flex |
| Replica Of              | Compatible with Active-Passive replication |
| Reshard/rebalance       | Compatible with database scaling for clustered databases, which redistributes data between the new shards. |

<!--
    Individual footnotes are rendered below the following heading.
    Thus, any additional sections need to be placed above this comment.
-->

Footnotes


Gears

Install RedisGears

Summary: The RedisGears Python plugin allows you to run RedisGears functions with Python.

RedisGears Python plugin

Summary: The RedisGears JVM plugin allows you to run RedisGears functions in the Java virtual machine.

RedisGears JVM plugin

Summary: Register RedisGears functions to run when certain events occur in a Redis database.

Register events

What is RedisGears?

RedisGears is an engine for data processing in Redis. RedisGears supports batch and event-driven processing for Redis data. To use RedisGears, you write functions that describe how your data should be processed. You then submit this code to your Redis deployment for remote execution.

Supported languages

As of RedisGears v1.2, you can enable a plugin to select which programming language to use. It currently supports code written in either [Python]() or [Java]().

Prior to v1.2, RedisGears only supported Python. However, an internal C API exists and can be used by other Redis modules. Support for other languages is being planned.

Getting started with RedisGears

RedisGears is implemented by a Redis module. To use RedisGears, you'll need to make sure that your Redis deployment has the module installed. [Redis Software](https://docs.redislabs.com/latest/rs/) supports the module natively.

If you're running Redis Open Source, you'll also need to [install the RedisGears module]() before using it.

To get started with RedisGears, see the quick start tutorial for [Python]() or [Java]().

Write-behind caching patterns

Redis users typically implement caching by using the look-aside pattern. However, with RedisGears, you can implement write-behind caching strategies as well.

Redis publishes RedisGears recipes to support write-behind. You can learn how to use these recipes in our write-behind caching guides for [Python]() and [Java]().

More info

- [RedisGears source](https://github.com/RedisGears/RedisGears)


Install RedisGears

Overview

Before you can use RedisGears, you have to install the RedisGears module on your Redis Software cluster.

Minimum requirements

- Redis Software 6.0.12 or later
- The [cluster is setup]() and all of the nodes are joined to the cluster

Install RedisGears

If your cluster uses Redis Software v6.0.12 or later and has internet access, you only need to download the RedisGears package. It automatically fetches dependencies like the Python and JVM plugins during online installation.

Offline installation requires you to manually upload dependencies to the primary node.

Install RedisGears and dependencies

1. Download the **RedisGears** package from the Redis Software [download center](https://cloud.redis.io/#/rlec-downloads).

For offline installation of RedisGears v1.2 and later, you also need to download the **RedisGears Dependencies** packages for both Python and Java.
<br/>
For RedisGears v1.0, you only need the Python dependency package.

1. Upload the RedisGears package to a node in the cluster.

1. For offline installation only, copy the dependencies to the primary node.

Skip this step unless your cluster does not have internet access.

    1. For versions 7.2.4 and later, copy the dependencies to `$modulesdatadir/rg/<version-integer>/<OS_name>/<architecture>/deps/`:

        [code example]

    1. For versions 6.4.2 and earlier, copy the dependencies to `$modulesdatadir/rg/<version-integer>/deps/`:

        [code example]

    Replace these fields with your own values:

    - `<OS>`: the operating system running Redis Software
    - `<version>`: the RedisGears version `(x.y.z)`
    - `<version-integer>`: the RedisGears version as an integer, calculated as <nobr>`(x*10000 + y*100 + z)`</nobr>

        For example, the `<version-integer>` for RedisGears version 1.2.5 is 10205.

    - `<OS_name>`: the operating system's name
    - `<architecture>`: the node's architecture

1. Add RedisGears to the cluster with a `POST` request to the primary node's [`/v2/modules`]() REST API endpoint:

    [code example]

Here, the *module* parameter specifies the full path of the module package and must be submitted as form-data. In addition, the package must be available and accessible to the server processing the request.

After the install is complete, RedisGears will appear in the list of available modules on the **settings** and **create database** pages of the Redis Software admin console.

Enable RedisGears for a database

After installation, create a new database and enable RedisGears:

- [With Python]()

- [With the JVM]()

Upgrade RedisGears for existing databases

To upgrade RedisGears for an existing database after installing a new version, use [`rladmin upgrade db`]():

[code example]

The following example shows how to upgrade a database named `shopping-cart` to RedisGears version 1.2.9 without changing its configuration:

[code example]

These command examples also upgrade the database to the latest Redis version on the cluster. For more module upgrade information and examples, see [Upgrade modules]().

Uninstall RedisGears

To uninstall RedisGears, make a [`DELETE` request to the `/v2/modules` REST API endpoint]().

POST https://[host][:port]/v2/modules
    {"module=@/tmp/redisgears.<OS>.<version>.zip"}

rladmin upgrade db <database-name-or-ID> and module module_name rg version <new_version_integer> module_args "<module arguments>"

rladmin upgrade db shopping-cart and module module_name rg version 10209 module_args keep_args


Accumulate

Summary: Reduces many records in the pipe to a single record.

Overview

[code example]

Accumulate is a many-to-one function that iterates through the records in the pipe and reduces them to a single record.

You can provide a parameter to set the initial accumulator value. Otherwise, the initial accumulator object is null.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| accumulator | <nobr>AccumulateOperation<T,I></nobr> | A function with logic to update the accumulator value with each record |
| initialValue | template type I | The initial value of the accumulated object |

Returns

Returns a GearsBuilder object with a new template type.

Examples

Both of the following examples count the number of records in the pipeline.

Without the `initialValue` parameter:

[code example]

With the `initialValue` parameter set to 0:

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> accumulate(
    gears.operations.AccumulateOperation<T,I> accumulator)

public <I extends java.io.Serializable> GearsBuilder<I> accumulate(
    I initialValue,
    gears.operations.AccumulateOperation<T,I> accumulator)

GearsBuilder.CreateGearsBuilder(reader).accumulate((a, r)->{
    Integer ret = null;
    if (a == null) {
	    ret = 1;
    } else {
	    ret = (Integer)a;
    }
    return ret + 1;
});

GearsBuilder.CreateGearsBuilder(reader).accumulate(0, (a, r)->{
   	return a + 1;
});


AccumulateBy

Summary: Groups records and reduces each group to a single record per group.

Overview

[code example]

Iterates through the records in the pipe, groups them based on the provided extractor, and then reduces each group to a single record per group with the accumulator function.

The initial value of the accumulator is null unless you provide a value initializer operation as a parameter.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| accumulator | <nobr>AccumulateByOperation<T,I></nobr> | A function with logic to update the accumulator value with each record |
| extractor | ExtractorOperation<T> | Extracts a specific value from each record |
| valueInitializer | ValueInitializerOperation<I> | Whenever the accumulated value is null, use this function to initialize it |

Returns

Examples

Both of the following examples count the number of unique values.

Without the `valueInitializer` parameter:

[code example]

With the `valueInitializer` parameter:

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> accumulateBy(
	gears.operations.ExtractorOperation<T> extractor,
	gears.operations.AccumulateByOperation<T,I> accumulator)

public <I extends java.io.Serializable> GearsBuilder<I> accumulateBy(
	gears.operations.ValueInitializerOperation<I> valueInitializer,
	gears.operations.ExtractorOperation<T> extractor,
	gears.operations.AccumulateByOperation<T,I> accumulator)

GearsBuilder.CreateGearsBuilder(reader).
   	accumulateBy(r->{
   		return r.getStringVal();
   	},(k, a, r)->{
   		Integer ret = null;
   		if(a == null) {
   			ret = 0;
   		}else {
   			ret = (Integer)a;
   		}
   		return ret + 1;
});

GearsBuilder.CreateGearsBuilder(reader).
    	accumulateBy(()->{
   		return 0;
   	},r->{
   		return r.getStringVal();
   	},(k, a, r)->{
   		return a + 1;
   	});


AsyncFilter

Summary: Asynchronously filters out records in the pipe based on a given condition.

Overview

[code example]

The `asyncFilter` function allows you to use a [`GearsFuture`]() object to asynchronously filter out records in the pipe based on a given condition.

The filter operation should contain a conditional statement and return a boolean for each record:
- If `true`, the record will continue through the pipe.
- If `false`, it filters out the record.

Parameters

| Name | Type | Description |
|------|------|-------------|
| filter | AsyncFilterOperation<T> | A function that checks a condition for each record in the pipe. Returns a boolean. |

Returns

Returns a GearsBuilder object with the same template type as the input builder.

Example

[code example]

public GearsBuilder<T> asyncFilter(
    gears.operations.AsyncFilterOperation<T> filter)

GearsBuilder.CreateGearsBuilder(reader).map(r->r.getKey()).
	asyncFilter(r->{
		GearsFuture<Boolean> f = new GearsFuture<Boolean>();
		new Thread(new Runnable() {

			@Override
			public void run() {
				try {
					Thread.sleep(1);

					f.setResult(r.equals("x"));
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		}).start();
		return f;
});


AsyncForeach

Summary: For each record in the pipe, asynchronously runs some operations.

Overview

[code example]

The `asyncForeach` function allows you to use a [`GearsFuture`]() object to define a set of operations and run them asynchronously for each record in the pipe.

Parameters

| Name | Type | Description |
|------|------|-------------|
| foreach | AsyncForeachOperation<T> | The set of operations to run for each record |

Returns

Example

[code example]

public GearsBuilder<T> asyncForeach(
    gears.operations.AsyncForeachOperation<T> foreach)

GearsBuilder.CreateGearsBuilder(reader).map(r->r.getKey()).
	asyncForeach(r->{
		GearsFuture<Serializable> f = new GearsFuture<Serializable>();
		new Thread(new Runnable() {

			@Override
			public void run() {
				try {
					Thread.sleep(1);

					f.setResult(r);
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		}).start();
		return f;
});


AsyncMap

Summary: Asynchronously maps records one-to-one.

Overview

[code example]

The `asyncMap` function allows you to use a [`GearsFuture`]() object to asynchronously map each input record in the pipe to an output record, one-to-one.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| mapper | <nobr>AsyncMapOperation<T,I></nobr> | For each input record, returns a new output record |

Returns

Example

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> asyncMap(
	gears.operations.AsyncMapOperation<T,I> mapper)

GearsBuilder.CreateGearsBuilder(reader).
	asyncMap(r->{
		GearsFuture<String> f = new GearsFuture<String>();
		new Thread(new Runnable() {

			@Override
			public void run() {
				try {
					Thread.sleep(1);

					f.setResult("done");
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		}).start();
		return f;
});


CallNext

Summary: Calls the next execution that overrides the command or the original command itself. A more flexible version of callNextArray.

Overview

[code example]

When you override a Redis command with the [`CommandOverrider`](), use `callNext` to run the next execution that overrides the command or the original command itself.

It is a more flexible version of [`callNextArray`]() since the list of string arguments does not have to be an explicit `String[]` object. This allows function calls like: <nobr>`callNext("key", "value")`.</nobr>

Parameters

| Name | Type | Description |
|------|------|-------------|
| args | string | Redis command arguments |

Returns

Returns the command result. It could be a string or an array of strings, depending on the command.

Examples

Without `String[]`:

[code example]

With `String[]`:

[code example]

public static java.lang.Object callNext(java.lang.String... args)

GearsBuilder.callNext(new String[]{"restaurant:1", "reviews", "50"});


CallNextArray

Summary: Calls the next execution that overrides the command or the original command itself.

Overview

[code example]

When you override a Redis command with the [`CommandOverrider`](), use `callNextArray` to run the next execution that overrides the command or the original command itself.

It accepts an array of strings, which represents the command arguments.

Parameters

| Name | Type | Description |
|------|------|-------------|
| args | array of strings | Redis command arguments |

Returns

Example

[code example]

public static native java.lang.Object callNextArray(
    java.lang.String[] command)

GearsBuilder.callNextArray(new String[]{"restaurant:1", "reviews", "50"});


Collect

Summary: Collects all records to the origin shard.

Overview

[code example]

Collects all of the records to the shard where the RedisGears job started.

Parameters

Returns

Example

[code example]


ConfigGet

Summary: Gets the value of a RedisGears configuration setting.

Overview

[code example]

Gets the value of a RedisGears [configuration setting]().

You can set configuration values when you load the module or use the `RG.CONFIGSET` command.

Parameters

| Name | Type | Description |
|------|------|-------------|
| key | string | The configuration setting to get |

Returns

Returns the configuration value of a RedisGears configuration setting.

Example

[code example]

public static java.lang.String configGet(java.lang.String key)


Count

Summary: Counts the number of records in the pipe.

Overview

[code example]

Counts the number of records in the pipe and returns the total as a single record.

Parameters

Returns

Returns a GearsBuilder object with a new template type of `Integer`.

Example

[code example]


CreateGearsBuilder

Summary: Creates a new GearsBuilder object.

Overview

[code example]

Creates a new `GearsBuilder` object. Use this function instead of a `GearsBuilder` constructor to avoid type warnings.

Parameters

Type Parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder. The reader determines the type. |

Parameters:

| Name | Type | Description |
|------|------|-------------|
| desc | string | The description |
| reader | BaseReader<I> | The pipe reader |

Returns

Returns a new GearsBuilder object.

Example

[code example]

public static <I extends java.io.Serializable> GearsBuilder<I> CreateGearsBuilder(
    gears.readers.BaseReader<I> reader)

public static <I extends java.io.Serializable> GearsBuilder<I> CreateGearsBuilder(
    gears.readers.BaseReader<I> reader,
    java.lang.String desc)


Execute

Summary: Runs a Redis command. A more flexible version of executeArray.

Overview

[code example]

Runs a Redis command, similar to [`executeArray`](). However, the `execute` function is more flexible. Unlike `executeArray`, the list of string arguments does not have to be an explicit `String[]` object. It allows function calls like this: <nobr>`execute("SET", "key", "value")`.</nobr>

Parameters

| Name | Type | Description |
|------|------|-------------|
| command | string | A Redis command |

Returns

Examples

public static java.lang.Object execute(java.lang.String... command)

GearsBuilder.execute(new String[]{"SET", "age:maximum", "100"});


ExecuteArray

Summary: Runs a Redis command.

Overview

[code example]

Runs a Redis command. It accepts an array of strings, which represents the command to execute.

Parameters

| Name | Type | Description |
|------|------|-------------|
| command | array of strings | A Redis command |

Returns

Example

[code example]

public static native java.lang.Object executeArray(
    java.lang.String[] command)

GearsBuilder.executeArray(new String[]{"SET", "age:maximum", "100"});


Filter

Overview

[code example]

Filters out records in the pipe based on a given condition.

The filter operation should contain a conditional statement and return a boolean for each record:
- If `true`, the record will continue through the pipe.
- If `false`, it filters out the record.

Parameters

| Name | Type | Description |
|------|------|-------------|
| filter | FilterOperation<T> | A function that checks a condition for each record in the pipe. Returns a boolean. |

Returns

Returns a GearsBuilder object with the same template type as the input builder.

Example

Get all records that contain the substring "person:":

[code example]

public GearsBuilder<T> filter(
    gears.operations.FilterOperation<T> filter)

GearsBuilder.CreateGearsBuilder(reader).
    filter(r->{
        return r.getKey().contains("person:");
});


FlatMap

Overview

[code example]

Maps a single input record to one or more output records.

The FlatMap operation must return an [`Iterable`](https://docs.oracle.com/javase/8/docs/api/java/lang/Iterable.html). RedisGears
splits the elements from the `Iterable` object and processes them as individual records.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder object |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| flatmapper | <nobr>FlatMapOperation<T,I></nobr> | For each input record, returns one or more output records |

Returns

Returns a GearsBuilder object with a new template type.

Example

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> flatMap(
	gears.operations.FlatMapOperation<T,I> flatmapper)

GearsBuilder.CreateGearsBuilder(reader).flatMap(r->{
   	return r.getListVal();
});


Foreach

Summary: For each record in the pipe, runs some operations.

Overview

[code example]

Defines a set of operations to run for each record in the pipe.

Parameters

| Name | Type | Description |
|------|------|-------------|
| foreach | ForeachOperation<T> | The set of operations to run for each record |

Returns

Example

For each person hash, add a new full_name field that combines their first and last names:

[code example]

public GearsBuilder<T> foreach(
    gears.operations.ForeachOperation<T> foreach)

GearsBuilder.CreateGearsBuilder(reader).foreach(r->{
    String firstName = r.getHashVal().get("first_name");
    String lastName = r.getHashVal().get("last_name");
   	r.getHashVal().put("full_name", firstName + lastName);
});


Hashtag

Summary: Returns a string that maps to the current shard.

Overview

[code example]

Returns a string that maps to the current shard according to the cluster slot mapping.

You can use the `hashtag` function when you need to create a key that resides on the current shard.

Parameters

Returns

Example

The following example uses the `hashtag` function to calculate the hslot. The string maps to the current shard.

[code example]

public static java.lang.String hashtag()

GearsBuilder.execute(
    "SET",
    String.format("key{%s}", GearsBuilder.hashtag()),
    "1"
);


GearsBuilder

Summary: Creates a RedisGears pipeline of operations to transform data.

Accumulate

AccumulateBy

AsyncFilter

AsyncForeach

Summary: Asynchronously maps records one-to-one.

AsyncMap

CallNext

CallNextArray

Collect

ConfigGet

Count

Summary: Creates a new GearsBuilder object.

CreateGearsBuilder

Execute

Summary: Runs a Redis command.

ExecuteArray

Filter

FlatMap

Foreach

Hashtag

Summary: Groups records and reduces each group to a single record per group locally on each shard.

LocalAccumulateBy

Summary: Writes a log message to the Redis log file.

Log

Summary: Maps records one-to-one.

Map

Summary: Registers the pipeline of functions to run when certain events occur.

Register

Summary: Moves records between shards according to the extracted data.

Repartition

Summary: Runs the pipeline of functions immediately.

Run

Overview

The `GearsBuilder` class allows you to create a pipeline of RedisGears functions that transform data.

It requires a reader to supply data to the pipe.

To create a `GearsBuilder` object, follow this example code:

[code example]

Functions

BaseReader reader = ...; // Initialize the reader
builder = GearsBuilder.CreateGearsBuilder(reader);


LocalAccumulateBy

Overview

[code example]

The `localAccumulateBy` function is similar to [`accumulateBy`](), except it performs the operation locally on each shard without moving data between shards.

On each shard, it iterates through the records in the pipe, groups them based on the provided extractor, and then reduces each group to a single record per group with the accumulator function.

The initial value of the accumulator is null.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| accumulator | <nobr>AccumulateByOperation<T,I></nobr> | A function with logic to update the accumulator value with each record |
| extractor | ExtractorOperation<T> | Extracts a specific value from each record |

Returns

Example

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> localAccumulateBy(
	gears.operations.ExtractorOperation<T> extractor,
	gears.operations.AccumulateByOperation<T,I> accumulator)

GearsBuilder.CreateGearsBuilder(reader).
   	localAccumulateBy(r->{
   		return r.getStringVal();
   	},(k, a, r)->{
   		Integer ret = null;
   		if(a == null) {
   			ret = 0;
   		}else {
   			ret = (Integer)a;
   		}
   		return ret + 1;
});


Log

Overview

[code example]

Writes a log message to the Redis log file. If you do not specify a `LogLevel`, it will default to `NOTICE`.

Parameters

| Name | Type | Description |
|------|------|-------------|
| msg | string | The message to write to the log |
| level | LogLevel | The log level (DEBUG, NOTICE, VERBOSE, WARNING) |

Returns

Example

[code example]

public static void log(java.lang.String msg)

public static void log(java.lang.String msg, LogLevel level)

GearsBuilder.log(
    "Setting keys to expire after 1 month",
    LogLevel.WARNING
);


Map

Summary: Maps records one-to-one.

Overview

[code example]

Maps each input record in the pipe to an output record, one-to-one.

Parameters

Type parameters:

| Name | Description |
|------|-------------|
| I | The template type of the returned builder |

Function parameters:

| Name | Type | Description |
|------|------|-------------|
| mapper | <nobr>MapOperation<T,I></nobr> | For each input record, returns a new output record |

Returns

Example

Map each record to its string value:

[code example]

public <I extends java.io.Serializable> GearsBuilder<I> map(
	gears.operations.MapOperation<T,I> mapper)

GearsBuilder.CreateGearsBuilder(reader).
 		map(r->{
    		return r.getStringVal();
});


Register

Overview

[code example]

Registers the pipeline of functions to run when certain [events]() occur. The registered functions will run each time the event occurs.

Execution modes:

| Name | Description |
|------|-------------|
| ASYNC | Runs asynchronously on all of the shards. |
| ASYNC_LOCAL | Runs asynchronously but only on the current shard that generated the event. |
| SYNC | Runs synchronously only on the same shard that generated the event. |

If you call `register()` without specifying an execution mode, it will default to `ASYNC`.

Parameters

| Name | Type | Description |
|------|------|-------------|
| mode | ExecutionMode | The execution mode to use (ASYNC/ASYNC_LOCAL/SYNC) |
| onRegister | OnRegisteredOperation | Register callback that will be called on each shard upon register |
| onUnregistered | OnUnregisteredOperation | Unregister callback that will be called on each shard upon unregister |

Returns

Returns a registration ID.

Example

[code example]

public java.lang.String register()

public java.lang.String register(ExecutionMode mode)

public java.lang.String register(
    ExecutionMode mode,
    gears.operations.OnRegisteredOperation onRegister,
    gears.operations.OnUnregisteredOperation onUnregistered)


Repartition

Overview

[code example]

Moves records between the shards. The extracted data determines the new shard location for each record.

Parameters

| Name | Type | Description |
|------|------|-------------|
| extractor | <nobr>ExtractorOperation<T></nobr> | Extracts a specific value from each record |

Returns

Returns a GearsBuilder object with a new template type.

Example

Repartition by value:

[code example]

public GearsBuilder<T> repartition(
	gears.operations.ExtractorOperation<T> extractor)

GearsBuilder.CreateGearsBuilder(reader).
   	repartition(r->{
   		return r.getStringVal();
});


Run

Overview

[code example]

Runs the pipeline of functions immediately upon execution. It will only run once.

Parameters

| Name | Type | Description |
|------|------|-------------|
| collect | boolean | Whether or not to collect the results from the entire cluster before returning them |
| jsonSerialize | boolean | Whether or not to serialize the results to JSON before returning them |

Returns

Example

[code example]

public void run()

public void run(boolean jsonSerialize, boolean collect)


GearsFuture

Summary: Allows asynchronous processing of records.

Summary: Sets an error message.

SetError

Summary: Sets a computation to run asynchronously.

SetResult

Overview

The `GearsFuture` class allows asynchronous processing of records in another thread.

You can use a `GearsFuture` object with the following [`GearsBuilder`]() functions:

- [`asyncFilter`]()
- [`asyncForeach`]()
- [`asyncMap`]()

Functions


SetError

Summary: Sets an error message.

Overview

[code example]

Sets an error message for an asynchronous computation.

Parameters

| Name | Type | Description |
|------|------|-------------|
| error | string | An error message |

Returns

Example

[code example]

public void setError(java.lang.String error)
	throws java.lang.Exception

GearsFuture<Boolean> f = new GearsFuture<Boolean>();
try {
	f.setError("An error has occurred during asyncForeach");
} catch (Exception e) {
	e.printStackTrace();
}


SetResult

Overview

[code example]

Sets a computation to run asynchronously.

Parameters

| Name | Type | Description |
|------|------|-------------|
| result | template type I | The result of a computation |

Returns

Example

[code example]

public void setResult(I result)
	throws java.lang.Exception

GearsBuilder.CreateGearsBuilder(reader).map(r->r.getKey()).
	asyncFilter(r->{
		GearsFuture<Boolean> f = new GearsFuture<Boolean>();
		try {
			f.setResult(r.equals("x"));
		} catch (Exception e) {
			e.printStackTrace();
		}
		return f;
});


RedisGears Java classes and functions

GearsBuilder

GearsFuture

Summary: Extracts data from the database and creates records to pass through a RedisGears pipeline.

RedisGears readers

Overview

The RedisGears JVM plugin provides a set of classes and functions for you to use in your Java code.

Classes


CommandOverrider

Summary: Override a Redis command.

Overview

The `CommandOverrider` allows you to override and customize Redis commands.

1. Pass the `CommandOverrider` to the [`GearsBuilder.CreateGearsBuilder()`]() function in your Java code.
1. Call the [`register()`]() function.
1. Run `RG.JEXECUTE` to register your code.

If you register code that uses `CommandOverrider`, its `reader` value is `"CommandReader"` when you run the `RG.DUMPREGISTRATIONS` command, not `"CommandOverrider"`.

Parameters

| Name | Type | Description |
|------|------|-------------|
| command | string | The command to override |
| prefix | string | Only override the command for keys that start with this string |

Output records

Outputs a record with the command's name and arguments.

Example

The following example shows how to override the `HSET` command so that it also adds a `last_modified` timestamp for "user:" hashes.

Add some user data as a hash:

[code example]

Example code:

[code example]

After you register the previous code with the `RG.JEXECUTE` command, try to update the user's data with `HSET` to test it:

[code example]

Now the user's profile should have the updated value for `posts` and a `last_modified` timestamp:

[code example]

redis> HSET user:1 name "morgan" posts 201
(integer) 2

// Create the reader that will pass data to the pipe
CommandOverrider overrider = new CommandOverrider();
// Override the HSET command
overrider.setCommand("HSET");
// Only override HSET for keys that start with "user:"
overrider.setPrefix("user:");

// Create the data pipe builder
GearsBuilder.CreateGearsBuilder(overrider).map(r-> {
    // Get the operation arguments
    ArrayList<String> operationArguments = new ArrayList<String>();
    for (int i = 1; i < r.length; i++) {
        operationArguments.add(new String((byte[]) r[i], StandardCharsets.UTF_8));
    }

    // Add a last-modified field and a corresponding timestamp to the operation arguments
    operationArguments.add("last-modified");
    operationArguments.add(Long.toString(System.currentTimeMillis()));

    // Run the enriched HSET operation
    Object reply = GearsBuilder.callNext(operationArguments.toArray(new String[0]));

    return reply.toString();
}).register(ExecutionMode.SYNC);


CommandReader

Summary: Run RedisGears functions on command.

Overview

The `CommandReader` allows you to run RedisGears functions on command when you:

1. Pass the `CommandReader` to the [`GearsBuilder.CreateGearsBuilder()`]() function in your Java code.
1. Call the [`register()`]() function.
1. Run `RG.JEXECUTE` to register your code.
1. Use `RG.TRIGGER` to run your code on command:

    [code example]

Parameters

| Name | Type | Description |
|------|------|-------------|
| trigger | string | The command name that triggers the registered RedisGears functions to run |

Output records

Outputs a record with the command trigger's name and arguments.

Example

The following example shows how to create a custom command to update an item's stock. It also adds a timestamp to track when the last restock occurred.

Add a hash to the database that represents an inventory item:

[code example]

Example code:

[code example]

After you register the previous code with the `RG.JEXECUTE` command, run `RG.TRIGGER` to test it:

[code example]

The item now has the updated value for `stock` and a `last_restocked` timestamp:

[code example]

RG.TRIGGER <Trigger name> [arg1 arg2 ...]

redis> HSET inventory:headphones:1 color "blue" stock 5 price 30.00
(integer) 3

// Create the reader that will pass data to the pipe
CommandReader reader = new CommandReader();
// Set the name of the custom command
reader.setTrigger("Restock");

// Create the data pipe builder
GearsBuilder.CreateGearsBuilder(reader).map(r-> {
    // Parse the command arguments to get the key name and new stock value
    String itemKey = new String((byte[]) r[1], StandardCharsets.UTF_8);
    String newStock = new String((byte[]) r[2], StandardCharsets.UTF_8);

    // Update the item's stock and add a timestamp
    GearsBuilder.execute("HSET", itemKey , "stock", newStock,
        			"last_restocked", Long.toString(System.currentTimeMillis()));

    return "OK restocked " + itemKey;
}).register();

redis> RG.TRIGGER Restock inventory:headphones:1 20
1) "OK restocked inventory:headphones:1"

redis> HGETALL inventory:headphones:1
1) "color"
2) "blue"
3) "stock"
4) "20"
5) "price"
6) "30.00"
7) "last_restocked"
8) "1643232394078"


RedisGears readers

Summary: Override a Redis command.

CommandOverrider

Summary: Run RedisGears functions on command.

CommandReader

Summary: A template for creating custom readers.

JavaReader

Summary: Gets key names from a database.

KeysOnlyReader

Summary: Gets keys and their values from a database.

KeysReader

Summary: Gets the shard ID for each shard in a database.

ShardsIDReader

Summary: Reads Redis stream data.

StreamReader

Overview

A reader extracts data from the database and creates records.

The [`GearsBuilder.CreateGearsBuilder(reader)`]() function takes a reader as a parameter and passes the generated records through a pipeline of RedisGears functions.

Classes


JavaReader

Summary: A template for creating custom readers.

Overview

The `JavaReader` is an abstract class that allows you to create a custom reader in Java.

Create a custom reader

To create a custom reader:

- Extend the `JavaReader` class
- Override the `iterator()` function

Custom reader example

The implementation of the [`KeysOnlyReader`]() class shows how to create a custom reader with `JavaReader`:

[code example]

import java.util.Iterator;

import gears.GearsBuilder;

/**
 * A reader that only reads key names from the key space
 *
 */
public class KeysOnlyReader extends JavaReader<String> {

	/**
	 *
	 */
	private static final long serialVersionUID = 1L;

	private String scanSize;
	private String pattern;

	/**
	 * Create a new KeysOnlyReader reader
	 * @param scanSize - the size to use with the scan command
	 * @param pattern - the pattern of the keys to read
	 */
	public KeysOnlyReader(int scanSize, String pattern) {
		this.scanSize = Integer.toString(scanSize);
		this.pattern = pattern;
	}

	/**
	 * Create a new KeysOnlyReader reader with default pattern (*) and default
	 * scan size (10000)
	 */
	public KeysOnlyReader() {
		this(10000, "*");
	}

	@Override
	public Iterator<String> iterator() {
		return new Iterator<String>() {

			String cursor = "0";
			int currIndex = 0;
			Object[] keys = null;
			boolean isDone = false;
			String nextKey = null;

			private String innerNext() {
				while(!isDone) {
					if(keys == null) {
						Object[] res = (Object[]) GearsBuilder.execute("scan",
										cursor == null ? "0" : cursor,
										"MATCH", pattern, "COUNT", scanSize);
						keys = (Object[])res[1];
						cursor = (String)res[0];
						currIndex = 0;
					}
					if(currIndex < keys.length) {
						return (String) keys[currIndex++];
					}

					keys = null;
					if(cursor.charAt(0) == '0') {
						isDone = true;
					}
				}
				return null;
			}

			@Override
			public boolean hasNext() {
				if(nextKey == null) {
					nextKey = innerNext();
				}
				return !isDone;
			}

			@Override
			public String next() {
				String temp = nextKey != null ? nextKey : innerNext();
				nextKey = innerNext();
				return temp;
			}

		};
	}

}


KeysOnlyReader

Summary: Gets key names from a database.

Overview

The `KeysOnlyReader` only extracts the key names from a database.

Constructors

You can use one of these constructors to create a new `KeysOnlyReader` object:

[code example]

Parameters

| Name | Type | Default value | Description |
|------|------|---------------|-------------|
| pattern | string | "\*" (match all keys) | Get all keys that match this pattern |
| scanSize | integer | 10000 | The scan command's size limit |

Output records

Each output record is a string that represents the key's name.

Examples

Get all keys in the database:

[code example]

Only get keys that start with "user:":

[code example]

public KeysOnlyReader()

public KeysOnlyReader(int scanSize, String pattern)

KeysOnlyReader reader = new KeysOnlyReader();

KeysOnlyReader reader = new KeysOnlyReader(1000, "user:*");


KeysReader

Overview

Creates records from the keys and values stored in a Redis database.

 Currently only supports string and hash data types. For other data types, it will only extract the key name.

Constructors

You can use one of these constructors to create a new `KeysReader` object:

[code example]

Parameters

| Name | Type | Default value | Description |
|------|------|---------------|-------------|
| commands | array of strings | null | The commands that this reader is registered on |
| eventTypes | array of strings | null | The [event types]() to register on (usually the command name) |
| keyTypes | array of strings | null | The key types to register on |
| noScan | boolean | false | Whether or not to scan the key space or just read the pattern as is |
| pattern/prefix | string | "\*" (match all keys) | The reader will get all keys that match this pattern |
| readValues | boolean | true | Whether or not to read the keys' values |

Output records

Creates a `KeysReaderRecord` for each matching key in the database.

| Name | Type | Description |
|------|------|-------------|
| key | string | The name of the key |
| type | long | The core Redis type: 'string', 'hash', 'list', 'set', 'zset', or 'stream' |
| event | string | The event that triggered the execution (null if using the run function) |
| stringVal | string | The key's value for string data types |
| hashVal | Map<String,String> | The key's value for hash data types |
| listVal | List<String> | The key's value for list data types |
| setVal | Set<String> | The key's value for set data types |

Examples

Here's a basic example of a `KeysReader` that creates records for all keys in the database:

[code example]

In the following example, the `KeysReader` creates records for all keys in the database that start with "person:". When registered, it only runs for hashes after `HSET` and `DEL` events occur.

[code example]

public KeysReader()

public KeysReader(String pattern)

public KeysReader(String prefix, boolean readValues)

public KeysReader(String pattern, boolean noScan, boolean readValues)

public KeysReader(String prefix,
                  boolean readValues,
                  String[] eventTypes,
                  String[] keyTypes)

public KeysReader(String pattern,
                  boolean noScan,
                  boolean readValues,
                  String[] eventTypes,
                  String[] keyTypes)

String[] eventTypes = {"HSET", "DEL"};
String[] keyTypes = {"HASH"};
KeysReader reader = new KeysReader("person:*", false, true, eventTypes, keyTypes);