Configure the query performance factor for Redis Query Engine in Redis Enterprise

Configure the query performance factor for Redis Query Engine in Redis Enterprise to increase the performance of queries.

Query performance factors are intended to increase the performance of queries, including vector search. When enabled, it allows you to increase a database's compute capacity and query throughput by allocating more virtual CPUs per shard. This is in addition to horizontal scaling with more shards which enables a higher throughput of key value operations. This document describes how to configure the query performance factor.

Note:
Some use cases might not scale effectively. Redis experts can help determine if vertical scaling with the Redis Query Engine will boost performance for your use case and guide you on whether to use vertical scaling, horizontal scaling, or both.

Prerequisites

Redis Query Engine requires a cluster running Redis Enterprise Software version 7.4.2-54 or later.

If you do not have a cluster that supports Redis Query Engine, install Redis Enterprise Software version 7.4.2-54 or later on a new cluster, or upgrade an existing cluster.

Sizing

  1. Calculate the hardware requirements for your Redis database:

    1. Use the hardware requirements documentation to derive the overall cluster architecture.

    2. Calculate the RAM requirements using the Index Size Calculator. The total RAM required is the sum of the dataset and index sizes.

  2. Determine the query performance factor you want and the required number of CPUs. Unused CPUs, above the 20% necessary for Redis, can be used for the scalable Redis Query Engine.

  3. Create a new Redis database with the number of CPUs configured for the query performance factor.

Calculate query performance factor

CPUs for query performance factor

Vertical scaling of the Redis Query Engine is achieved by provisioning additional CPUs for the RediSearch module. At least 20% of the available CPUs must be reserved for Redis internal processing. Use the following formula to define the maximum number of CPUs that can be allocated to search.

Variable Value
CPUs per node x
Redis internals 20%
Available CPUs for Redis Query Engine floor(0.8 * x)

Query performance factor versus CPUs

The following table shows the number of CPUs required for each performance factor. This calculation is sensitive to how the search index and queries are defined. Certain scenarios might yield less throughput than the ratios in the following table.

Scale factor Minimum CPUs required for Redis Query Engine
None (default) 1
2 3
4 6
6 9
8 12
10 15
12 18
14 21
16 24

Example performance factor calculation

Variable Value
CPUs per node 8
Available CPUs floor(0.8 * 8)=6
Scale factor 4x
Minimum CPUs required for scale factor 6

Configure query performance factor manually

To manually configure the query performance factor in Redis Enterprise Software:

  1. Configure query performance factor parameters when you create a new database or edit an existing database's configuration in the Cluster Manager UI.

  2. If you configure the query performance factor for an existing database, you also need to restart shards. Newly created databases can skip this step.

Configure query performance factor parameters in the Cluster Manager UI

You can use the Cluster Manager UI to configure the query performance factor when you create a new database or edit an existing database with search enabled.

  1. In the Capabilities section of the database configuration screen, click Parameters.

  2. If you are creating a new database, select Search and query.

  3. Adjust the RediSearch parameters to include:

    MT_MODE MT_MODE_FULL WORKER_THREADS <NUMBER_OF_THREADS>

    See Calculate query performance factor to determine the minimum CPUs required to use for <NUMBER_OF_THREADS>.

  4. Expand the Query Performance Factor section and enter the following values:

    • mnp for Connections routing

    • 32 for Connections limit

    Configure search parameters and query performance factor.

  5. Click Done to close the parameter editor.

  6. Click Create or Save.

Restart shards

After you update the query performance factor for an existing database, restart all shards to apply the new settings. You can migrate shards to restart them. Newly created databases can skip this step.

  1. Use rladmin status shards db <db-name> to list all shards for your database:

    rladmin status shards db db-name

    Example output:

    SHARDS:
DB:ID   NAME                ID      NODE   ROLE   SLOTS   USED_MEMORY STATUS
db:2    db-name             redis:1 node:1 master 0-16383 1.95MB      OK    
db:2    db-name             redis:2 node:2 slave  0-16383 1.95MB      OK

    Note the following fields for the next steps:

    • ID: the Redis shard's ID.
    • NODE: the node on which the shard currently resides.
    • ROLE: master is a primary shard; slave is a replica shard.

  2. For each replica shard, use rladmin migrate shard to move it to a different node and restart it:

    rladmin migrate shard <shard_id> target_node <node_id>

  3. After you migrate the replica shards, migrate the original primary shards.

  4. Rerun rladmin status shards db <db-name> to verify the shards migrated to different nodes:

    rladmin status shards db db-name

    Example output:

    SHARDS:
DB:ID   NAME                ID      NODE   ROLE   SLOTS   USED_MEMORY STATUS
db:2    db-name             redis:1 node:2 master 0-16383 1.95MB      OK    
db:2    db-name             redis:2 node:1 slave  0-16383 1.95MB      OK

Configure query performance factor with the REST API

You can configure the query performance factor when you create a new database or update an existing database using the Redis Enterprise Software REST API.

Create new database with the REST API

To create a database and configure the query performance factor, use the create database REST API endpoint with a BDB object that includes the following parameters:

{
    "sched_policy": "mnp",
    "conns": 32,
    "module_list": [{
        "module_name": "search",
        "module_args": "MT_MODE MT_MODE_FULL WORKER_THREADS <NUMBER_OF_CPUS>"
    }]
}

See Calculate performance factor to determine the value to use for <NUMBER_OF_CPUS>.

Example REST API request for a new database

The following JSON is an example request body used to create a new database with a 4x query performance factor configured:

{
    "name": "scalable-search-db",
    "type": "redis",
    "memory_size": 10000000,
    "port": 13000,
    "authentication_redis_pass": "<your default db pwd>",
    "proxy_policy": "all-master-shards",
    "sched_policy": "mnp",
    "conns": 32,
    "sharding": true,
    "shards_count": 3,
    "shards_placement": "sparse",
    "shard_key_regex": [{"regex": ".*\\{(?<tag>.*)\\}.*"}, {"regex": "(?<tag>.*)"}],
    "replication": false,
    "module_list": [{
        "module_name": "search",
        "module_args": "MT_MODE MT_MODE_FULL WORKER_THREADS 6"
    }]
}

The following cURL request creates a new database from the JSON example:

curl -k -u "<user>:<password>" https://<host>:9443/v1/bdbs -H "Content-Type:application/json" -d @scalable-search-db.json

Update existing database with the REST API

To configure the query performance factor for an existing database, use the following REST API requests:

Note:
  • Because this procedure also restarts the database shards, you should perform it during a maintenance period.
  • This procedure overwrites any existing module configuration parameters.

The following example script uses both endpoints to configure a 4x query performance factor:

#!/bin/bash
export DB_ID=1
export CPU=6
export MODULE_ID=`curl -s -k -u "<user>:<password>" https://<host>:9443/v1/bdbs/$DB_ID | jq '.module_list[] | select(.module_name=="search").module_id' | tr -d '"'`

curl -o /dev/null -s -k -u "<user>:<password>" -X PUT https://<host>:9443/v1/bdbs/$DB_ID -H "Content-Type:application/json" -d '{
    "sched_policy": "mnp",
    "conns": 32
}'

sleep 1

curl -o /dev/null -s -k -u "<user>:<password>" https://<host>:9443/v1/bdbs/$DB_ID/modules/upgrade -H "Content-Type:application/json" -d '{
    "modules": [
      {
        "module_name": "search",
        "new_module_args": "MT_MODE MT_MODE_FULL WORKER_THREADS '$CPU'",
        "current_module": "'$MODULE_ID'",
        "new_module": "'$MODULE_ID'"
      }
    ]
}'

Monitoring Redis Query Engine

To monitor a database with a query performance factor configured:

  1. Integrate your Redis Enterprise deployment with Prometheus. See Prometheus and Grafana with Redis Enterprise for instructions.

  2. Monitor the redis_process_cpu_usage_percent shard metric.

    The following Prometheus UI screenshot shows redis_process_cpu_usage_percent spikes for a database with two shards:

    • 1st 100% spike: memtier_benchmark search test at the default (no additional CPUs for search).

    • 2nd 100% spike: reconfiguration and shard restart for a 4x query performance factor.

    • 3rd 600% spike: memtier_benchmark search test with threading at a 4x query performance factor (6 CPUs per shard).

    The Prometheus graph shows three spikes for redis_process_cpu_usage_percent: 100%, another 100%, then 600%.
RATE THIS PAGE
Back to top ↑