Extend existing search and query features

Summary: Details about extensions for query expanders and scoring functions

Overview

RediSearch supports an extension mechanism, much like Redis supports modules. The API is very minimal at the moment, and it does not yet support dynamic loading of extensions on a running server. Instead, extensions must be written in C (or a language that has an interface with C) and compiled into dynamic libraries that can be loaded at start up.

There are two kinds of extension APIs at the moment:

1. **Query expanders**, whose role is to expand query tokens (i.e., stemmers).
2. **Scoring functions**, whose role is to rank search results at query time.

Registering and loading extensions

Extensions should be compiled into dynamic library files (e.g., `.so` files), and loaded into the RediSearch module during initialization.

Compiling

Extensions should be compiled and linked as dynamic libraries. An example Makefile for an extension [can be found here](https://github.com/RediSearch/RediSearch/blob/master/tests/ctests/ext-example/Makefile).

    That folder also contains an example extension that is used for testing and can be taken as a skeleton for implementing your own extension.

Loading

Loading an extension is done by appending `EXTLOAD {path/to/ext.so}` after the `loadmodule` configuration directive when loading the RediSearch module. For example:

    [code example]

    This causes the RediSearch module to automatically load the extension and register its expanders and scorers.

Initializing an extension

The entry point of an extension is a function with the signature:

[code example]

When loading an extension, RediSearch looks for this function and calls it. This function is responsible for registering and initializing the expanders and scorers.

It should return REDISEARCH_ERR on error or REDISEARCH_OK on success.

Example init function

[code example]

Calling your custom functions

When performing a query, you can use your scorers or expanders by specifying the SCORER or EXPANDER arguments with the given alias. For example:

[code example]

**NOTE**: Expander and scorer aliases are **case sensitive**.

The query expander API

Only basic query expansion is supported, one token at a time. An expander can decide to expand any given token with as many tokens it wishes, which will be union-merged in query time.

The API for an expander is the following:

[code example]

RSQueryExpanderCtx

`RSQueryExpanderCtx` is a context that contains private data of the extension, and a callback method to expand the query. It is defined as:

[code example]

RSToken

`RSToken` represents a single query token to be expanded, and is defined as:

[code example]

The scoring function API

For the final ranking, the scoring function analyzes each document retrieved by the query, taking into account not only the terms that triggered the document's retrieval but also metadata like its prior score, length, and so on.

Since the scoring function is evaluated for each document, potentially millions of times, and since
redis is single threaded, it is important that it works as fast as possible and be heavily optimized.

A scoring function is applied to each potential result for each document and is implemented with the following signature:

[code example]

`RSScoringFunctionCtx` is a context that implements some helper methods.

`RSIndexResult` is the result information containing the document id, frequency, terms, and offsets.

`RSDocumentMetadata` is an object holding global information about the document, such as its presumptive score.

`minScore` is the minimal score that will yield a result that is relevant to the search. It can be used to stop processing midway or before or even before it starts.

The return value of the function is a `double` representing the final score of the result.
Returning 0 causes the result to be counted, but if there are results with a score greater than 0, they will appear above it.
To completely filter out a result and not count it in the totals, the scorer should return the special value `RS_SCORE_FILTEROUT`, which is internally set to negative infinity, or -1/0.

RSScoringFunctionCtx

This is an object containing the following members:

* `void *privdata`: a pointer to an object set by the extension on initialization time.
* `RSPayload payload*`: A Payload object set either by the query expander or the client.
* `int GetSlop(RSIndexResult *res)*`: A callback method that yields the total minimal distance between the query terms. This can be used to prefer results where the slop is smaller and the terms are nearer to each other.

RSIndexResult

This is an object holding the information about the current result in the index, which is an aggregate of all the terms that resulted in the current document being considered a valid result. See `redisearch.h` for details.

RSDocumentMetadata

This is an object describing global information, unrelated to the current query, about the document being evaluated by the scoring function.

Example query expander

This example query expander expands each token with the term foo:

[code example]

Example scoring function

This is an actual scoring function, which calculates TF-IDF for the document, multiplies it by the document score, and divides it by the slop:

[code example]

$ redis-server --loadmodule ./redisearch.so EXTLOAD ./ext/my_extension.so

#include <redisearch.h> //must be in the include path

int RS_ExtensionInit(RSExtensionCtx *ctx) {

  /* Register  a scoring function with an alias my_scorer and no special private data and free function */
  if (ctx->RegisterScoringFunction("my_scorer", MyCustomScorer, NULL, NULL) == REDISEARCH_ERR) {
    return REDISEARCH_ERR;
  }

  /* Register a query expander  */
  if (ctx->RegisterQueryExpander("my_expander", MyExpander, NULL, NULL) ==
      REDISEARCH_ERR) {
    return REDISEARCH_ERR;
  }

  return REDISEARCH_OK;
}

FT.SEARCH my_index "foo bar" EXPANDER my_expander SCORER my_scorer

#include <redisearch.h> //must be in the include path

void MyQueryExpander(RSQueryExpanderCtx *ctx, RSToken *token) {
    ...
}

typedef struct RSQueryExpanderCtx {

  /* Opaque query object used internally by the engine, and should not be accessed */
  struct RSQuery *query;

  /* Opaque query node object used internally by the engine, and should not be accessed */
  struct RSQueryNode **currentNode;

  /* Private data of the extension, set on extension initialization */
  void *privdata;

  /* The language of the query, defaults to "english" */
  const char *language;

  /* ExpandToken allows the user to add an expansion of the token in the query, that will be
   * union-merged with the given token in query time. str is the expanded string, len is its length,
   * and flags is a 32 bit flag mask that can be used by the extension to set private information on
   * the token */
  void (*ExpandToken)(struct RSQueryExpanderCtx *ctx, const char *str, size_t len,
                      RSTokenFlags flags);

  /* SetPayload allows the query expander to set GLOBAL payload on the query (not unique per token)
   */
  void (*SetPayload)(struct RSQueryExpanderCtx *ctx, RSPayload payload);

} RSQueryExpanderCtx;

/* A token in the query. The expanders receive query tokens and can expand the query with more query
 * tokens */
typedef struct {
  /* The token string - which may or may not be NULL terminated */
  const char *str;
  /* The token length */
  size_t len;

  /* 1 if the token is the result of query expansion */
  uint8_t expanded:1;

  /* Extension specific token flags that can be examined later by the scoring function */
  RSTokenFlags flags;
} RSToken;

double MyScoringFunction(RSScoringFunctionCtx *ctx, RSIndexResult *res,
                                    RSDocumentMetadata *dmd, double minScore);

#include <redisearch.h> //must be in the include path

void DummyExpander(RSQueryExpanderCtx *ctx, RSToken *token) {
    ctx->ExpandToken(ctx, strdup("foo"), strlen("foo"), 0x1337);
}

#include <redisearch.h> //must be in the include path

double TFIDFScorer(RSScoringFunctionCtx *ctx, RSIndexResult *h, RSDocumentMetadata *dmd,
                   double minScore) {
  // no need to evaluate documents with score 0
  if (dmd->score == 0) return 0;

  // calculate sum(tf-idf) for each term in the result
  double tfidf = 0;
  for (int i = 0; i < h->numRecords; i++) {
    // take the term frequency and multiply by the term IDF, add that to the total
    tfidf += (float)h->records[i].freq * (h->records[i].term ? h->records[i].term->idf : 0);
  }
  // normalize by the maximal frequency of any term in the document
  tfidf /=  (double)dmd->maxFreq;

  // multiply by the document score (between 0 and 1)
  tfidf *= dmd->score;

  // no need to factor the slop if tfidf is already below minimal score
  if (tfidf < minScore) {
    return 0;
  }

  // get the slop and divide the result by it, making sure we prefer results with closer terms
  tfidf /= (double)ctx->GetSlop(h);

  return tfidf;
}


Garbage collection

Summary: Details about garbage collection

The need for garbage collection

* When documents are deleted by the user, Redis only marks them as deleted in the global document table rather than deleting them outright. This is done for efficiency. Depending on the length of a document, deletion can be a long operation.
* This means that it is no longer the case that an internal numeric id is assigned to a deleted document. When the index is traversed, a check is made for deletion.
* All inverted index entries belonging to deleted document ids are garbage.
* Updating a document is basically the same as deleting it and then adding it again with a new incremental internal ID. No diffing is performed, and the indexes are appended, so the IDs remain incremental, and the updates fast.

All of the above means that if there are a lot of updates and deletes, a large portion of our inverted index will become garbage, both slowing things down and consuming unnecessary memory.

You want to optimize the index, but you also don't want to disturb normal operation. This means that optimization or garbage collection should be a background process that's non-intrusive. It only needs to be faster than the deletion rate over a sufficient period of time so that you don't create more garbage than you can collect.

Garbage collecting a single-term index

A single-term inverted index is an array of blocks, each of which contains an encoded list of records; e.g., a document id delta plus other data depending on the index encoding scheme. When some of these records refer to deleted documents this is called garbage.

The algorithm is simple:

0. Create a reader and writer for each block.
1. Read each block's records one by one.
2. If no record is invalid, do nothing.
3. When a garbage record is found, the reader is advanced, but not the writer.
4. When at least one garbage record is found, the next records are encoded to the writer, recalculating the deltas.

Pseudo code:

[code example]

GC on numeric indexes

Numeric indexes are a tree of inverted indexes with a special encoding of (docId delta, value). This means the same algorithm can be applied to them, only traversing each inverted index object in the tree.

FORK GC

Information about FORK GC can be found in this [blog](https://redislabs.com/blog/increased-garbage-collection-performance-redisearch-1-4-1/).

Since v1.6, the FORK GC is the default GC policy and was proven very efficient both in cleaning the index and not reducing query and indexing performance, even for very write-internsive use cases.

foreach index_block as block:

   reader = new_reader(block)
   writer = new_write(block)
   garbage = 0
   while not reader.end():
        record = reader.decode_next()
        if record.is_valid():
            if garbage != 0:
                # Write the record at the writer's tip with a newly calculated delta
                writer.write_record(record)
            else:
                writer.advance(record.length)
        else:
            garbage += record.length


Administration

Summary: Redis Search Administration

Summary: Redis Search can be tuned through multiple configuration parameters. Some of these parameters can only be set at load-time, while other parameters can be set either at load-time or at run-time.

Configuration parameters

Summary: Details about design choices and implementations

Internal design

Summary: Technical overview of the search and query features of Redis Open Source

Technical overview

Summary: Details about garbage collection

Garbage collection

Summary: This document describes how documents are added to the index.

Document Indexing


Document Indexing

Components

* `Document` - contains the actual document and its fields.
* `RSAddDocumentCtx` - the per-document state that is used while it
  is being indexed. The state is discarded after the indexing is complete.
* `ForwardIndex` - contains terms found in the document. The forward index
  is used to write the `InvertedIndex`.
* `InvertedIndex` - an index that maps terms to occurrences within applicable
  documents.

Architecture

The indexing process begins by creating a new `RSAddDocumentCtx` and adding a
document to it. Internally, this is divided into several steps.

1. Submission

   A `DocumentContext` is created, and is associated with a document (as received)
   from input. The submission process will also perform some preliminary caching.

2. Preprocessing

   After a document has been submitted, it is preprocessed. Preprocessing performs
   stateless processing on all document input fields. For text fields, this
   means tokenizing the document and creating a forward index. The preprocesors
   will store this information in per-field variables within the `AddDocumentCtx`.
   This computed result is then written to the persistent index later on during
   the indexing phase.

   If the document is sufficiently large, the preprocessing is done in a separate
   thread, which allows concurrent preprocessing and also avoids blocking other
   threads. If the document is smaller, the preprocessing is done within the main
   thread, avoiding the overhead of additional context switching.
   The `SELF_EXC_THRESHOLD` macro contains the threshold for 'sufficiently large'.

   After the document is preprocessed, it is submitted to be indexed.

3. Indexing

   Indexing proper consists of committing the precomputed results of the
   preprocessing phase. It is done in a single thread, and is in the form
   of a queue.

   Because documents must be written to the index in the exact order of their
   document ID assignment, and because the indexing process must also yield to other potential
   indexing processes, you may end up in a situation where document IDs are written
   to the index out-of-order. To solve that, the order in which documents
   are actually written must be well-defined. If there is only one thread writing
   documents, then this thread will not need to worry about out-of-order IDs
   while writing.

   Having a single background thread also helps optimize in several areas, as
   will be seen later on. The basic idea is that when there are a lot of
   documents queued for the indexing thread, the indexing thread may treat them
   as batch commands, greatly reducing the number of locks/unlocks of the GIL
   and the number of times term keys need to be opened and closed.

4. Skipping already indexed documents

   The phases below may operate on more than one document at a time. When a document
   is fully indexed, it is marked as done. When the thread iterates over the queue
   it will only perform processing/indexing on items not yet marked as done.

5. Term merging

   Term merging, or forward index merging, is done when there is more than a
   single document in the queue. The forward index of each document in the queue
   is scanned, and a larger, master forward index is constructed in its place.
   Each entry in the forward index contains a reference to the origin document
   as well as the normal offset/score/frequency information.

   Creating a master forward index avoids opening common term keys more than once per
   document.

   If there is only one document within the queue, a master forward index
   is not created.

   Note that the internal type of the master forward index is not actually
   `ForwardIndex`.

6. Document ID assignment

   At this point, the GIL is locked and every document in the queue is assigned
   a document ID. The assignment is done immediately before writing to the index
   so as to reduce the number of times the GIL is locked; thus, the GIL is
   locked only once, right before the index is written.

7. Writing to Indexes

   With the GIL being locked, any pending index data is written to the indexes.
   This usually involves opening one or more Redis keys, and writing/copying
   computed data into those keys.

   After this is done, the reply for the given document is sent, and the
   `AddDocumentCtx` freed.


Technical overview

Abstract

Redis Search is a powerful text search and secondary indexing engine that is built on top of Redis Open Source.

Unlike other Redis search libraries, it does not use the internal data structures of Redis such as sorted sets. Using its own highly optimized data structures and algorithms, it allows for advanced search features, high performance, and a low memory footprint. It can perform simple text searches, as well as complex structured queries, filtering by numeric properties and geographical distances.

Redis Search supports continuous indexing with no performance degradation, maintaining concurrent loads of querying and indexing. This makes it ideal for searching frequently updated databases without the need for batch indexing and service interrupts.

The Enterprise version of Redis Search supports scaling the search engine across many servers, allowing it to easily grow to billions of documents on hundreds of servers.

All of this is done while taking advantage of Redis's robust architecture and infrastructure. Using Redis's protocol, replication, persistence, and clustering, Redis Search delivers a powerful yet simple to manage and maintain search and indexing engine that can be used as a standalone database, or to augment existing Redis databases with advanced powerful indexing capabilities.

---

Main features

* Full-Text indexing of multiple fields in a document, including:
    * Exact phrase matching.
    * Stemming in many languages.
    * Chinese tokenization support.
    * Prefix queries.
    * Optional, negative, and union queries.
* Distributed search on billions of documents.
* Numeric property indexing.
* Geographical indexing and radius filters.
* Incremental indexing without performance loss.
* A structured query language for advanced queries:
    * Unions and intersections
    * Optional and negative queries
    * Tag filtering
    * Prefix matching
* A powerful autocomplete engine with fuzzy matching.
* Multiple scoring models and sorting by values.
* Concurrent, low-latency insertion and updates of documents.
* Concurrent searches allowing long-running queries without blocking Redis.
* An extension mechanism allowing custom scoring models and query extension.
* Support for indexing existing Hash objects in Redis databases.

---

Indexing documents

Redis needs to know how to index documents in order to search effectively. A document may have several fields, each with its own weight. For example, a title is usually more important than the text itself. The engine can also use numeric or geographical fields for filtering. Hence, the first step is to create the index definition, which tells Redis how to treat the documents that will be added. For example, to define an index of products, indexing their title, description, brand, and price fields, the index creation would look like:

[code example]

When a document is added to this index, as in the following example, each field of the document is broken into its terms (tokenization), and indexed by marking the index for each of the terms in the index. As a result, the product is added immediately to the index and can now be found in future searches.

[code example]

This tells Redis to take the document, break each field into its terms (tokenization), and index it by marking the index for each of the terms in the index as contained in this document. Thus, the product is added immediately to the index and can now be found in future searches.

Searching

Now that the products have been added to our index, searching is very simple:

[code example]

This will tell Redis to intersect the lists of documents for each term and return all documents containing the three terms. Of course, more complex queries can be performed, and the full syntax of the query language is detailed below.

Data structures

Redis uses its own custom data structures and uses Redis' native structures only for storing the actual document content (using Hash objects).

Using specialized data structures allows faster searches and more memory effective storage of index records, utilizing compression techniques like delta encoding.

These are the data structures Redis uses under the hood:

Index and document metadata

For each search _index_, there is a root data structure containing the schema, statistics, etc., but most importantly, compact metadata about each document indexed.

Internally, inside the index, Redis uses delta encoded lists of numeric, incremental, 32-bit document ids. This means that the user given keys or ids for documents, need to be replaced with the internal ids on indexing, and back to the original ids on search.

For that, Redis saves two tables, mapping the two kinds of ids in two ways (one table uses a compact trie, the other is simply an array where the internal document ID is the array index). On top of that, for each document, its user given presumptive score is stored, along with some status bits and any optional payload attached to the document by the user.

Accessing the document metadata table is an order of magnitude faster than accessing the hash object where the document is actually saved, so scoring functions that need to access metadata about the document can operate fast enough.

Inverted index

For each term appearing in at least one document, an inverted index is kept, which is basically a list of all the documents where this term appears. The list is compressed using delta coding, and the document ids are always incrementing.

For example, when a user indexes the documents "foo", "bar", and "baz", they are assigned incrementing ids, e.g., `1025, 1045, 1080`. When encoding them into the index,  only the first ID is encoded, followed by the deltas between each entry and the previous one, e.g., `1025, 20, 35`.

Using variable-width encoding, one byte is used to express numbers under 255, two bytes for numbers between 256 and 16,383, and so on. This can compress the index by up to 75%.

On top of the IDs, the frequency of each term in each document, a bit mask representing the fields in which the term appeared in the document, and a list of the positions in which the term appeared is saved.

The structure of the default search record is as follows. Usually, all the entries are one byte long:

[code example]

Optionally, you can choose not to save any one of those attributes besides the ID, degrading the features available to the engine.

Numeric index

Numeric properties are indexed in a special data structure that enables filtering by numeric ranges in an efficient way. One could view a numeric value as a term operating just like an inverted index. For example, all the products with the price $100 are in a specific list, which is intersected with the rest of the query. See [query execution engine]() for more information.

However, in order to filter by a range of prices, you would have to intersect the query with all the distinct prices within that range, or perform a union query. If the range has many values in it, this becomes highly inefficient.

To avoid this, numeric entries are grouped, with close values together, in a single range node. These nodes are stored in a binary range tree, which allows the engine to select the relevant nodes and union them together. Each entry in a range node contains a document Id and the actual numeric value for that document. To further optimize, the tree uses an adaptive algorithm to try to merge as many nodes as possible within the same range node.

Tag index

Tag indexes are similar to full-text indexes, but use simpler tokenization and encoding in the index. The values in these fields cannot be accessed by general field-less search and can be used only with a special syntax.

The main differences between tag fields and full-text fields are:

1. The tokenization is simpler. The user can determine a separator (defaults to a comma) for multiple tags. Whitespace trimming is done only at the end of tags. Thus, tags can contain spaces, punctuation marks, accents, etc. The only two transformations that are performed are lower-casing (for latin languages only as of now) and whitespace trimming.

2. Tags cannot be found from a general full-text search. If a document has a field called *tags* with the values *foo* and *bar*, searching for foo or bar without a special tag modifier (see below) will not return this document.

3. The index is much simpler and more compressed. Only the document IDs are stored in the index, usually resulting in 1-2 bytes per index entry.

Geo index

Geo indexes utilize Redis's own geo-indexing capabilities. At query time, the geographical part of the query (a radius filter) is sent to Redis, returning only the ids of documents that are within that radius. Longitude and latitude should be passed as a string `lon,lat`. For example, `1.23,4.56`.

Autocomplete

The autocomplete engine (see below for a fuller description) uses a compact trie or prefix tree to encode terms and search them by prefix.

Query language

Simple syntax is supported for complex queries that can be combined together to express complex filtering and matching rules. The query is a text string in the [`FT.SEARCH`]() request that is parsed using a complex query processor.

* Multi-word phrases are lists of tokens, e.g., `foo bar baz`, and imply intersection (logical AND) of the terms.
* Exact phrases are wrapped in quotes, e.g `"hello world"`.
* OR unions (e.g., `word1 OR word2`), are expressed with a pipe (`|`) character. For example, `hello|hallo|shalom|hola`.
* NOT negation (e.g., `word1 NOT word2`) of expressions or sub-queries use the dash (`-`) character. For example, `hello -world`.
* Prefix matches (all terms starting with a prefix) are expressed with a `*` following a 2-letter or longer prefix.
* Selection of specific fields using the syntax `@field:hello world`.
* Numeric Range matches on numeric fields with the syntax `@field:[{min} {max}]`.
* Geo radius matches on geo fields with the syntax `@field:[{lon} {lat} {radius} {m|km|mi|ft}]`
* Tag field filters with the syntax `@field:{tag | tag | ...}`. See the [full documentation on tag fields]().
* Optional terms or clauses: `foo ~bar` means bar is optional but documents with bar in them will rank higher.

Complex query examples

Expressions can be combined together to express complex rules. For example, given a database of products, where each entity has the fields `title`, `brand`, `tags` and `price`, expressing a generic search would be simply:

[code example]

This would return documents containing these terms in any field. Limiting the search to specific fields (title only in this case) is expressed as:

[code example]

Numeric filters can be combined to filter by price within a given price range:

[code example]

Multiple text fields can be accessed in different query clauses. For example, to select products of multiple brands:

[code example]

Tag fields can be used to index multi-term properties without actual full-text tokenization:

[code example]

And negative clauses can also be added to filter out plasma and CRT TVs:

[code example]

Scoring model

Redis comes with a few very basic scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use sortable fields (see below). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search request.

If you prefer a custom scoring function, it is possible to add more functions using the [extension API]().

These are the pre-bundled scoring functions available in Redis:

* **TFIDF** (default)

    Basic [TF-IDF scoring](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) with document score and proximity boosting factored in.

* **TFIDF.DOCNORM**
*
    Identical to the default TFIDF scorer, with one important distinction:

* **BM25STD (default)**

    A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25).

    The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`.

    The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated.

* **BM25STD.NORM**

    A variation of `BM25STD`, where the scores are normalized by the minimum and maximum score.

* **BM25STD.TANH**

    A variation of `BM25STD.NORM`, where the scores are normalised by the linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4.

* **DISMAX**

    A simple scorer that sums up the frequencies of the matched terms. In the case of union clauses, it will give the maximum value of those matches.

* **DOCSCORE**

    A scoring function that just returns the presumptive score of the document without applying any calculations to it. Since document scores can be updated, this can be useful if you'd like to use an external score and nothing further.

Sortable fields

It is possible to bypass the scoring function mechanism and order search results by the value of different document properties (fields) directly, even if the sorting field is not used by the query. For example, you can search for first name and sort by the last name.

When creating the index with [`FT.CREATE`](), you can declare `TEXT`, `TAG`, `NUMERIC`, and `GEO` properties as `SORTABLE`. When a property is sortable, you can later decide to order the results by its values with relatively low latency. When a property is not sortable, it can still be sorted by its values, but may increase latency. For example, the following schema:

[code example]

Would allow the following query:

[code example]

Result highlighting and summarization

Redis uses advanced algorithms for highlighting and summarizing, which enable only the relevant portions of a document to appear in response to a search query. This feature allows users to immediately understand the relevance of a document to their search criteria, typically highlighting the matching terms in bold text. The syntax is as follows:

[code example]

Summarization will fragment the text into smaller sized snippets. Each snippet will contain the found term(s) and some additional surrounding context.

Highlighting will highlight the found term and its variants with a user-defined tag. This may be used to display the matched text in a different typeface using a markup language, or to otherwise make the text appear differently.

Autocomplete

Another important feature for Redis Open Source is its autocomplete engine. This allows users to create dictionaries of weighted terms, and then query them for completion suggestions to a given user prefix. Completions can have payloads, which are user-provided pieces of data that can be used for display. For example, completing the names of users, it is possible to add extra metadata about users to be displayed.

For example, if a user starts to put the term "lcd tv" into a dictionary, sending the prefix "lc" will return the full term as a result. The dictionary is modeled as a compact trie (prefix tree) with weights, which is traversed to find the top suffixes of a prefix.

Redis also allows fuzzy suggestions, meaning you can get suggestions to prefixes even if the user makes a typo in their prefix. This is enabled using a Levenshtein automaton, allowing efficient searching of the dictionary for all terms within a maximal Levenshtein distance of a term or prefix. Suggestions are then weighted based on both their original score and their distance from the prefix typed by the user.

However, searching for fuzzy prefixes (especially very short ones) will traverse an enormous number of suggestions. In fact, fuzzy suggestions for any single letter will traverse the entire dictionary, so the recommendation is to use this feature carefully and in full consideration of the performance penalty it incurs.

The autocomplete engine supports Unicode, allowing for fuzzy matches in non-latin languages as well.

See the [autocomplete page]() for more information and examples.

Search engine internals

The Redis module API

Redis Search is implemented using the [Redis module API]() and is loaded into Redis as an extension module at start-up.

Redis modules make it possible to extend Redis's core functionality, implementing new Redis commands, data structures, and capabilities with similar performance to native core Redis itself. Redis modules are dynamic libraries that can be loaded into Redis at start-up or loaded at run-time using the [`MODULE LOAD`]() command. Redis exports a C API, in the form of a single C header file called `redismodule.h`.

While the logic of Redis Search and its algorithms are mostly independent, and it could be ported quite easily to run as a stand-alone server, it still takes advantage of Redis as a robust infrastructure for a database server. Building on top of Redis means that, by default, modules are afforded:

* A high performance network protocol server
* Robust replication
* Highly durable persistence as snapshots of transaction logs
* Cluster mode

Query execution engine

Redis uses a high-performance flexible query processing engine that can evaluate very complex queries in real time.

The above query language is compiled into an execution plan that consists of a tree of index iterators or filters. These can be any of:

* Numeric filter
* Tag filter
* Text filter
* Geo filter
* Intersection operation (combining 2 or more filters)
* Union operation (combining 2 or more filters)
* NOT operation (negating the results of an underlying filter)
* Optional operation (wrapping an underlying filter in an optional matching filter)

The query parser generates a tree of these filters. For example, a multi-word search would be resolved into an intersect operation of multiple text filters, each traversing an inverted index of a different term. Simple optimizations such as removing redundant layers in the tree are applied.

Each of the filters in the resulting tree evaluates one match at a time. This means that at any given moment, the query processor is busy evaluating and scoring one matching document. This means that very little memory allocation is done at run-time, resulting in higher performance.

The resulting matching documents are then fed to a post-processing chain of result processors that are responsible for scoring them, extracting the top-N results, loading the documents from storage, and sending them to the client. That chain is dynamic as well, which adapts based on the attributes of the query. For example, a query that only needs to return document ids will not include a stage for loading documents from storage.

Concurrent updates and searches

While Redis Search is extremely fast and uses highly optimized data structures and algorithms, it was facing the same problem with regards to concurrency. Depending on the size of your data set and the cardinality of search queries, queries can take anywhere between a few microseconds to hundreds of milliseconds, or even seconds in extreme cases. When that happens, the entire Redis server process is blocked.

Think, for example, of a full-text query intersecting the terms "hello" and "world", each with a million entries, and a half-million common intersection points. To perform that query in a millisecond, Redis would have to scan, intersect, and rank each result in one nanosecond, [which is impossible with current hardware](https://gist.github.com/jboner/2841832). The same goes for indexing a 1,000 word document. It blocks Redis entirely for the duration of the query.

Redis Search uses the Redis Module API's concurrency features to avoid stalling the server for long periods of time. The idea is simple - while Redis itself is single-threaded, a module can run many threads, and any one of those threads can acquire the **Global Lock** when it needs to access Redis data, operate on it, and release it.

Redis cannot be queried in parallel, as only one thread can acquire the lock, including the Redis main thread, but care is taken to make sure that a long-running query will give other queries time to run by yielding this lock from time to time.

The following design principles were adopted to allow concurrency:

1. Redis Search has a thread pool for running concurrent search queries.

2. When a search request arrives, it is passed to the handler, parsed on the main thread, and then a request object is passed to the thread pool via a queue.

3. The thread pool runs a query processing function in its own thread.

4. The function locks the Redis Global lock and starts executing the query.

5. Since the search execution is basically an iterator running in a cycle, the elapsed time is sampled every several iterations (sampling on each iteration would slow things down as it has a cost of its own).

6. If enough time has elapsed, the query processor releases the Global Lock and immediately tries to acquire it again. When the lock is released, the kernel will schedule another thread to run - be it Redis's main thread, or another query thread.

7. When the lock is acquired again, all Redis resources that were held before releasing the lock are re-opened (keys might have been deleted while the thread has been sleeping) and work resumes from the previous state.

Thus the operating system's scheduler makes sure all query threads get CPU time to run. While one is running the rest wait idly, but since execution is yielded about 5,000 times a second, it creates the effect of concurrency. Fast queries will finish in one go without yielding execution, slow ones will take many iterations to finish, but will allow other queries to run concurrently.

Index garbage collection

Redis Search is optimized for high write, update, and delete throughput. One of the main design choices dictated by this goal is that deleting and updating documents do not actually delete anything from the index:

1. Deletion simply marks the document deleted in a global document metadata table using a single bit.
2. Updating, on the other hand, marks a document as deleted, assigns it a new incremental document ID, and re-indexes the document under a new ID, without performing a comparison of the change.

What this means, is that index entries belonging to deleted documents are not removed from the index, and can be seen as garbage. Over time, an index with many deletes and updates will contain mostly garbage, both slowing things down and consuming unnecessary memory.

To overcome this, Redis Search employs a background garbage collection (GC) mechanism. During normal operation of the index, a special thread randomly samples indexes, traverses them, and looks for garbage. Index sections containing garbage are cleaned and memory is reclaimed. This is done in a non- intrusive way, operating on very small amounts of data per scan, and utilizing Redis's concurrency mechanism (see above) to avoid interrupting searches and indexing. The algorithm also tries to adapt to the state of the index, increasing the GC's frequency if the index contains a lot of garbage, and decreasing it if it doesn't, to the point of hardly scanning if the index does not contain garbage.

Extension model

RedisSearch supports an extension mechanism, much like Redis supports modules. The API is very minimal at the moment, and it does not yet support dynamic loading of extensions at run-time. Instead, extensions must be written in C (or a language that has an interface with C) and compiled into dynamic libraries that will be loaded at start-up.

There are two kinds of extension APIs at the moment:

1. **Query expanders**, whose role is to expand query tokens (i.e., stemmers).
2. **Scoring functions**, whose role is to rank search results at query time.

Extensions are compiled into dynamic libraries and loaded into Redis Search on initialization of the module. The mechanism is based on the code of Redis's own module system, albeit far simpler.

---

Scalable distributed search

While Redis Search is very fast and memory efficient, if an index is big enough, at some point it will be too slow or consume too much memory. It must then be scaled out and partitioned over several machines, each of which will hold a small part of the complete search index.

Traditional clusters map different keys to different shards to achieve this. However, with search indexes this approach is not practical. If each word’s index was mapped to a different shard, it would be necessary to intersect records from different servers for multi-term queries.

The way to address this challenge is to employ a technique called index partitioning, which is very simple at its core:

* The index is split across many machines/partitions by document ID.
* Every partition has a complete index of all the documents mapped to it.
* All shards are queried concurrently and the results from each shard are merged into a single result.

To facilitate this, a new component called a coordinator is added to the cluster. When searching for documents, the coordinator receives the query and sends it to N partitions, each holding a sub-index of 1/N documents. Since we’re only interested in the top K results of all partitions, each partition returns just its own top K results. Then, the N lists of K elements are merged and the top K elements are extracted from the merged list.

FT.CREATE idx PREFIX 1 doc: SCHEMA
    title TEXT WEIGHT 5
    description TEXT
    brand TEXT
    PRICE numeric

HSET doc:1
    title "Acme 42 inch LCD TV"
    description "42 inch brand new Full-HD tv with smart tv capabilities"
    brand "Acme"
    price 300

@title:(lcd tv)
    @brand:(sony | samsung | lg)
    @price:[100 500.2]

@title:(lcd tv)
    @brand:(sony | samsung | lg)
    @tags:{42 inch | smart tv}
    @price:[100 500.2]

    -@tags:{plasma | crt}

FT.CREATE users SCHEMA first_name TEXT last_name TEXT SORTABLE age NUMERIC SORTABLE

FT.SEARCH users "john lennon" SORTBY age DESC

FT.SEARCH ...
    SUMMARIZE [FIELDS {num} {field}] [FRAGS {numFrags}] [LEN {fragLen}] [SEPARATOR {separator}]
    HIGHLIGHT [FIELDS {num} {field}] [TAGS {openTag} {closeTag}]


Aggregations

Summary: Groupings, projections, and aggregation functions

Overview

Aggregations are a way to process the results of a search query. Aggregation allows you to group, sort, and transform your result data, and to extract analytic insights from it. Much like aggregation queries in other databases and search engines, they can be used to create analytics reports, or perform [faceted search](https://en.wikipedia.org/wiki/Faceted_search) style queries.

For example, indexing a web-server's logs, you can create a report for unique users by hour, country, or any other breakdown. Or you can create different reports for errors, warnings, etc.

Core concepts

The basic idea of an aggregate query is this:

* Perform a search query, filtering for records you wish to process.
* Build a pipeline of operations that transform the results by zero or more sequences of:
  * **Group and reduce**: grouping by fields in the results, and applying reducer functions to each group.
  * **Sort**: sort the results based on one or more fields.
  * **Apply transformations**: apply mathematical and string functions on fields in the pipeline, optionally creating new fields or replacing existing ones.
  * **Limit**: limit the result, regardless of how the result is sorted.
  * **Filter**: filter the results (post-query) based on predicates relating to its values.

The pipeline is dynamic and re-entrant, and every operation can be repeated. For example, you can group by property X, sort the top 100 results by group size, then group by property Y and sort the results by some other property, then apply a transformation on the output.

Figure 1: Aggregation Pipeline Example

<img src="../../img/pipeline.png">

Aggregate request format

The aggregate request's syntax is defined as follows:

[code example]

Parameters in detail

Parameters that can take a variable number of arguments are expressed in the
form of `param {nargs} {property_1... property_N}`. The first argument to the
parameter is the number of arguments following the parameter. This allows
Redis to avoid a parsing ambiguity in case one of your arguments has the
name of another parameter. For example, to sort by first name, last name, and
country, one would specify `SORTBY 6 firstName ASC lastName DESC country ASC`.

* **index_name**: The index the query is executed against.

* **query_string**: The base filtering query that retrieves the documents. It follows the exact same syntax as the search query, including filters, unions, not, optional, etc.

* **LOAD {nargs} {property} ...** : Load document fields from the document HASH objects. This should be avoided as a general rule of thumb. Fields needed for aggregations should be stored as SORTABLE (and optionally UNF to avoid any normalization), where they are available to the aggregation pipeline with very low latency. LOAD hurts the performance of aggregate queries considerably since every processed record needs to execute the equivalent of HMGET against a Redis key, which when executed over millions of keys, amounts to very high processing times.
The document ID can be loaded using `@__key`.

* **GROUPBY {nargs} {property} ...** : Group the results in the pipeline based on one or more properties. Each group should have at least one reducer (See below), a function that handles the group entries, either counting them or performing multiple aggregate operations (see below).

* **REDUCE {func} {nargs} {arg} ... [AS {name}]**: Reduce the matching results in each group into a single record, using a reduction function. For example, COUNT will count the number of records in the group. See the Reducers section below for more details on available reducers.

    The reducers can have their own property names using the `AS {name}` optional argument. If a name is not given, the resulting name will be the name of the reduce function and the group properties. For example, if a name is not given to COUNT_DISTINCT by property `@foo`, the resulting name will be `count_distinct(@foo)`.

* **SORTBY {nargs} {property} {ASC|DESC} [MAX {num}]**: Sort the pipeline up until the point of SORTBY, using a list of properties. By default, sorting is ascending, but `ASC` or `DESC ` can be added for each property. `nargs` is the number of sorting parameters, including ASC and DESC. for example: `SORTBY 4 @foo ASC @bar DESC`.

    `MAX` is used to optimized sorting, by sorting only for the n-largest elements. Although it is not connected to `LIMIT`, you usually need just `SORTBY … MAX` for common queries.

* **APPLY {expr} AS {name}**: Apply a one-to-one transformation on one or more properties, and either store the result as a new property down the pipeline, or replace any property using this transformation. `expr` is an expression that can be used to perform arithmetic operations on numeric properties, or functions that can be applied on properties depending on their types (see below), or any combination thereof. For example: `APPLY "sqrt(@foo)/log(@bar) + 5" AS baz` will evaluate this expression dynamically for each record in the pipeline and store the result as a new property called baz, that can be referenced by further APPLY / SORTBY / GROUPBY / REDUCE operations down the pipeline.

* **LIMIT {offset} {num}**. Limit the number of results to return just `num` results starting at index `offset` (zero based). AS mentioned above, it is much more efficient to use `SORTBY … MAX` if you are interested in just limiting the output of a sort operation.

     However, limit can be used to limit results without sorting, or for paging the n-largest results as determined by `SORTBY MAX`. For example, getting results 50-100 of the top 100 results is most efficiently expressed as `SORTBY 1 @foo MAX 100 LIMIT 50 50`. Removing the MAX from SORTBY will result in the pipeline sorting all the records and then paging over results 50-100.

* **FILTER {expr}**. Filter the results using predicate expressions relating to values in each result. The expressions are applied post-query and relate to the current state of the pipeline. See FILTER Expressions below for full details.

* **PARAMS {nargs} {name} {value}**. Define one or more value parameters. Each parameter has a name and a value. Parameters can be referenced in the query string by a `$`, followed by the parameter name, e.g., `$user`, and each such reference in the search query to a parameter name is substituted by the corresponding parameter value. For example, with parameter definition `PARAMS 4 lon 29.69465 lat 34.95126`, the expression `@loc:[$lon $lat 10 km]` would be evaluated to `@loc:[29.69465 34.95126 10 km]`. Parameters cannot be referenced in the query string where concrete values are not allowed, such as in field names, e.g., `@loc`
*

Example

A log of visits to a website might look like the following, each record of which has the following fields/properties:

* **url** (text, sortable)
* **timestamp** (numeric, sortable) - Unix timestamp of visit entry.
* **country** (tag, sortable)
* **user_id** (text, sortable, not indexed)

Example 1: unique users by hour, ordered chronologically.

The first step is to determine the index name and the filtering query. A filter query of `*` means "get all records":

[code example]

Next, group the results by hour. The data contains visit times as unix timestamps in second resolution, so you'll need to extract the hour component of the timestamp. To do so, add an APPLY step that strips the sub-hour information from the timestamp and stores is as a new property, `hour`:

[code example]

Next, group the results by hour and count the distinct user ids in each hour. This is done by a GROUPBY/REDUCE step:

[code example]

Next, sort the results by hour, ascending:

[code example]

And as a final step, format the hour as a human readable timestamp. This is done by calling the transformation function `timefmt` that formats Unix timestamps. You can specify a format to be passed to the system's `strftime` function ([see documentation](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html)), but not specifying one is equivalent to specifying `%FT%TZ` to `strftime`.

[code example]

Example 2: Sort visits to a specific URL by day and country:

The next example filters by the url, transforms the timestamp to its day part, and groups by the day and country, counting the number of visits per group, sorting by day ascending and country descending.

[code example]

GROUPBY reducers

`GROUPBY` works similarly to SQL `GROUP BY` clauses, and creates groups of results based on one or more properties in each record. For each group, Redis returns the group keys, or the values common to all records in the group, and the results of zero or more `REDUCE` clauses.

Each `GROUPBY` step in the pipeline may be accompanied by zero or more `REDUCE` clauses. Reducers apply an accumulation function to each record in the group and reduces them into a single record representing the group. When the processing is complete, all the records upstream of the `GROUPBY` step emit their reduced record.

For example, the simplest reducer is COUNT, which simply counts the number of records in each group.

If multiple `REDUCE` clauses exist for a single `GROUPBY` step, each reducer works independently on each result and writes its final output once. Each reducer may have its own alias determined using the `AS` optional parameter. If `AS` is not specified, the alias is the reduce function and its parameters, e.g. `count_distinct(foo,bar)`.

Supported GROUPBY reducers

#### COUNT

**Format**

[code example]

**Description**

Count the number of records in each group

#### COUNT_DISTINCT

**Format**

[code example]`

**Description**

Count the number of distinct values for `property`.

The reducer creates a hash-set per group, and hashes each record. This can be memory heavy if the groups are big.

#### COUNT_DISTINCTISH

**Format**

[code example]

**Description**

Same as COUNT_DISTINCT, provides an approximation instead of an exact count, which consumes less memory and CPU for big groups.

The reducer uses [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) counters per group, at ~3% error rate, and 1024 bytes of constant space  per group. This means it is ideal for a few huge groups and not ideal for many small groups. In the former case, it can be an order of magnitude faster and consume much less memory than COUNT_DISTINCT, but again, it does not fit every use case.

#### SUM

**Format**

[code example]

**Description**

Return the sum of all numeric values of a given property in a group. Non-numeric values in the group are counted as 0.

#### MIN

**Format**

[code example]

**Description**

Return the minimal value of a property, whether it is a string, number, or NULL.

#### MAX

**Format**

[code example]

**Description**

Return the maximal value of a property, whether it is a string, number or NULL.

#### AVG

**Format**

[code example]

**Description**

Return the average value of a numeric property. This is equivalent to reducing by sum and count, and later, applying the ratio of them as an APPLY step.

#### STDDEV

**Format**

[code example]

**Description**

Return the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of a numeric property in the group.

#### QUANTILE

**Format**

[code example]

**Description**

Return the value of a numeric property at a given quantile of the results. Quantile is expressed as a number between 0 and 1. For example, the median can be expressed as the quantile at 0.5, e.g. `REDUCE QUANTILE 2 @foo 0.5 AS median` .

If multiple quantiles are required, just repeat the QUANTILE reducer for each quantile. For example, `REDUCE QUANTILE 2 @foo 0.5 AS median REDUCE QUANTILE 2 @foo 0.99 AS p99`.

#### TOLIST

**Format**

[code example]

**Description**

Merge all distinct values of a given property into a single array.

#### FIRST_VALUE

**Format**

[code example]

**Description**

Return the first or top value of a given property in the group, optionally by comparing it to another property. For example, you can extract the name of the oldest user in the group:

[code example]

If no `BY` is specified, the first value encountered in the group is returned.

If you wish to get the top or bottom value in the group sorted by the same value, you are better off using the `MIN/MAX` reducers, but the same effect will be achieved by doing `REDUCE FIRST_VALUE 4 @foo BY @foo DESC`.

#### RANDOM_SAMPLE

**Format**

[code example]

**Description**

Perform a reservoir sampling of the group elements with a given size, and return an array of the sampled items with an even distribution.

APPLY expressions

`APPLY` performs a one-to-one transformation on one or more properties in each record. It either stores the result as a new property down the pipeline, or replaces any property using this transformation.

The transformations are expressed as a combination of arithmetic expressions and built in functions. Evaluating functions and expressions is recursively nested and can be composed without limit. For example: `sqrt(log(foo) * floor(@bar/baz)) + (3^@qaz % 6)` or simply `@foo/@bar`.

If an expression or a function is applied to values that do not match the expected types, no error is emitted and a NULL value is set as the result.

APPLY steps must have an explicit alias determined by the `AS` parameter.

Literals inside expressions

* Numbers are expressed as integers or floating point numbers, e.g., `2`, `3.141`, and `-34`. `inf` and `-inf` are acceptable as well.
* Strings are quoted with either single or double quotes. Single quotes are acceptable inside strings quoted with double quotes and vice versa. Punctuation marks can be escaped with backslashes. e.g. `"foo's bar"` ,`'foo\'s bar'`, `"foo \"bar\""` .
* Any literal or sub-expression can be wrapped in parentheses to resolve ambiguities of operator precedence.

Arithmetic operations

For numeric expressions and properties, addition (`+`), subtraction (`-`), multiplication (`*`), division (`/`), modulo (`%`), and power (`^`) are supported. Bitwise logical operators are not supported.

Note that these operators apply only to numeric values and numeric sub-expressions. Any attempt to multiply a string by a number, for instance, will result in a NULL output.

List of field APPLY functions

| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| exists(s)| Checks whether a field exists in a document.                 | `exists(@field)`   |
| case(condition, if_true, if_false) | If condition is non-zero, return if_true, otherwise return if_false. | `case(exists(@foo), @foo, "no foo")` |

List of numeric APPLY functions

| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| log(x)   | Return the logarithm of a number, property or subexpression  | `log(@foo)`        |
| abs(x)   | Return the absolute number of a numeric expression           | `abs(@foo-@bar)`   |
| ceil(x)  | Round to the smallest value not less than x                  | `ceil(@foo/3.14)`  |
| floor(x) | Round to largest value not greater than x                    | `floor(@foo/3.14)` |
| log2(x)  | Return the logarithm of x to base 2                          | `log2(2^@foo)`     |
| exp(x)   | Return the exponent of x, e.g., `e^x`                        | `exp(@foo)`        |
| sqrt(x)  | Return the square root of x                                  | `sqrt(@foo)`       |

List of string APPLY functions

| Function                         |                                                              |                                                          |
| -------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |
| upper(s)                         | Return the uppercase conversion of s                         | `upper('hello world')`                                   |
| lower(s)                         | Return the lowercase conversion of s                         | `lower("HELLO WORLD")`                                   |
| startswith(s1,s2)                | Return `1` if s2 is the prefix of s1, `0` otherwise.         | `startswith(@field, "company")`                          |
| contains(s1,s2)                  | Return the number of occurrences of s2 in s1, `0` otherwise. If s2 is an empty string, return `length(s1) + 1`.  | `contains(@field, "pa")`                                 |
| strlen(s)                        | Return the length of s                                       | `strlen(@t)`                                             |
| substr(s, offset, count)         | Return the substring of s, starting at _offset_ and having _count_ characters.
If offset is negative, it represents the distance from the end of the string.
If count is -1, it means "the rest of the string starting at offset". | `substr("hello", 0, 3)`
 `substr("hello", -2, -1)`  |
| format( fmt, ...)                | Use the arguments following `fmt` to format a string.
Currently the only format argument supported is `%s` and it applies to all types of arguments. | `format("Hello, %s, you are %s years old", @name, @age)` |
| matched_terms([max_terms=100])   | Return the query terms that matched for each record (up to 100), as a list. If a limit is specified, Redis will return the first N matches found, based on query order. | `matched_terms()`                                        |
| split(s, [sep=","], [strip=" "]) | Split a string by any character in the string sep, and strip any characters in strip. If only s is specified, it is split by commas and spaces are stripped. The output is an array. | split("foo,bar")                                         |

List of date/time APPLY functions

| Function            | Description                                                  |
| ------------------- | ------------------------------------------------------------ |
| timefmt(x, [fmt])      | Return a formatted time string based on a numeric timestamp value x.
 See [strftime](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html) for formatting options.
Not specifying `fmt` is equivalent to `%FT%TZ`. |
| parsetime(timesharing, [fmt]) | The opposite of timefmt() - parse a time format using a given format string |
| day(timestamp) | Round a Unix timestamp to midnight (00:00) start of the current day. |
| hour(timestamp) | Round a Unix timestamp to the beginning of the current hour. |
| minute(timestamp) | Round a Unix timestamp to the beginning of the current minute. |
| month(timestamp) | Round a unix timestamp to the beginning of the current month. |
| dayofweek(timestamp) | Convert a Unix timestamp to the day number (Sunday = 0). |
| dayofmonth(timestamp) | Convert a Unix timestamp to the day of month number (1 .. 31). |
| dayofyear(timestamp) | Convert a Unix timestamp to the day of year number (0 .. 365). |
| year(timestamp) | Convert a Unix timestamp to the current year (e.g. 2018). |
| monthofyear(timestamp) | Convert a Unix timestamp to the current month (0 .. 11). |

List of geo APPLY functions

| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| geodistance(field,field)        | Return distance in meters.    | `geodistance(@field1,@field2)`       |
| geodistance(field,"lon,lat")    | Return distance in meters.    | `geodistance(@field,"1.2,-3.4")`     |
| geodistance(field,lon,lat)      | Return distance in meters.    | `geodistance(@field,1.2,-3.4)`       |
| geodistance("lon,lat",field)    | Return distance in meters.    | `geodistance("1.2,-3.4",@field)`     |
| geodistance("lon,lat","lon,lat")| Return distance in meters.    | `geodistance("1.2,-3.4","5.6,-7.8")` |
| geodistance("lon,lat",lon,lat)  | Return distance in meters.    | `geodistance("1.2,-3.4",5.6,-7.8)`   |
| geodistance(lon,lat,field)      | Return distance in meters.    | `geodistance(1.2,-3.4,@field)`       |
| geodistance(lon,lat,"lon,lat")  | Return distance in meters.    | `geodistance(1.2,-3.4,"5.6,-7.8")`   |
| geodistance(lon,lat,lon,lat)    | Return distance in meters.    | `geodistance(1.2,-3.4,5.6,-7.8)`     |

[code example]

To retrieve the distance:

[code example]

Note: the geo field must be preloaded using `LOAD`.

Results can also be sorted by distance:

[code example]

Note: Make sure no location is missing, otherwise the SORTBY will not return any results.
Use FILTER to make sure you do the sorting on all valid locations.

FILTER expressions

FILTER expressions filter the results using predicates relating to values in the result set.

The FILTER expressions are evaluated post-query and relate to the current state of the pipeline. Thus they can be useful to prune the results based on group calculations. Note that the filters are not indexed and will not speed up processing.

Filter expressions follow the syntax of APPLY expressions, with the addition of the conditions `==`, `!=`, `<`, `<=`, `>`, `>=`. Two or more predicates can be combined with logical AND (`&&`) and OR (`||`). A single predicate can be negated with a NOT prefix (`!`).

For example, filtering all results where the user name is 'foo' and the age is less than 20 is expressed as:

[code example]

Several filter steps can be added, although at the same stage in the pipeline, it is more efficient to combine several predicates into a single filter step.

Cursor API

[code example]

You can use cursors with [`FT.AGGREGATE`](), with the `WITHCURSOR` keyword. Cursors allow you to
consume only part of the response, allowing you to fetch additional results as needed.
This is much quicker than using `LIMIT` with offset, since the query is executed only
once, and its state is stored on the server.

To use cursors, specify the `WITHCURSOR` keyword in [`FT.AGGREGATE`](). For example:

[code example]

This will return a response of an array with two elements. The first element is
the actual (partial) result, and the second is the cursor ID. The cursor ID
can then be fed to [`FT.CURSOR READ`]() repeatedly until the cursor ID is 0, in
which case all results have been returned.

To read from an existing cursor, use [`FT.CURSOR READ`](). For example:

[code example]

Assuming `342459320` is the cursor ID returned from the [`FT.AGGREGATE`]() request, here is an example in pseudo-code:

[code example]

Note that even if the cursor is 0, a partial result may still be returned.

Cursor settings

#### Read size

You can control how many rows are read for each cursor fetch by using the
`COUNT` parameter. This parameter can be specified both in [`FT.AGGREGATE`]()
(immediately after `WITHCURSOR`) or in [`FT.CURSOR READ`]().

The following example will read 10 rows at a time:
[code example]

You can override this setting by also specifying `COUNT` in `CURSOR READ`. The following example will return at most 50 results:

[code example]

The default read size is 1000.

#### Timeouts and limits

Because cursors are stateful resources that occupy memory on the server, they
have a limited lifetime. To safeguard against orphaned/stale cursors,
cursors have an idle timeout value. If no activity occurs on the cursor before
the idle timeout, the cursor is deleted. The idle timer resets to 0 whenever
the cursor is read from using `CURSOR READ`.

The default idle timeout is 300000 milliseconds (or 300 seconds). You can modify
the idle timeout using the `MAXIDLE` keyword when creating the cursor. Note that
the value cannot exceed the default 300s.

For example, to set a limit of ten seconds:

[code example]

During a cursor’s lifetime, CURSOR READ may return results that exclude newer updates.

Also, note that in case of any load rebalancing operations (such as Atomic Slot Migration) occurring during the lifetime of a cursor, some results may be missing. In a Redis Software deployment, these operations may happen behind the scenes without any user intervention.

Other cursor commands

Cursors can be explicitly deleted using the `CURSOR DEL` command. For example:

[code example]

Note that cursors are automatically deleted if all their results have been
returned, or if they have timed out.

All idle cursors can be forcefully purged at the same time using `FT.CURSOR GC idx 0` command.
By default, Redis uses a lazy throttled approach to garbage collection, which
collects idle cursors every 500 operations, or every second, whichever is later.

FT.AGGREGATE
  {index_name:string}
  {query_string:string}
  [VERBATIM]
  [LOAD {nargs:integer} {property:string} ...]
  [GROUPBY
    {nargs:integer} {property:string} ...
    REDUCE
      {FUNC:string}
      {nargs:integer} {arg:string} ...
      [AS {name:string}]
    ...
  ] ...
  [SORTBY
    {nargs:integer} {string} ...
    [MAX {num:integer}] ...
  ] ...
  [APPLY
    {EXPR:string}
    AS {name:string}
  ] ...
  [FILTER {EXPR:string}] ...
  [LIMIT {offset:integer} {num:integer} ] ...
  [PARAMS {nargs} {name} {value} ... ]

FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour

  GROUPBY 1 @hour
  	REDUCE COUNT_DISTINCT 1 @user_id AS num_users

  SORTBY 2 @hour ASC

  APPLY timefmt(@hour) AS hour

FT.AGGREGATE myIndex "@url:\"about.html\""
    APPLY "@timestamp - (@timestamp % 86400)" AS day
    GROUPBY 2 @day @country
    	REDUCE count 0 AS num_visits
    SORTBY 4 @day ASC @country DESC

REDUCE FIRST_VALUE {nargs} {property} [BY {property} [ASC|DESC]]

REDUCE RANDOM_SAMPLE {nargs} {property} {sample_size}

FT.AGGREGATE myIdx "*"  LOAD 1 location  APPLY "geodistance(@location,\"-1.1,2.2\")" AS dist

FT.AGGREGATE idx "*" LOAD 1 @location FILTER "exists(@location)" APPLY "geodistance(@location,-117.824722,33.68590)" AS dist SORTBY 2 @dist DESC

FT.AGGREGATE
  ...
  FILTER "@name=='foo' && @age < 20"
  ...

FT.AGGREGATE ... WITHCURSOR [COUNT {read size} MAXIDLE {idle timeout}]
FT.CURSOR READ {idx} {cid} [COUNT {read size}]
FT.CURSOR DEL {idx} {cid}

response, cursor = FT.AGGREGATE "idx" "redis" "WITHCURSOR";
while (1) {
  processResponse(response)
  if (!cursor) {
    break;
  }
  response, cursor = FT.CURSOR read "idx" cursor
}

FT.AGGREGATE idx query WITHCURSOR COUNT 10

FT.AGGREGATE idx query WITHCURSOR MAXIDLE 10000


FT.AGGREGATE order of operations

Summary: Order of operations for the FT.AGGREGATE command

Overview

[`FT.AGGREGATE`]() is a powerful Redis Search command for performing advanced data aggregation, filtering, sorting, and transformations on indexed hash or JSON documents. This reference page provides a structured breakdown of syntax, ordering rules, and best practices.

The [main aggregations page]() has a simple diagram showing how `FT.AGGREGATE` pipelines are constructed, but it doesn't tell the whole story. For example, you can create more complex aggregation pipelines by applying multiple `REDUCE` functions under a single `GROUPBY` clause, or you can chain groupings and mix in additional mapping steps:

`GROUPBY` ... `REDUCE` ... `APPLY` ... `GROUPBY` ... `REDUCE`

The examples on this page are based on a hypothetical "products" data set, which you can [download here](./data/products.txt).

Syntax and expression ordering

The `FT.AGGREGATE` command processes multiple expressions in a pipeline. Below is the recommended order:

1. `index` – the name of your index, which must be the first argument.
1. `query` – your query, which must be the second argument.
1. `FILTER` – filters raw documents before transformations or aggregation.
1. `LOAD` – loads document fields.
1. `APPLY` – applies transformations on fields.
1. `GROUPBY` – groups results by specific fields.
1. `REDUCE` – performs aggregations. For example, `SUM`, `COUNT`, and `AVG`.
1. `SORTBY` – orders the results based on specified fields.
1. `LIMIT` – restricts the number of results returned.
1. `DIALECT 2` - provides for more comprehensive query syntax, for example using parameters in `FILTER` expressions.

Other keywords will be discussed toward the end of this page.

When to use `@`

You must add `@` at the start of a field name in the following circumstances:

- When referencing fields loaded from documents. In the following example, `price` is a document field and must be prefixed with `@`.

[code example]

- When referencing fields inside a `FILTER` clause that were loaded from documents.

[code example]

- When referencing fields inside `GROUPBY` or `REDUCE` clauses.

[code example]

- When referencing fields created by `REDUCE` in `APPLY` or `FILTER` clauses.

[code example]

- When referencing fields created by `APPLY` in another `APPLY` or `FILTER` clause.

[code example]

- When referencing fields created by `APPLY` in a `SORTBY` clause.

[code example]

GROUPBY with multiple REDUCE operations

You can use multiple `REDUCE` operations after `GROUPBY` for different aggregations.

[code example]

Multiple APPLY operations followed by GROUPBY and REDUCE

You can use `APPLY` in various ways before and after `GROUPBY` and `REDUCE`.

[code example]

FILTER and PARAMS

Use `FILTER` to remove unwanted records, and `PARAMS` to pass values to parameterized queries.

[code example]

Placement of FILTER before and after GROUPBY/APPLY

- **Before GROUPBY:** Removes records before aggregation.
- **After GROUPBY:** Filters based on aggregated results.

LOAD after GROUPBY/REDUCE

This is not allowed and you'll get a syntax error.

Placement rules for specific parameters

| Parameter    | Placement                                                 |
|-----         |-----                                                      |
| `TIMEOUT`    | Can be placed anywhere.                                   |
| `LIMIT`      | Must be at the end, before `DIALECT`.                     |
| `WITHCURSOR` | Must be at the end, before `DIALECT`.                     |
| `SCORER`     | Must be placed between the query and pipeline operations. |
| `ADDSCORES`  | Must be placed between the query and pipeline operations. |
| `DIALECT`    | Must be at the end.                                       |

LIMIT and WITHCURSOR used together

While you wouldn't ordinarily use `LIMIT` and `WITHCURSOR` together in the same query, you can do so if necessary.
`LIMIT`, as the name suggests, will limit the total number of results returned for the given query. `WITHCURSOR` will paginate the results in chunks of size `COUNT`. You can use the [cursor API]() to retrieve more results, as shown below.

[code example]

See the following resources for more information:

- [Aggregations]() discussion page.
- [`FT.AGGREGATE` command page](https://redis.io/docs/latest/commands/ft.aggregate/)
- [Redis Search source code](https://github.com/RediSearch/RediSearch/tree/master/src/aggregate)

FT.AGGREGATE products "*"
  LOAD 1 @price
  APPLY "@price * 1.1" AS adjusted_price
  SORTBY 2 @adjusted_price DESC
  LIMIT 0 10

 1) (integer) 200
 2) 1) "price"
    2) "623"
    3) "adjusted_price"
    4) "685.3"
 3) 1) "price"
    2) "619.75"
    3) "adjusted_price"
    4) "681.725"
  .
  .
  .

FT.AGGREGATE products "*"
  LOAD 1 @rating
  FILTER "@rating >= 4.5"
  LIMIT 0 10

 1) (integer) 5
 2) 1) "rating"
    2) "4.5"
 3) 1) "rating"
    2) "4.8"
 4) 1) "rating"
    2) "4.5"
  .
  .
  .

FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE SUM 1 @price AS total_price
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Toys"
   3) "total_price"
   4) "9799.25"
3) 1) "category"
   2) "Electronics"
   3) "total_price"
   4) "10683.5"
4) 1) "category"
   2) "Apparel"
   3) "total_price"
   4) "10273.5"
  .
  .
  .

FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE SUM 1 @price AS total_price
  APPLY "@total_price * 1.2" AS boosted_price
  FILTER "@total_price > 1000"
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Toys"
   3) "total_price"
   4) "9799.25"
   5) "boosted_price"
   6) "11759.1"
3) 1) "category"
   2) "Electronics"
   3) "total_price"
   4) "10683.5"
   5) "boosted_price"
   6) "12820.2"
  .
  .
  .

FT.AGGREGATE products "*"
  LOAD 2 @price @discount
  APPLY "@price - @discount" AS net_price
  APPLY "@net_price * 1.1" AS marked_up
  FILTER "@net_price > 200"
  LIMIT 0 10

1) (integer) 60
2) 1) "price"
   2) "220"
   3) "discount"
   4) "0"
   5) "net_price"
   6) "220"
   7) "marked_up"
   8) "242"
3) 1) "price"
   2) "223.25"
   3) "discount"
   4) "1.5"
   5) "net_price"
   6) "221.75"
   7) "marked_up"
   8) "243.925"
  .
  .
  .

FT.AGGREGATE products "*"
  LOAD 2 @price @discount
  APPLY "@price - @discount" AS net_price
  SORTBY 2 @net_price DESC
  LIMIT 0 10

 1) (integer) 200
 2) 1) "price"
    2) "623"
    3) "discount"
    4) "6"
    5) "net_price"
    6) "617"
 3) 1) "price"
    2) "619.75"
    3) "discount"
    4) "4.5"
    5) "net_price"
    6) "615.25"
  .
  .
  .

FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS product_count
  REDUCE SUM 1 @price AS total_price
  REDUCE AVG 1 @rating AS avg_rating
  SORTBY 2 @total_price DESC
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Groceries"
   3) "product_count"
   4) "44"
   5) "total_price"
   6) "13495.25"
   7) "avg_rating"
   8) "3.94090909091"
3) 1) "category"
   2) "Home"
   3) "product_count"
   4) "40"
   5) "total_price"
   6) "11026.75"
   7) "avg_rating"
   8) "3.78"
  .
  .
  .

FT.AGGREGATE products "*"
  LOAD 3 @price @discount @quantity
  APPLY "@price - @discount" AS final_price
  APPLY "@final_price * @quantity" AS total_revenue
  GROUPBY 1 @category
  REDUCE SUM 1 @total_revenue AS total_category_revenue
  SORTBY 2 @total_category_revenue DESC
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Groceries"
   3) "total_category_revenue"
   4) "81373"
3) 1) "category"
   2) "Home"
   3) "total_category_revenue"
   4) "55797.5"
  .
  .
  .

FT.AGGREGATE products "*"
  LOAD 3 @price @rating @quantity
  FILTER "@price >= 500"
  FILTER "@rating >= 4.0"
  APPLY "@price * @quantity" AS total_value
  SORTBY 2 @total_value DESC
  LIMIT 0 10
  DIALECT 2

1) (integer) 200
 2) 1) "price"
    2) "606.75"
    3) "rating"
    4) "4.2"
    5) "quantity"
    6) "10"
    7) "total_value"
    8) "6067.5"
 3) 1) "price"
    2) "541.75"
    3) "rating"
    4) "4.5"
    5) "quantity"
    6) "10"
    7) "total_value"
    8) "5417.5"
  .
  .
  .

FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS product_count
  LIMIT 0 100
  WITHCURSOR COUNT 3

1) 1) (integer) 6
   2) 1) "category"
      2) "Toys"
      3) "product_count"
      4) "28"
   3) 1) "category"
      2) "Electronics"
      3) "product_count"
      4) "31"
   4) 1) "category"
      2) "Apparel"
      3) "product_count"
      4) "36"
2) (integer) 89400486
127.0.0.1:6379> FT.CURSOR READ products 89400486 COUNT 3
1) 1) (integer) 0
   2) 1) "category"
      2) "Home"
      3) "product_count"
      4) "40"
   3) 1) "category"
      2) "Groceries"
      3) "product_count"
      4) "44"
   4) 1) "category"
      2) "Books"
      3) "product_count"
      4) "21"
2) (integer) 89400486
  .
  .
  .


Autocomplete with Redis

Summary: Learn how to use the autocomplete feature of Redis for efficient prefix-based suggestion retrieval.

Overview

Redis Search provides an autocomplete feature using suggestions that are stored in a [trie-based](https://en.wikipedia.org/wiki/Trie) data structure.
This feature allows you to store and retrieve ranked suggestions based on user input prefixes, making it useful for applications like search boxes, command completion, and chatbot responses.

This guide covers how to use the [`FT.SUGADD`](), [`FT.SUGGET`](), [`FT.SUGDEL`](), and [`FT.SUGLEN`]() commands to implement autocomplete, and some examples of how you can use these commands with [`FT.SEARCH`]().

Add autocomplete suggestions

To add phrases or words to a suggestions dictionary, use the [`FT.SUGADD`]() command.
You will assign a score to each entry, which determines its ranking in the results.

[code example]

Integer scores were used in the above examples, but the scores argument is described as being floating point.
Integer scores are converted to floating point internally.
Also, "`autocomplete`" in the above examples is just the name of the key; you can use any key name you wish.

Optional arguments

The `FT.SUGADD` command can take two optional arguments:

* `INCR`: increments the existing entry of the suggestion by the given score instead of replacing the score. This is useful for updating the dictionary based on user queries in real time.
* `PAYLOAD`: saves a string with the suggestion, which can be fetched by adding the `WITHPAYLOADS` argument to `FT.SUGGET`.

Retrieve suggestions

To get autocomplete suggestions for a given prefix, use the [`FT.SUGGET`]() command.

[code example]

If you wish to see the scores, use the `WITHSCORES` option:

[code example]

Enable fuzzy matching

If you want to allow for small spelling mistakes or typos, use the `FUZZY` option. This option performs a fuzzy prefix search, including prefixes at a [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) of 1 from the provided prefix.

[code example]

Optional arguments

There are three additional arguments you can use with `FT.SUGGET`:

* `MAX num`: limits the results to a maximum of `num`. The default for `MAX` is 5.
* `WITHSCORES`: returns the score of each suggestion.
* `WITHPAYLOADS`: returns optional payloads saved with the suggestions. If no payload is present for an entry, a `nil` reply is returned.
    [code example]

Delete suggestions

To remove a specific suggestion from the dictionary, use the `FT.SUGDEL` command.

[code example]

After deletion, running `FT.SUGGET autocomplete hell FUZZY` will no longer return "help me".

Check the number of suggestions

To get a count of the number of entries in a given suggestions dictionary, use the `FT.SUGLEN` command.

[code example]

Use autocomplete with search

A common approach is to:

1. Use FT.SUGGET to suggest query completions as users type in a text field.
1. Once the user selects a suggestion, run FT.SEARCH using the selected term to get full search results.

Example workflow

1. Get suggestions for a given user input.

    [code example]
1. Capture the user's selection.
1. Use the selected suggestion in a full-text search.

    [code example]

When to use autocomplete versus full-text search

* Use `FT.SUGGET` when you need fast, real-time prefix-based suggestion retrieval.
* Use `FT.SEARCH` when you need document retrieval, filtering, and ranking based on relevance.

Autocomplete use cases

The autocomplete feature in Redis Search is useful for:

- **Search box suggestions**: providing live suggestions as users type.
- **Command completion**: offering autocompletion for CLI tools.
- **Product search**: suggesting product names in e-commerce applications.
- **Chatbot responses**: recommending common phrases dynamically.

FT.SUGADD autocomplete "hello world" 100
FT.SUGADD autocomplete "hello there" 90
FT.SUGADD autocomplete "help me" 80
FT.SUGADD autocomplete "hero" 70

redis> FT.SUGGET autocomplete "he"
1) "hero"
2) "help me"
3) "hello world"
4) "hello there"

redis> FT.SUGGET autocomplete hell FUZZY
1) "hello world"
2) "hello there"
3) "help me"

redis> FT.SUGADD autocomplete hero 70 PAYLOAD "you're no hero"
    (integer) 4
    redis> FT.SUGGET autocomplete "hr" FUZZY WITHPAYLOADS
    1) "hero"
    2) "you're no hero"
    3) "help me"
    4) (nil)
    5) "hello world"
    6) (nil)
    7) "hello there"
    8) (nil)

redis> FT.SUGDEL autocomplete "help me"
(integer) 1

redis> FT.SUGLEN autocomplete
(integer) 3


Chinese support

Summary: Chinese support for searching and querying in Redis Open Source

Overview

Support for adding documents in Chinese is available starting at version 0.99.0.

Chinese support allows Chinese documents to be added and tokenized using segmentation
rather than simple tokenization using whitespace and/or punctuation.

Indexing a Chinese document is different than indexing a document in most other
languages because of how tokens are extracted. While most languages can have
their tokens distinguished by separation characters and whitespace, this
is not common in Chinese.

Chinese tokenization is done by scanning the input text and checking every
character or sequence of characters against a dictionary of predefined terms,
and determining the most likely match based on the surrounding terms and characters.

Redis makes use of the [Friso](https://github.com/lionsoul2014/friso)
Chinese tokenization library for this purpose. This is largely transparent to
the user and often no additional configuration is required.

Example: using chinese in queries

In pseudo-code:

[code example]

Using the Python client:

[code example]

Using custom dictionaries

If you wish to use a custom dictionary, you can do so at the module level when
loading the module. The `FRISOINI` setting can point to the location of a
`friso.ini` file that contains the relevant settings and paths to the dictionary
files.

Note that there is no default `friso.ini` file location. RediSearch comes with
its own `friso.ini` and dictionary files, which are compiled into the module
binary at build-time.

FT.CREATE idx ON HASH SCHEMA txt TEXT
HSET docCn txt "Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步，从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]"
FT.SEARCH idx "数据" LANGUAGE chinese HIGHLIGHT SUMMARIZE
# Outputs:
# 数据?... 数据进行写操作。由于完全实现了发布... 数据冗余很有帮助。[8...

# -*- coding: utf-8 -*-

from redisearch.client import Client, Query
from redisearch import TextField

client = Client('idx')
try:
    client.drop_index()
except:
    pass

client.create_index([TextField('txt')])

# Add a document
client.add_document('docCn1',
                    txt='Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]',
                    language='chinese')
print client.search(Query('数据').summarize().highlight().language('chinese')).docs[0].txt
# Outputs:
# 数据?... 数据进行写操作。由于完全实现了发布... 数据冗余很有帮助。[8...


Query dialects

Summary: Learn how to use query dialects

Overview

Redis Open Source currently supports four query dialects for use with the [`FT.SEARCH`](), [`FT.AGGREGATE`](), and other Redis Search commands.
Dialects provide for enhancing the query API incrementally, introducing innovative behaviors and new features that support new use cases in a way that does not break the API for existing applications.

Dialects 1, 3, and 4 are deprecated in Redis 8 in Redis Open Source. However, DIALECT 1 remains the default.

`DIALECT 1` (Deprecated)

Dialect version 1 was the default query syntax dialect from the first release of search and query until dialect version 2 was introduced with version [2.4](https://github.com/RediSearch/RediSearch/releases/tag/v2.4.3).
This dialect is also the default dialect. See below for information about changing the default dialect.

`DIALECT 2`

Dialect version 2 was introduced in the [2.4](https://github.com/RediSearch/RediSearch/releases/tag/v2.4.3) release to address query parser inconsistencies found in previous versions of Redis. Dialect version 1 remains the default dialect. To use dialect version 2, append `DIALECT 2` to your query command.
Support for vector search also was introduced in the 2.4 release and requires `DIALECT 2`. See [here]() for more details.
`FT.SEARCH ... DIALECT 2`

It was determined that under certain conditions some query parsing rules did not behave as originally intended.
Particularly, some queries containing the operators below could return unexpected results.

1. AND, multi-word phrases that imply intersection
1. `"..."` (exact), `~` (optional), `-` (negation), and `%` (fuzzy)
1. OR, words separated by the `|` (pipe) character that imply union
1. wildcard characters

Existing queries that used dialect 1 may behave differently using dialect 2 if they fall into any of the following categories:

1. Your query has a field modifier followed by multiple words. Consider the sample query:

    `@name:James Brown`

    Here, the field modifier `@name` is followed by two words, `James` and `Brown`.

    In `DIALECT 1`, this query would be interpreted as find `James Brown` in the `@name` field.
    In `DIALECT 2`, this query would be interpreted as find `James` in the `@name` field, and `Brown` in any text field. In other words, it would be interpreted as `(@name:James) Brown`.
    In `DIALECT 2`, you could achieve the dialect 1 behavior by updating your query to `@name:(James Brown)`.

1. Your query uses `"..."`, `~`, `-`, and/or `%`. Consider a simple query with negation:

    `-hello world`

    In `DIALECT 1`, this query is interpreted as find values in any field that do not contain `hello` and do not contain `world`; the equivalent of `-(hello world)` or `-hello -world`.
    In `DIALECT 2`, this query is interpreted as `-hello` and `world` (only `hello` is negated).
    In `DIALECT 2`, you could achieve the dialect 1 behavior by updating your query to `-(hello world)`.

1. Your query used `|`. Consider the simple query:

    `hello world | "goodbye" moon`

    In `DIALECT 1`, this query is interpreted as searching for `(hello world | "goodbye") moon`.
    In `DIALECT 2`, this query is interpreted as searching for either `hello world` `"goodbye" moon`.

1. Your query uses a wildcard pattern. Consider the simple query:

    `"w'foo*bar?'"`

    As shown above, you must use double quotes to contain the `w` pattern.

With `DIALECT 2` you can use un-escaped spaces in tag queries, even with stopwords.

`DIALECT 2` is required with vector searches.

`DIALECT 2` functionality was enhanced in the 2.10 release.
It introduces support for new comparison operators for `NUMERIC` fields:

* `==` (equal).

  `FT.SEARCH idx "@numeric==3456" DIALECT 2`

  and

  `FT.SEARCH idx "@numeric:[3456]" DIALECT 2`
* `!=` (not equal).

  `FT.SEARCH idx "@numeric!=3456" DIALECT 2`
* `>` (greater than).

  `FT.SEARCH idx "@numeric>3456" DIALECT 2`
* `>=` (greater than or equal).

  `FT.SEARCH idx "@numeric>=3456" DIALECT 2`
* `<` (less than).

  `FT.SEARCH idx "@numeric<3456" DIALECT 2`
* `<=` (less than or equal).

  `FT.SEARCH idx "@numeric<=3456" DIALECT 2`

The Dialect version 2 enhancements also introduce simplified syntax for logical operations:

* `|` (or).

  `FT.SEARCH idx "@tag:{3d3586fe-0416-4572-8ce1 | 3d3586fe-0416-6758-4ri8}" DIALECT 2`

  which is equivalent to

  `FT.SEARCH idx "(@tag:{3d3586fe-0416-4572-8ce1} | @tag{3d3586fe-0416-6758-4ri8})" DIALECT 2`

* `<space>` (and).

  `FT.SEARCH idx "(@tag:{3d3586fe-0416-4572-8ce1} @tag{3d3586fe-0416-6758-4ri8})" DIALECT 2`

* `-` (negation).

  `FT.SEARCH idx "(@tag:{3d3586fe-0416-4572-8ce1} -@tag{3d3586fe-0416-6758-4ri8})" DIALECT 2`

* `~` (optional/proximity).

  `FT.SEARCH idx "(@tag:{3d3586fe-0416-4572-8ce1} ~@tag{3d3586fe-0416-6758-4ri8})" DIALECT 2`

`DIALECT 3` (Deprecated)

Dialect version 3 was introduced in the [2.6](https://github.com/RediSearch/RediSearch/releases/tag/v2.6.3) release. This version introduced support for multi-value indexing and querying of attributes for any attribute type ( [TEXT](), [TAG](), [NUMERIC](), [GEO]() and [VECTOR]()) defined by a [JSONPath]() leading to an array or multiple scalar values. Support for [GEOSHAPE]() queries was also introduced in this version.

The primary difference between dialects version 2 and version 3 is that JSON is returned rather than scalars for multi-value attributes. Apart from specifying `DIALECT 3` at the end of a [`FT.SEARCH`]() command, there are no other syntactic changes. Dialect version 1 remains the default dialect. To use dialect version 3, append `DIALECT 3` to your query command.

`FT.SEARCH ... DIALECT 3`

**Example**

Sample JSON:

[code example]

Create an index:

[code example]

Now search, with and without `DIALECT 3`.

- With dialect 1 (default):

    [code example]

    Only the first element of the expected two elements is returned.

- With dialect 3:

    [code example]

    Both elements are returned.

`DIALECT 4` (Deprecated)

Dialect version 4 was introduced in the [2.8](https://github.com/RediSearch/RediSearch/releases/tag/v2.8.4) release. It introduces performance optimizations for sorting operations on [`FT.SEARCH`]() and [`FT.AGGREGATE`](). Apart from specifying `DIALECT 4` at the end of a [`FT.SEARCH`]() command, there are no other syntactic changes. Dialect version 1 remains the default dialect. To use dialect version 4, append `DIALECT 4` to your query command.

`FT.SEARCH ... DIALECT 4`

Dialect version 4 will improve performance in four different scenarios:

1. **Skip sorter** - applied when there is no sorting to be done. The query can return once it reaches the `LIMIT` of requested results.
1. **Partial range** - applied when there is a `SORTBY` on a numeric field, either with no filter or with a filter by the same numeric field. Such queries will iterate on a range large enough to satisfy the `LIMIT` of requested results.
1. **Hybrid** - applied when there is a `SORTBY` on a numeric field in addition to another non-numeric filter. It could be the case that some results will get filtered, leaving too small a range to satisfy any specified `LIMIT`. In such cases, the iterator then is re-wound and additional iterations occur to collect result up to the requested `LIMIT`.
1. **No optimization** - If there is a sort by score or by a non-numeric field, there is no other option but to retrieve all results and compare their values to the search parameters.

You can also use `WITHOUTCOUNT` in place of `DIALECT 4` when used with either FT.SEARCH or FT.AGGREGATE.

Use `FT.EXPLAINCLI` to compare dialects

The [`FT.EXPLAINCLI`]() command is a powerful tool that provides a window into the inner workings of your queries. It's like a roadmap that details your query's journey from start to finish.

When you run [`FT.EXPLAINCLI`](), it returns an array representing the execution plan of a complex query. This plan is a step-by-step guide of how Redis interprets your query and how it plans to fetch results. It's a behind-the-scenes look at the process, giving you insights into how the search engine works.

The [`FT.EXPLAINCLI`]() accepts a `DIALECT` argument, allowing you to execute the query using different dialect versions, allowing you to compare the resulting query plans.

To use [`FT.EXPLAINCLI`](), you need to provide an index and a query predicate. The index is the name of the index you created using [`FT.CREATE`](), and the query predicate is the same as if you were sending it to [`FT.SEARCH`]() or [`FT.AGGREGATE`]().

Here's an example of how to use [`FT.EXPLAINCLI`]() to understand differences in dialect versions 1 and 2.

Negation of the intersection between tokens `hello` and `world`:

[code example]

Intersection of the negation of the token `hello` together with token `world`:

[code example]

Same result as `DIALECT 1`:

[code example]

[`FT.EXPLAIN`]() doesn't execute the query. It only explains the plan. It's a way to understand how your query is interpreted by Redis Search, which can be invaluable when you're trying to optimize your searches.

Change the default dialect

The default dialect is `DIALECT 1`. If you wish to change that, you can do so by using the `DEFAULT_DIALECT` parameter when loading the RediSearch module:

[code example]

You can also change the query dialect on an already running server using the `FT.CONFIG` command:

[code example]

FT.CREATE js_idx ON JSON PREFIX 1 js: SCHEMA $.underlyings[*].underlier AS und TAG

ft.search js_idx * return 1 und
      1) (integer) 1
      2) "js:1"
      3) 1) "und"
         2) "AAPL UW"

ft.search js_idx * return 1 und DIALECT 3
        1) (integer) 1
        2) "js:1"
        3) 1) "und"
           2) "[\"AAPL UW\",\"NFLX UW\"]"

FT.EXPLAINCLI idx:dialects "-hello world" DIALECT 1

FT.EXPLAINCLI idx:dialects "-hello world" DIALECT 2
 1) INTERSECT {
 2)   NOT {
 3)     hello
 4)   }
 5)   UNION {
 6)     world
 7)     +world(expanded)
 8)   }
 9) }
10)

FT.EXPLAINCLI idx:dialects "-(hello world)" DIALECT 2
1) NOT {
2)   INTERSECT {
3)     hello
4)     world
5)   }
6) }
7)

$ redis-server --loadmodule ./redisearch.so DEFAULT_DIALECT 2


Tokenization

Summary: Controlling text tokenization and escaping

Overview

Full-text search works by comparing words, URLs, numbers, and other elements of the query
against the text in the searchable fields of each document. However,
it would be very inefficient to compare the entire text of the query against the
entire text of each field over and over again, so the search system doesn't do this.
Instead, it splits the document text into short, significant sections
called *tokens* during the indexing process and stores the tokens as part of the document's
index data.

During a search, the query system also tokenizes the
query text and then simply compares the tokens from the query against the tokens stored
for each document. Finding a match like this is much more efficient than pattern-matching on
the whole text and also lets you use
[stemming]() and
[stop words]()
to improve the search even further. See this article about
[Tokenization](https://queryunderstanding.com/tokenization-c8cdd6aef7ff)
for a general introduction to the concepts.

Redis uses a very simple tokenizer for documents and a slightly more sophisticated tokenizer for queries. Both allow a degree of control over string escaping and tokenization.

The sections below describe the rules for tokenizing text fields and queries.
Note that
[Tag fields]()
are essentially text fields but they use a simpler form of tokenization, as described
separately in the
[Tokenization rules for tag fields](#tokenization-rules-for-tag-fields) section.

Tokenization rules for text fields

1. All punctuation marks and whitespace (besides underscores) separate the document and queries into tokens. For example, any character of `,.<>{}[]"':;!@#$%^&*()-+=~` will break the text into terms, so the text `foo-bar.baz...bag` will be tokenized into `[foo, bar, baz, bag]`

2. Escaping separators in both queries and documents is done by prepending a backslash to any separator. For example, the text `hello\-world hello-world` will be tokenized as `[hello-world, hello, world]`. In most languages you will need an extra backslash to signify an actual backslash when formatting the document or query, so the actual text entered into redis-cli will be `hello\\-world`.

3. Underscores (`_`) are not used as separators in either document or query, so the text `hello_world` will remain as is after tokenization.

4. Repeating spaces or punctuation marks are stripped.

5. Latin characters are converted to lowercase.

6. A backslash before the first digit will tokenize it as a term. This will translate the `-` sign as NOT, which otherwise would make the number negative. Add a backslash before `.` if you are searching for a float. For example, `-20 -> {-20} vs -\20 -> {NOT{20}}`.

Tokenization rules for tag fields

[Tag fields]() interpret
a text field as a list of *tags* delimited by a
[separator]()
character (which is a comma "," by
default). The tokenizer simply splits the text wherever it finds the separator and so most
punctuation marks and whitespace are valid characters within each tag token. The only
changes that the tokenizer makes to the tags are:

-   Trimming whitespace at the start and end of the tag. Other whitespace in the tag text is left intact.
-   Converting Latin alphabet characters to lowercase. You can override this by adding the
    [`CASESENSITIVE`]() option in the indexing schema for the tag field.

This means that when you define a tag field, you don't need to escape any characters, except
in the unusual case where you want leading or trailing spaces to be part of the tag text.
However, you do need to escape certain characters in a *query* against a tag field. See the
[Query syntax]() and
[Exact match]() pages for more information about escaping
and how to use [DIALECT 2](), which is required for
exact match queries involving tags.