title: Internal design

summary: Details about design choices and implementations

title: Overview

text:
Redis Open Source implements inverted indexes on top of Redis, but unlike previous implementations of Redis inverted indexes, it uses a custom data encoding that allows more memory and CPU efficient searches, and more advanced search features.

This document details some of the design choices and how these features are implemented.

title: Intro: Redis String DMA

text:
The main feature that this module takes advantage of is Redis Modules Strings Direct Memory Access (DMA).

This feature is simple, yet very powerful. It allows modules to allocate data on Redis string keys, and then get direct pointers to the data allocated by the keys without copying or serializing.

This allows very fast access to huge amounts of memory. From the module's perspective, the string value is exposed simply as `char *`, meaning it can be cast to any data structure.

You simply call `RedisModule_StringTruncate` to resize a memory chunk to the size needed. Then you call `RedisModule_StringDMA` to get direct access to the memory in that key. See [https://github.com/RedisLabs/RedisModulesSDK/blob/master/FUNCTIONS.md#redismodule_stringdma](https://github.com/RedisLabs/RedisModulesSDK/blob/master/FUNCTIONS.md#redismodule_stringdma)

This API is used in the module mainly to encode inverted indexes, and also for other auxiliary data structures.

A generic "Buffer" implementation using DMA strings can be found in [buffer.c](https://github.com/RediSearch/RediSearch/blob/master/src/buffer.c). It automatically resizes the Redis string it uses as raw memory when the capacity needs to grow.

title: Inverted index encoding

text:
An [inverted index](https://en.wikipedia.org/wiki/Inverted_index) is the data structure at the heart of all search engines. The idea is simple. For each word or search term, a list of all the documents it appears in is kept. Other data is kept as well, such as term frequency, and the offsets where a term appeared in the document. Offsets are used for exact match type searches, or for ranking of results.

When a search is performed, either a single index is traversed, or the intersection or union of two or more indexes is traversed. Classic Redis implementations of search engines use sorted sets as inverted indexes. This works but has significant memory overhead, and it also does not allow for encoding of offsets, as explained above.

Redis Open Source uses string DMA (see above) to efficiently encode inverted indexes. It combines [delta encoding](https://en.wikipedia.org/wiki/Delta_encoding) and [varint encoding](https://developers.google.com/protocol-buffers/docs/encoding#varints) to encode entries, minimizing space used for indexes, while keeping decompression and traversal efficient.

For each hit (document/word entry), the following items are encoded:

* The document ID as a delta from the previous document.
* The term frequency, factored by the document's rank (see below).
* Flags, that can be used to filter only specific fields or other user-defined properties.
* An offset vector of all the document offsets of the word.

Document IDs as entered by the user are converted to internal incremental document IDs, that allow delta encoding to be efficient and let the inverted indexes be sorted by document ID.

This allows for a single index hit entry to be encoded in as little as 6 bytes. Note: this is the best case. Depending on the number of occurrences of the word in the document, this can get much higher.

To optimize searches, two additional auxiliary data structures are kept in different DMA string keys:

1. **Skip index**: a table of the index offset of 1/50th of the index entries. This allows faster lookup when intersecting inverted indexes, as the entire list doesn't need to be traversed.
2. **Score index**: In simple single-word searches, there is no real need to traverse all the results, just the top N results the user is interested in. So an auxiliary index of the top 20 or so entries is stored for each term, which are used when applicable.

title: Document and result ranking

text:
Each document entered to the engine has a [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) scoring of each word to rank the results.

As an optimization, each inverted index hit is encoded with `TF * Document_rank` as its score, and only IDF is applied during searches. This may change in the future.

On top of that, in the case of intersection queries, the minimal distance between the terms in the query is factored into the ranking. The closest the terms are to each other, the better the result.

When searching, priority queue of the top N results requested is maintained, which are eventually returned, sorted by rank.

title: Index ppecs and field weights

text:
When creating an "index" using [`FT.CREATE`](), the user specifies the fields to be indexed and their respective weights. This can be used to give some document fields, like a title, more weight in ranking results.

For example:

[code example]

will create an index on fields named title, body, and url, with scores of 10, 1, and 2 respectively.

When documents are indexed, the weights are taken from the saved *index Spec* that is stored in a special Redis key, and only fields that appear in this spec are indexed.

title: Query execution engine

text:
A chained-iterator based approach is used as part of query execution, which is similar to [Python generators](https://wiki.python.org/moin/Generators) in concept.

Iterators that yield index hits are chained together. Those can be:

1. **Read Iterators**, reading hits one by one from an inverted index. For example, `hello`.
2. **Intersect Iterators**, aggregating two or more iterators, yielding only their intersection points. For example, `hello AND world`.
3. **Exact Intersect Iterators** - same as above, but yielding results only if the intersection is an exact phrase. For example,  `hello NEAR world`.
4. **Union Iterators** - combining two or more iterators, and yielding a union of their hits. For example,  `hello OR world`.

These are combined based on the query as an execution plan that is evaluated lazily. For example:

[code example]

All these iterators are lazily evaluated, entry by entry, with constant memory overhead.

The root iterator is read by the query execution engine and filtered for the top N results contained in it.

title: Numeric filters

text:
It's possible to define a field in the index schema as `NUMERIC`, meaning you will be able to limit search results only to those where the given value falls within a specific range. Filtering is done by adding `FILTER` predicates (more than one is supported) to your query. For example:

[code example]

The filter syntax follows the ZRANGEBYSCORE semantics of Redis, meaning `-inf` and `+inf` are supported, and prepending `(` to a number means an exclusive range.

As of release 0.6, the implementation uses a multi-level range tree, saving ranges at multiple resolutions to allow efficient range scanning. Adding numeric filters can accelerate slow queries if the numeric range is small relative to the entire span of the filtered field. For example, a filter on dates focusing on a few days out of years of data can speed a heavy query by an order of magnitude.

title: Auto-complete and fuzzy suggestions

text:
Another important feature for searching and querying is auto-completion or suggestion. It allows you to create dictionaries of weighted terms, and then query them for completion suggestions to a given user prefix.  For example, if you put the term "lcd tv" into a dictionary, sending the prefix "lc" will return it as a result. The dictionary is modeled as a compressed trie (prefix tree) with weights, that is traversed to find the top suffixes of a prefix.

Redis Open Source also allows for fuzzy suggestions, meaning you can get suggestions to user prefixes even if the user has a typo in the prefix. This is enabled using a Levenshtein automaton, allowing efficient searching of a dictionary for all terms within a maximal Levenshtein distance of a term or prefix. Suggestions are weighted based on both their original score and their distance from a prefix typed by the user. Only suggestions where the prefix is up to one Levenshtein distance away from the typed prefix are supported for performance reasons.

However, since searching for fuzzy prefixes, especially very short ones, will traverse an enormous amount of suggestions (in fact, fuzzy suggestions for any single letter will traverse the entire dictionary!), it is recommended that you use this feature carefully, and only when considering the performance penalty it incurs. Since Redis is single threaded, blocking it for any amount of time means no other queries can be processed at that time.

To support unicode fuzzy matching, 16-bit runes are used inside the trie and not bytes. This increases memory consumption if the text is purely ASCII, but allows completion with the same level of support to all modern languages. This is done in the following manner:

1. Assume all input to `FT.SUG*` commands is valid UTF-8.
2. Input strings are converted to 32-bit Unicode, optionally normalizing, case-folding, and removing accents on the way. If the conversion fails it's because the input is not valid UTF-8.
3. The 32-bit runes are trimmed to 16-bit runes using the lower 16 bits. These can be used for insertion, deletion, and search.
4. The output of searches is converted back to UTF-8.

code: FT.CREATE my_index title 10.0 body 1.0 url 2.0

section_id: index-ppecs-and-field-weights

code:
hello ==> read("hello")

hello world ==> intersect( read("hello"), read("world") )

"hello world" ==> exact_intersect( read("hello"), read("world") )

"hello world" foo ==> intersect(
                            exact_intersect(
                                read("hello"),
                                read("world")
                            ),
                            read("foo")
                      )

section_id: query-execution-engine

code: FT.SEARCH products "hd tv" FILTER price 100 (300

section_id: numeric-filters


title: Extend existing search and query features

summary: Details about extensions for query expanders and scoring functions

page_type: content

title: Overview

role: overview

text:
RediSearch supports an extension mechanism, much like Redis supports modules. The API is very minimal at the moment, and it does not yet support dynamic loading of extensions on a running server. Instead, extensions must be written in C (or a language that has an interface with C) and compiled into dynamic libraries that can be loaded at start up.

There are two kinds of extension APIs at the moment:

1. **Query expanders**, whose role is to expand query tokens (i.e., stemmers).
2. **Scoring functions**, whose role is to rank search results at query time.

title: Registering and loading extensions

role: content

text: Extensions should be compiled into dynamic library files (e.g., `.so` files), and loaded into the RediSearch module during initialization.

title: Compiling

text:
Extensions should be compiled and linked as dynamic libraries. An example Makefile for an extension [can be found here](https://github.com/RediSearch/RediSearch/blob/master/tests/ctests/ext-example/Makefile).

    That folder also contains an example extension that is used for testing and can be taken as a skeleton for implementing your own extension.

title: Loading

text:
Loading an extension is done by appending `EXTLOAD {path/to/ext.so}` after the `loadmodule` configuration directive when loading the RediSearch module. For example:

    [code example]

    This causes the RediSearch module to automatically load the extension and register its expanders and scorers.

title: Initializing an extension

text:
The entry point of an extension is a function with the signature:

[code example]

When loading an extension, RediSearch looks for this function and calls it. This function is responsible for registering and initializing the expanders and scorers.

It should return REDISEARCH_ERR on error or REDISEARCH_OK on success.

title: Example init function

role: example

text: [code example]

title: Calling your custom functions

text:
When performing a query, you can use your scorers or expanders by specifying the SCORER or EXPANDER arguments with the given alias. For example:

[code example]

**NOTE**: Expander and scorer aliases are **case sensitive**.

title: The query expander API

text:
Only basic query expansion is supported, one token at a time. An expander can decide to expand any given token with as many tokens it wishes, which will be union-merged in query time.

The API for an expander is the following:

[code example]

title: RSQueryExpanderCtx

text:
`RSQueryExpanderCtx` is a context that contains private data of the extension, and a callback method to expand the query. It is defined as:

[code example]

title: RSToken

text:
`RSToken` represents a single query token to be expanded, and is defined as:

[code example]

title: The scoring function API

text:
For the final ranking, the scoring function analyzes each document retrieved by the query, taking into account not only the terms that triggered the document's retrieval but also metadata like its prior score, length, and so on.

Since the scoring function is evaluated for each document, potentially millions of times, and since
redis is single threaded, it is important that it works as fast as possible and be heavily optimized.

A scoring function is applied to each potential result for each document and is implemented with the following signature:

[code example]

`RSScoringFunctionCtx` is a context that implements some helper methods.

`RSIndexResult` is the result information containing the document id, frequency, terms, and offsets.

`RSDocumentMetadata` is an object holding global information about the document, such as its presumptive score.

`minScore` is the minimal score that will yield a result that is relevant to the search. It can be used to stop processing midway or before or even before it starts.

The return value of the function is a `double` representing the final score of the result.
Returning 0 causes the result to be counted, but if there are results with a score greater than 0, they will appear above it.
To completely filter out a result and not count it in the totals, the scorer should return the special value `RS_SCORE_FILTEROUT`, which is internally set to negative infinity, or -1/0.

title: RSScoringFunctionCtx

text:
This is an object containing the following members:

* `void *privdata`: a pointer to an object set by the extension on initialization time.
* `RSPayload payload*`: A Payload object set either by the query expander or the client.
* `int GetSlop(RSIndexResult *res)*`: A callback method that yields the total minimal distance between the query terms. This can be used to prefer results where the slop is smaller and the terms are nearer to each other.

title: RSIndexResult

text: This is an object holding the information about the current result in the index, which is an aggregate of all the terms that resulted in the current document being considered a valid result. See `redisearch.h` for details.

title: RSDocumentMetadata

text: This is an object describing global information, unrelated to the current query, about the document being evaluated by the scoring function.

title: Example query expander

text:
This example query expander expands each token with the term foo:

[code example]

title: Example scoring function

text:
This is an actual scoring function, which calculates TF-IDF for the document, multiplies it by the document score, and divides it by the slop:

[code example]

code: $ redis-server --loadmodule ./redisearch.so EXTLOAD ./ext/my_extension.so

section_id: loading

section_id: initializing-an-extension

code:
#include <redisearch.h> //must be in the include path

int RS_ExtensionInit(RSExtensionCtx *ctx) {

  /* Register  a scoring function with an alias my_scorer and no special private data and free function */
  if (ctx->RegisterScoringFunction("my_scorer", MyCustomScorer, NULL, NULL) == REDISEARCH_ERR) {
    return REDISEARCH_ERR;
  }

  /* Register a query expander  */
  if (ctx->RegisterQueryExpander("my_expander", MyExpander, NULL, NULL) ==
      REDISEARCH_ERR) {
    return REDISEARCH_ERR;
  }

  return REDISEARCH_OK;
}

section_id: example-init-function

code: FT.SEARCH my_index "foo bar" EXPANDER my_expander SCORER my_scorer

section_id: calling-your-custom-functions

code:
#include <redisearch.h> //must be in the include path

void MyQueryExpander(RSQueryExpanderCtx *ctx, RSToken *token) {
    ...
}

section_id: the-query-expander-api

code:
typedef struct RSQueryExpanderCtx {

  /* Opaque query object used internally by the engine, and should not be accessed */
  struct RSQuery *query;

  /* Opaque query node object used internally by the engine, and should not be accessed */
  struct RSQueryNode **currentNode;

  /* Private data of the extension, set on extension initialization */
  void *privdata;

  /* The language of the query, defaults to "english" */
  const char *language;

  /* ExpandToken allows the user to add an expansion of the token in the query, that will be
   * union-merged with the given token in query time. str is the expanded string, len is its length,
   * and flags is a 32 bit flag mask that can be used by the extension to set private information on
   * the token */
  void (*ExpandToken)(struct RSQueryExpanderCtx *ctx, const char *str, size_t len,
                      RSTokenFlags flags);

  /* SetPayload allows the query expander to set GLOBAL payload on the query (not unique per token)
   */
  void (*SetPayload)(struct RSQueryExpanderCtx *ctx, RSPayload payload);

} RSQueryExpanderCtx;

section_id: rsqueryexpanderctx

code:
/* A token in the query. The expanders receive query tokens and can expand the query with more query
 * tokens */
typedef struct {
  /* The token string - which may or may not be NULL terminated */
  const char *str;
  /* The token length */
  size_t len;

  /* 1 if the token is the result of query expansion */
  uint8_t expanded:1;

  /* Extension specific token flags that can be examined later by the scoring function */
  RSTokenFlags flags;
} RSToken;

section_id: rstoken

code:
double MyScoringFunction(RSScoringFunctionCtx *ctx, RSIndexResult *res,
                                    RSDocumentMetadata *dmd, double minScore);

section_id: the-scoring-function-api

code:
#include <redisearch.h> //must be in the include path

void DummyExpander(RSQueryExpanderCtx *ctx, RSToken *token) {
    ctx->ExpandToken(ctx, strdup("foo"), strlen("foo"), 0x1337);
}

section_id: example-query-expander

code:
#include <redisearch.h> //must be in the include path

double TFIDFScorer(RSScoringFunctionCtx *ctx, RSIndexResult *h, RSDocumentMetadata *dmd,
                   double minScore) {
  // no need to evaluate documents with score 0
  if (dmd->score == 0) return 0;

  // calculate sum(tf-idf) for each term in the result
  double tfidf = 0;
  for (int i = 0; i < h->numRecords; i++) {
    // take the term frequency and multiply by the term IDF, add that to the total
    tfidf += (float)h->records[i].freq * (h->records[i].term ? h->records[i].term->idf : 0);
  }
  // normalize by the maximal frequency of any term in the document
  tfidf /=  (double)dmd->maxFreq;

  // multiply by the document score (between 0 and 1)
  tfidf *= dmd->score;

  // no need to factor the slop if tfidf is already below minimal score
  if (tfidf < minScore) {
    return 0;
  }

  // get the slop and divide the result by it, making sure we prefer results with closer terms
  tfidf /= (double)ctx->GetSlop(h);

  return tfidf;
}

section_id: example-scoring-function


title: Garbage collection

summary: Details about garbage collection

title: The need for garbage collection

text:
* When documents are deleted by the user, Redis only marks them as deleted in the global document table rather than deleting them outright. This is done for efficiency. Depending on the length of a document, deletion can be a long operation.
* This means that it is no longer the case that an internal numeric id is assigned to a deleted document. When the index is traversed, a check is made for deletion.
* All inverted index entries belonging to deleted document ids are garbage.
* Updating a document is basically the same as deleting it and then adding it again with a new incremental internal ID. No diffing is performed, and the indexes are appended, so the IDs remain incremental, and the updates fast.

All of the above means that if there are a lot of updates and deletes, a large portion of our inverted index will become garbage, both slowing things down and consuming unnecessary memory.

You want to optimize the index, but you also don't want to disturb normal operation. This means that optimization or garbage collection should be a background process that's non-intrusive. It only needs to be faster than the deletion rate over a sufficient period of time so that you don't create more garbage than you can collect.

title: Garbage collecting a single-term index

text:
A single-term inverted index is an array of blocks, each of which contains an encoded list of records; e.g., a document id delta plus other data depending on the index encoding scheme. When some of these records refer to deleted documents this is called garbage.

The algorithm is simple:

0. Create a reader and writer for each block.
1. Read each block's records one by one.
2. If no record is invalid, do nothing.
3. When a garbage record is found, the reader is advanced, but not the writer.
4. When at least one garbage record is found, the next records are encoded to the writer, recalculating the deltas.

Pseudo code:

[code example]

title: GC on numeric indexes

text: Numeric indexes are a tree of inverted indexes with a special encoding of (docId delta, value). This means the same algorithm can be applied to them, only traversing each inverted index object in the tree.

title: FORK GC

text:
Information about FORK GC can be found in this [blog](https://redislabs.com/blog/increased-garbage-collection-performance-redisearch-1-4-1/).

Since v1.6, the FORK GC is the default GC policy and was proven very efficient both in cleaning the index and not reducing query and indexing performance, even for very write-internsive use cases.

code:
foreach index_block as block:

   reader = new_reader(block)
   writer = new_write(block)
   garbage = 0
   while not reader.end():
        record = reader.decode_next()
        if record.is_valid():
            if garbage != 0:
                # Write the record at the writer's tip with a newly calculated delta
                writer.write_record(record)
            else:
                writer.advance(record.length)
        else:
            garbage += record.length

section_id: garbage-collecting-a-single-term-index


title: Administration

summary: Redis Search Administration

summary: Redis Search can be tuned through multiple configuration parameters. Some of these parameters can only be set at load-time, while other parameters can be set either at load-time or at run-time.

title: Configuration parameters

title: Internal design

summary: Technical overview of the search and query features of Redis Open Source

title: Technical overview

summary: Details about garbage collection

title: Garbage collection

summary: This document describes how documents are added to the index.

title: Document Indexing

page_type: index


title: Document Indexing

title: Components

text:
* `Document` - contains the actual document and its fields.
* `RSAddDocumentCtx` - the per-document state that is used while it
  is being indexed. The state is discarded after the indexing is complete.
* `ForwardIndex` - contains terms found in the document. The forward index
  is used to write the `InvertedIndex`.
* `InvertedIndex` - an index that maps terms to occurrences within applicable
  documents.

title: Architecture

text:
The indexing process begins by creating a new `RSAddDocumentCtx` and adding a
document to it. Internally, this is divided into several steps.

1. Submission

   A `DocumentContext` is created, and is associated with a document (as received)
   from input. The submission process will also perform some preliminary caching.

2. Preprocessing

   After a document has been submitted, it is preprocessed. Preprocessing performs
   stateless processing on all document input fields. For text fields, this
   means tokenizing the document and creating a forward index. The preprocesors
   will store this information in per-field variables within the `AddDocumentCtx`.
   This computed result is then written to the persistent index later on during
   the indexing phase.

   If the document is sufficiently large, the preprocessing is done in a separate
   thread, which allows concurrent preprocessing and also avoids blocking other
   threads. If the document is smaller, the preprocessing is done within the main
   thread, avoiding the overhead of additional context switching.
   The `SELF_EXC_THRESHOLD` macro contains the threshold for 'sufficiently large'.

   After the document is preprocessed, it is submitted to be indexed.

3. Indexing

   Indexing proper consists of committing the precomputed results of the
   preprocessing phase. It is done in a single thread, and is in the form
   of a queue.

   Because documents must be written to the index in the exact order of their
   document ID assignment, and because the indexing process must also yield to other potential
   indexing processes, you may end up in a situation where document IDs are written
   to the index out-of-order. To solve that, the order in which documents
   are actually written must be well-defined. If there is only one thread writing
   documents, then this thread will not need to worry about out-of-order IDs
   while writing.

   Having a single background thread also helps optimize in several areas, as
   will be seen later on. The basic idea is that when there are a lot of
   documents queued for the indexing thread, the indexing thread may treat them
   as batch commands, greatly reducing the number of locks/unlocks of the GIL
   and the number of times term keys need to be opened and closed.

4. Skipping already indexed documents

   The phases below may operate on more than one document at a time. When a document
   is fully indexed, it is marked as done. When the thread iterates over the queue
   it will only perform processing/indexing on items not yet marked as done.

5. Term merging

   Term merging, or forward index merging, is done when there is more than a
   single document in the queue. The forward index of each document in the queue
   is scanned, and a larger, master forward index is constructed in its place.
   Each entry in the forward index contains a reference to the origin document
   as well as the normal offset/score/frequency information.

   Creating a master forward index avoids opening common term keys more than once per
   document.

   If there is only one document within the queue, a master forward index
   is not created.

   Note that the internal type of the master forward index is not actually
   `ForwardIndex`.

6. Document ID assignment

   At this point, the GIL is locked and every document in the queue is assigned
   a document ID. The assignment is done immediately before writing to the index
   so as to reduce the number of times the GIL is locked; thus, the GIL is
   locked only once, right before the index is written.

7. Writing to Indexes

   With the GIL being locked, any pending index data is written to the indexes.
   This usually involves opening one or more Redis keys, and writing/copying
   computed data into those keys.

   After this is done, the reply for the given document is sent, and the
   `AddDocumentCtx` freed.


title: Technical overview

title: Abstract

text:
Redis Search is a powerful text search and secondary indexing engine that is built on top of Redis Open Source.

Unlike other Redis search libraries, it does not use the internal data structures of Redis such as sorted sets. Using its own highly optimized data structures and algorithms, it allows for advanced search features, high performance, and a low memory footprint. It can perform simple text searches, as well as complex structured queries, filtering by numeric properties and geographical distances.

Redis Search supports continuous indexing with no performance degradation, maintaining concurrent loads of querying and indexing. This makes it ideal for searching frequently updated databases without the need for batch indexing and service interrupts.

The Enterprise version of Redis Search supports scaling the search engine across many servers, allowing it to easily grow to billions of documents on hundreds of servers.

All of this is done while taking advantage of Redis's robust architecture and infrastructure. Using Redis's protocol, replication, persistence, and clustering, Redis Search delivers a powerful yet simple to manage and maintain search and indexing engine that can be used as a standalone database, or to augment existing Redis databases with advanced powerful indexing capabilities.

---

title: Main features

text:
* Full-Text indexing of multiple fields in a document, including:
    * Exact phrase matching.
    * Stemming in many languages.
    * Chinese tokenization support.
    * Prefix queries.
    * Optional, negative, and union queries.
* Distributed search on billions of documents.
* Numeric property indexing.
* Geographical indexing and radius filters.
* Incremental indexing without performance loss.
* A structured query language for advanced queries:
    * Unions and intersections
    * Optional and negative queries
    * Tag filtering
    * Prefix matching
* A powerful autocomplete engine with fuzzy matching.
* Multiple scoring models and sorting by values.
* Concurrent, low-latency insertion and updates of documents.
* Concurrent searches allowing long-running queries without blocking Redis.
* An extension mechanism allowing custom scoring models and query extension.
* Support for indexing existing Hash objects in Redis databases.

---

title: Indexing documents

text:
Redis needs to know how to index documents in order to search effectively. A document may have several fields, each with its own weight. For example, a title is usually more important than the text itself. The engine can also use numeric or geographical fields for filtering. Hence, the first step is to create the index definition, which tells Redis how to treat the documents that will be added. For example, to define an index of products, indexing their title, description, brand, and price fields, the index creation would look like:

[code example]

When a document is added to this index, as in the following example, each field of the document is broken into its terms (tokenization), and indexed by marking the index for each of the terms in the index. As a result, the product is added immediately to the index and can now be found in future searches.

[code example]

This tells Redis to take the document, break each field into its terms (tokenization), and index it by marking the index for each of the terms in the index as contained in this document. Thus, the product is added immediately to the index and can now be found in future searches.

title: Searching

text:
Now that the products have been added to our index, searching is very simple:

[code example]

This will tell Redis to intersect the lists of documents for each term and return all documents containing the three terms. Of course, more complex queries can be performed, and the full syntax of the query language is detailed below.

title: Data structures

text:
Redis uses its own custom data structures and uses Redis' native structures only for storing the actual document content (using Hash objects).

Using specialized data structures allows faster searches and more memory effective storage of index records, utilizing compression techniques like delta encoding.

These are the data structures Redis uses under the hood:

title: Index and document metadata

text:
For each search _index_, there is a root data structure containing the schema, statistics, etc., but most importantly, compact metadata about each document indexed.

Internally, inside the index, Redis uses delta encoded lists of numeric, incremental, 32-bit document ids. This means that the user given keys or ids for documents, need to be replaced with the internal ids on indexing, and back to the original ids on search.

For that, Redis saves two tables, mapping the two kinds of ids in two ways (one table uses a compact trie, the other is simply an array where the internal document ID is the array index). On top of that, for each document, its user given presumptive score is stored, along with some status bits and any optional payload attached to the document by the user.

Accessing the document metadata table is an order of magnitude faster than accessing the hash object where the document is actually saved, so scoring functions that need to access metadata about the document can operate fast enough.

title: Inverted index

text:
For each term appearing in at least one document, an inverted index is kept, which is basically a list of all the documents where this term appears. The list is compressed using delta coding, and the document ids are always incrementing.

For example, when a user indexes the documents "foo", "bar", and "baz", they are assigned incrementing ids, e.g., `1025, 1045, 1080`. When encoding them into the index,  only the first ID is encoded, followed by the deltas between each entry and the previous one, e.g., `1025, 20, 35`.

Using variable-width encoding, one byte is used to express numbers under 255, two bytes for numbers between 256 and 16,383, and so on. This can compress the index by up to 75%.

On top of the IDs, the frequency of each term in each document, a bit mask representing the fields in which the term appeared in the document, and a list of the positions in which the term appeared is saved.

The structure of the default search record is as follows. Usually, all the entries are one byte long:

[code example]

Optionally, you can choose not to save any one of those attributes besides the ID, degrading the features available to the engine.

title: Numeric index

text:
Numeric properties are indexed in a special data structure that enables filtering by numeric ranges in an efficient way. One could view a numeric value as a term operating just like an inverted index. For example, all the products with the price $100 are in a specific list, which is intersected with the rest of the query. See [query execution engine]() for more information.

However, in order to filter by a range of prices, you would have to intersect the query with all the distinct prices within that range, or perform a union query. If the range has many values in it, this becomes highly inefficient.

To avoid this, numeric entries are grouped, with close values together, in a single range node. These nodes are stored in a binary range tree, which allows the engine to select the relevant nodes and union them together. Each entry in a range node contains a document Id and the actual numeric value for that document. To further optimize, the tree uses an adaptive algorithm to try to merge as many nodes as possible within the same range node.

title: Tag index

text:
Tag indexes are similar to full-text indexes, but use simpler tokenization and encoding in the index. The values in these fields cannot be accessed by general field-less search and can be used only with a special syntax.

The main differences between tag fields and full-text fields are:

1. The tokenization is simpler. The user can determine a separator (defaults to a comma) for multiple tags. Whitespace trimming is done only at the end of tags. Thus, tags can contain spaces, punctuation marks, accents, etc. The only two transformations that are performed are lower-casing (for latin languages only as of now) and whitespace trimming.

2. Tags cannot be found from a general full-text search. If a document has a field called *tags* with the values *foo* and *bar*, searching for foo or bar without a special tag modifier (see below) will not return this document.

3. The index is much simpler and more compressed. Only the document IDs are stored in the index, usually resulting in 1-2 bytes per index entry.

title: Geo index

text: Geo indexes utilize Redis's own geo-indexing capabilities. At query time, the geographical part of the query (a radius filter) is sent to Redis, returning only the ids of documents that are within that radius. Longitude and latitude should be passed as a string `lon,lat`. For example, `1.23,4.56`.

title: Autocomplete

text: The autocomplete engine (see below for a fuller description) uses a compact trie or prefix tree to encode terms and search them by prefix.

title: Query language

text:
Simple syntax is supported for complex queries that can be combined together to express complex filtering and matching rules. The query is a text string in the [`FT.SEARCH`]() request that is parsed using a complex query processor.

* Multi-word phrases are lists of tokens, e.g., `foo bar baz`, and imply intersection (logical AND) of the terms.
* Exact phrases are wrapped in quotes, e.g `"hello world"`.
* OR unions (e.g., `word1 OR word2`), are expressed with a pipe (`|`) character. For example, `hello|hallo|shalom|hola`.
* NOT negation (e.g., `word1 NOT word2`) of expressions or sub-queries use the dash (`-`) character. For example, `hello -world`.
* Prefix matches (all terms starting with a prefix) are expressed with a `*` following a 2-letter or longer prefix.
* Selection of specific fields using the syntax `@field:hello world`.
* Numeric Range matches on numeric fields with the syntax `@field:[{min} {max}]`.
* Geo radius matches on geo fields with the syntax `@field:[{lon} {lat} {radius} {m|km|mi|ft}]`
* Tag field filters with the syntax `@field:{tag | tag | ...}`. See the [full documentation on tag fields]().
* Optional terms or clauses: `foo ~bar` means bar is optional but documents with bar in them will rank higher.

title: Complex query examples

text:
Expressions can be combined together to express complex rules. For example, given a database of products, where each entity has the fields `title`, `brand`, `tags` and `price`, expressing a generic search would be simply:

[code example]

This would return documents containing these terms in any field. Limiting the search to specific fields (title only in this case) is expressed as:

[code example]

Numeric filters can be combined to filter by price within a given price range:

[code example]

Multiple text fields can be accessed in different query clauses. For example, to select products of multiple brands:

[code example]

Tag fields can be used to index multi-term properties without actual full-text tokenization:

[code example]

And negative clauses can also be added to filter out plasma and CRT TVs:

[code example]

title: Scoring model

text:
Redis comes with a few very basic scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use sortable fields (see below). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search request.

If you prefer a custom scoring function, it is possible to add more functions using the [extension API]().

These are the pre-bundled scoring functions available in Redis:

* **TFIDF** (default)

    Basic [TF-IDF scoring](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) with document score and proximity boosting factored in.

* **TFIDF.DOCNORM**
*
    Identical to the default TFIDF scorer, with one important distinction:

* **BM25STD (default)**

    A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25).

    The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`.

    The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated.

* **BM25STD.NORM**

    A variation of `BM25STD`, where the scores are normalized by the minimum and maximum score.

* **BM25STD.TANH**

    A variation of `BM25STD.NORM`, where the scores are normalised by the linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4.

* **DISMAX**

    A simple scorer that sums up the frequencies of the matched terms. In the case of union clauses, it will give the maximum value of those matches.

* **DOCSCORE**

    A scoring function that just returns the presumptive score of the document without applying any calculations to it. Since document scores can be updated, this can be useful if you'd like to use an external score and nothing further.

title: Sortable fields

text:
It is possible to bypass the scoring function mechanism and order search results by the value of different document properties (fields) directly, even if the sorting field is not used by the query. For example, you can search for first name and sort by the last name.

When creating the index with [`FT.CREATE`](), you can declare `TEXT`, `TAG`, `NUMERIC`, and `GEO` properties as `SORTABLE`. When a property is sortable, you can later decide to order the results by its values with relatively low latency. When a property is not sortable, it can still be sorted by its values, but may increase latency. For example, the following schema:

[code example]

Would allow the following query:

[code example]

title: Result highlighting and summarization

role: returns

text:
Redis uses advanced algorithms for highlighting and summarizing, which enable only the relevant portions of a document to appear in response to a search query. This feature allows users to immediately understand the relevance of a document to their search criteria, typically highlighting the matching terms in bold text. The syntax is as follows:

[code example]

Summarization will fragment the text into smaller sized snippets. Each snippet will contain the found term(s) and some additional surrounding context.

Highlighting will highlight the found term and its variants with a user-defined tag. This may be used to display the matched text in a different typeface using a markup language, or to otherwise make the text appear differently.

title: Autocomplete

text:
Another important feature for Redis Open Source is its autocomplete engine. This allows users to create dictionaries of weighted terms, and then query them for completion suggestions to a given user prefix. Completions can have payloads, which are user-provided pieces of data that can be used for display. For example, completing the names of users, it is possible to add extra metadata about users to be displayed.

For example, if a user starts to put the term "lcd tv" into a dictionary, sending the prefix "lc" will return the full term as a result. The dictionary is modeled as a compact trie (prefix tree) with weights, which is traversed to find the top suffixes of a prefix.

Redis also allows fuzzy suggestions, meaning you can get suggestions to prefixes even if the user makes a typo in their prefix. This is enabled using a Levenshtein automaton, allowing efficient searching of the dictionary for all terms within a maximal Levenshtein distance of a term or prefix. Suggestions are then weighted based on both their original score and their distance from the prefix typed by the user.

However, searching for fuzzy prefixes (especially very short ones) will traverse an enormous number of suggestions. In fact, fuzzy suggestions for any single letter will traverse the entire dictionary, so the recommendation is to use this feature carefully and in full consideration of the performance penalty it incurs.

The autocomplete engine supports Unicode, allowing for fuzzy matches in non-latin languages as well.

See the [autocomplete page]() for more information and examples.

title: Search engine internals

title: The Redis module API

text:
Redis Search is implemented using the [Redis module API]() and is loaded into Redis as an extension module at start-up.

Redis modules make it possible to extend Redis's core functionality, implementing new Redis commands, data structures, and capabilities with similar performance to native core Redis itself. Redis modules are dynamic libraries that can be loaded into Redis at start-up or loaded at run-time using the [`MODULE LOAD`]() command. Redis exports a C API, in the form of a single C header file called `redismodule.h`.

While the logic of Redis Search and its algorithms are mostly independent, and it could be ported quite easily to run as a stand-alone server, it still takes advantage of Redis as a robust infrastructure for a database server. Building on top of Redis means that, by default, modules are afforded:

* A high performance network protocol server
* Robust replication
* Highly durable persistence as snapshots of transaction logs
* Cluster mode

title: Query execution engine

text:
Redis uses a high-performance flexible query processing engine that can evaluate very complex queries in real time.

The above query language is compiled into an execution plan that consists of a tree of index iterators or filters. These can be any of:

* Numeric filter
* Tag filter
* Text filter
* Geo filter
* Intersection operation (combining 2 or more filters)
* Union operation (combining 2 or more filters)
* NOT operation (negating the results of an underlying filter)
* Optional operation (wrapping an underlying filter in an optional matching filter)

The query parser generates a tree of these filters. For example, a multi-word search would be resolved into an intersect operation of multiple text filters, each traversing an inverted index of a different term. Simple optimizations such as removing redundant layers in the tree are applied.

Each of the filters in the resulting tree evaluates one match at a time. This means that at any given moment, the query processor is busy evaluating and scoring one matching document. This means that very little memory allocation is done at run-time, resulting in higher performance.

The resulting matching documents are then fed to a post-processing chain of result processors that are responsible for scoring them, extracting the top-N results, loading the documents from storage, and sending them to the client. That chain is dynamic as well, which adapts based on the attributes of the query. For example, a query that only needs to return document ids will not include a stage for loading documents from storage.

title: Concurrent updates and searches

text:
While Redis Search is extremely fast and uses highly optimized data structures and algorithms, it was facing the same problem with regards to concurrency. Depending on the size of your data set and the cardinality of search queries, queries can take anywhere between a few microseconds to hundreds of milliseconds, or even seconds in extreme cases. When that happens, the entire Redis server process is blocked.

Think, for example, of a full-text query intersecting the terms "hello" and "world", each with a million entries, and a half-million common intersection points. To perform that query in a millisecond, Redis would have to scan, intersect, and rank each result in one nanosecond, [which is impossible with current hardware](https://gist.github.com/jboner/2841832). The same goes for indexing a 1,000 word document. It blocks Redis entirely for the duration of the query.

Redis Search uses the Redis Module API's concurrency features to avoid stalling the server for long periods of time. The idea is simple - while Redis itself is single-threaded, a module can run many threads, and any one of those threads can acquire the **Global Lock** when it needs to access Redis data, operate on it, and release it.

Redis cannot be queried in parallel, as only one thread can acquire the lock, including the Redis main thread, but care is taken to make sure that a long-running query will give other queries time to run by yielding this lock from time to time.

The following design principles were adopted to allow concurrency:

1. Redis Search has a thread pool for running concurrent search queries.

2. When a search request arrives, it is passed to the handler, parsed on the main thread, and then a request object is passed to the thread pool via a queue.

3. The thread pool runs a query processing function in its own thread.

4. The function locks the Redis Global lock and starts executing the query.

5. Since the search execution is basically an iterator running in a cycle, the elapsed time is sampled every several iterations (sampling on each iteration would slow things down as it has a cost of its own).

6. If enough time has elapsed, the query processor releases the Global Lock and immediately tries to acquire it again. When the lock is released, the kernel will schedule another thread to run - be it Redis's main thread, or another query thread.

7. When the lock is acquired again, all Redis resources that were held before releasing the lock are re-opened (keys might have been deleted while the thread has been sleeping) and work resumes from the previous state.

Thus the operating system's scheduler makes sure all query threads get CPU time to run. While one is running the rest wait idly, but since execution is yielded about 5,000 times a second, it creates the effect of concurrency. Fast queries will finish in one go without yielding execution, slow ones will take many iterations to finish, but will allow other queries to run concurrently.

title: Index garbage collection

text:
Redis Search is optimized for high write, update, and delete throughput. One of the main design choices dictated by this goal is that deleting and updating documents do not actually delete anything from the index:

1. Deletion simply marks the document deleted in a global document metadata table using a single bit.
2. Updating, on the other hand, marks a document as deleted, assigns it a new incremental document ID, and re-indexes the document under a new ID, without performing a comparison of the change.

What this means, is that index entries belonging to deleted documents are not removed from the index, and can be seen as garbage. Over time, an index with many deletes and updates will contain mostly garbage, both slowing things down and consuming unnecessary memory.

To overcome this, Redis Search employs a background garbage collection (GC) mechanism. During normal operation of the index, a special thread randomly samples indexes, traverses them, and looks for garbage. Index sections containing garbage are cleaned and memory is reclaimed. This is done in a non- intrusive way, operating on very small amounts of data per scan, and utilizing Redis's concurrency mechanism (see above) to avoid interrupting searches and indexing. The algorithm also tries to adapt to the state of the index, increasing the GC's frequency if the index contains a lot of garbage, and decreasing it if it doesn't, to the point of hardly scanning if the index does not contain garbage.

title: Extension model

text:
RedisSearch supports an extension mechanism, much like Redis supports modules. The API is very minimal at the moment, and it does not yet support dynamic loading of extensions at run-time. Instead, extensions must be written in C (or a language that has an interface with C) and compiled into dynamic libraries that will be loaded at start-up.

There are two kinds of extension APIs at the moment:

1. **Query expanders**, whose role is to expand query tokens (i.e., stemmers).
2. **Scoring functions**, whose role is to rank search results at query time.

Extensions are compiled into dynamic libraries and loaded into Redis Search on initialization of the module. The mechanism is based on the code of Redis's own module system, albeit far simpler.

---

title: Scalable distributed search

text:
While Redis Search is very fast and memory efficient, if an index is big enough, at some point it will be too slow or consume too much memory. It must then be scaled out and partitioned over several machines, each of which will hold a small part of the complete search index.

Traditional clusters map different keys to different shards to achieve this. However, with search indexes this approach is not practical. If each word’s index was mapped to a different shard, it would be necessary to intersect records from different servers for multi-term queries.

The way to address this challenge is to employ a technique called index partitioning, which is very simple at its core:

* The index is split across many machines/partitions by document ID.
* Every partition has a complete index of all the documents mapped to it.
* All shards are queried concurrently and the results from each shard are merged into a single result.

To facilitate this, a new component called a coordinator is added to the cluster. When searching for documents, the coordinator receives the query and sends it to N partitions, each holding a sub-index of 1/N documents. Since we’re only interested in the top K results of all partitions, each partition returns just its own top K results. Then, the N lists of K elements are merged and the top K elements are extracted from the merged list.

code:
FT.CREATE idx PREFIX 1 doc: SCHEMA
    title TEXT WEIGHT 5
    description TEXT
    brand TEXT
    PRICE numeric

section_id: indexing-documents

code:
HSET doc:1
    title "Acme 42 inch LCD TV"
    description "42 inch brand new Full-HD tv with smart tv capabilities"
    brand "Acme"
    price 300

code: FT.SEARCH idx "full hd tv"

section_id: searching

section_id: inverted-index

code: lcd tv

section_id: complex-query-examples

code: @title:(lcd tv)

code:
@title:(lcd tv)
    @brand:(sony | samsung | lg)
    @price:[100 500.2]

code:
@title:(lcd tv)
    @brand:(sony | samsung | lg)
    @tags:{42 inch | smart tv}
    @price:[100 500.2]

    -@tags:{plasma | crt}

code: FT.CREATE users SCHEMA first_name TEXT last_name TEXT SORTABLE age NUMERIC SORTABLE

section_id: sortable-fields

code: FT.SEARCH users "john lennon" SORTBY age DESC

code:
FT.SEARCH ...
    SUMMARIZE [FIELDS {num} {field}] [FRAGS {numFrags}] [LEN {fragLen}] [SEPARATOR {separator}]
    HIGHLIGHT [FIELDS {num} {field}] [TAGS {openTag} {closeTag}]

section_id: result-highlighting-and-summarization


title: Aggregations

summary: Groupings, projections, and aggregation functions

title: Overview

text:
Aggregations are a way to process the results of a search query. Aggregation allows you to group, sort, and transform your result data, and to extract analytic insights from it. Much like aggregation queries in other databases and search engines, they can be used to create analytics reports, or perform [faceted search](https://en.wikipedia.org/wiki/Faceted_search) style queries.

For example, indexing a web-server's logs, you can create a report for unique users by hour, country, or any other breakdown. Or you can create different reports for errors, warnings, etc.

title: Core concepts

text:
The basic idea of an aggregate query is this:

* Perform a search query, filtering for records you wish to process.
* Build a pipeline of operations that transform the results by zero or more sequences of:
  * **Group and reduce**: grouping by fields in the results, and applying reducer functions to each group.
  * **Sort**: sort the results based on one or more fields.
  * **Apply transformations**: apply mathematical and string functions on fields in the pipeline, optionally creating new fields or replacing existing ones.
  * **Limit**: limit the result, regardless of how the result is sorted.
  * **Filter**: filter the results (post-query) based on predicates relating to its values.

The pipeline is dynamic and re-entrant, and every operation can be repeated. For example, you can group by property X, sort the top 100 results by group size, then group by property Y and sort the results by some other property, then apply a transformation on the output.

Figure 1: Aggregation Pipeline Example

<img src="../../img/pipeline.png">

title: Aggregate request format

text:
The aggregate request's syntax is defined as follows:

[code example]

title: Parameters in detail

role: parameters

text:
Parameters that can take a variable number of arguments are expressed in the
form of `param {nargs} {property_1... property_N}`. The first argument to the
parameter is the number of arguments following the parameter. This allows
Redis to avoid a parsing ambiguity in case one of your arguments has the
name of another parameter. For example, to sort by first name, last name, and
country, one would specify `SORTBY 6 firstName ASC lastName DESC country ASC`.

* **index_name**: The index the query is executed against.

* **query_string**: The base filtering query that retrieves the documents. It follows the exact same syntax as the search query, including filters, unions, not, optional, etc.

* **LOAD {nargs} {property} ...** : Load document fields from the document HASH objects. This should be avoided as a general rule of thumb. Fields needed for aggregations should be stored as SORTABLE (and optionally UNF to avoid any normalization), where they are available to the aggregation pipeline with very low latency. LOAD hurts the performance of aggregate queries considerably since every processed record needs to execute the equivalent of HMGET against a Redis key, which when executed over millions of keys, amounts to very high processing times.
The document ID can be loaded using `@__key`.

* **GROUPBY {nargs} {property} ...** : Group the results in the pipeline based on one or more properties. Each group should have at least one reducer (See below), a function that handles the group entries, either counting them or performing multiple aggregate operations (see below).

* **REDUCE {func} {nargs} {arg} ... [AS {name}]**: Reduce the matching results in each group into a single record, using a reduction function. For example, COUNT will count the number of records in the group. See the Reducers section below for more details on available reducers.

    The reducers can have their own property names using the `AS {name}` optional argument. If a name is not given, the resulting name will be the name of the reduce function and the group properties. For example, if a name is not given to COUNT_DISTINCT by property `@foo`, the resulting name will be `count_distinct(@foo)`.

* **SORTBY {nargs} {property} {ASC|DESC} [MAX {num}]**: Sort the pipeline up until the point of SORTBY, using a list of properties. By default, sorting is ascending, but `ASC` or `DESC ` can be added for each property. `nargs` is the number of sorting parameters, including ASC and DESC. for example: `SORTBY 4 @foo ASC @bar DESC`.

    `MAX` is used to optimized sorting, by sorting only for the n-largest elements. Although it is not connected to `LIMIT`, you usually need just `SORTBY … MAX` for common queries.

* **APPLY {expr} AS {name}**: Apply a one-to-one transformation on one or more properties, and either store the result as a new property down the pipeline, or replace any property using this transformation. `expr` is an expression that can be used to perform arithmetic operations on numeric properties, or functions that can be applied on properties depending on their types (see below), or any combination thereof. For example: `APPLY "sqrt(@foo)/log(@bar) + 5" AS baz` will evaluate this expression dynamically for each record in the pipeline and store the result as a new property called baz, that can be referenced by further APPLY / SORTBY / GROUPBY / REDUCE operations down the pipeline.

* **LIMIT {offset} {num}**. Limit the number of results to return just `num` results starting at index `offset` (zero based). AS mentioned above, it is much more efficient to use `SORTBY … MAX` if you are interested in just limiting the output of a sort operation.

     However, limit can be used to limit results without sorting, or for paging the n-largest results as determined by `SORTBY MAX`. For example, getting results 50-100 of the top 100 results is most efficiently expressed as `SORTBY 1 @foo MAX 100 LIMIT 50 50`. Removing the MAX from SORTBY will result in the pipeline sorting all the records and then paging over results 50-100.

* **FILTER {expr}**. Filter the results using predicate expressions relating to values in each result. The expressions are applied post-query and relate to the current state of the pipeline. See FILTER Expressions below for full details.

* **PARAMS {nargs} {name} {value}**. Define one or more value parameters. Each parameter has a name and a value. Parameters can be referenced in the query string by a `$`, followed by the parameter name, e.g., `$user`, and each such reference in the search query to a parameter name is substituted by the corresponding parameter value. For example, with parameter definition `PARAMS 4 lon 29.69465 lat 34.95126`, the expression `@loc:[$lon $lat 10 km]` would be evaluated to `@loc:[29.69465 34.95126 10 km]`. Parameters cannot be referenced in the query string where concrete values are not allowed, such as in field names, e.g., `@loc`
*

title: Example

text:
A log of visits to a website might look like the following, each record of which has the following fields/properties:

* **url** (text, sortable)
* **timestamp** (numeric, sortable) - Unix timestamp of visit entry.
* **country** (tag, sortable)
* **user_id** (text, sortable, not indexed)

title: Example 1: unique users by hour, ordered chronologically.

text:
The first step is to determine the index name and the filtering query. A filter query of `*` means "get all records":

[code example]

Next, group the results by hour. The data contains visit times as unix timestamps in second resolution, so you'll need to extract the hour component of the timestamp. To do so, add an APPLY step that strips the sub-hour information from the timestamp and stores is as a new property, `hour`:

[code example]

Next, group the results by hour and count the distinct user ids in each hour. This is done by a GROUPBY/REDUCE step:

[code example]

Next, sort the results by hour, ascending:

[code example]

And as a final step, format the hour as a human readable timestamp. This is done by calling the transformation function `timefmt` that formats Unix timestamps. You can specify a format to be passed to the system's `strftime` function ([see documentation](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html)), but not specifying one is equivalent to specifying `%FT%TZ` to `strftime`.

[code example]

title: Example 2: Sort visits to a specific URL by day and country:

text:
The next example filters by the url, transforms the timestamp to its day part, and groups by the day and country, counting the number of visits per group, sorting by day ascending and country descending.

[code example]

title: GROUPBY reducers

text:
`GROUPBY` works similarly to SQL `GROUP BY` clauses, and creates groups of results based on one or more properties in each record. For each group, Redis returns the group keys, or the values common to all records in the group, and the results of zero or more `REDUCE` clauses.

Each `GROUPBY` step in the pipeline may be accompanied by zero or more `REDUCE` clauses. Reducers apply an accumulation function to each record in the group and reduces them into a single record representing the group. When the processing is complete, all the records upstream of the `GROUPBY` step emit their reduced record.

For example, the simplest reducer is COUNT, which simply counts the number of records in each group.

If multiple `REDUCE` clauses exist for a single `GROUPBY` step, each reducer works independently on each result and writes its final output once. Each reducer may have its own alias determined using the `AS` optional parameter. If `AS` is not specified, the alias is the reduce function and its parameters, e.g. `count_distinct(foo,bar)`.

title: Supported GROUPBY reducers

role: compatibility

text:
#### COUNT

**Format**

[code example]

**Description**

Count the number of records in each group

#### COUNT_DISTINCT

**Format**

[code example]`

**Description**

Count the number of distinct values for `property`.

The reducer creates a hash-set per group, and hashes each record. This can be memory heavy if the groups are big.

#### COUNT_DISTINCTISH

**Format**

[code example]

**Description**

Same as COUNT_DISTINCT, provides an approximation instead of an exact count, which consumes less memory and CPU for big groups.

The reducer uses [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) counters per group, at ~3% error rate, and 1024 bytes of constant space  per group. This means it is ideal for a few huge groups and not ideal for many small groups. In the former case, it can be an order of magnitude faster and consume much less memory than COUNT_DISTINCT, but again, it does not fit every use case.

#### SUM

**Format**

[code example]

**Description**

Return the sum of all numeric values of a given property in a group. Non-numeric values in the group are counted as 0.

#### MIN

**Format**

[code example]

**Description**

Return the minimal value of a property, whether it is a string, number, or NULL.

#### MAX

**Format**

[code example]

**Description**

Return the maximal value of a property, whether it is a string, number or NULL.

#### AVG

**Format**

[code example]

**Description**

Return the average value of a numeric property. This is equivalent to reducing by sum and count, and later, applying the ratio of them as an APPLY step.

#### STDDEV

**Format**

[code example]

**Description**

Return the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of a numeric property in the group.

#### QUANTILE

**Format**

[code example]

**Description**

Return the value of a numeric property at a given quantile of the results. Quantile is expressed as a number between 0 and 1. For example, the median can be expressed as the quantile at 0.5, e.g. `REDUCE QUANTILE 2 @foo 0.5 AS median` .

If multiple quantiles are required, just repeat the QUANTILE reducer for each quantile. For example, `REDUCE QUANTILE 2 @foo 0.5 AS median REDUCE QUANTILE 2 @foo 0.99 AS p99`.

#### TOLIST

**Format**

[code example]

**Description**

Merge all distinct values of a given property into a single array.

#### FIRST_VALUE

**Format**

[code example]

**Description**

Return the first or top value of a given property in the group, optionally by comparing it to another property. For example, you can extract the name of the oldest user in the group:

[code example]

If no `BY` is specified, the first value encountered in the group is returned.

If you wish to get the top or bottom value in the group sorted by the same value, you are better off using the `MIN/MAX` reducers, but the same effect will be achieved by doing `REDUCE FIRST_VALUE 4 @foo BY @foo DESC`.

#### RANDOM_SAMPLE

**Format**

[code example]

**Description**

Perform a reservoir sampling of the group elements with a given size, and return an array of the sampled items with an even distribution.

title: APPLY expressions

text:
`APPLY` performs a one-to-one transformation on one or more properties in each record. It either stores the result as a new property down the pipeline, or replaces any property using this transformation.

The transformations are expressed as a combination of arithmetic expressions and built in functions. Evaluating functions and expressions is recursively nested and can be composed without limit. For example: `sqrt(log(foo) * floor(@bar/baz)) + (3^@qaz % 6)` or simply `@foo/@bar`.

If an expression or a function is applied to values that do not match the expected types, no error is emitted and a NULL value is set as the result.

APPLY steps must have an explicit alias determined by the `AS` parameter.

title: Literals inside expressions

text:
* Numbers are expressed as integers or floating point numbers, e.g., `2`, `3.141`, and `-34`. `inf` and `-inf` are acceptable as well.
* Strings are quoted with either single or double quotes. Single quotes are acceptable inside strings quoted with double quotes and vice versa. Punctuation marks can be escaped with backslashes. e.g. `"foo's bar"` ,`'foo\'s bar'`, `"foo \"bar\""` .
* Any literal or sub-expression can be wrapped in parentheses to resolve ambiguities of operator precedence.

title: Arithmetic operations

text:
For numeric expressions and properties, addition (`+`), subtraction (`-`), multiplication (`*`), division (`/`), modulo (`%`), and power (`^`) are supported. Bitwise logical operators are not supported.

Note that these operators apply only to numeric values and numeric sub-expressions. Any attempt to multiply a string by a number, for instance, will result in a NULL output.

title: List of field APPLY functions

text:
| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| exists(s)| Checks whether a field exists in a document.                 | `exists(@field)`   |
| case(condition, if_true, if_false) | If condition is non-zero, return if_true, otherwise return if_false. | `case(exists(@foo), @foo, "no foo")` |

title: List of numeric APPLY functions

text:
| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| log(x)   | Return the logarithm of a number, property or subexpression  | `log(@foo)`        |
| abs(x)   | Return the absolute number of a numeric expression           | `abs(@foo-@bar)`   |
| ceil(x)  | Round to the smallest value not less than x                  | `ceil(@foo/3.14)`  |
| floor(x) | Round to largest value not greater than x                    | `floor(@foo/3.14)` |
| log2(x)  | Return the logarithm of x to base 2                          | `log2(2^@foo)`     |
| exp(x)   | Return the exponent of x, e.g., `e^x`                        | `exp(@foo)`        |
| sqrt(x)  | Return the square root of x                                  | `sqrt(@foo)`       |

title: List of string APPLY functions

text:
| Function                         |                                                              |                                                          |
| -------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |
| upper(s)                         | Return the uppercase conversion of s                         | `upper('hello world')`                                   |
| lower(s)                         | Return the lowercase conversion of s                         | `lower("HELLO WORLD")`                                   |
| startswith(s1,s2)                | Return `1` if s2 is the prefix of s1, `0` otherwise.         | `startswith(@field, "company")`                          |
| contains(s1,s2)                  | Return the number of occurrences of s2 in s1, `0` otherwise. If s2 is an empty string, return `length(s1) + 1`.  | `contains(@field, "pa")`                                 |
| strlen(s)                        | Return the length of s                                       | `strlen(@t)`                                             |
| substr(s, offset, count)         | Return the substring of s, starting at _offset_ and having _count_ characters.
If offset is negative, it represents the distance from the end of the string.
If count is -1, it means "the rest of the string starting at offset". | `substr("hello", 0, 3)`
 `substr("hello", -2, -1)`  |
| format( fmt, ...)                | Use the arguments following `fmt` to format a string.
Currently the only format argument supported is `%s` and it applies to all types of arguments. | `format("Hello, %s, you are %s years old", @name, @age)` |
| matched_terms([max_terms=100])   | Return the query terms that matched for each record (up to 100), as a list. If a limit is specified, Redis will return the first N matches found, based on query order. | `matched_terms()`                                        |
| split(s, [sep=","], [strip=" "]) | Split a string by any character in the string sep, and strip any characters in strip. If only s is specified, it is split by commas and spaces are stripped. The output is an array. | split("foo,bar")                                         |

title: List of date/time APPLY functions

text:
| Function            | Description                                                  |
| ------------------- | ------------------------------------------------------------ |
| timefmt(x, [fmt])      | Return a formatted time string based on a numeric timestamp value x.
 See [strftime](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html) for formatting options.
Not specifying `fmt` is equivalent to `%FT%TZ`. |
| parsetime(timesharing, [fmt]) | The opposite of timefmt() - parse a time format using a given format string |
| day(timestamp) | Round a Unix timestamp to midnight (00:00) start of the current day. |
| hour(timestamp) | Round a Unix timestamp to the beginning of the current hour. |
| minute(timestamp) | Round a Unix timestamp to the beginning of the current minute. |
| month(timestamp) | Round a unix timestamp to the beginning of the current month. |
| dayofweek(timestamp) | Convert a Unix timestamp to the day number (Sunday = 0). |
| dayofmonth(timestamp) | Convert a Unix timestamp to the day of month number (1 .. 31). |
| dayofyear(timestamp) | Convert a Unix timestamp to the day of year number (0 .. 365). |
| year(timestamp) | Convert a Unix timestamp to the current year (e.g. 2018). |
| monthofyear(timestamp) | Convert a Unix timestamp to the current month (0 .. 11). |

title: List of geo APPLY functions

text:
| Function | Description                                                  | Example            |
| -------- | ------------------------------------------------------------ | ------------------ |
| geodistance(field,field)        | Return distance in meters.    | `geodistance(@field1,@field2)`       |
| geodistance(field,"lon,lat")    | Return distance in meters.    | `geodistance(@field,"1.2,-3.4")`     |
| geodistance(field,lon,lat)      | Return distance in meters.    | `geodistance(@field,1.2,-3.4)`       |
| geodistance("lon,lat",field)    | Return distance in meters.    | `geodistance("1.2,-3.4",@field)`     |
| geodistance("lon,lat","lon,lat")| Return distance in meters.    | `geodistance("1.2,-3.4","5.6,-7.8")` |
| geodistance("lon,lat",lon,lat)  | Return distance in meters.    | `geodistance("1.2,-3.4",5.6,-7.8)`   |
| geodistance(lon,lat,field)      | Return distance in meters.    | `geodistance(1.2,-3.4,@field)`       |
| geodistance(lon,lat,"lon,lat")  | Return distance in meters.    | `geodistance(1.2,-3.4,"5.6,-7.8")`   |
| geodistance(lon,lat,lon,lat)    | Return distance in meters.    | `geodistance(1.2,-3.4,5.6,-7.8)`     |

[code example]

To retrieve the distance:

[code example]

Note: the geo field must be preloaded using `LOAD`.

Results can also be sorted by distance:

[code example]

Note: Make sure no location is missing, otherwise the SORTBY will not return any results.
Use FILTER to make sure you do the sorting on all valid locations.

title: FILTER expressions

text:
FILTER expressions filter the results using predicates relating to values in the result set.

The FILTER expressions are evaluated post-query and relate to the current state of the pipeline. Thus they can be useful to prune the results based on group calculations. Note that the filters are not indexed and will not speed up processing.

Filter expressions follow the syntax of APPLY expressions, with the addition of the conditions `==`, `!=`, `<`, `<=`, `>`, `>=`. Two or more predicates can be combined with logical AND (`&&`) and OR (`||`). A single predicate can be negated with a NOT prefix (`!`).

For example, filtering all results where the user name is 'foo' and the age is less than 20 is expressed as:

[code example]

Several filter steps can be added, although at the same stage in the pipeline, it is more efficient to combine several predicates into a single filter step.

title: Cursor API

text:
[code example]

You can use cursors with [`FT.AGGREGATE`](), with the `WITHCURSOR` keyword. Cursors allow you to
consume only part of the response, allowing you to fetch additional results as needed.
This is much quicker than using `LIMIT` with offset, since the query is executed only
once, and its state is stored on the server.

To use cursors, specify the `WITHCURSOR` keyword in [`FT.AGGREGATE`](). For example:

[code example]

This will return a response of an array with two elements. The first element is
the actual (partial) result, and the second is the cursor ID. The cursor ID
can then be fed to [`FT.CURSOR READ`]() repeatedly until the cursor ID is 0, in
which case all results have been returned.

To read from an existing cursor, use [`FT.CURSOR READ`](). For example:

[code example]

Assuming `342459320` is the cursor ID returned from the [`FT.AGGREGATE`]() request, here is an example in pseudo-code:

[code example]

Note that even if the cursor is 0, a partial result may still be returned.

title: Cursor settings

text:
#### Read size

You can control how many rows are read for each cursor fetch by using the
`COUNT` parameter. This parameter can be specified both in [`FT.AGGREGATE`]()
(immediately after `WITHCURSOR`) or in [`FT.CURSOR READ`]().

The following example will read 10 rows at a time:
[code example]

You can override this setting by also specifying `COUNT` in `CURSOR READ`. The following example will return at most 50 results:

[code example]

The default read size is 1000.

#### Timeouts and limits

Because cursors are stateful resources that occupy memory on the server, they
have a limited lifetime. To safeguard against orphaned/stale cursors,
cursors have an idle timeout value. If no activity occurs on the cursor before
the idle timeout, the cursor is deleted. The idle timer resets to 0 whenever
the cursor is read from using `CURSOR READ`.

The default idle timeout is 300000 milliseconds (or 300 seconds). You can modify
the idle timeout using the `MAXIDLE` keyword when creating the cursor. Note that
the value cannot exceed the default 300s.

For example, to set a limit of ten seconds:

[code example]

During a cursor’s lifetime, CURSOR READ may return results that exclude newer updates.

Also, note that in case of any load rebalancing operations (such as Atomic Slot Migration) occurring during the lifetime of a cursor, some results may be missing. In a Redis Software deployment, these operations may happen behind the scenes without any user intervention.

title: Other cursor commands

text:
Cursors can be explicitly deleted using the `CURSOR DEL` command. For example:

[code example]

Note that cursors are automatically deleted if all their results have been
returned, or if they have timed out.

All idle cursors can be forcefully purged at the same time using `FT.CURSOR GC idx 0` command.
By default, Redis uses a lazy throttled approach to garbage collection, which
collects idle cursors every 500 operations, or every second, whichever is later.

code:
FT.AGGREGATE
  {index_name:string}
  {query_string:string}
  [VERBATIM]
  [LOAD {nargs:integer} {property:string} ...]
  [GROUPBY
    {nargs:integer} {property:string} ...
    REDUCE
      {FUNC:string}
      {nargs:integer} {arg:string} ...
      [AS {name:string}]
    ...
  ] ...
  [SORTBY
    {nargs:integer} {string} ...
    [MAX {num:integer}] ...
  ] ...
  [APPLY
    {EXPR:string}
    AS {name:string}
  ] ...
  [FILTER {EXPR:string}] ...
  [LIMIT {offset:integer} {num:integer} ] ...
  [PARAMS {nargs} {name} {value} ... ]

section_id: aggregate-request-format

code: FT.AGGREGATE myIndex "*"

code:
FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour

  GROUPBY 1 @hour
  	REDUCE COUNT_DISTINCT 1 @user_id AS num_users

  SORTBY 2 @hour ASC

  APPLY timefmt(@hour) AS hour

code:
FT.AGGREGATE myIndex "@url:\"about.html\""
    APPLY "@timestamp - (@timestamp % 86400)" AS day
    GROUPBY 2 @day @country
    	REDUCE count 0 AS num_visits
    SORTBY 4 @day ASC @country DESC

code: REDUCE COUNT 0

section_id: supported-groupby-reducers

code: REDUCE COUNT_DISTINCT 1 {property}

code: REDUCE COUNT_DISTINCTISH 1 {property}

code: REDUCE SUM 1 {property}

code: REDUCE MIN 1 {property}

code: REDUCE MAX 1 {property}

code: REDUCE AVG 1 {property}

code: REDUCE STDDEV 1 {property}

code: REDUCE QUANTILE 2 {property} {quantile}

code: REDUCE TOLIST 1 {property}

code: REDUCE FIRST_VALUE {nargs} {property} [BY {property} [ASC|DESC]]

code: REDUCE FIRST_VALUE 4 @name BY @age DESC

code: REDUCE RANDOM_SAMPLE {nargs} {property} {sample_size}

code: FT.AGGREGATE myIdx "*"  LOAD 1 location  APPLY "geodistance(@location,\"-1.1,2.2\")" AS dist

section_id: list-of-geo-apply-functions

code: FT.AGGREGATE idx "*" LOAD 1 @location FILTER "exists(@location)" APPLY "geodistance(@location,-117.824722,33.68590)" AS dist SORTBY 2 @dist DESC

code:
FT.AGGREGATE
  ...
  FILTER "@name=='foo' && @age < 20"
  ...

section_id: filter-expressions

code:
FT.AGGREGATE ... WITHCURSOR [COUNT {read size} MAXIDLE {idle timeout}]
FT.CURSOR READ {idx} {cid} [COUNT {read size}]
FT.CURSOR DEL {idx} {cid}

section_id: cursor-api

code: FT.AGGREGATE idx * WITHCURSOR

code: FT.CURSOR READ idx 342459320

code:
response, cursor = FT.AGGREGATE "idx" "redis" "WITHCURSOR";
while (1) {
  processResponse(response)
  if (!cursor) {
    break;
  }
  response, cursor = FT.CURSOR read "idx" cursor
}

code: FT.AGGREGATE idx query WITHCURSOR COUNT 10

section_id: cursor-settings

code: FT.CURSOR READ idx 342459320 COUNT 50

code: FT.AGGREGATE idx query WITHCURSOR MAXIDLE 10000

code: FT.CURSOR DEL idx 342459320

section_id: other-cursor-commands


title: FT.AGGREGATE order of operations

summary: Order of operations for the FT.AGGREGATE command

title: Overview

text:
[`FT.AGGREGATE`]() is a powerful Redis Search command for performing advanced data aggregation, filtering, sorting, and transformations on indexed hash or JSON documents. This reference page provides a structured breakdown of syntax, ordering rules, and best practices.

The [main aggregations page]() has a simple diagram showing how `FT.AGGREGATE` pipelines are constructed, but it doesn't tell the whole story. For example, you can create more complex aggregation pipelines by applying multiple `REDUCE` functions under a single `GROUPBY` clause, or you can chain groupings and mix in additional mapping steps:

`GROUPBY` ... `REDUCE` ... `APPLY` ... `GROUPBY` ... `REDUCE`

The examples on this page are based on a hypothetical "products" data set, which you can [download here](./data/products.txt).

title: Syntax and expression ordering

role: syntax

text:
The `FT.AGGREGATE` command processes multiple expressions in a pipeline. Below is the recommended order:

1. `index` – the name of your index, which must be the first argument.
1. `query` – your query, which must be the second argument.
1. `FILTER` – filters raw documents before transformations or aggregation.
1. `LOAD` – loads document fields.
1. `APPLY` – applies transformations on fields.
1. `GROUPBY` – groups results by specific fields.
1. `REDUCE` – performs aggregations. For example, `SUM`, `COUNT`, and `AVG`.
1. `SORTBY` – orders the results based on specified fields.
1. `LIMIT` – restricts the number of results returned.
1. `DIALECT 2` - provides for more comprehensive query syntax, for example using parameters in `FILTER` expressions.

Other keywords will be discussed toward the end of this page.

title: When to use `@`

text:
You must add `@` at the start of a field name in the following circumstances:

- When referencing fields loaded from documents. In the following example, `price` is a document field and must be prefixed with `@`.

[code example]

- When referencing fields inside a `FILTER` clause that were loaded from documents.

[code example]

- When referencing fields inside `GROUPBY` or `REDUCE` clauses.

[code example]

- When referencing fields created by `REDUCE` in `APPLY` or `FILTER` clauses.

[code example]

- When referencing fields created by `APPLY` in another `APPLY` or `FILTER` clause.

[code example]

- When referencing fields created by `APPLY` in a `SORTBY` clause.

[code example]

title: GROUPBY with multiple REDUCE operations

text:
You can use multiple `REDUCE` operations after `GROUPBY` for different aggregations.

[code example]

title: Multiple APPLY operations followed by GROUPBY and REDUCE

text:
You can use `APPLY` in various ways before and after `GROUPBY` and `REDUCE`.

[code example]

title: FILTER and PARAMS

text:
Use `FILTER` to remove unwanted records, and `PARAMS` to pass values to parameterized queries.

[code example]

title: Placement of FILTER before and after GROUPBY/APPLY

text:
- **Before GROUPBY:** Removes records before aggregation.
- **After GROUPBY:** Filters based on aggregated results.

title: LOAD after GROUPBY/REDUCE

text: This is not allowed and you'll get a syntax error.

title: Placement rules for specific parameters

text:
| Parameter    | Placement                                                 |
|-----         |-----                                                      |
| `TIMEOUT`    | Can be placed anywhere.                                   |
| `LIMIT`      | Must be at the end, before `DIALECT`.                     |
| `WITHCURSOR` | Must be at the end, before `DIALECT`.                     |
| `SCORER`     | Must be placed between the query and pipeline operations. |
| `ADDSCORES`  | Must be placed between the query and pipeline operations. |
| `DIALECT`    | Must be at the end.                                       |

title: LIMIT and WITHCURSOR used together

role: limits

text:
While you wouldn't ordinarily use `LIMIT` and `WITHCURSOR` together in the same query, you can do so if necessary.
`LIMIT`, as the name suggests, will limit the total number of results returned for the given query. `WITHCURSOR` will paginate the results in chunks of size `COUNT`. You can use the [cursor API]() to retrieve more results, as shown below.

[code example]

See the following resources for more information:

- [Aggregations]() discussion page.
- [`FT.AGGREGATE` command page](https://redis.io/docs/latest/commands/ft.aggregate/)
- [Redis Search source code](https://github.com/RediSearch/RediSearch/tree/master/src/aggregate)

code:
FT.AGGREGATE products "*"
  LOAD 1 @price
  APPLY "@price * 1.1" AS adjusted_price
  SORTBY 2 @adjusted_price DESC
  LIMIT 0 10

 1) (integer) 200
 2) 1) "price"
    2) "623"
    3) "adjusted_price"
    4) "685.3"
 3) 1) "price"
    2) "619.75"
    3) "adjusted_price"
    4) "681.725"
  .
  .
  .

section_id: when-to-use

code:
FT.AGGREGATE products "*"
  LOAD 1 @rating
  FILTER "@rating >= 4.5"
  LIMIT 0 10

 1) (integer) 5
 2) 1) "rating"
    2) "4.5"
 3) 1) "rating"
    2) "4.8"
 4) 1) "rating"
    2) "4.5"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE SUM 1 @price AS total_price
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Toys"
   3) "total_price"
   4) "9799.25"
3) 1) "category"
   2) "Electronics"
   3) "total_price"
   4) "10683.5"
4) 1) "category"
   2) "Apparel"
   3) "total_price"
   4) "10273.5"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE SUM 1 @price AS total_price
  APPLY "@total_price * 1.2" AS boosted_price
  FILTER "@total_price > 1000"
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Toys"
   3) "total_price"
   4) "9799.25"
   5) "boosted_price"
   6) "11759.1"
3) 1) "category"
   2) "Electronics"
   3) "total_price"
   4) "10683.5"
   5) "boosted_price"
   6) "12820.2"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  LOAD 2 @price @discount
  APPLY "@price - @discount" AS net_price
  APPLY "@net_price * 1.1" AS marked_up
  FILTER "@net_price > 200"
  LIMIT 0 10

1) (integer) 60
2) 1) "price"
   2) "220"
   3) "discount"
   4) "0"
   5) "net_price"
   6) "220"
   7) "marked_up"
   8) "242"
3) 1) "price"
   2) "223.25"
   3) "discount"
   4) "1.5"
   5) "net_price"
   6) "221.75"
   7) "marked_up"
   8) "243.925"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  LOAD 2 @price @discount
  APPLY "@price - @discount" AS net_price
  SORTBY 2 @net_price DESC
  LIMIT 0 10

 1) (integer) 200
 2) 1) "price"
    2) "623"
    3) "discount"
    4) "6"
    5) "net_price"
    6) "617"
 3) 1) "price"
    2) "619.75"
    3) "discount"
    4) "4.5"
    5) "net_price"
    6) "615.25"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS product_count
  REDUCE SUM 1 @price AS total_price
  REDUCE AVG 1 @rating AS avg_rating
  SORTBY 2 @total_price DESC
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Groceries"
   3) "product_count"
   4) "44"
   5) "total_price"
   6) "13495.25"
   7) "avg_rating"
   8) "3.94090909091"
3) 1) "category"
   2) "Home"
   3) "product_count"
   4) "40"
   5) "total_price"
   6) "11026.75"
   7) "avg_rating"
   8) "3.78"
  .
  .
  .

section_id: groupby-with-multiple-reduce-operations

code:
FT.AGGREGATE products "*"
  LOAD 3 @price @discount @quantity
  APPLY "@price - @discount" AS final_price
  APPLY "@final_price * @quantity" AS total_revenue
  GROUPBY 1 @category
  REDUCE SUM 1 @total_revenue AS total_category_revenue
  SORTBY 2 @total_category_revenue DESC
  LIMIT 0 10

1) (integer) 6
2) 1) "category"
   2) "Groceries"
   3) "total_category_revenue"
   4) "81373"
3) 1) "category"
   2) "Home"
   3) "total_category_revenue"
   4) "55797.5"
  .
  .
  .

code:
FT.AGGREGATE products "*"
  LOAD 3 @price @rating @quantity
  FILTER "@price >= 500"
  FILTER "@rating >= 4.0"
  APPLY "@price * @quantity" AS total_value
  SORTBY 2 @total_value DESC
  LIMIT 0 10
  DIALECT 2

1) (integer) 200
 2) 1) "price"
    2) "606.75"
    3) "rating"
    4) "4.2"
    5) "quantity"
    6) "10"
    7) "total_value"
    8) "6067.5"
 3) 1) "price"
    2) "541.75"
    3) "rating"
    4) "4.5"
    5) "quantity"
    6) "10"
    7) "total_value"
    8) "5417.5"
  .
  .
  .

section_id: filter-and-params

code:
FT.AGGREGATE products "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS product_count
  LIMIT 0 100
  WITHCURSOR COUNT 3

1) 1) (integer) 6
   2) 1) "category"
      2) "Toys"
      3) "product_count"
      4) "28"
   3) 1) "category"
      2) "Electronics"
      3) "product_count"
      4) "31"
   4) 1) "category"
      2) "Apparel"
      3) "product_count"
      4) "36"
2) (integer) 89400486
127.0.0.1:6379> FT.CURSOR READ products 89400486 COUNT 3
1) 1) (integer) 0
   2) 1) "category"
      2) "Home"
      3) "product_count"
      4) "40"
   3) 1) "category"
      2) "Groceries"
      3) "product_count"
      4) "44"
   4) 1) "category"
      2) "Books"
      3) "product_count"
      4) "21"
2) (integer) 89400486
  .
  .
  .

section_id: limit-and-withcursor-used-together


title: Autocomplete with Redis

summary: Learn how to use the autocomplete feature of Redis for efficient prefix-based suggestion retrieval.

title: Overview

text:
Redis Search provides an autocomplete feature using suggestions that are stored in a [trie-based](https://en.wikipedia.org/wiki/Trie) data structure.
This feature allows you to store and retrieve ranked suggestions based on user input prefixes, making it useful for applications like search boxes, command completion, and chatbot responses.

This guide covers how to use the [`FT.SUGADD`](), [`FT.SUGGET`](), [`FT.SUGDEL`](), and [`FT.SUGLEN`]() commands to implement autocomplete, and some examples of how you can use these commands with [`FT.SEARCH`]().

title: Add autocomplete suggestions

text:
To add phrases or words to a suggestions dictionary, use the [`FT.SUGADD`]() command.
You will assign a score to each entry, which determines its ranking in the results.

[code example]

Integer scores were used in the above examples, but the scores argument is described as being floating point.
Integer scores are converted to floating point internally.
Also, "`autocomplete`" in the above examples is just the name of the key; you can use any key name you wish.

title: Optional arguments

text:
The `FT.SUGADD` command can take two optional arguments:

* `INCR`: increments the existing entry of the suggestion by the given score instead of replacing the score. This is useful for updating the dictionary based on user queries in real time.
* `PAYLOAD`: saves a string with the suggestion, which can be fetched by adding the `WITHPAYLOADS` argument to `FT.SUGGET`.

title: Retrieve suggestions

text:
To get autocomplete suggestions for a given prefix, use the [`FT.SUGGET`]() command.

[code example]

If you wish to see the scores, use the `WITHSCORES` option:

[code example]

title: Enable fuzzy matching

text:
If you want to allow for small spelling mistakes or typos, use the `FUZZY` option. This option performs a fuzzy prefix search, including prefixes at a [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) of 1 from the provided prefix.

[code example]

title: Optional arguments

text:
There are three additional arguments you can use with `FT.SUGGET`:

* `MAX num`: limits the results to a maximum of `num`. The default for `MAX` is 5.
* `WITHSCORES`: returns the score of each suggestion.
* `WITHPAYLOADS`: returns optional payloads saved with the suggestions. If no payload is present for an entry, a `nil` reply is returned.
    [code example]

title: Delete suggestions

text:
To remove a specific suggestion from the dictionary, use the `FT.SUGDEL` command.

[code example]

After deletion, running `FT.SUGGET autocomplete hell FUZZY` will no longer return "help me".

title: Check the number of suggestions

text:
To get a count of the number of entries in a given suggestions dictionary, use the `FT.SUGLEN` command.

[code example]

title: Use autocomplete with search

text:
A common approach is to:

1. Use FT.SUGGET to suggest query completions as users type in a text field.
1. Once the user selects a suggestion, run FT.SEARCH using the selected term to get full search results.

Example workflow

1. Get suggestions for a given user input.

    [code example]
1. Capture the user's selection.
1. Use the selected suggestion in a full-text search.

    [code example]

title: When to use autocomplete versus full-text search

text:
* Use `FT.SUGGET` when you need fast, real-time prefix-based suggestion retrieval.
* Use `FT.SEARCH` when you need document retrieval, filtering, and ranking based on relevance.

title: Autocomplete use cases

text:
The autocomplete feature in Redis Search is useful for:

- **Search box suggestions**: providing live suggestions as users type.
- **Command completion**: offering autocompletion for CLI tools.
- **Product search**: suggesting product names in e-commerce applications.
- **Chatbot responses**: recommending common phrases dynamically.

code:
FT.SUGADD autocomplete "hello world" 100
FT.SUGADD autocomplete "hello there" 90
FT.SUGADD autocomplete "help me" 80
FT.SUGADD autocomplete "hero" 70

section_id: add-autocomplete-suggestions

code:
redis> FT.SUGGET autocomplete "he"
1) "hero"
2) "help me"
3) "hello world"
4) "hello there"

section_id: retrieve-suggestions

code:
redis> FT.SUGGET autocomplete hell FUZZY
1) "hello world"
2) "hello there"
3) "help me"

section_id: enable-fuzzy-matching

code:
redis> FT.SUGADD autocomplete hero 70 PAYLOAD "you're no hero"
    (integer) 4
    redis> FT.SUGGET autocomplete "hr" FUZZY WITHPAYLOADS
    1) "hero"
    2) "you're no hero"
    3) "help me"
    4) (nil)
    5) "hello world"
    6) (nil)
    7) "hello there"
    8) (nil)

section_id: optional-arguments

code:
redis> FT.SUGDEL autocomplete "help me"
(integer) 1

section_id: delete-suggestions

code:
redis> FT.SUGLEN autocomplete
(integer) 3

section_id: check-the-number-of-suggestions

code: FT.SUGGET autocomplete "hel"

section_id: use-autocomplete-with-search

code: FT.SEARCH index "hello world"


title: Chinese support

summary: Chinese support for searching and querying in Redis Open Source

title: Overview

text:
Support for adding documents in Chinese is available starting at version 0.99.0.

Chinese support allows Chinese documents to be added and tokenized using segmentation
rather than simple tokenization using whitespace and/or punctuation.

Indexing a Chinese document is different than indexing a document in most other
languages because of how tokens are extracted. While most languages can have
their tokens distinguished by separation characters and whitespace, this
is not common in Chinese.

Chinese tokenization is done by scanning the input text and checking every
character or sequence of characters against a dictionary of predefined terms,
and determining the most likely match based on the surrounding terms and characters.

Redis makes use of the [Friso](https://github.com/lionsoul2014/friso)
Chinese tokenization library for this purpose. This is largely transparent to
the user and often no additional configuration is required.

title: Example: using chinese in queries

text:
In pseudo-code:

[code example]

Using the Python client:

[code example]

title: Using custom dictionaries

text:
If you wish to use a custom dictionary, you can do so at the module level when
loading the module. The `FRISOINI` setting can point to the location of a
`friso.ini` file that contains the relevant settings and paths to the dictionary
files.

Note that there is no default `friso.ini` file location. RediSearch comes with
its own `friso.ini` and dictionary files, which are compiled into the module
binary at build-time.

code:
FT.CREATE idx ON HASH SCHEMA txt TEXT
HSET docCn txt "Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步，从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]"
FT.SEARCH idx "数据" LANGUAGE chinese HIGHLIGHT SUMMARIZE
# Outputs:
# 数据?... 数据进行写操作。由于完全实现了发布... 数据冗余很有帮助。[8...

section_id: example-using-chinese-in-queries

code:
# -*- coding: utf-8 -*-

from redisearch.client import Client, Query
from redisearch import TextField

client = Client('idx')
try:
    client.drop_index()
except:
    pass

client.create_index([TextField('txt')])

# Add a document
client.add_document('docCn1',
                    txt='Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]',
                    language='chinese')
print client.search(Query('数据').summarize().highlight().language('chinese')).docs[0].txt
# Outputs:
# 数据?... 数据进行写操作。由于完全实现了发布... 数据冗余很有帮助。[8...