Index and query vectors

Learn how to index and query vector embeddings with Redis

Redis Query Engine lets you index vector fields in hash or JSON objects (see the Vectors reference page for more information). Among other things, vector fields can store text embeddings, which are AI-generated vector representations of the semantic information in pieces of text. The vector distance between two embeddings indicates how similar they are semantically. By comparing the similarity of an embedding generated from some query text with embeddings stored in hash or JSON fields, Redis can retrieve documents that closely match the query in terms of their meaning.

In the example below, we use Microsoft.ML to generate the vector embeddings to store and index with Redis Query Engine. We also show how to adapt the code to use Azure OpenAI for the embeddings.

Initialize

The example is probably easiest to follow if you start with a new console app, which you can create using the following command:

dotnet new console -n VecQueryExample

In the app's project folder, add NRedisStack:

dotnet add package NRedisStack

Then, add the Microsoft.ML package.

dotnet add package Microsoft.ML

If you want to try the optional Azure embedding described below, you should also add Azure.AI.OpenAI:

dotnet add package Azure.AI.OpenAI --prerelease

Import dependencies

Add the following imports to your source file:

// Redis connection and Query Engine.
using NRedisStack.RedisStackCommands;
using StackExchange.Redis;
using NRedisStack.Search;
using static NRedisStack.Search.Schema;
using NRedisStack.Search.Literals.Enums;

// Text embeddings.
using Microsoft.ML;
using Microsoft.ML.Transforms.Text;

If you are using the Azure embeddings, also add:

// Azure embeddings.
using Azure;
using Azure.AI.OpenAI;

Define a function to obtain the embedding model

A few steps are involved in initializing the embedding model (known as a PredictionEngine, in Microsoft terminology), so we declare a function to contain those steps together. (See the Microsoft.ML docs for more information about the ApplyWordEmbedding method, including example code.)

Note that we use two classes, TextData and TransformedTextData, to specify the PredictionEngine model. C# syntax requires us to place these classes after the main code in a console app source file. The section Declare TextData and TransformedTextData below shows how to declare them.

static PredictionEngine<TextData, TransformedTextData> GetPredictionEngine(){
    // Create a new ML context, for ML.NET operations. It can be used for
    // exception tracking and logging, as well as the source of randomness.
    var mlContext = new MLContext();

    // Create an empty list as the dataset
    var emptySamples = new List<TextData>();

    // Convert sample list to an empty IDataView.
    var emptyDataView = mlContext.Data.LoadFromEnumerable(emptySamples);

    // A pipeline for converting text into a 150-dimension embedding vector
    var textPipeline = mlContext.Transforms.Text.NormalizeText("Text")
        .Append(mlContext.Transforms.Text.TokenizeIntoWords("Tokens",
            "Text"))
        .Append(mlContext.Transforms.Text.ApplyWordEmbedding("Features",
            "Tokens", WordEmbeddingEstimator.PretrainedModelKind
            .SentimentSpecificWordEmbedding));

    // Fit to data.
    var textTransformer = textPipeline.Fit(emptyDataView);

    // Create the prediction engine to get the embedding vector from the input text/string.
    var predictionEngine = mlContext.Model.CreatePredictionEngine<TextData,
        TransformedTextData>(textTransformer);

    return predictionEngine;
}

Define a function to generate an embedding

Our embedding model represents the vectors as an array of float values, but when you store vectors in a Redis hash object, you must encode the vector array as a byte string. To simplify this, we declare a GetEmbedding() function that applies the PredictionEngine model described above, and then encodes the returned float array as a byte string. If you are storing your documents as JSON objects instead of hashes, then you should use the float array for the embedding directly, without first converting it to a byte string.

static byte[] GetEmbedding(
    PredictionEngine<TextData, TransformedTextData> model, string sentence
)
{
    // Call the prediction API to convert the text into embedding vector.
    var data = new TextData()
    {
        Text = sentence
    };

    var prediction = model.Predict(data);

    // Convert prediction.Features to a binary blob
    float[] floatArray = Array.ConvertAll(prediction.Features, x => (float)x);
    byte[] byteArray = new byte[floatArray.Length * sizeof(float)];
    Buffer.BlockCopy(floatArray, 0, byteArray, 0, byteArray.Length);

    return byteArray;
}

Generate an embedding from Azure OpenAI

Azure OpenAI can be a convenient way to access an embedding model, because you don't need to manage and scale the server infrastructure yourself.

You can create an Azure OpenAI service and deployment to serve embeddings of whatever type you need. Select your region, note the service endpoint and key, and add them where you see placeholders in the function below. See Learn how to generate embeddings with Azure OpenAI for more information.

private static byte[] GetEmbeddingFromAzure(string sentence){
	Uri oaiEndpoint = new ("your-azure-openai-endpoint”);
	string oaiKey = "your-openai-key";

	AzureKeyCredential credentials = new (oaiKey);
	OpenAIClient openAIClient = new (oaiEndpoint, credentials);

	EmbeddingsOptions embeddingOptions = new() {
    	     DeploymentName = "your-deployment-name",
    	     Input = { sentence },
	};

	// Generate the vector embedding
	var returnValue = openAIClient.GetEmbeddings(embeddingOptions);

	// Convert the array of floats to binary blob
	float[] floatArray = Array.ConvertAll(returnValue.Value.Data[0].Embedding.ToArray(), x => (float)x);
	byte[] byteArray = new byte[floatArray.Length * sizeof(float)];
	Buffer.BlockCopy(floatArray, 0, byteArray, 0, byteArray.Length);
	return byteArray;
}

Create the index

Connect to Redis and delete any index previously created with the name vector_idx. (The DropIndex() call throws an exception if the index doesn't already exist, which is why you need the try...catch block.)

var muxer = ConnectionMultiplexer.Connect("localhost:6379");
var db = muxer.GetDatabase();

try { db.FT().DropIndex("vector_idx");} catch {}

Next, create the index. The schema in the example below includes three fields: the text content to index, a tag field to represent the "genre" of the text, and the embedding vector generated from the original text content. The embedding field specifies HNSW indexing, the L2 vector distance metric, Float32 values to represent the vector's components, and 150 dimensions, as required by our embedding model.

The FTCreateParams object specifies hash objects for storage and a prefix doc: that identifies the hash objects we want to index.

var schema = new Schema()
    .AddTextField(new FieldName("content", "content"))
    .AddTagField(new FieldName("genre", "genre"))
    .AddVectorField("embedding", VectorField.VectorAlgo.HNSW,
        new Dictionary<string, object>()
        {
            ["TYPE"] = "FLOAT32",
            ["DIM"] = "150",
            ["DISTANCE_METRIC"] = "L2"
        }
    );

db.FT().Create(
    "vector_idx",
    new FTCreateParams()
        .On(IndexDataType.HASH)
        .Prefix("doc:"),
    schema
);

Add data

You can now supply the data objects, which will be indexed automatically when you add them with HashSet(), as long as you use the doc: prefix specified in the index definition.

Firstly, create an instance of the PredictionEngine model using our GetPredictionEngine() function. You can then pass this to the GetEmbedding() function to create the embedding that represents the content field, as shown below .

(If you are using an Azure OpenAI model for the embeddings, then use GetEmbeddingFromAzure() instead of GetEmbedding(), and note that the PredictionModel is managed by the server, so you don't need to create an instance yourself.)

var predEngine = GetPredictionEngine();

var sentence1 = "That is a very happy person";

HashEntry[] doc1 = {
    new("content", sentence1),
    new("genre", "persons"),
    new("embedding", GetEmbedding(predEngine, sentence1))
};

db.HashSet("doc:1", doc1);

var sentence2 = "That is a happy dog";

HashEntry[] doc2 = {
    new("content", sentence2),
    new("genre", "pets"),
    new("embedding", GetEmbedding(predEngine, sentence2))
};

db.HashSet("doc:2", doc2);

var sentence3 = "Today is a sunny day";

HashEntry[] doc3 = {
    new("content", sentence3),
    new("genre", "weather"),
    new("embedding", GetEmbedding(predEngine, sentence3))
};

db.HashSet("doc:3", doc3);

Run a query

After you have created the index and added the data, you are ready to run a query. To do this, you must create another embedding vector from your chosen query text. Redis calculates the vector distance between the query vector and each embedding vector in the index as it runs the query. We can request the results to be sorted to rank them in order of ascending distance.

The code below creates the query embedding using the GetEmbedding() method, as with the indexing, and passes it as a parameter when the query executes (see Vector search for more information about using query parameters with embeddings). The query is a K nearest neighbors (KNN) search that sorts the results in order of vector distance from the query vector.

(As before, replace GetEmbedding() with GetEmbeddingFromAzure() if you are using Azure OpenAI.)

var res = db.FT().Search("vector_idx",
    new Query("*=>[KNN 3 @embedding $query_vec AS score]")
    .AddParam("query_vec", GetEmbedding(predEngine, "That is a happy person"))
    .ReturnFields(
        new FieldName("content", "content"),
        new FieldName("score", "score")
    )
    .SetSortBy("score")
    .Dialect(2));

foreach (var doc in res.Documents) {
    var props = doc.GetProperties();
    var propText = string.Join(
        ", ",
        props.Select(p => $"{p.Key}: '{p.Value}'")
    );

    Console.WriteLine(
        $"ID: {doc.Id}, Properties: [\n  {propText}\n]"
    );
}

Declare TextData and TransformedTextData

As we noted in the section above about the embedding model, we must declare two very simple classes at the end of the source file. These are required because the API that generates the model expects classes with named fields for the input string and output float array.

class TextData
{
    public string Text { get; set; }
}

class TransformedTextData : TextData
{
    public float[] Features { get; set; }
}

Run the code

Assuming you have added the code from the steps above to your source file, it is now ready to run, but note that it may take a while to complete when you run it for the first time (which happens because the tokenizer must download the embedding model data before it can generate the embeddings). When you run the code, it outputs the following result text:

ID: doc:1, Properties: [
  score: '4.30777168274', content: 'That is a very happy person'
]
ID: doc:2, Properties: [
  score: '25.9752807617', content: 'That is a happy dog'
]
ID: doc:3, Properties: [
  score: '68.8638000488', content: 'Today is a sunny day'
]

The results are ordered according to the value of the score field, which represents the vector distance here. The lowest distance indicates the greatest similarity to the query. As you would expect, the result for doc:1 with the content text "That is a very happy person" is the result that is most similar in meaning to the query text "That is a happy person".

Learn more

See Vector search for more information about the indexing options, distance metrics, and query format for vectors.