For developersUsing Redis OM .NET to work with JSON and Hashes in Redis
Redis OM provides high-level abstractions for using Redis in .NET, making it easy to model and query your Redis domain objects.
Redis OM contains the following features:
- Declarative object mapping for Redis objects
- Declarative secondary-index generation
- Fluent APIs for querying Redis
- Fluent APIs for performing Redis aggregations
#Add and retrieve objects using Redis OM .NET
The Redis OM library supports declarative storage and retrieval of objects from Redis. You will still use the Document Attribute to decorate a class you'd like to store in Redis. From there, all you need to do is either call Insert or InsertAsync on the RedisCollection or Set or SetAsync on the RedisConnection, passing in the object you want to set in Redis. You can then retrieve those objects with Get or GetAsync with the RedisConnection or with FindById or FindByIdAsync in the RedisCollection.
The Code above will declare an Employee class, and allow you to add employees to Redis, and then retrieve Employees from Redis the output from this method will look like this:
If you wanted to find them in Redis directly you could run HGETALL Employee:01FHDFE115DKRWZW0XNF17V2RK and that will retrieve the Employee object as a Hash from Redis. If you do not specify a prefix, the prefix will be the fully-qualified class name.
#Creating an index with Redis OM .NET
To unlock some of the nicest functionality of Redis OM, e.g., running searches, matches, aggregations, reductions, mappings, etc... You will need to tell Redis how you want data to be stored and how you want it indexed. One of the features the Redis OM library provides is creating indices that map directly to your objects by declaring the indices as attributes on your class.
Let's start with an example class.
As shown above, you can declare a class as being indexed with the Document Attribute. In the Document attribute, you can set a few fields to help build the index:
| Property Name | Description | Default | Optional |
| -------------------- | :------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | ------------------ | ---- |
| StorageType | Defines the underlying data structure used to store the object in Redis, options are HASH and JSON | HASH | true |
| IndexName | The name of the index |
$"{SimpleClassName | .ToLower()}-idx} | true |
| Prefixes | The key prefixes for redis to build an index off of | new string[]{$"{FullyQualifiedClassName}:"} | true |
| Language | Language to use for full-text search indexing | null | true |
| LanguageField | The name of the field in which the document stores its Language | null | true |
| Filter | The filter to use to determine whether a particular item is indexed, e.g. @Age>=18 | null | true |
| IdGenerationStrategy | The strategy used to generate Ids for documents, if left blank it will use a ULID generation strategy | UlidGenerationStrategy | true |#Field level declarations
#Id fields
Every class indexed by Redis must contain an Id Field marked with the
RedisIdField.#Indexed fields
In addition to declaring an Id Field, you can also declare indexed fields, which will let you search for values within those fields afterward. There are two types of Field level attributes.
- Indexed - This type of index is valid for fields that are of the type string, a Numeric type (double/int/float etc. . .), or can be decorated for fields that are of the type GeoLoc, the exact way that the indexed field is interpreted depends on the indexed type
- Searchable - This type is only valid for string fields, but this enables full-text search on the decorated fields.
#IndexedAttribute properties
There are properties inside the IndexedAttribute that let you further customize how things are stored & queried.
| PropertyName | type | Description | Default | Optional |
|---|---|---|---|---|
| PropertyName | string | The name of the property to be indexed | The name of the property being indexed | true |
| Sortable | bool | Whether to index the item so it can be sorted on in queries, enables use of OrderBy & OrderByDescending -> collection.OrderBy(x=>x.Email) | false | true |
| Normalize | bool | Only applicable for string type fields Determines whether the text in a field is normalized (sent to lower case) for purposes of sorting | true | true |
| Separator | char | Only applicable for string type fields Character to use for separating tag field, allows the application of multiple tags fo the same item e.g. article.Category = technology,parenting is delineated by a , means that collection.Where(x=>x.Category == "technology") and collection.Where(x=>x.Category == "parenting") will both match the record | , | true |
| CaseSensitive | bool | Only applicable for string type fields - Determines whether case is considered when performing matches on tags | false | true |
#SearchableAttribute properties
There are properties for the SearchableAttribute that let you further customize how the full-text search determines matches
| PropertyName | type | Description | Default | Optional |
|---|---|---|---|---|
| PropertyName | string | The name of the property to be indexed | The name of the indexed property | true |
| Sortable | bool | Whether to index the item so it can be sorted on in queries, enables use of OrderBy & OrderByDescending -> collection.OrderBy(x=>x.Email) | false | true |
| NoStem | bool | Determines whether to use stemming, in other words adding the stem of the word to the index, setting to true will stop the Redis from indexing the stems of words | false | true |
| PhoneticMatcher | string | The phonetic matcher to use if you'd like the index to use (PhoneticMatching)[https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/phonetic_matching/] with the index | null | true |
| Weight | double | determines the importance of the field for checking result accuracy | 1.0 | true |
#Creating the index
After declaring the index, the creation of the index is pretty straightforward. All you have to do is call CreateIndex for the decorated type. The library will take care of serializing the provided type into a searchable index. The library does not try to be particularly clever, so if the index already exists it will the creation request will be rejected, and you will have to drop and re-add the index (migrations is a feature that may be added in the future)
#Text searches in Redis OM .NET
The
RedisCollection provides a fluent interface for querying objects stored in redis. This means that if you store an object in Redis with the Redis OM library you can query objects stored in Redis with ease using the LINQ syntax you're used to.#Define the model
Let's start off by defining a model that we will be using for querying, we will use a
Employee Class which will have some basic stuff we may want to query in it#Connect to Redis
Now we will initialize a RedisConnectionProvider, and grab a handle to a RedisCollection for Employee
#Create our index
Next we'll create the index, so next in our Main method, let's take our type and condense it into an index
#Seed some data
Next we'll seed a few piece of data in our database to play around with:
#Simple text query of an indexed field
With these data inserted into our database, we can now go ahead and begin querying. Let's start out by trying to query people by name. We can search for all employees named
Susan with a simple Where predicate:The
Where Predicates also support and/or operators, e.g. to find all employees named Alice or Bob you can use:#Limiting result object fields
When you are querying larger Documents in Redis, you may not want to have to drag back the entire object over the network, in that case you can limit the results to only what you want using a
Select predicate. E.g. if you only wanted to find out the ages of employees, all you would need to do is select the age of employees:Or if you want to select more than one field you can create a new anonymous object:
#Limiting returned objects
You can limit the size of your result (in the number of objects returned) with
Skip & Take predicates. Skip will skip over the specified number of records, and Take will take only the number of records provided (at most);#Full text search
There are two types of attributes that can decorate strings,
Indexed, which we've gone over and Searchable which we've yet to discuss. The Searchable attribute considers equality slightly differently than Indexed, it operates off a full-text search. In expressions involving Searchable fields, equality—==— means a match. A match in the context of a searchable field is not necessarily a full exact match but rather that the string contains the search text. Let's look at some examples.#Find employee's in sales
So we have a
Department string which is marked as Searchable in our Employee class. Notice how we've named our departments. They contain a region and a department type. If we wanted only to find all employee's in Sales we could do so with:This will produce:
Because they are all folks in departments called
salesIf you wanted to search for everyone in a department in
EMEA you could search with:Which of course would produce:
#Sorting
If a
Searchable or Indexed field is marked as Sortable, or Aggregatable, you can order by that field using OrderBy predicates.#Performing numeric queries with Redis OM .NET
In addition to providing capabilities for text queries, Redis OM also provides you the ability to perform numeric equality and numeric range queries. Let us assume a model of:
Assume that we've connected to Redis already and retrieved a
RedisCollection and seeded some data as such:We can now perform queries against the numeric values in our data as you would with any other collection using LINQ expressions.
You can of course also pair numeric queries with Text Queries:
#Sorting
If an
Indexed field is marked as Sortable, or Aggregatable, you can order by that field using OrderBy predicates.#Aggregations with Redis OM .NET
Aggregations are a method of grouping documents together and run processing on them on the server to transform them into data that you need in your application, without having to perform the computation client-side.
#Anatomy of a Pipeline
Aggregations in Redis are build around an aggregation pipeline, you will start off with a RedisAggregationSet of objects that you have indexed in Redis. From there you can
- Query to filter down the results you want
- Apply functions to them to combine functions to them
- Group like features together
- Run reductions on groups
- Sort records
- Further filter down records
#Setting up for Aggregations
Redis OM .NET provides an RedisAggregationSet class that will let you perform aggregations on employees, let's start off with a trivial aggregation. Let's start off by defining a model:
We'll then create the index for that model, pull out a RedisAggregationSet from our provider, and initialize the index, and seed some data into our database
#The AggregationResult
The Aggregations pipeline is all built around the
RedisAggregationSet<T> this Set is generic, so you can provide the model that you want to build your aggregations around (an Indexed type), but you will notice that the return type from queries to the RedisAggregationSet is the generic type passed into it. Rather it is an AggregationResult<T> where T is the generic type you passed into it. This is a really important concept, when results are returned from aggregations, they are not hydrated into an object like they are with queries. That's because Aggregations aren't meant to pull out your model data from the database, rather they are meant to pull out aggregated results. The AggregationResult has a RecordShell field, which is ALWAYS null outside of the pipeline. It can be used to build expressions for querying objects in Redis, but when the AggregationResult lands, it will not contain a hydrated record, rather it will contain a dictionary of Aggregations built by the Aggregation pipeline. This means that you can access the results of your aggregations by indexing into the AggregationResult.#Simple Aggregations
Let's try running an aggregation where we find the Sum of the sales for all our employees in EMEA. So the Aggregations Pipeline will use the
RecordShell object, which is a reference to the generic type of the aggregation set, for something as simple as a group-less SUM, you will simply get back a numeric type from the aggregation.The
Where expression tells the aggregation pipeline which records to consider, and subsequently the SUM expression indicates which field to sum. Aggregations are a rich feature and this only scratches the surface of it, these pipelines are remarkably flexible and provide you the ability to do all sorts of neat operations on your Data in Redis.#Apply functions
Apply functions are functions that you can define as expressions to apply to your data in Redis. In essence, they allow you to combine your data together, and extract the information you want.
#Apply functions data model
For the remainder of this article we will be using this data model:
#Anatomy of an apply function
Apply is a method on the RedisAggregationSet<T> class which takes two arguments, each of which is a component of the apply function.First it takes the expression that you want Redis to execute on every record in the pipeline, this expression takes a single parameter, an
AggregationResult<T>, where T is the generic type of your RedisAggregationSet. This AggregationResult has two things we should think about, first it contains a RecordShell which is a placeholder for the generic type, and secondly it has an Aggregations property - which is a dictionary containing the results from your pipeline. Both of these can be used in apply functions.The second component is the alias, that's the name the result of the function is stored in when the pipeline executes.
#Adjusted sales
Our data model has two properties related to sales,
Sales, how much the employee has sold, and SalesAdjustment, a figure used to adjust sales based off various factors, perhaps territory covered, experience, etc. . . The idea being that perhaps a fair way to analyze an employee's performance is a combination of these two fields rather than each individually. So let's say we wanted to find what everyone's adjusted sales were, we could do that by creating an apply function to calculate it.#Arithmetic apply functions
Functions that use arithmetic and math can use the mathematical operators
+ for addition, - for subtraction, * for multiplication, / for division, and % for modular division, also the ^ operator, which is typically used for bitiwise exclusive-or operations, has been reserved for power functions. Additionally, you can use many System.Math library operations within Apply functions, and those will be translated to the appropriate methods for use by Redis.#Available math functions
| Function | Type | Description | Example |
|---|---|---|---|
| Log10 | Math | yields the 10 base log for the number | Math.Log10(x["AdjustedSales"]) |
| Abs | Math | yields the absolute value of the provided number | Math.Abs(x["AdjustedSales"]) |
| Ceil | Math | yields the smallest integer not less than the provided number | Math.Ceil(x["AdjustedSales"]) |
| Floor | Math | yields the smallest integer not greater than the provided number | Math.Floor(x["AdjustedSales"]) |
| Log | Math | yields the Log base 2 for the provided number | Math.Log(x["AdjustedSales"]) |
| Exp | Math | yields the natural exponent for the provided number (e^y) | Math.Exp(x["AdjustedSales"]) |
| Sqrt | Math | yields the Square root for the provided number | Math.Sqrt(x["AdjustedSales"]) |
#String functions
You can also apply multiple string functions to your data, if for example you wanted to create a birthday message for each employee you could do so by calling String.Format on your records:
#List of string functions
| Function | Type | Description | Example |
|---|---|---|---|
| ToUpper | String | yields the provided string to upper case | x.RecordShell.Name.ToUpper() |
| ToLower | String | yields the provided string to lower case | x.RecordShell.Name.ToLower() |
| StartsWith | String | Boolean expression - yields 1 if the string starts with the argument | x.RecordShell.Name.StartsWith("bob") |
| Contains | String | Boolean expression - yields 1 if the string contains the argument | x.RecordShell.Name.Contains("bob") |
| Substring | String | yields the substring starting at the given 0 based index, the length of the second argument, if the second argument is not provided, it will simply return the balance of the string | x.RecordShell.Name.Substring(4, 10) |
| Format | String | Formats the string based off the provided pattern | string.Format("Hello {0} You are {1} years old", x.RecordShell.Name, x.RecordShell.Age) |
| Split | String | Split's the string with the provided string - unfortunately if you are only passing in a single splitter, because of how expressions work, you'll need to provide string split options so that no optional parameters exist when building the expression, just pass StringSplitOptions.None | x.RecordShell.Name.Split(",", StringSplitOptions.None) |
#Time functions
You can also perform functions on time data in Redis. If you have a timestamp stored in a useable format, a unix timestamp or a timestamp string that can be translated from strftime, you can operate on them. For example if you wanted to translate a unix timestamp to YYYY-MM-DDTHH:MM::SSZ you can do so by just calling ApplyFunctions.FormatTimestamp on the record inside of your apply function. E.g.
#Time functions available
| Function | Type | Description | Example |
|---|---|---|---|
| ApplyFunctions.FormatTimestamp | time | transforms a unix timestamp to a formatted time string based off strftime conventions | ApplyFunctions.FormatTimestamp(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.ParseTime | time | Parsers the provided formatted timestamp to a unix timestamp | ApplyFunctions.ParseTime(x.RecordShell.TimeString, "%FT%ZT") |
| ApplyFunctions.Day | time | Rounds a unix timestamp to the beginning of the day | ApplyFunctions.Day(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.Hour | time | Rounds a unix timestamp to the beginning of current hour | ApplyFunctions.Hour(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.Minute | time | Round a unix timestamp to the beginning of the current minute | ApplyFunctions.Minute(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.Month | time | Rounds a unix timestamp to the beginning of the current month | ApplyFunctions.Month(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.DayOfWeek | time | Converts the unix timestamp to the day number with Sunday being 0 | ApplyFunctions.DayOfWeek(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.DayOfMonth | time | Converts the unix timestamp to the current day of the month (1..31) | ApplyFunctions.DayOfMonth(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.DayOfYear | time | Converts the unix timestamp to the current day of the year (1..31) | ApplyFunctions.DayOfYear(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.Year | time | Converts the unix timestamp to the current year | ApplyFunctions.Year(x.RecordShell.LastTimeOnline) |
| ApplyFunctions.MonthOfYear | time | Converts the unix timestamp to the current year | ApplyFunctions.MonthOfYear(x.RecordShell.LastTimeOnline) |
#Geo distance
Another useful function is the
GeoDistance function, which allows you computer the distance between two points, e.g. if you wanted to see how far away from the office each employee was you could use the ApplyFunctions.GeoDistance function inside your pipeline:#Grouping and reductions with Redis OM .NET
Grouping and reducing operations using aggregations can be extremely powerful.
#What Is a Group
A group is simply a group of like records in Redis.
e.g.
If grouped together by
Department would be one group. When grouped by Name, they would be two groups.#Reductions
What makes groups so useful in Redis Aggregations is that you can run reductions on them to aggregate items within the group. For example, you can calculate summary statistics on numeric fields, retrieve random samples, distinct counts, approximate distinct counts of any aggregatable field in the set.
#Using Groups and Reductions with Redis OM .NET
You can run reductions against an
RedisAggregationSet either with or without a group. If you run a reduction without a group, the result of the reduction will materialize immediately as the desired type. If you run a reduction against a group, the results will materialize when they are enumerated.#Reductions without a Group
If you wanted to calculate a reduction on all the records indexed by Redis in the collection, you would simply call the reduction on the
RedisAggregationSet#Reductions with a Group
If you want to build a group to run reductions on, e.g. you wanted to calculate the average sales in a department, you would use a
GroupBy predicate to specify which field or fields to group by. If you want to group by 1 field, your lambda function for the group by will yield just the field you want to group by. If you want to group by multiple fields, new up an anonymous type in line:From here you can run reductions on your groups. To run a Reduction, execute a reduction function. When the collection materializes the
AggregationResult<T> will have the reduction stored in a formatted string which is the PropertyName_COMMAND_POSTFIX, see supported operations table below for postfixes. If you wanted to calculate the sum of the sales of all the departments you could:| Command Name | Command Postfix | Description |
|---|---|---|
| Count | COUNT | number of records meeting the query, or in the group |
| CountDistinct | COUNT_DISTINCT | Counts the distinct occurrences of a given property in a group |
| CountDistinctish | COUNT_DISTINCTISH | Provides an approximate count of distinct occurrences of a given property in each group - less expensive computationally but does have a small 3% error rate |
| Sum | SUM | The sum of all occurrences of the provided field in each group |
| Min | MIN | Minimum occurrence for the provided field in each group |
| Max | MAX | Maximum occurrence for the provided field in each group |
| Average | AVG | Arithmetic mean of all the occurrences for the provided field in a group |
| StandardDeviation | STDDEV | Standard deviation from the arithmetic mean of all the occurrences for the provided field in each group |
| Quantile | QUANTLE | The value of a record at the provided quantile for a field in each group, e.g., the Median of the field would be sitting at quantile .5 |
| Distinct | TOLIST | Enumerates all the distinct values of a given field in each group |
| FirstValue | FIRST_VALUE | Retrieves the first occurrence of a given field in each group |
| RandomSample | RANDOMSAMPLE{NumRecords} | Random sample of the given field in each group |
#Closing Groups
When you invoke a
GroupBy the type of return type changes from RedisAggregationSet to a GroupedAggregationSet. In some instances you may need to close a group out and use its results further down the pipeline. To do this, all you need to do is call CloseGroup on the GroupedAggregationSet - that will end the group predicates and allow you to use the results further down the pipeline.#Using geo filters in Redis OM .NET
A really nifty bit of indexing you can do with Redis OM is geo-indexing. To GeoIndex, all you need to do is to mark a
GeoLoc field in your model as Indexed and create the indexSo let's create the index and seed some data.
#Querying Based off Location
With our data seeded, we can now run geo-filters on our restaurants data, let's say we had an office (e.g. Redis's offices in Mountain View at
-122.064224,37.377266) and we wanted to find nearby restaurants, we could do so by using a GeoFilter query restaurants within a certain radius, say 1 mile we can: