{ "id": "data-denormalization", "title": "Data denormalization", "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/data-denormalization/", "summary": "Learn about denormalization strategies", "tags": [ "docs", "integrate", "rs", "rdi" ], "last_updated": "2026-04-01T08:10:08-05:00", "page_type": "content", "content_hash": "c317b3a11bf9c02c49190ba577afb82038739b18c1d3cefd47a7df824bcec185", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "The data in the source database is often\n[*normalized*](https://en.wikipedia.org/wiki/Database_normalization).\nThis means that columns can't have composite values (such as arrays) and relationships between entities\nare expressed as mappings of primary keys to foreign keys between different tables.\nNormalized data models reduce redundancy and improve data integrity for write queries but this comes\nat the expense of speed.\nA Redis cache, on the other hand, is focused on making *read* queries fast, so RDI provides data\n*denormalization* to help with this." }, { "id": "joining-one-to-one-relationships", "title": "Joining one-to-one relationships", "role": "content", "text": "You can join one-to-one relationships by making more than one job to write to the same Redis key.\n\nFirst, you must configure the parent entity to use `merge` as the `on_update` strategy.\n\n[code example]\n\nThen, you can configure the child entity to write to the same Redis key as the parent entity. You can do this by using the `key` attribute in the `with` block of the job, as shown in this example:\n\n[code example]\n\nThe joined data will look like this in Redis:\n\n[code example]\n\n\nIf you don't set `merge` as the `on_update` strategy for all jobs targeting the same key, the entire parent record in Redis will be overwritten whenever any related record in the source database is updated. This will result in the loss of values written by other jobs.\n\n\nWhen using this approach, you must ensure that the `key` expression in the child job matches the key expression in the parent job. If you use a different key expression, the child data will not be written to the same Redis key as the parent data.\n\nIn the example above, the `addresses` job uses the default key pattern to write to the same Redis key as the `customers` job. You can find more information about the default key pattern [here]().\n\nYou can also use custom keys for the parent entity, as long as you use the same key for all jobs that write to the same Redis key." }, { "id": "joining-one-to-many-relationships", "title": "Joining one-to-many relationships", "role": "content", "text": "To join one-to-many relationships, you can use the *Nesting* strategy.\nWith this, the parent object (the \"one\") is represented as a JSON document with the children (the \"many\") nested inside it as a JSON map attribute. The diagram below shows a nesting with the child objects in a map called `InvoiceLineItems`:\n\n\n\n\nTo configure normalization, you must first configure the parent entity to use JSON as the target data type. Add `data_type: json` to the parent job as shown in the example below:\n\n[code example]\n\nAfter you have configured the parent entity, you can then configure the child entities to be nested under it, based on their relation type. To do this, use the `nest` block, as shown in this example:\n\n[code example]\n\nThe job has a `with` section under `output` that includes the `nest` block.\nThe job must include the following attributes in the `nest` block:\n\n- `parent`: This specifies the config of the parent entities. You only\n need to supply the parent `table` name. Note that this attribute refers to a Redis *key* that will be added to the target\n database, not to a table you can access from the pipeline. See [Using nesting](#using-nesting) below\n for the format of the key that is generated.\n- `nesting_key`: The unique key of each child entry in the JSON map that will be created under the path.\n- `parent_key`: The field in the parent entity that stores the unique ID (foreign key) of the parent entity. This can't be a composite key.\n- `child_key`: The field in the child entity that stores the unique ID (foreign key) to the parent entity. You only need to add this attribute if the name of the child's foreign key field is different from the parent's. This can't be a composite key.\n- `path`: The [JSONPath](https://goessner.net/articles/JsonPath/)\n for the map where you want to store the child entities. The path must start with the `$` character, which denotes\n the document root.\n- `structure`: (Optional) The type of JSON nesting structure for the child entities. Currently, only a JSON map\n is supported so if you supply this attribute then the value must be `map`." }, { "id": "using-nesting", "title": "Using nesting", "role": "content", "text": "There are several important things to note when you use nesting:\n\n- When you specify `nest` in the job, you must also set the `data_type` attribute to `json` and\n the `on_update` attribute to `merge` in the surrounding `output` block.\n- Key expressions are *not* supported for the `nest` output blocks. The parent key is always calculated\n using the following template:\n\n [code example]\n \n For example:\n \n [code example]\n\n- If you specify `expire` in the `nest` output block then this will set the expiration on the *parent* object.\n- You can only use one level of nesting.\n- If you are using PostgreSQL then you must make the following change for all child tables that you want to nest:\n \n [code example]\n \n This configuration affects the information written to the write-ahead log (WAL) and whether it is available\n for RDI to capture. By default, PostgreSQL only records\n modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the\n Redis key in the destination database.\n See the\n [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity)\n for more information about this.\n- Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated." } ], "examples": [ { "id": "joining-one-to-one-relationships-ex0", "language": "yaml", "code": "# jobs/customers.yaml\nsource:\n table: customers\n\noutput:\n - uses: redis.write\n with:\n data_type: json\n on_update: merge", "section_id": "joining-one-to-one-relationships" }, { "id": "joining-one-to-one-relationships-ex1", "language": "yaml", "code": "# jobs/addresses.yaml\nsource:\n table: addresses\n\ntransform:\n - uses: add_field\n with:\n field: customer_address\n language: jmespath\n # You can use the following JMESPath expression to create a JSON object and combine the address fields into a single object.\n expression: |\n {\n \"street\": street,\n \"city\": city,\n \"state\": state,\n \"zip\": zip\n }\n\noutput:\n - uses: redis.write\n with:\n data_type: json\n # We specify the key to write to the same key as the parent entity.\n key:\n expression: concat(['customers:id:', customer_id])\n language: jmespath\n on_update: merge\n mapping:\n # You can specify one or more fields to write to the parent entity.\n - customer_address: customer_address", "section_id": "joining-one-to-one-relationships" }, { "id": "joining-one-to-one-relationships-ex2", "language": "json", "code": "{\n \"id\": \"1\",\n \"first_name\": \"John\",\n \"last_name\": \"Doe\",\n \"customer_address\": {\n \"street\": \"123 Main St\",\n \"city\": \"Anytown\",\n \"state\": \"CA\",\n \"zip\": \"12345\"\n }\n}", "section_id": "joining-one-to-one-relationships" }, { "id": "joining-one-to-many-relationships-ex0", "language": "yaml", "code": "# jobs/invoice.yaml\nsource:\n schema: public\n table: Invoice\n\noutput:\n - uses: redis.write\n with:\n # Setting the data type to json ensures that the parent object will be created in a way that supports nesting.\n data_type: json\n # Important: do not set a custom key for the parent entity.\n # When nesting the child object under the parent, the parent key is automatically calculated based on\n # the parent table name and the parent key field and if a custom key is set, it will cause a mismatch\n # between the key used to write the parent and the key used to write the child.", "section_id": "joining-one-to-many-relationships" }, { "id": "joining-one-to-many-relationships-ex1", "language": "yaml", "code": "# jobs/invoice_line.yaml\nsource:\n schema: public\n table: InvoiceLine\noutput:\n - uses: redis.write\n with:\n nest: # cannot co-exist with other parameters such as 'key'\n parent:\n # schema: public\n table: Invoice\n nesting_key: InvoiceLineId # the unique key in the composite structure under which the child data will be stored\n parent_key: InvoiceId\n child_key: InvoiceId # optional, if different from parent_key\n path: $.InvoiceLineItems # path must start from document root ($)\n structure: map # only map supported for now\n on_update: merge # only merge supported for now\n data_type: json # only json supported for now", "section_id": "joining-one-to-many-relationships" }, { "id": "using-nesting-ex0", "language": "bash", "code": "::", "section_id": "using-nesting" }, { "id": "using-nesting-ex1", "language": "bash", "code": "Invoice:InvoiceId:1", "section_id": "using-nesting" }, { "id": "using-nesting-ex2", "language": "sql", "code": "ALTER TABLE REPLICA IDENTITY FULL;", "section_id": "using-nesting" } ] }