{
  "id": "transform-examples",
  "title": "Job files",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/",
  "summary": "Learn how to configure job files for data transformation.",
  "tags": [
    "docs",
    "integrate",
    "rs",
    "rdi"
  ],
  "last_updated": "2026-04-01T08:10:08-05:00",
  "children": [
    {
      "id": "redis-row-format",
      "summary": "",
      "title": "Row Format",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format/"
    },
    {
      "id": "redis-sql-case-example",
      "summary": "",
      "title": "Using SQL CASE",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-sql-case-example/"
    },
    {
      "id": "redis-hash-example",
      "summary": "",
      "title": "Write to a Redis hash",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-hash-example/"
    },
    {
      "id": "redis-json-example",
      "summary": "",
      "title": "Write to a Redis JSON document",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example/"
    },
    {
      "id": "redis-set-example",
      "summary": "",
      "title": "Write to a Redis set",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-set-example/"
    },
    {
      "id": "redis-sorted-set-example",
      "summary": "",
      "title": "Write to a Redis sorted set",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-sorted-set-example/"
    },
    {
      "id": "redis-stream-example",
      "summary": "",
      "title": "Write to a Redis stream",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-stream-example/"
    },
    {
      "id": "redis-string-example",
      "summary": "",
      "title": "Write to a Redis string",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-string-example/"
    },
    {
      "id": "redis-add-field-example",
      "summary": "",
      "title": "Add new fields to a key",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example/"
    },
    {
      "id": "formatting-date-and-time-values",
      "summary": "",
      "title": "Formatting date and time values",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values/"
    },
    {
      "id": "remapping-the-output",
      "summary": "",
      "title": "Remapping the fields in the output",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output/"
    },
    {
      "id": "redis-remove-field-example",
      "summary": "",
      "title": "Remove fields from a key",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example/"
    },
    {
      "id": "map-example",
      "summary": "",
      "title": "Restructure JSON or hash objects",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/map-example/"
    },
    {
      "id": "redis-expiration-example",
      "summary": "",
      "title": "Set custom expiration times / TTL",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example/"
    },
    {
      "id": "redis-set-key-name",
      "summary": "",
      "title": "Set the key name in the target database",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-set-key-name/"
    },
    {
      "id": "redis-opcode-example",
      "summary": "",
      "title": "Using the operation code",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example/"
    },
    {
      "id": "redis-writing-to-multiple-keys",
      "summary": "",
      "title": "Writing to multiple keys",
      "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-writing-to-multiple-keys/"
    }
  ],
  "page_type": "content",
  "content_hash": "e4fea8a6abe878691235aa4657e2fa66ca2707e798c9d5965ad8f78be9cef9b0",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "You can optionally supply one or more job files that specify how you want to\ntransform the captured data before writing it to the target.\nEach job file contains a YAML\nconfiguration that controls the transformation for a particular table from the source\ndatabase. You can also add a `default-job.yaml` file to provide\na default transformation for tables that don't have a specific job file of their own.\n\nThe job files have a structure like the following example. This configures a default\njob that:\n\n- Writes the data to a Redis hash\n- Adds a field `app_code` to the hash with a value of `foo`\n- Adds a prefix of `aws` and a suffix of `gcp` to the key\n\n[code example]\n\nThe main sections of these files are:\n\n- `source`: This is a mandatory section that specifies the data items that you want to \n  use. You can add the following properties here:\n  - `server_name`: Logical server name (optional).\n  - `db`: Database name (optional). This refers to a database name you supplied in\n    [config.yaml]().\n  - `schema`: Database schema (optional). This refers to a schema name you supplied in\n    [config.yaml]().\n  - `table`: Database table name. This refers to a table name you supplied in `config.yaml`. The default\n  job doesn't apply to a specific table, so use \"*\" in place of the table name for this job only.\n  - `row_format`: Format of the data to be transformed. This can take the values `partial` (default) to\n  use only the payload data, or `full` to use the complete change record. See the `transform` section below\n  for details of the extra data you can access when you use the `full` option.\n  - `case_insensitive`: This applies to the `server_name`, `db`, `schema`, and `table` properties\n  and is set to `true` by default. Set it to `false` if you need to use case-sensitive values for these\n  properties.\n\n- `transform`: This is an optional section describing the transformation that the pipeline\n  applies to the data before writing it to the target. The `uses` property specifies a\n  *transformation block* that will use the parameters supplied in the `with` section. See the \n  [data transformation reference]()\n  for more details about the supported transformation blocks. See also the\n  [JMESPath custom functions]()\n  reference and [JMESPath functions proposal](https://jmespath.org/proposals/functions.html) for \n  full details of the functions available for the `expression` field.\n  You can test your transformation logic using the\n  [dry run]()\n  feature in the API.\n\n  If you set `row_format` to `full` under the `source` settings, you can access extra data from the\n  change record in the transformation:\n  - Use the `key` object to access the attributes of the key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.\n  - Use `before.<FIELD_NAME>` to get the value of a field *before* it was updated in the source database\n  - Use `after.<FIELD_NAME>` to get the value of a field *after* it was updated in the source database\n  - Use `after.<FIELD_NAME>` when adding new fields during transformations\n  \n  See [Row Format]() for a more detailed explanation of the full format.\n  \n \n- `output`: This is a mandatory section to specify the data structure(s) that\n  RDI will write to\n  the target along with the text pattern for the key(s) that will access it.\n  Note that you can map one record to more than one key in Redis or nest\n  a record as a field of a JSON structure (see\n  [Data denormalization]()\n  for more information about nesting). You can add the following properties in the `output` section:\n  - `uses`: This must have the value `redis.write` to specify writing to a Redis data\n  structure. You can add more than one block of this type in the same job.\n  - `with`:\n    - `connection`: Connection name as defined in `config.yaml` (by default, the connection named `target` is used).\n    - `data_type`: Target data structure when writing data to Redis. The supported types are `hash`, `json`, `set`,\n   `sorted_set`, `stream` and `string`.\n    - `key`: This lets you override the default key for the data structure with custom logic:\n      - `expression`: Expression to generate the key.\n      - `language`: Expression language, which must be `jmespath` or `sql`.\n    - `expire`: Positive integer value or SQL/JMESPath expression indicating a number of seconds\n      for the key to expire. If you don't specify this property, the key will never expire.\n      See [Set custom expiration times / TTL]() for more information and examples.\n\nIn a job file, the `transform` section is optional, but if you don't specify\na `transform`, you must specify custom key logic in `output.with.key`. You can include\nboth of these sections if you want both a custom transform and a custom key.\n\nAnother example below shows how you can rename the `fname` field to `first_name` in the table `emp`\nusing the\n[`rename_field`]() block. It also demonstrates how you can set the key of this record instead of relying on\nthe default logic.\n\n[code example]\n\nSee the\n[RDI configuration file]()\nreference for full details about the\navailable source, transform, and target configuration options and see\nalso the\n[data transformation reference]()\nfor details of all the available transformation blocks.\n\nWhen using the `sql` option as language for the expressions keep in mind that RDI uses the SQL syntax and\nfunctions supported by SQLite and those may differ from the ANSI-SQL ones. You can find more details in SQLite's\n[official documentation](https://sqlite.org/lang.html)."
    },
    {
      "id": "examples",
      "title": "Examples",
      "role": "example",
      "text": "The pages listed below show examples of typical job files for different use cases."
    }
  ],
  "examples": [
    {
      "id": "overview-ex0",
      "language": "yaml",
      "code": "name: Default job with prefix and suffix\nsource:\n  table: \"*\"\n  row_format: full\ntransform:\n  - uses: add_field\n    with:\n      fields:\n        - field: after.app_code\n          expression: \"`foo`\"\n          language: jmespath\noutput:\n  - uses: redis.write\n    with:\n      data_type: hash\n      key:\n        expression: concat(['aws', '#', table, '#', keys(key)[0], '#', values(key)[0], '#gcp'])\n        language: jmespath",
      "section_id": "overview"
    },
    {
      "id": "overview-ex1",
      "language": "yaml",
      "code": "name: Rename field example\nsource:\n  server_name: redislabs\n  schema: dbo\n  table: emp\ntransform:\n  - uses: rename_field\n    with:\n      from_field: fname\n      to_field: first_name\noutput:\n  - uses: redis.write\n    with:\n      connection: target\n      key:\n        expression: concat(['emp:fname:',fname,':lname:',lname])\n        language: jmespath",
      "section_id": "overview"
    }
  ]
}