Caching expression results

The Flink processor can cache the result of any expression that produces a value (for example, an add_field expression, a map expression, the arguments to a redis.lookup, or a custom output key/expire expression). Caching is useful when the same expression is evaluated repeatedly with the same input field values, for example when many incoming records share a common foreign key.

The cache: block

You enable caching by adding a cache: block next to the expression you want to cache. Cache keys are derived from the values of the input fields referenced by the expression, not from the full record. See cache for the full property list.

Caching an add_field expression

The example below adds a country field whose value is derived from country_code and country_name. When the same combination of input values appears repeatedly (for example, many customers from the same country), caching the result avoids re-evaluating the expression.

name: Cached country field
source:
  schema: dbo
  table: customer
transform:
  - uses: add_field
    with:
      field: country
      language: sql
      expression: country_code || ' - ' || UPPER(country_name)
      cache:
        enabled: true
        max_size: 500
        ttl_seconds: 300

Caching a map expression

name: Cached map expression
source:
  table: customer
transform:
  - uses: map
    with:
      language: jmespath
      expression: |
        {
          "CustomerId": customer_id,
          "Country": country_code
        }
      cache:
        enabled: true

Caching redis.lookup arguments and results

redis.lookup supports two independent caches. The cache: block caches the argument expressions (the JMESPath or SQL expressions that produce the Redis command arguments). The lookup_cache: block caches the result of the Redis command itself, keyed by the resolved arguments. Both blocks accept the same properties as the cache: block above.

name: Cached lookup
source:
  table: order
transform:
  - uses: redis.lookup
    with:
      connection: target
      cmd: HGETALL
      args:
        - concat(['customer:', customer_id])
      language: jmespath
      field: customer
      cache:
        enabled: true
        ttl_seconds: 60
      lookup_cache:
        enabled: true
        max_size: 10000
        ttl_seconds: 300

Caching key and expire output expressions

A cache: block can also be added to the output key and expire expressions when those are dynamic. The properties are the same as above.

name: Cached key expression
source:
  table: order
output:
  - uses: redis.write
    with:
      data_type: hash
      key:
        expression: concat(['order:', order_id])
        language: jmespath
        cache:
          enabled: true