{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake",
  "title": "Prepare Snowflake for RDI",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/snowflake/",
  "summary": "Prepare Snowflake databases to work with RDI",
  "content": "\nThis guide describes the steps required to prepare a Snowflake database as a source for Redis Data Integration (RDI) pipelines.\n\nDuring both the [snapshot](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines#pipeline-lifecycle) and\n[Change data capture (CDC)](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines#pipeline-lifecycle)\nphases, RDI uses [Snowflake Streams](https://docs.snowflake.com/en/user-guide/streams) to read data from the monitored\ntables. For the initial snapshot, RDI creates the stream with `SHOW_INITIAL_ROWS = TRUE` so it can read the current\ntable contents before continuing with ongoing CDC. RDI automatically creates and manages the required streams.\n\n## Setup\n\nThe following checklist shows the steps to prepare a Snowflake database for RDI,\nwith links to the sections that explain the steps in full detail.\nYou may find it helpful to track your progress with the checklist as you\ncomplete each step.\n\n\u003e [!NOTE]\n\u003e Snowflake is only supported with RDI deployed on Kubernetes/Helm. RDI VM mode does not support Snowflake as a source database.\n\n```checklist {id=\"snowflakelist\"}\n- [ ] [Set up Snowflake permissions](#1-set-up-snowflake-permissions)\n- [ ] [Configure authentication](#2-configure-authentication)\n- [ ] [Set up secrets for Kubernetes deployment](#3-set-up-secrets-for-kubernetes-deployment)\n- [ ] [Configure RDI for Snowflake](#4-configure-rdi-for-snowflake)\n```\n\n## 1. Set up Snowflake permissions\n\nThe following are the minimum runtime permissions for the RDI role to read the source tables and create the Snowflake\nobjects RDI uses for CDC:\n\n- `USAGE`, `OPERATE` on the warehouse used for RDI reads\n- `USAGE` on the source database and source schema\n- `SELECT` on the source tables\n- `USAGE` on the CDC schema used by RDI\n- `CREATE STREAM`, `CREATE TABLE` on the CDC schema used by RDI\n\nIf you configure `cdcDatabase` and `cdcSchema`, grant the CDC permissions there. Otherwise, grant them in the source\nschema. If your Snowflake setup requires it, also grant any additional cross-database privileges needed for the CDC\nschema to reference the source tables.\n\n\u003e [!NOTE]\n\u003e RDI manages the Snowflake streams it uses for snapshot and CDC. The collector creates the stream in the configured CDC\n\u003e schema and later issues `CREATE OR REPLACE STREAM` statements to keep the stream aligned with the expected offset, so\n\u003e the RDI role must be able to create and own those stream objects in the CDC schema.\n\u003e\n\u003e There is one stricter bootstrap requirement for the first stream created on a source table: if Snowflake change\n\u003e tracking is not already enabled on that table, only the table owner can create that initial stream. If the source\n\u003e tables are not owned by the RDI role, ask a Snowflake administrator or table owner to enable change tracking first:\n\u003e\n\u003e ```sql\n\u003e ALTER TABLE MYDB.PUBLIC.customers SET CHANGE_TRACKING = TRUE;\n\u003e ALTER TABLE MYDB.PUBLIC.orders SET CHANGE_TRACKING = TRUE;\n\u003e ```\n\nGrant the required permissions to your RDI user:\n\n```sql\n-- Grant usage on the warehouse\nGRANT USAGE, OPERATE ON WAREHOUSE COMPUTE_WH TO ROLE rdi_role;\n\n-- Grant usage on the source database and schema\nGRANT USAGE ON DATABASE MYDB TO ROLE rdi_role;\nGRANT USAGE ON SCHEMA MYDB.PUBLIC TO ROLE rdi_role;\n\n-- Grant SELECT on tables to capture\nGRANT SELECT ON TABLE MYDB.PUBLIC.customers TO ROLE rdi_role;\nGRANT SELECT ON TABLE MYDB.PUBLIC.orders TO ROLE rdi_role;\n\n-- Grant permissions on the schema RDI uses for CDC objects\nGRANT USAGE ON SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;\nGRANT CREATE STREAM, CREATE TABLE ON SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;\n\n-- Assign the role to your RDI user\nGRANT ROLE rdi_role TO USER rdi_user;\n```\n\nIf you use centralized grant management, you can also add future grants in the CDC schema so newly created tables and\nstreams automatically receive the desired privileges. These grants are optional and are not part of the minimum runtime\npermissions:\n\n```sql\nGRANT SELECT ON FUTURE TABLES IN SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;\nGRANT SELECT ON FUTURE STREAMS IN SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;\n```\n\n## 2. Configure authentication\n\nRDI supports two authentication methods for Snowflake. You must configure one of these methods.\n\n### Password authentication\n\nUse standard username and password credentials. Store these securely using Kubernetes secrets (see step 3).\n\n\u003e [!NOTE]\n\u003e Many Snowflake accounts require MFA for password-based sign-ins. If you want to use password authentication for RDI,\n\u003e configure the Snowflake user as a service user that is allowed to authenticate non-interactively. Otherwise, use\n\u003e private key authentication instead. For more information, see the Snowflake\n\u003e [MFA rollout documentation](https://docs.snowflake.com/en/user-guide/security-mfa-rollout).\n\n### Private key authentication\n\nFor enhanced security, use key-pair authentication:\n\n1. Generate a private key:\n\n    ```bash\n    openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt\n    ```\n\n1. Generate the public key:\n\n    ```bash\n    openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub\n    ```\n\n1. Register the public key with your Snowflake user:\n\n    ```sql\n    ALTER USER rdi_user SET RSA_PUBLIC_KEY='\u003cpublic_key_content\u003e';\n    ```\n\n## 3. Set up secrets for Kubernetes deployment\n\nBefore deploying the RDI pipeline, configure the necessary secrets.\n\n### Password authentication\n\n```bash\nkubectl create secret generic source-db \\\n  --namespace=rdi \\\n  --from-literal=SOURCE_DB_USERNAME=your_username \\\n  --from-literal=SOURCE_DB_PASSWORD=your_password\n```\n\n### Private key authentication\n\nCreate a secret with the private key file:\n\n```bash\nkubectl create secret generic source-db-ssl \\\n  --namespace=rdi \\\n  --from-file=client.key=/path/to/rsa_key.p8\n```\n\nAlso create the source-db secret with the username:\n\n```bash\nkubectl create secret generic source-db \\\n  --namespace=rdi \\\n  --from-literal=SOURCE_DB_USERNAME=your_username\n```\n\n## 4. Configure RDI for Snowflake\n\nUse the following example configuration in your `config.yaml` file:\n\n```yaml\nsources:\n  snowflake:\n    type: riotx\n    connection:\n      type: snowflake\n      url: \"jdbc:snowflake://myaccount.snowflakecomputing.com/\"\n      user: \"${SOURCE_DB_USERNAME}\"\n      password: \"${SOURCE_DB_PASSWORD}\"  # Omit for key-pair auth\n      database: \"MYDB\"\n      warehouse: \"COMPUTE_WH\"\n      # role: \"RDI_ROLE\"                 # Optional: Snowflake role\n      # cdcDatabase: \"CDC_DB\"            # Optional: Separate database for CDC streams\n      # cdcSchema: \"CDC_SCHEMA\"          # Optional: Separate schema for CDC streams\n    schemas:\n      - PUBLIC\n    tables:\n      PUBLIC.customers: {}\n      PUBLIC.orders: {}\n    advanced:\n      riotx:\n        poll: \"30s\"\n        snapshot: \"INITIAL\"              # Or \"NEVER\" to skip initial snapshot\n        # streamPrefix: \"data:\"          # Optional: Redis stream prefix\n        # streamLimit: 100000            # Optional: Max stream length\n        # keyColumns:                    # Recommended: stable key columns\n        #   - \"id\"\n        # clearOffset: false             # Optional: Clear offset on start\n\ntargets:\n  target:\n    connection:\n      type: redis\n      host: ${TARGET_DB_HOST}\n      port: ${TARGET_DB_PORT}\n      user: ${TARGET_DB_USERNAME}\n      password: ${TARGET_DB_PASSWORD}\n\nprocessors:\n  target_data_type: json\n```\n\n\u003e [!NOTE]\n\u003e Snowflake uses one configured `database` and one or more source-level `schemas`. In the `tables` section, specify each\n\u003e table as `SCHEMA.table`. Even when you configure only one schema, explicit `SCHEMA.table` names are recommended for\n\u003e clarity.\n\n### Snowflake connection properties\n\n| Property      | Type   | Required | Description                                                    |\n|---------------|--------|----------|----------------------------------------------------------------|\n| `type`        | string | Yes      | Must be `\"snowflake\"`                                          |\n| `url`         | string | Yes      | JDBC URL: `jdbc:snowflake://\u003caccount\u003e.snowflakecomputing.com/` |\n| `user`        | string | Yes      | Snowflake user                                                 |\n| `password`    | string | No*      | Snowflake password                                             |\n| `database`    | string | Yes      | Snowflake database name                                        |\n| `warehouse`   | string | Yes      | Snowflake warehouse name                                       |\n| `role`        | string | No       | Snowflake role name                                            |\n| `cdcDatabase` | string | No       | Database for CDC streams (if different from source)            |\n| `cdcSchema`   | string | No       | Schema for CDC streams (if different from source)              |\n\n* Either `password` or private key authentication is required. See [Configure authentication](#2-configure-authentication) for details.\n\n### Snowflake source properties\n\n| Property   | Type   | Required | Description                                                      |\n|------------|--------|----------|------------------------------------------------------------------|\n| `schemas`  | array  | Yes      | Schema names to capture from                                     |\n| `tables`   | object | Yes      | Tables to capture, keyed as `SCHEMA.table`                       |\n\n### Advanced configuration options\n\nConfigure under `sources.\u003cname\u003e.advanced.riotx`:\n\n| Property       | Type    | Default     | Description                                  |\n|----------------|---------|-------------|----------------------------------------------|\n| `poll`         | string  | `\"30s\"`     | Polling interval for stream changes          |\n| `snapshot`     | string  | `\"INITIAL\"` | Snapshot mode: `INITIAL` or `NEVER`          |\n| `streamPrefix` | string  | `\"data:\"`   | Prefix for the Redis stream written by RDI   |\n| `streamLimit`  | integer | -           | Maximum stream length (XTRIM MAXLEN)         |\n| `keyColumns`   | array   | -           | Stable source columns to use as message keys |\n| `clearOffset`  | boolean | `false`     | Clear existing offset on start               |\n| `count`        | integer | `0`         | Limit records per poll (0 = unlimited)       |\n\nFor reliable update and delete handling, define `keyColumns` with a stable business key or surrogate key when possible.\n\n## Troubleshooting\n\n### Connection issues\n\n**Error: \"Failed to connect to Snowflake\"**\n\n- Verify the account URL is correct (format: `\u003caccount\u003e.snowflakecomputing.com`)\n- Check network connectivity to Snowflake\n- Verify the warehouse is running and accessible\n- Check firewall rules allow outbound HTTPS (port 443)\n\n**Error: \"Authentication failed\"**\n\n- For password auth: verify username and password are correct\n- For key-pair auth: verify the private key matches the public key registered in Snowflake\n- Ensure the user has appropriate permissions\n\n**Error: \"Warehouse not found\"**\n\n- Verify the warehouse name is correct\n- Ensure the user has USAGE permission on the warehouse\n\n**Error: \"Network policy is required\"**\n\nIf the collector logs show an error like the following:\n\n```\nFailed to open new session for user: USERNAME, host: \u003caccount\u003e.snowflakecomputing.com. Error: Fail : Network policy is required.\nFailed to initialize pool: Fail : Network policy is required.\n```\n\nYour Snowflake account enforces a network policy, so you must whitelist the RDI egress IP addresses in Snowflake.\n\nThe following example creates a network policy and applies it to the RDI user. Replace the example IP addresses with your actual RDI egress IPs and replace `\"USERNAME\"` with your RDI user:\n\n```sql\nUSE ROLE SECURITYADMIN;\n\nCREATE NETWORK POLICY rdi_policy\n  ALLOWED_IP_LIST = (\n    '203.0.113.10/32',\n    '203.0.113.20/32',\n    '198.51.100.30/32',\n    '198.51.100.40/32'\n  );\n\nALTER USER \"USERNAME\"\nSET NETWORK_POLICY = rdi_policy;\n```\n\n### CDC issues\n\n**No data appearing in Redis**\n\n1. Verify Snowflake Streams exist in the CDC schema:\n\n    ```sql\n    SHOW STREAMS IN SCHEMA my_cdc_database.my_cdc_schema;\n    ```\n\n1. Check the polling interval configuration\n1. Verify Redis connection is working\n1. Check the collector logs:\n\n    ```bash\n    kubectl get deployments -n rdi | grep riotx-collector\n    kubectl logs -n rdi deployment/\u003criotx-collector-deployment\u003e\n    ```\n\n**Stale or missing changes**\n\n- Snowflake Streams depend on Snowflake change tracking and retention settings\n- If the collector was offline longer than the available retention window, changes may be lost\n- Consider using `clearOffset: true` to restart from current state\n\n### Performance tuning\n\n**High Snowflake warehouse usage**\n\n- Increase `poll` interval (e.g., `\"60s\"` or `\"120s\"`)\n- Use a dedicated warehouse for CDC operations\n- Each poll first calls Snowflake's `SYSTEM$STREAM_HAS_DATA` function to check whether the stream has new data. This\n  check does not start the warehouse; warehouse compute starts only when RDI reads rows from the stream.\n\n**Redis memory concerns**\n\n- Set `streamLimit` to cap stream length\n- Use `count` to limit records per poll batch\n\n**Initial snapshot too slow**\n\n- Use `snapshot: \"NEVER\"` to skip initial snapshot\n- Pre-load data using other methods if needed\n\n### Enable debug logging\n\nEnable debug logging in the source configuration:\n\n```yaml\nsources:\n  snowflake:\n    type: riotx\n    logging:\n      level: debug\n    # ... rest of configuration\n```\n\nView collector logs:\n\n```bash\nkubectl get deployments -n rdi | grep riotx-collector\nkubectl logs -n rdi deployment/\u003criotx-collector-deployment\u003e -f\n```\n\n## 5. Configuration is complete\n\nOnce you have followed the steps above, your Snowflake database is ready for RDI to use.\n\n## See also\n\n- [Snowflake Streams Documentation](https://docs.snowflake.com/en/user-guide/streams)\n- [Snowflake Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth)\n- [Snowflake MFA rollout documentation](https://docs.snowflake.com/en/user-guide/security-mfa-rollout)\n- [RDI Deployment Guide](https://redis.io/docs/latest/integrate/redis-data-integration/1.19.1/data-pipelines/deploy)\n",
  "tags": ["docs","integrate","rs","rdi"],
  "last_updated": "2026-09-19T17:55:58-07:00"
}