{ "schema_version": 2, "id": "develop/ai/featureform/register-providers/s3", "title": "Register an S3 provider", "url": "https://redis.io/docs/latest/develop/ai/featureform/register-providers/s3/", "summary": "Register Amazon S3 and S3-compatible object storage with Redis Feature Form.", "tags": [], "last_updated": "2026-08-25T13:17:04-07:00", "page_type": "content", "content_hash": "a3546d8270524b32ceb333da8c321f7f9819057f930e2c849600deb8a46e1c2c", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "Register an `s3` provider when Redis Feature Form needs a named Amazon Simple Storage Service (S3) bucket and prefix for supported batch-data or Spark workflows.\n\nThe provider fills the `offline-store` role. It isn't a compute engine, catalog, or online serving provider. Each resource and compute provider still determines whether it accepts S3." }, { "id": "before-you-begin", "title": "Before you begin", "role": "content", "text": "Make sure you have:\n\n- A Feature Form [workspace](https://redis.io/docs/latest/develop/ai/featureform/manage-workspace).\n- An existing bucket and Amazon Web Services (AWS) region.\n- Network access from the Feature Form server and relevant workers to the S3 endpoint.\n- Either a usable default AWS credential chain in the Feature Form runtime or access-key credentials in a registered [secret provider](https://redis.io/docs/latest/develop/ai/featureform/register-providers#configure-secret-providers).\n- Permission to write, read, and delete objects under the configured prefix.\n\nPass the bucket name without an `s3://` prefix. Set `path_prefix` in Python, or use `--s3-path-prefix` with the CLI, for a directory-like prefix within the bucket.\n\nThe Python examples use this workspace-scoped provider client:\n\n[code example]" }, { "id": "choose-authentication", "title": "Choose authentication", "role": "content", "text": "If you omit the Python access-key fields or the corresponding CLI flags, Feature Form uses the runtime's default AWS credential chain. This is the usual choice when the server runs with an assigned identity.\n\nIn Python, set both `access_key_id_secret` and `secret_access_key_secret`. With the CLI, supply both references:\n\n[code example]\n\nFeature Form rejects a configuration that supplies only one member of the pair." }, { "id": "register-amazon-s3", "title": "Register Amazon S3", "role": "content", "text": "This example uses the runtime's default AWS credential chain:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nTo use access-key secret references instead:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nThe environment references are resolved by the Feature Form server. Replace them with references to another registered secret backend when appropriate." }, { "id": "register-s3-compatible-storage", "title": "Register S3-compatible storage", "role": "content", "text": "Set a full Hypertext Transfer Protocol (HTTP) or secure HTTP (HTTPS) endpoint and enable path-style access for an S3-compatible service:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nThe endpoint must include `http://` or `https://` and a hostname." }, { "id": "grant-health-check-permissions", "title": "Grant health-check permissions", "role": "content", "text": "The default registration health check writes a small object under the configured prefix, reads it back, and deletes it. The identity needs permission for all three operations.\n\nUse `skip_health_check=True` in Python or `--skip-health-check` with the CLI only to defer this validation intentionally. A read-only bucket can't pass the default check and won't support workflows that need to publish or clean up objects." }, { "id": "use-s3-with-spark", "title": "Use S3 with Spark", "role": "content", "text": "A generic Spark provider can refer to an S3 provider for two separate purposes:\n\n- Hadoop S3A access supplies the region, endpoint, path style, and credentials used to read or write S3 data.\n- Remote execution staging publishes temporary configuration and runtime assets for the Spark driver.\n\nRegister the S3 provider before the Spark provider that refers to it. Remote Spark staging currently requires an S3 provider configured with access-key authentication. A provider that relies on the default AWS credential chain can still be used for supported S3A data access, but not for this staging path.\n\nKeep staging and long-lived data in separate prefixes or providers when they need different permissions or retention policies." }, { "id": "verify-registration", "title": "Verify registration", "role": "content", "text": "[code example]\n\nIn Python, use `providers.get(\"\")` or `providers.list()`." }, { "id": "update-safely", "title": "Update safely", "role": "content", "text": "| Field | Update behavior |\n| --- | --- |\n| Bucket | Immutable |\n| Region, endpoint, and path-style access | Requires `force=True` or `--force` |\n| Path prefix and credential references | Mutable |\n\nChanging a prefix changes where later operations read and write. It doesn't move existing objects. Before updating or deleting the provider, review every dataset, catalog, compute, and staging configuration that refers to it." }, { "id": "troubleshoot-registration", "title": "Troubleshoot registration", "role": "errors", "text": "| Symptom | What to check |\n| --- | --- |\n| Bucket validation fails | Pass a bucket name without `s3://`; put the remaining path in `path_prefix` or `--s3-path-prefix` |\n| Credential validation fails | Supply both access-key secret references or omit both to use the default AWS credential chain |\n| Health check is denied | Grant write, read, and delete permission under the configured prefix |\n| A custom endpoint can't be reached | Check its scheme, hostname, network route, certificate trust, and path-style behavior from the Feature Form server |\n| Spark can't access data | Confirm that the Spark provider refers to this S3 provider for Hadoop S3A access and that the runtime has the required credentials |\n| Remote Spark staging rejects the provider | Configure the S3 provider with access-key secret references; staging doesn't use default-chain authentication |" } ], "examples": [ { "id": "before-you-begin-ex0", "language": "python", "code": "import featureform as ff\n\nclient = ff.Client.from_env()\nproviders = client.providers(\"\")", "section_id": "before-you-begin" }, { "id": "choose-authentication-ex0", "language": "text", "code": "--s3-access-key-id-secret env:AWS_ACCESS_KEY_ID\n--s3-secret-access-key-secret env:AWS_SECRET_ACCESS_KEY", "section_id": "choose-authentication" }, { "id": "register-amazon-s3-ex0", "language": "python", "code": "from featureform.types import ProviderType, S3Config\n\nproviders.register(\n name=\"\",\n provider_type=ProviderType.S3,\n config=S3Config(\n bucket=\"\",\n region=\"\",\n path_prefix=\"\",\n ),\n)", "section_id": "register-amazon-s3" }, { "id": "register-amazon-s3-ex1", "language": "bash", "code": "ff provider register \\\n --workspace \\\n --type s3 \\\n --s3-bucket \\\n --s3-region \\\n --s3-path-prefix ", "section_id": "register-amazon-s3" }, { "id": "register-amazon-s3-ex2", "language": "python", "code": "from featureform.types import EnvSecretRef, ProviderType, S3Config\n\nproviders.register(\n name=\"\",\n provider_type=ProviderType.S3,\n config=S3Config(\n bucket=\"\",\n region=\"\",\n path_prefix=\"\",\n access_key_id_secret=EnvSecretRef(name=\"AWS_ACCESS_KEY_ID\"),\n secret_access_key_secret=EnvSecretRef(name=\"AWS_SECRET_ACCESS_KEY\"),\n ),\n)", "section_id": "register-amazon-s3" }, { "id": "register-amazon-s3-ex3", "language": "bash", "code": "ff provider register \\\n --workspace \\\n --type s3 \\\n --s3-bucket \\\n --s3-region \\\n --s3-path-prefix \\\n --s3-access-key-id-secret env:AWS_ACCESS_KEY_ID \\\n --s3-secret-access-key-secret env:AWS_SECRET_ACCESS_KEY", "section_id": "register-amazon-s3" }, { "id": "register-s3-compatible-storage-ex0", "language": "python", "code": "from featureform.types import EnvSecretRef, ProviderType, S3Config\n\nproviders.register(\n name=\"\",\n provider_type=ProviderType.S3,\n config=S3Config(\n bucket=\"\",\n region=\"\",\n endpoint=\"https://\",\n path_style_access=True,\n access_key_id_secret=EnvSecretRef(name=\"S3_ACCESS_KEY_ID\"),\n secret_access_key_secret=EnvSecretRef(name=\"S3_SECRET_ACCESS_KEY\"),\n ),\n)", "section_id": "register-s3-compatible-storage" }, { "id": "register-s3-compatible-storage-ex1", "language": "bash", "code": "ff provider register \\\n --workspace \\\n --type s3 \\\n --s3-bucket \\\n --s3-region \\\n --s3-endpoint https:// \\\n --s3-path-style-access \\\n --s3-access-key-id-secret env:S3_ACCESS_KEY_ID \\\n --s3-secret-access-key-secret env:S3_SECRET_ACCESS_KEY", "section_id": "register-s3-compatible-storage" }, { "id": "verify-registration-ex0", "language": "bash", "code": "ff provider get --workspace \nff provider list --workspace ", "section_id": "verify-registration" } ] }