Register an S3 provider
Register Amazon S3 and S3-compatible object storage with Redis Feature Form.
Register an s3 provider when Redis Feature Form needs a named Amazon Simple Storage Service (S3) bucket and prefix for supported batch-data or Spark workflows.
The provider fills the offline-store role. It isn't a compute engine, catalog, or online serving provider. Each resource and compute provider still determines whether it accepts S3.
Before you begin
Make sure you have:
- A Feature Form workspace.
- An existing bucket and Amazon Web Services (AWS) region.
- Network access from the Feature Form server and relevant workers to the S3 endpoint.
- Either a usable default AWS credential chain in the Feature Form runtime or access-key credentials in a registered secret provider.
- Permission to write, read, and delete objects under the configured prefix.
Pass the bucket name without an s3:// prefix. Set path_prefix in Python, or use --s3-path-prefix with the CLI, for a directory-like prefix within the bucket.
The Python examples use this workspace-scoped provider client:
import featureform as ff
client = ff.Client.from_env()
providers = client.providers("<workspace-id>")
Choose authentication
If you omit the Python access-key fields or the corresponding CLI flags, Feature Form uses the runtime's default AWS credential chain. This is the usual choice when the server runs with an assigned identity.
In Python, set both access_key_id_secret and secret_access_key_secret. With the CLI, supply both references:
--s3-access-key-id-secret env:AWS_ACCESS_KEY_ID
--s3-secret-access-key-secret env:AWS_SECRET_ACCESS_KEY
Feature Form rejects a configuration that supplies only one member of the pair.
Register Amazon S3
This example uses the runtime's default AWS credential chain:
from featureform.types import ProviderType, S3Config
providers.register(
name="<s3-provider-name>",
provider_type=ProviderType.S3,
config=S3Config(
bucket="<bucket-name>",
region="<aws-region>",
path_prefix="<path-prefix>",
),
)
To use access-key secret references instead:
from featureform.types import EnvSecretRef, ProviderType, S3Config
providers.register(
name="<s3-provider-name>",
provider_type=ProviderType.S3,
config=S3Config(
bucket="<bucket-name>",
region="<aws-region>",
path_prefix="<path-prefix>",
access_key_id_secret=EnvSecretRef(name="AWS_ACCESS_KEY_ID"),
secret_access_key_secret=EnvSecretRef(name="AWS_SECRET_ACCESS_KEY"),
),
)
The environment references are resolved by the Feature Form server. Replace them with references to another registered secret backend when appropriate.
Register S3-compatible storage
Set a full Hypertext Transfer Protocol (HTTP) or secure HTTP (HTTPS) endpoint and enable path-style access for an S3-compatible service:
from featureform.types import EnvSecretRef, ProviderType, S3Config
providers.register(
name="<s3-provider-name>",
provider_type=ProviderType.S3,
config=S3Config(
bucket="<bucket-name>",
region="<region>",
endpoint="https://<object-storage-host>",
path_style_access=True,
access_key_id_secret=EnvSecretRef(name="S3_ACCESS_KEY_ID"),
secret_access_key_secret=EnvSecretRef(name="S3_SECRET_ACCESS_KEY"),
),
)
The endpoint must include http:// or https:// and a hostname.
Grant health-check permissions
The default registration health check writes a small object under the configured prefix, reads it back, and deletes it. The identity needs permission for all three operations.
Use skip_health_check=True in Python or --skip-health-check with the CLI only to defer this validation intentionally. A read-only bucket can't pass the default check and won't support workflows that need to publish or clean up objects.
Use S3 with Spark
A generic Spark provider can refer to an S3 provider for two separate purposes:
- Hadoop S3A access supplies the region, endpoint, path style, and credentials used to read or write S3 data.
- Remote execution staging publishes temporary configuration and runtime assets for the Spark driver.
Register the S3 provider before the Spark provider that refers to it. Remote Spark staging currently requires an S3 provider configured with access-key authentication. A provider that relies on the default AWS credential chain can still be used for supported S3A data access, but not for this staging path.
Keep staging and long-lived data in separate prefixes or providers when they need different permissions or retention policies.
Verify registration
ff provider get <s3-provider-name> --workspace <workspace-id>
ff provider list --workspace <workspace-id>
In Python, use providers.get("<s3-provider-name>") or providers.list().
Update safely
| Field | Update behavior |
|---|---|
| Bucket | Immutable |
| Region, endpoint, and path-style access | Requires force=True or --force |
| Path prefix and credential references | Mutable |
Changing a prefix changes where later operations read and write. It doesn't move existing objects. Before updating or deleting the provider, review every dataset, catalog, compute, and staging configuration that refers to it.
Troubleshoot registration
| Symptom | What to check |
|---|---|
| Bucket validation fails | Pass a bucket name without s3://; put the remaining path in path_prefix or --s3-path-prefix |
| Credential validation fails | Supply both access-key secret references or omit both to use the default AWS credential chain |
| Health check is denied | Grant write, read, and delete permission under the configured prefix |
| A custom endpoint can't be reached | Check its scheme, hostname, network route, certificate trust, and path-style behavior from the Feature Form server |
| Spark can't access data | Confirm that the Spark provider refers to this S3 provider for Hadoop S3A access and that the runtime has the required credentials |
| Remote Spark staging rejects the provider | Configure the S3 provider with access-key secret references; staging doesn't use default-chain authentication |