{
  "schema_version": 2,
  "id": "develop/ai/featureform/register-providers/databricks",
  "title": "Register Databricks providers",
  "url": "https://redis.io/docs/latest/develop/ai/featureform/register-providers/databricks/",
  "summary": "Register Databricks compute, Unity Catalog, and secret providers with Redis Feature Form.",
  "tags": [],
  "last_updated": "2026-08-25T13:17:04-07:00",
  "page_type": "content",
  "content_hash": "84ec6227f97ab0a8e125fc1f2ca328b286d2563b65bea0027b6ba210e4c6133f",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "Use Databricks with Redis Feature Form by registering each required provider record separately. A typical Spark workflow needs at least one `unity-catalog` provider and one `databricks` compute provider. Register an additional provider for each physical catalog, secret backend, or online store the workflow uses."
    },
    {
      "id": "understand-the-databricks-providers",
      "title": "Understand the Databricks providers",
      "role": "content",
      "text": "| Provider type | Purpose | Required for Databricks compute |\n| --- | --- | --- |\n| `unity-catalog` | Identifies a catalog and provides credentials for catalog health, direct reads, and optional Volume artifact storage | Yes |\n| `databricks` | Selects Databricks compute, the managed-output catalog, table format defaults, and artifact storage | Yes |\n| `databricks-secret` | Resolves existing Databricks secret scope and key values | No |\n\nThe `databricks` provider fills the `offline-store` and `compute` roles. It doesn't provide online serving. A feature view also needs a supported `online-store` provider such as Redis."
    },
    {
      "id": "before-you-begin",
      "title": "Before you begin",
      "role": "content",
      "text": "Make sure you have:\n\n- A Feature Form [workspace](https://redis.io/docs/latest/develop/ai/featureform/manage-workspace).\n- An HTTPS [Databricks workspace URL](https://docs.databricks.com/aws/en/workspace/workspace-details).\n- A personal access token (PAT), or an OAuth machine-to-machine (M2M) client ID and client secret.\n- A non-Databricks [secret provider](https://redis.io/docs/latest/develop/ai/featureform/register-providers#configure-secret-providers) that can resolve the PAT or OAuth client secret.\n- Every Unity Catalog catalog the workflow needs, plus a schema and Volume for execution artifacts. One catalog can serve multiple purposes.\n- Either an existing cluster or the configuration required to create Jobs compute.\n\nThe compute principal needs permission to inspect or create its selected compute, submit Databricks Jobs work, access the output catalog, and delete Feature Form-managed tables. Workloads need read and write access to the required catalog locations.\n\n\nDon't use a `databricks@<provider>:<scope>#<key>` reference to authenticate a `databricks` or `unity-catalog` provider. Feature Form rejects that bootstrap cycle. Use `env`, Vault, Kubernetes secrets, or AWS Secrets Manager for these credentials."
    },
    {
      "id": "register-providers-in-dependency-order",
      "title": "Register providers in dependency order",
      "role": "content",
      "text": "For a new Feature Form workspace, register providers in this order:\n\n1. Register a non-Databricks secret provider for the Databricks personal access token (PAT) or OAuth machine-to-machine (M2M) client secret.\n2. Register one `unity-catalog` provider for each physical input, output, or artifact catalog.\n3. Register the `databricks` compute provider.\n4. Register `databricks-secret` if another provider needs a value from a Databricks secret scope.\n5. Register the Redis `online-store` provider used by feature views.\n\nThe Python `DatabricksWorkspaceConfig` object in the examples is a reusable configuration object. It isn't a separate provider record."
    },
    {
      "id": "plan-permissions-and-network-access",
      "title": "Plan permissions and network access",
      "role": "content",
      "text": "Provider registration and workload execution use different identities and network paths:\n\n| Component or identity | Required access |\n| --- | --- |\n| Feature Form server | The bootstrap secret backend, Databricks workspace APIs, configured catalogs and clusters, and Redis for provider health checks and inference-time feature reads |\n| Unity Catalog provider principal | Catalog lookup and direct reads; create, read, and delete access to a configured artifact Volume when used |\n| Databricks provider principal | Inspect existing compute or submit Jobs compute; access and delete Feature Form-managed output tables |\n| Databricks workload identity | Read input catalogs, write output catalogs, read execution artifacts, and connect to Redis during feature-view materialization |\n\nAllow each connection independently. A successful connection from the Feature Form server doesn't prove that Databricks compute can reach the same endpoint. See the [Unity Catalog privilege reference](https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference) for catalog permissions and the [Databricks authentication documentation](https://docs.databricks.com/aws/en/dev-tools/auth) for workspace authentication.\n\nThe Python examples use a workspace-scoped provider client and this shared Databricks workspace configuration. Each tab also shows the equivalent command-line interface (CLI) command.\n\n[code example]"
    },
    {
      "id": "register-unity-catalog",
      "title": "Register Unity Catalog",
      "role": "content",
      "text": "Register the catalog before the compute provider that refers to it:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nFor OAuth M2M authentication with a Kubernetes-backed client secret, first register the Kubernetes secret provider as described in [Configure secret providers](https://redis.io/docs/latest/develop/ai/featureform/register-providers#register-kubernetes-secrets). Then replace the shared Python workspace configuration or the PAT flags with:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nOne `unity-catalog` provider represents one physical Databricks catalog. You can reference the same provider for input tables, managed outputs, and execution artifacts when they use that catalog. Register separate providers when they use different physical catalogs or need different credentials:\n\n| Catalog use | Where to reference the provider name |\n| --- | --- |\n| Existing input tables | The dataset location's `catalog_provider` field |\n| Feature Form-managed outputs | `DatabricksConfig.output_catalog` or `--databricks-output-catalog` |\n| Unity Catalog Volume artifacts | `artifact_store.provider` or `--databricks-artifact-store-provider` |\n\nAll catalog and compute providers used by one workload must be in the same Feature Form workspace and identify the same Databricks workspace host. They can use different Databricks credentials. Register every referenced catalog provider before the compute provider. The compute examples on this page use the same `<catalog-provider-name>` for managed outputs and artifact storage."
    },
    {
      "id": "choose-compute",
      "title": "Choose compute",
      "role": "content",
      "text": "Feature Form exposes two Databricks compute targets:\n\n| Target | Use it when | Required configuration |\n| --- | --- | --- |\n| `existing_cluster` | You manage a long-running cluster outside Feature Form | An accessible cluster ID |\n| `job_cluster` | Each managed job should use Jobs compute | A runtime version, worker node type or instance pool, and fixed workers or autoscaling |\n\nAn existing-cluster health check authenticates and verifies that Feature Form can inspect the cluster. A Jobs compute health check authenticates and validates the configuration, but it doesn't create a cluster or submit a representative job."
    },
    {
      "id": "register-an-existing-cluster",
      "title": "Register an existing cluster",
      "role": "content",
      "text": "**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nIf you author Python callable transformations, import `SparkCallableRuntime` from `featureform.types` and set `callable_runtime=SparkCallableRuntime(python_version=\"<python-version>\", cloudpickle_version=\"<cloudpickle-version>\")` on the Python `DatabricksConfig`. With the CLI, add:\n\n[code example]\n\nThe versions must match the environment that serializes and executes the callable."
    },
    {
      "id": "register-jobs-compute",
      "title": "Register Jobs compute",
      "role": "content",
      "text": "This example uses a fixed worker count:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]"
    },
    {
      "id": "apply-a-compute-policy-and-tags",
      "title": "Apply a compute policy and tags",
      "role": "content",
      "text": "Use a compute policy to enforce workspace requirements. Set `apply_policy_default_values` when the Databricks Jobs application programming interface (API) should apply defaults defined by the policy. Registration validates these fields but doesn't create a cluster or prove that the principal can use the policy.\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nDatabricks allows at most 45 custom cluster tags, and Feature Form enforces that platform limit during registration. Repeat `--databricks-job-custom-tag` for each `key=value` pair. See the [Databricks Jobs API reference](https://docs.databricks.com/api/jobs/v2) for cluster-tag constraints and the [Databricks compute policy reference](https://docs.databricks.com/aws/en/admin/clusters/policy-definition) for policy rules and default-value behavior.\n\nIn Python, set `instance_pool_id=\"<instance-pool-id>\"` instead of `node_type_id` when the workspace uses an instance pool. With the CLI, use `--databricks-job-instance-pool-id <instance-pool-id>` instead of a worker node type.\n\nTo autoscale in Python, import `DatabricksAutoscaleConfig` from `featureform.types`, omit `num_workers`, and set `autoscale=DatabricksAutoscaleConfig(min_workers=int(\"<minimum-workers>\"), max_workers=int(\"<maximum-workers>\"))`. With the CLI, omit `--databricks-job-num-workers` and set both of these flags:\n\n[code example]\n\nFixed workers and autoscaling are mutually exclusive. Common optional settings include a driver node type or pool, cluster policy, data security mode, runtime engine, custom tags, log delivery, init scripts, and disk settings. Run `ff provider register --help` for the current flags."
    },
    {
      "id": "configure-aws-attributes",
      "title": "Configure AWS attributes",
      "role": "configuration",
      "text": "Use Amazon Web Services (AWS) attributes only for Jobs compute in a Databricks workspace on AWS. In Python, create the following configuration and set `aws_attributes=aws_attributes` on `DatabricksJobClusterConfig`. With the CLI, append the equivalent flags to the registration command:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nThis example keeps the driver on an on-demand instance, uses Spot instances with on-demand fallback for subsequent nodes, limits the Spot bid to the corresponding on-demand price, and attaches a preconfigured instance profile. The instance profile must already be added to the Databricks workspace.\n\n| Python field | CLI flag | Meaning or constraint |\n| --- | --- | --- |\n| `availability` | `--databricks-job-aws-availability` | Databricks availability mode |\n| `zone_id` | `--databricks-job-aws-zone-id` | AWS availability zone or Databricks automatic-zone value |\n| `first_on_demand` | `--databricks-job-aws-first-on-demand` | Nonnegative number of initial on-demand nodes |\n| `spot_bid_price_percent` | `--databricks-job-aws-spot-bid-price-percent` | Nonnegative percentage of the corresponding on-demand price |\n| `ebs_volume_type` | `--databricks-job-aws-ebs-volume-type` | Databricks-supported Amazon Elastic Block Store (EBS) volume type |\n| `ebs_volume_count` | `--databricks-job-aws-ebs-volume-count` | Nonnegative number of EBS volumes per instance |\n| `ebs_volume_size` | `--databricks-job-aws-ebs-volume-size` | Nonnegative EBS volume size in gigabytes |\n| `ebs_volume_iops` | `--databricks-job-aws-ebs-volume-iops` | Nonnegative provisioned input/output operations per second for supported volume types |\n| `ebs_volume_throughput` | `--databricks-job-aws-ebs-volume-throughput` | Nonnegative throughput for supported volume types |\n| `instance_profile_arn` | `--databricks-job-aws-instance-profile-arn` | Preconfigured AWS instance-profile Amazon Resource Name (ARN) |\n\nSee the [Databricks Jobs API reference](https://docs.databricks.com/aws/en/reference/jobs-2.0-api) for the allowed values, units, interactions, and cloud-specific constraints.\n\nFor a Python callable transformation on Jobs compute, import `SparkCallableRuntime` from `featureform.types` and set `callable_runtime=SparkCallableRuntime(python_version=\"<python-version>\")`. With the CLI, set `--databricks-callable-python-version`. Don't set the provider-level cloudpickle version for `job_cluster`."
    },
    {
      "id": "choose-artifact-storage",
      "title": "Choose artifact storage",
      "role": "content",
      "text": "Feature Form publishes temporary execution artifacts for Databricks jobs. Explicitly configure a Unity Catalog Volume for every new Databricks compute provider. If you omit `artifact_store`, Feature Form uses the legacy Workspace Files and DBFS policy for compatibility; this policy isn't recommended for new registrations.\n\nTo use a [Unity Catalog Volume](https://docs.databricks.com/aws/en/volumes/), create the schema and Volume first. The Unity Catalog provider's principal needs create, read, and delete access through the Databricks Files API. The Databricks run principal needs read access.\n\nIn Python, import `DatabricksUnityCatalogArtifactStoreConfig` from `featureform.types` and set `artifact_store=DatabricksUnityCatalogArtifactStoreConfig(provider=\"<catalog-provider-name>\", schema=\"<schema-name>\", volume=\"<volume-name>\")` on `DatabricksConfig`. With the CLI, add these flags to the compute registration command:\n\n[code example]\n\nThe artifact catalog can use the same `unity-catalog` provider as the output catalog, as shown in the compute examples, or another provider. It must target the same Databricks workspace host. The output catalog and artifact-store policy are immutable; changing either requires a new Databricks compute provider."
    },
    {
      "id": "choose-a-managed-table-format",
      "title": "Choose a managed table format",
      "role": "content",
      "text": "The provider default applies when a managed transformation, feature view, or training set doesn't select a format.\n\n| Value | Use with Feature Form dataframe reads | Important boundary |\n| --- | --- | --- |\n| `delta_uniform` | Yes | Default; exposes Iceberg-compatible metadata |\n| `iceberg` | Yes | Not supported for incremental Databricks transformations |\n| `delta` | No | Remains usable by Databricks compute, but not by the current dataframe path |\n\nChanging the provider default affects new managed output versions. It doesn't rewrite existing outputs."
    },
    {
      "id": "use-databricks-secrets",
      "title": "Use Databricks secrets",
      "role": "content",
      "text": "Feature Form doesn't infer access to Databricks secret scopes from a `databricks` compute or `unity-catalog` provider. To use any value stored in a Databricks secret scope, register a separate `databricks-secret` provider. Every reference names that provider, the scope, and the key: `databricks@<databricks-secret-provider-name>:<scope>#<key>`.\n\nA `databricks-secret` provider must authenticate before it can call the Databricks Secrets API. Feature Form therefore has to resolve the provider's PAT or OAuth client secret before it can retrieve any Databricks secret. The provider can't use a secret behind itself as that bootstrap credential.\n\nUse this bootstrap order:\n\n1. Register a non-Databricks secret provider, such as `env`, Vault, Kubernetes secrets, or AWS Secrets Manager.\n2. Register `databricks-secret` with a PAT or OAuth client-secret reference from that provider.\n3. Use the registered `databricks-secret` provider in references to existing Databricks scopes and keys.\n\nThe following PAT example uses the built-in `env` provider for the bootstrap credential:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nFor OAuth M2M authentication, use the same non-Databricks bootstrap secret provider used by the catalog and compute providers:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nThis secret-provider type doesn't have a live health probe, so registration success doesn't prove that the scope and key can be read. Test a representative reference before relying on the provider."
    },
    {
      "id": "connect-databricks-compute-to-redis",
      "title": "Connect Databricks compute to Redis",
      "role": "content",
      "text": "A feature view needs a Redis `online-store` provider for materialization and inference-time reads. Register Redis as described in [Register Redis providers](https://redis.io/docs/latest/develop/ai/featureform/register-providers/redis).\n\nIf the Redis password is stored in a Databricks secret scope, configure the provider with the registered `databricks-secret` provider:\n\n**Python:**\n\n[code example]\n\n**ff CLI:**\n\n[code example]\n\n\n\nFor both `job_cluster` and `existing_cluster` Spark feature-view materialization, the Redis password must use a `databricks@<provider>:<scope>#<key>` reference. With `job_cluster`, Feature Form adds the reference to the job environment so Databricks can resolve it. With `existing_cluster`, the cluster resolves the scope and key directly.\n\nKeep Transport Layer Security (TLS) enabled when the Redis deployment requires it. For Spark materialization, Redis certificate paths must also be available to the Databricks runtime; secret-backed Redis TLS certificate fields aren't projected to Spark. Verify Redis connectivity separately from the Feature Form server and the Databricks compute environment."
    },
    {
      "id": "databricks-configuration-reference",
      "title": "Databricks configuration reference",
      "role": "content",
      "text": "The tables in this section map the Python configuration fields to their `ff` CLI flags. All secret fields accept a registered secret reference, not a credential value."
    },
    {
      "id": "workspace-authentication-and-related-providers",
      "title": "Workspace authentication and related providers",
      "role": "content",
      "text": "`unity-catalog` and `databricks` use `DatabricksWorkspaceConfig`:\n\n| Python field | CLI flag | Requirement |\n| --- | --- | --- |\n| `workspace.workspace_url` | `--databricks-workspace-url` | Required HTTPS workspace URL |\n| `workspace.auth.type` | `--databricks-auth-type` | `pat` or `oauth_m2m` |\n| `DatabricksPATAuth.token` | `--databricks-token-secret` | Required for PAT authentication |\n| `DatabricksOAuthM2MAuth.client_id` | `--databricks-oauth-client-id` | Required for OAuth M2M |\n| `DatabricksOAuthM2MAuth.client_secret` | `--databricks-oauth-client-secret` | Required for OAuth M2M; use a non-Databricks secret reference |\n| `UnityCatalogConfig.catalog_name` | `--unity-catalog-name` | Required for `unity-catalog` |"
    },
    {
      "id": "databricks-compute",
      "title": "Databricks compute",
      "role": "content",
      "text": "| Python field | CLI flag | Requirement or default |\n| --- | --- | --- |\n| `workspace` | Databricks workspace and authentication flags | Required |\n| `compute_target` | `--databricks-compute-target` | `existing_cluster` by default; also accepts `job_cluster` |\n| `cluster_id` | `--databricks-cluster-id` | Required for `existing_cluster` |\n| `job_cluster` | `--databricks-job-*` | Required for `job_cluster`; see the Jobs compute table |\n| `output_catalog` | `--databricks-output-catalog` | Required `unity-catalog` provider name |\n| `default_managed_table_format` | `--default-managed-table-format` | `delta_uniform` by default; accepts `delta_uniform`, `iceberg`, or `delta` |\n| `callable_runtime.python_version` | `--databricks-callable-python-version` | Required when you configure a callable runtime |\n| `callable_runtime.cloudpickle_version` | `--databricks-callable-cloudpickle-version` | Required for an existing-cluster callable runtime; invalid for Jobs compute |\n| `artifact_store.type` | `--databricks-artifact-store-type` | Use `unity_catalog` for new registrations; omission selects the legacy `legacy_workspace_dbfs` compatibility policy |\n| `artifact_store.provider` | `--databricks-artifact-store-provider` | Required `unity-catalog` provider for Volume storage |\n| `artifact_store.schema` | `--databricks-artifact-store-schema` | Required for Volume storage |\n| `artifact_store.volume` | `--databricks-artifact-store-volume` | Required for Volume storage |\n\n`output_catalog` and the artifact-store policy are immutable. Register a new compute provider to change either setting."
    },
    {
      "id": "jobs-compute",
      "title": "Jobs compute",
      "role": "content",
      "text": "| Python field | CLI flag | Requirement or constraint |\n| --- | --- | --- |\n| `spark_version` | `--databricks-job-spark-version` | Required Databricks Runtime version |\n| `node_type_id` | `--databricks-job-node-type-id` | A worker node type or worker instance pool is required |\n| `driver_node_type_id` | `--databricks-job-driver-node-type-id` | Optional driver node type |\n| `num_workers` | `--databricks-job-num-workers` | Positive fixed count; mutually exclusive with autoscaling |\n| `autoscale.min_workers` | `--databricks-job-autoscale-min-workers` | Nonnegative; set with maximum workers |\n| `autoscale.max_workers` | `--databricks-job-autoscale-max-workers` | Greater than minimum workers |\n| `data_security_mode` | `--databricks-job-data-security-mode` | Optional Databricks access mode |\n| `policy_id` | `--databricks-job-policy-id` | Optional compute policy ID |\n| `apply_policy_default_values` | `--databricks-job-apply-policy-default-values` | `false` by default |\n| `runtime_engine` | `--databricks-job-runtime-engine` | Optional runtime engine such as `PHOTON` |\n| `custom_tags` | Repeat `--databricks-job-custom-tag <key>=<value>` | At most 45 entries; keys must be nonempty |\n| `cluster_log_conf.s3` | `--databricks-job-cluster-log-s3-*` | Optional; see the log-delivery table |\n| `init_scripts` | Repeat a `--databricks-job-init-script-*` flag | Optional; see the init-script table |\n| `instance_pool_id` | `--databricks-job-instance-pool-id` | Worker pool; can replace `node_type_id` |\n| `driver_instance_pool_id` | `--databricks-job-driver-instance-pool-id` | Optional driver pool |\n| `enable_elastic_disk` | `--databricks-job-enable-elastic-disk` | `false` by default |\n| `enable_local_disk_encryption` | `--databricks-job-enable-local-disk-encryption` | `false` by default |\n| `single_user_name` | `--databricks-job-single-user-name` | Optional user for `SINGLE_USER` access mode |\n| `aws_attributes` | `--databricks-job-aws-*` | Optional for Databricks on AWS; see [Configure AWS attributes](#configure-aws-attributes) |"
    },
    {
      "id": "log-delivery-and-init-scripts",
      "title": "Log delivery and init scripts",
      "role": "content",
      "text": "Jobs compute supports Amazon Simple Storage Service (S3) cluster-log delivery:\n\n| Python field under `cluster_log_conf.s3` | CLI flag |\n| --- | --- |\n| `destination` | `--databricks-job-cluster-log-s3-destination` |\n| `region` | `--databricks-job-cluster-log-s3-region` |\n| `endpoint` | `--databricks-job-cluster-log-s3-endpoint` |\n| `canned_acl` | `--databricks-job-cluster-log-s3-canned-acl` |\n| `enable_encryption` | `--databricks-job-cluster-log-s3-enable-encryption` |\n| `encryption_type` | `--databricks-job-cluster-log-s3-encryption-type` |\n| `kms_key` | `--databricks-job-cluster-log-s3-kms-key` |\n\nSet `destination` and either `region` or `endpoint`. For init scripts, each `DatabricksInitScript` must configure exactly one destination:\n\n| Python destination | CLI flag |\n| --- | --- |\n| `s3=DatabricksS3StorageInfo(...)` | Repeat `--databricks-job-init-script-s3 destination=<s3-uri>,region=<aws-region>` |\n| `dbfs=DatabricksStorageInfo(...)` | Repeat `--databricks-job-init-script-dbfs <dbfs-path>` |\n| `workspace=DatabricksStorageInfo(...)` | Repeat `--databricks-job-init-script-workspace <workspace-path>` |\n| `volumes=DatabricksStorageInfo(...)` | Repeat `--databricks-job-init-script-volumes <volume-path>` |\n\nThe S3 init-script value also accepts `endpoint`, `canned_acl`, `enable_encryption`, `encryption_type`, and `kms_key` as comma-separated `key=value` entries. It requires either `region` or `endpoint`."
    },
    {
      "id": "databricks-secret-provider",
      "title": "Databricks secret provider",
      "role": "content",
      "text": "| Python field | CLI flag | Requirement |\n| --- | --- | --- |\n| `workspace_url` | `--workspace-url` or `--databricks-workspace-url` | Required HTTPS workspace URL |\n| `auth.type` | `--auth-type` or `--databricks-auth-type` | `pat` or `oauth_m2m` |\n| `DatabricksSecretPATAuth.token_secret` | `--token-secret` or `--databricks-token-secret` | Required for PAT authentication |\n| `DatabricksSecretOAuthM2MAuth.client_id` | `--client-id` or `--databricks-client-id` | Required for OAuth M2M |\n| `DatabricksSecretOAuthM2MAuth.client_secret_secret` | `--client-secret-secret` or `--databricks-client-secret-secret` | Required for OAuth M2M; use a non-Databricks secret reference |"
    },
    {
      "id": "understand-registration-health-checks",
      "title": "Understand registration health checks",
      "role": "content",
      "text": "| Provider type | Registration checks | Not checked |\n| --- | --- | --- |\n| `unity-catalog` | Workspace authentication and catalog lookup | Table writes, Volume access, or compute permissions |\n| `databricks` with `existing_cluster` | Workspace authentication and cluster lookup | Job submission, libraries, catalog writes, artifacts, or Redis connectivity |\n| `databricks` with `job_cluster` | Workspace authentication and static configuration | Cluster creation, policy authorization, runtime startup, catalog access, or Redis connectivity |\n| `databricks-secret` | Configuration validation | Live scope or key access |\n\nBecause registration doesn't run a representative workload, policy, permission, library, secret, and network errors can first appear during an apply operation."
    },
    {
      "id": "verify-the-providers",
      "title": "Verify the providers",
      "role": "content",
      "text": "[code example]\n\nIn Python, use `providers.get()` or `providers.list()` for data providers and `secret_providers.get()` for the Databricks secret provider."
    },
    {
      "id": "troubleshoot-registration",
      "title": "Troubleshoot registration",
      "role": "errors",
      "text": "| Symptom | What to check |\n| --- | --- |\n| Output catalog provider isn't found | Register it first and confirm its name and Feature Form workspace |\n| Provider is the wrong type | Use `unity-catalog`, not a generic Iceberg catalog, for `output_catalog` or `--databricks-output-catalog` |\n| Workspace hosts don't match | Use the same Databricks workspace host for compute, output, and artifact catalogs |\n| Existing-cluster health fails | Check the cluster ID, workspace authentication, server network access, and permission to inspect the cluster |\n| Jobs compute validation fails | Supply a runtime, a node type or pool, and either fixed workers or valid autoscaling bounds |\n| A Jobs run is rejected by a compute policy | Confirm that the provider principal can use the policy, required fixed values match, and policy defaults are applied when needed |\n| A Jobs run starts but can't access a catalog | Check the Databricks workload identity's grants on the input and output catalogs, schemas, tables, and storage locations |\n| A Databricks secret fails on first use | Check the bootstrap credential, Databricks secret scope and key, and the principal's secret access |\n| Redis health passes but materialization can't connect | Test name resolution, routing, firewall rules, and TLS from the Databricks compute environment, not only from the Feature Form server |\n| Existing-cluster materialization rejects the Redis password reference | Use a `databricks@<provider>:<scope>#<key>` reference for the Redis password |\n| Volume publication fails | Check that the schema and Volume exist and that both principals have the required Files API access |\n| A dataframe read rejects an output | Check the managed table format; `delta` isn't supported by the current dataframe path |\n\nIf registration succeeds but the first workload fails, start with the health-check boundaries, identity permissions, and network paths on this page. Then compare the submitted Jobs configuration with the [Databricks Jobs API reference](https://docs.databricks.com/aws/en/reference/jobs-2.0-api)."
    }
  ],
  "examples": [
    {
      "id": "plan-permissions-and-network-access-ex0",
      "language": "python",
      "code": "import featureform as ff\nfrom featureform.types import (\n    DatabricksPATAuth,\n    DatabricksWorkspaceConfig,\n    EnvSecretRef,\n)\n\nclient = ff.Client.from_env()\nproviders = client.providers(\"<workspace-id>\")\ndatabricks_workspace = DatabricksWorkspaceConfig(\n    workspace_url=\"https://<databricks-workspace-host>\",\n    auth=DatabricksPATAuth(\n        token=EnvSecretRef(name=\"DATABRICKS_TOKEN\"),\n    ),\n)",
      "section_id": "plan-permissions-and-network-access"
    },
    {
      "id": "register-unity-catalog-ex0",
      "language": "python",
      "code": "from featureform.types import ProviderType, UnityCatalogConfig\n\nproviders.register(\n    name=\"<catalog-provider-name>\",\n    provider_type=ProviderType.UNITY_CATALOG,\n    config=UnityCatalogConfig(\n        workspace=databricks_workspace,\n        catalog_name=\"<catalog-name>\",\n    ),\n)",
      "section_id": "register-unity-catalog"
    },
    {
      "id": "register-unity-catalog-ex1",
      "language": "bash",
      "code": "ff provider register <catalog-provider-name> \\\n  --workspace <workspace-id> \\\n  --type unity-catalog \\\n  --databricks-workspace-url https://<databricks-workspace-host> \\\n  --databricks-auth-type pat \\\n  --databricks-token-secret env:DATABRICKS_TOKEN \\\n  --unity-catalog-name <catalog-name>",
      "section_id": "register-unity-catalog"
    },
    {
      "id": "register-unity-catalog-ex2",
      "language": "python",
      "code": "from featureform.types import DatabricksOAuthM2MAuth, K8sSecretRef\n\ndatabricks_workspace = DatabricksWorkspaceConfig(\n    workspace_url=\"https://<databricks-workspace-host>\",\n    auth=DatabricksOAuthM2MAuth(\n        client_id=\"<client-id>\",\n        client_secret=K8sSecretRef(\n            provider_name=\"<k8s-secret-provider-name>\",\n            name=\"<secret-name>\",\n            key=\"<client-secret-key>\",\n        ),\n    ),\n)",
      "section_id": "register-unity-catalog"
    },
    {
      "id": "register-unity-catalog-ex3",
      "language": "text",
      "code": "--databricks-auth-type oauth_m2m\n--databricks-oauth-client-id <client-id>\n--databricks-oauth-client-secret k8s@<k8s-secret-provider-name>:<secret-name>#<client-secret-key>",
      "section_id": "register-unity-catalog"
    },
    {
      "id": "register-an-existing-cluster-ex0",
      "language": "python",
      "code": "from featureform.types import (\n    DatabricksConfig,\n    DatabricksUnityCatalogArtifactStoreConfig,\n    ManagedTableFormat,\n    ProviderType,\n)\n\nproviders.register(\n    name=\"<databricks-provider-name>\",\n    provider_type=ProviderType.DATABRICKS,\n    config=DatabricksConfig(\n        workspace=databricks_workspace,\n        compute_target=\"existing_cluster\",\n        cluster_id=\"<cluster-id>\",\n        output_catalog=\"<catalog-provider-name>\",\n        artifact_store=DatabricksUnityCatalogArtifactStoreConfig(\n            provider=\"<catalog-provider-name>\",\n            schema=\"<schema-name>\",\n            volume=\"<volume-name>\",\n        ),\n        default_managed_table_format=ManagedTableFormat.DELTA_UNIFORM,\n    ),\n)",
      "section_id": "register-an-existing-cluster"
    },
    {
      "id": "register-an-existing-cluster-ex1",
      "language": "bash",
      "code": "ff provider register <databricks-provider-name> \\\n  --workspace <workspace-id> \\\n  --type databricks \\\n  --databricks-workspace-url https://<databricks-workspace-host> \\\n  --databricks-auth-type pat \\\n  --databricks-token-secret env:DATABRICKS_TOKEN \\\n  --databricks-compute-target existing_cluster \\\n  --databricks-cluster-id <cluster-id> \\\n  --databricks-output-catalog <catalog-provider-name> \\\n  --databricks-artifact-store-type unity_catalog \\\n  --databricks-artifact-store-provider <catalog-provider-name> \\\n  --databricks-artifact-store-schema <schema-name> \\\n  --databricks-artifact-store-volume <volume-name> \\\n  --default-managed-table-format delta_uniform",
      "section_id": "register-an-existing-cluster"
    },
    {
      "id": "register-an-existing-cluster-ex2",
      "language": "text",
      "code": "--databricks-callable-python-version <python-version>\n--databricks-callable-cloudpickle-version <cloudpickle-version>",
      "section_id": "register-an-existing-cluster"
    },
    {
      "id": "register-jobs-compute-ex0",
      "language": "python",
      "code": "from featureform.types import (\n    DatabricksConfig,\n    DatabricksJobClusterConfig,\n    DatabricksUnityCatalogArtifactStoreConfig,\n    ManagedTableFormat,\n    ProviderType,\n)\n\nproviders.register(\n    name=\"<databricks-provider-name>\",\n    provider_type=ProviderType.DATABRICKS,\n    config=DatabricksConfig(\n        workspace=databricks_workspace,\n        compute_target=\"job_cluster\",\n        job_cluster=DatabricksJobClusterConfig(\n            spark_version=\"<databricks-runtime-version>\",\n            node_type_id=\"<worker-node-type-id>\",\n            num_workers=int(\"<worker-count>\"),\n        ),\n        output_catalog=\"<catalog-provider-name>\",\n        artifact_store=DatabricksUnityCatalogArtifactStoreConfig(\n            provider=\"<catalog-provider-name>\",\n            schema=\"<schema-name>\",\n            volume=\"<volume-name>\",\n        ),\n        default_managed_table_format=ManagedTableFormat.DELTA_UNIFORM,\n    ),\n)",
      "section_id": "register-jobs-compute"
    },
    {
      "id": "register-jobs-compute-ex1",
      "language": "bash",
      "code": "ff provider register <databricks-provider-name> \\\n  --workspace <workspace-id> \\\n  --type databricks \\\n  --databricks-workspace-url https://<databricks-workspace-host> \\\n  --databricks-auth-type pat \\\n  --databricks-token-secret env:DATABRICKS_TOKEN \\\n  --databricks-compute-target job_cluster \\\n  --databricks-job-spark-version <databricks-runtime-version> \\\n  --databricks-job-node-type-id <worker-node-type-id> \\\n  --databricks-job-num-workers <worker-count> \\\n  --databricks-output-catalog <catalog-provider-name> \\\n  --databricks-artifact-store-type unity_catalog \\\n  --databricks-artifact-store-provider <catalog-provider-name> \\\n  --databricks-artifact-store-schema <schema-name> \\\n  --databricks-artifact-store-volume <volume-name> \\\n  --default-managed-table-format delta_uniform",
      "section_id": "register-jobs-compute"
    },
    {
      "id": "apply-a-compute-policy-and-tags-ex0",
      "language": "python",
      "code": "job_cluster = DatabricksJobClusterConfig(\n    spark_version=\"<databricks-runtime-version>\",\n    node_type_id=\"<worker-node-type-id>\",\n    num_workers=int(\"<worker-count>\"),\n    policy_id=\"<cluster-policy-id>\",\n    apply_policy_default_values=True,\n    custom_tags={\n        \"team\": \"<team-name>\",\n        \"purpose\": \"featureform\",\n    },\n)",
      "section_id": "apply-a-compute-policy-and-tags"
    },
    {
      "id": "apply-a-compute-policy-and-tags-ex1",
      "language": "text",
      "code": "--databricks-job-policy-id <cluster-policy-id>\n--databricks-job-apply-policy-default-values\n--databricks-job-custom-tag team=<team-name>\n--databricks-job-custom-tag purpose=featureform",
      "section_id": "apply-a-compute-policy-and-tags"
    },
    {
      "id": "apply-a-compute-policy-and-tags-ex2",
      "language": "text",
      "code": "--databricks-job-autoscale-min-workers <minimum-workers>\n--databricks-job-autoscale-max-workers <maximum-workers>",
      "section_id": "apply-a-compute-policy-and-tags"
    },
    {
      "id": "configure-aws-attributes-ex0",
      "language": "python",
      "code": "from featureform.types import DatabricksAWSAttributesConfig\n\naws_attributes = DatabricksAWSAttributesConfig(\n    availability=\"SPOT_WITH_FALLBACK\",\n    first_on_demand=1,\n    spot_bid_price_percent=100,\n    instance_profile_arn=\"<instance-profile-arn>\",\n)",
      "section_id": "configure-aws-attributes"
    },
    {
      "id": "configure-aws-attributes-ex1",
      "language": "text",
      "code": "--databricks-job-aws-availability SPOT_WITH_FALLBACK\n--databricks-job-aws-first-on-demand 1\n--databricks-job-aws-spot-bid-price-percent 100\n--databricks-job-aws-instance-profile-arn <instance-profile-arn>",
      "section_id": "configure-aws-attributes"
    },
    {
      "id": "choose-artifact-storage-ex0",
      "language": "text",
      "code": "--databricks-artifact-store-type unity_catalog\n--databricks-artifact-store-provider <catalog-provider-name>\n--databricks-artifact-store-schema <schema-name>\n--databricks-artifact-store-volume <volume-name>",
      "section_id": "choose-artifact-storage"
    },
    {
      "id": "use-databricks-secrets-ex0",
      "language": "python",
      "code": "from featureform.types import (\n    DatabricksSecretConfig,\n    DatabricksSecretPATAuth,\n    SecretProviderType,\n)\n\nsecret_providers = client.secret_providers(\"<workspace-id>\")\nsecret_providers.register(\n    name=\"<databricks-secret-provider-name>\",\n    provider_type=SecretProviderType.DATABRICKS,\n    config=DatabricksSecretConfig(\n        workspace_url=\"https://<databricks-workspace-host>\",\n        auth=DatabricksSecretPATAuth(\n            token_secret=EnvSecretRef(name=\"DATABRICKS_TOKEN\"),\n        ),\n    ),\n)",
      "section_id": "use-databricks-secrets"
    },
    {
      "id": "use-databricks-secrets-ex1",
      "language": "bash",
      "code": "ff secret-provider register <databricks-secret-provider-name> \\\n  --workspace <workspace-id> \\\n  --type databricks-secret \\\n  --workspace-url https://<databricks-workspace-host> \\\n  --auth-type pat \\\n  --token-secret env:DATABRICKS_TOKEN",
      "section_id": "use-databricks-secrets"
    },
    {
      "id": "use-databricks-secrets-ex2",
      "language": "python",
      "code": "from featureform.types import (\n    DatabricksSecretConfig,\n    DatabricksSecretOAuthM2MAuth,\n    K8sSecretRef,\n    SecretProviderType,\n)\n\nsecret_providers.register(\n    name=\"<databricks-secret-provider-name>\",\n    provider_type=SecretProviderType.DATABRICKS,\n    config=DatabricksSecretConfig(\n        workspace_url=\"https://<databricks-workspace-host>\",\n        auth=DatabricksSecretOAuthM2MAuth(\n            client_id=\"<client-id>\",\n            client_secret_secret=K8sSecretRef(\n                provider_name=\"<k8s-secret-provider-name>\",\n                name=\"<secret-name>\",\n                key=\"<client-secret-key>\",\n            ),\n        ),\n    ),\n)",
      "section_id": "use-databricks-secrets"
    },
    {
      "id": "use-databricks-secrets-ex3",
      "language": "bash",
      "code": "ff secret-provider register <databricks-secret-provider-name> \\\n  --workspace <workspace-id> \\\n  --type databricks-secret \\\n  --workspace-url https://<databricks-workspace-host> \\\n  --auth-type oauth_m2m \\\n  --client-id <client-id> \\\n  --client-secret-secret k8s@<k8s-secret-provider-name>:<secret-name>#<client-secret-key>",
      "section_id": "use-databricks-secrets"
    },
    {
      "id": "connect-databricks-compute-to-redis-ex0",
      "language": "python",
      "code": "from featureform.types import DatabricksSecretRef, ProviderType, RedisConfig\n\nproviders.register(\n    name=\"<redis-provider-name>\",\n    provider_type=ProviderType.REDIS,\n    config=RedisConfig(\n        host=\"<redis-host>\",\n        port=int(\"<redis-port>\"),\n        username=\"<redis-username>\",\n        password_secret=DatabricksSecretRef(\n            provider_name=\"<databricks-secret-provider-name>\",\n            scope=\"<secret-scope>\",\n            key=\"<redis-password-key>\",\n        ),\n    ),\n)",
      "section_id": "connect-databricks-compute-to-redis"
    },
    {
      "id": "connect-databricks-compute-to-redis-ex1",
      "language": "bash",
      "code": "ff provider register <redis-provider-name> \\\n  --workspace <workspace-id> \\\n  --type redis \\\n  --redis-host <redis-host> \\\n  --redis-port <redis-port> \\\n  --redis-username <redis-username> \\\n  --redis-password-secret databricks@<databricks-secret-provider-name>:<secret-scope>#<redis-password-key>",
      "section_id": "connect-databricks-compute-to-redis"
    },
    {
      "id": "verify-the-providers-ex0",
      "language": "bash",
      "code": "ff provider get <catalog-provider-name> --workspace <workspace-id>\nff provider get <databricks-provider-name> --workspace <workspace-id>\nff provider get <redis-provider-name> --workspace <workspace-id>\nff secret-provider get <databricks-secret-provider-name> --workspace <workspace-id>\nff provider list --workspace <workspace-id>",
      "section_id": "verify-the-providers"
    }
  ]
}