Zero Copy Prerequisites: Databricks Connection

📘

Beta: Zero Copy is currently in Open Beta and is being rolled out incrementally to all customers.

Optimove Zero Copy reads data directly from your Databricks SQL warehouse using a dedicated service principal that authenticates with OAuth machine-to-machine (M2M) credentials. This guide covers the prerequisites your team completes in Databricks before the connection can be established: creating the service principal, granting workspace and Unity Catalog permissions, collecting the connection attributes, and verifying the setup.

This is a one-time setup performed by a Databricks administrator on your workspace. No customer data leaves your Databricks environment during these steps.

📘

You will need account admin access to the Databricks Account Console, workspace admin access to the target workspace, a provisioned SQL Warehouse, and a Unity Catalog catalog, schema, and table that already contain your customer data.

Prerequisites

  • A Databricks workspace with Unity Catalog enabled
  • Account admin access — required to create service principals in the Account Console
  • Workspace admin access — required to manage entitlements and SQL Warehouse permissions
  • A SQL Warehouse provisioned (Serverless, Pro, or Classic)
  • The target catalog, schema, and table already created and populated
  • Outbound connectivity over HTTPS (port 443) — no additional port configuration is required

Connection Attributes

Collect the following values before you configure the connection in the Optimove platform.

AttributeDescriptionRequiredExample
HostServer hostname of your Databricks workspaceAlwaysdbc-abc123.cloud.databricks.com
HTTP PathSQL Warehouse HTTP pathAlways/sql/1.0/warehouses/xyz789
CatalogUnity Catalog name containing your dataAlwaysproduction
Client IDService principal Application ID (UUID)If using OAuth12345678-abcd-efgh-ijkl-123456789012
Client SecretService principal client secretIf using OAuthGenerated during setup
TokenPersonal Access TokenIf using PATdapi_xxxxxxxxxxxxxxxx
💡

Authentication: OAuth M2M or PAT
Provide either OAuth service principal credentials (Client ID and Client Secret) or a Personal Access Token. OAuth M2M is the recommended approach for production use. For the PAT path, see Alternative: Personal Access Token (PAT) below.

Step 1: Create a Service Principal

  1. Go to your Databricks Account Console — the account-level admin console, not the workspace.
  2. Navigate to User management > Service principals.
  3. Click Add service principal and give it a descriptive name, for example optimove-zero-copy.
  4. Note the Application ID (UUID) — this is your client_id.
📘

The Unity Catalog GRANT statements in Step 5 reference the service principal by this name. If you choose a different name, substitute it consistently throughout.

Step 2: Generate the OAuth Client Secret

  1. In the Account Console, select the service principal you created.
  2. Open the Secrets tab.
  3. Click Generate secret.
  4. Copy and securely store both the Client ID and the Client Secret.
⚠️

The client secret is displayed only at creation time. Store it securely. If it is lost, generate a new one.

📘

OAuth secrets have a maximum lifetime of 730 days (2 years). Plan to rotate the secret before it expires to avoid connection failures. Each service principal supports up to 5 active secrets, so you can generate a replacement before the current one expires.

Step 3: Assign the Service Principal to the Workspace

The service principal is created at the account level. It must be explicitly added to the target workspace and granted the required entitlements.

  1. In the Account Console, go to User management > Service principals.
  2. Select your service principal.
  3. Open the Workspaces tab and verify the service principal is assigned to the workspace where your data resides. If it is not, click Add to workspace and select the target workspace.
  4. Under Workspace entitlements, enable:
    • Workspace access
    • Databricks SQL access
📘

Without Workspace access, the service principal cannot connect to the workspace at all. Without Databricks SQL access, it cannot execute SQL queries.

Step 4: Grant SQL Warehouse Access

The service principal needs CAN USE permission on the SQL Warehouse. This is configured through the workspace UI, not via SQL.

  1. Open your Databricks Workspace.
  2. Navigate to SQL Warehouses in the left sidebar.
  3. Select the warehouse you want Optimove to use.
  4. Open the Permissions tab.
  5. Add the service principal and grant Can Use.
💡

SQL Warehouse type

Classic and Pro SQL warehouses may have 2–5 minutes of warm-up time when a connection is first established or when a scheduled sync begins. A Serverless SQL warehouse minimizes warm-up delays and improves query throughput, at the cost of slightly higher compute spend.

📘

Auto-start
If the SQL Warehouse has auto-stopped due to inactivity, it starts automatically when Optimove connects. This may add a delay to the first query after an idle period. No manual intervention is required.

Step 5: Grant Unity Catalog Permissions

Run the following in the Databricks SQL Editor. Replace the placeholders with your actual catalog, schema, and service principal name.

Grant catalog access:

GRANT USE CATALOG ON CATALOG `your_catalog` TO `optimove-zero-copy`;

Grant schema access:

GRANT USE SCHEMA ON SCHEMA `your_catalog`.`your_schema` TO `optimove-zero-copy`;

Grant read access on tables:

-- Grant SELECT on all current tables in the schema
GRANT SELECT ON SCHEMA `your_catalog`.`your_schema` TO `optimove-zero-copy`;
💡

Granting SELECT ON SCHEMA covers all current and future tables in that schema — this is the Unity Catalog equivalent of Snowflake's FUTURE grants. If you prefer fine-grained control, grant SELECT on individual tables instead:

GRANT SELECT ON TABLE `your_catalog`.`your_schema`.`your_table` TO `optimove-zero-copy`;

Step 6: Collect the Host and HTTP Path

  1. In your Databricks Workspace, go to SQL Warehouses.
  2. Select the warehouse Optimove will connect to.
  3. Open the Connection Details tab.
  4. Copy:
    • Server hostname → the host value
    • HTTP path → the http_path value

Step 7: Verify the Setup

Confirm the grants are in place before testing the connection.

Check grants on catalog:

SHOW GRANTS ON CATALOG `your_catalog`;

Check grants on schema:

SHOW GRANTS ON SCHEMA `your_catalog`.`your_schema`;

Check grants for service principal:

SHOW GRANTS TO `optimove-zero-copy`;

Step 8: Test the Connection in Optimove

Once all steps are complete, run the connection test in the Optimove platform's Zero Copy configuration. The test validates:

  1. Network connectivity and authentication to the workspace
  2. Access to the specified catalog (USE CATALOG)
  3. The ability to execute queries on the SQL warehouse

Step 9: Configure the IP Access List

Apply network restrictions after you have confirmed the connection works in Step 8.

If your Databricks workspace has IP Access Lists enabled, you must allowlist Optimove's IP addresses.

  1. In your Databricks Workspace, go to Settings > Security > IP Access Lists.
  2. Add all Optimove IP addresses for your region — US or EU.
  3. Find the full list on the IP Allow List page.
⚠️

Include all Optimove IPs for your region. The service may connect from any IP in the regional pool, so a partial list causes intermittent connection failures.

📘

IP Access Lists are a Databricks Enterprise tier feature and are disabled by default. If your workspace does not have them enabled, no action is needed for this step. Changes to IP access lists may take several minutes to propagate.

Permissions Summary

PermissionScopeHow to grantPurpose
Workspace accessWorkspaceAccount Console UIConnect to the workspace
Databricks SQL accessWorkspaceAccount Console UIExecute SQL queries
CAN USESQL WarehouseWorkspace UI, Permissions tabUse the compute resource
USE CATALOGCatalogSQL GRANT statementAccess the catalog
USE SCHEMASchemaSQL GRANT statementAccess the schema
SELECTSchema or tableSQL GRANT statementRead table data

Alternative: Personal Access Token (PAT)

If OAuth M2M is not available in your environment, you can authenticate with a Personal Access Token instead.

⚠️

A PAT is tied to an identity and inherits the permissions of the user or service principal it was generated for. OAuth M2M with a service principal is recommended for production.

To generate a PAT for a user:

  1. In the Databricks Workspace, click your user icon in the top right.
  2. Go to Settings > Developer.
  3. Click Manage next to Access tokens.
  4. Click Generate new token.
  5. Set a description and an expiration (maximum 730 days), then copy the token.
⚠️

Databricks automatically revokes PATs that have not been used for 90 days. Ensure scheduled syncs run regularly to keep the token active. Workspace admins may also enforce a maximum lifetime for new tokens.

To generate a PAT for a service principal, use the REST API:

curl -X POST "https://YOUR_HOST/api/2.0/token-management/on-behalf-of/tokens" -H "Authorization: Bearer ADMIN_PAT" -H "Content-Type: application/json" -d '{"application_id": "SERVICE_PRINCIPAL_APP_ID", "comment": "Optimove ZCL", "lifetime_seconds": 7776000}'

When using a PAT, provide the token value instead of client_id and client_secret. All of the catalog, schema, and table grants in Step 5 are still required — the PAT inherits permissions from the identity it was generated for.

Troubleshooting

ErrorCauseFix
403 FORBIDDENService principal lacks CAN USE on the SQL Warehouse, or is missing workspace entitlementsVerify Workspace access and Databricks SQL access are enabled (Step 3) and that CAN USE is granted on the warehouse (Step 4)
User does not have USE CATALOG on CatalogMissing USE CATALOG grantRun the catalog grant in Step 5
User does not have USE SCHEMA on SchemaMissing USE SCHEMA grantRun the schema grant in Step 5
User does not have SELECT on TableMissing SELECT grant on the table or schemaRun the SELECT grant in Step 5
Invalid access tokenThe PAT or OAuth client secret is incorrect or expiredRegenerate the token or client secret and update the connection
CATALOG_NOT_FOUNDThe specified catalog name does not existVerify the catalog name in Catalog Explorer
IP is not allowedThe workspace IP access list is blocking the serviceEnsure all Optimove IPs for your region are allowlisted (Step 9)



Did this page help you?