Zero Copy Prerequisites: Databricks Connection
Optimove Zero Copy reads data directly from your Databricks SQL warehouse using a dedicated service principal that authenticates with OAuth machine-to-machine (M2M) credentials. This guide covers the prerequisites your team completes in Databricks before the connection can be established: creating the service principal, granting workspace and Unity Catalog permissions, collecting the connection attributes, and verifying the setup.
This is a one-time setup performed by a Databricks administrator on your workspace. No customer data leaves your Databricks environment during these steps.
You will need account admin access to the Databricks Account Console, workspace admin access to the target workspace, a provisioned SQL Warehouse, and a Unity Catalog catalog, schema, and table that already contain your customer data.
Prerequisites
- A Databricks workspace with Unity Catalog enabled
- Account admin access — required to create service principals in the Account Console
- Workspace admin access — required to manage entitlements and SQL Warehouse permissions
- A SQL Warehouse provisioned (Serverless, Pro, or Classic)
- The target catalog, schema, and table already created and populated
- Outbound connectivity over HTTPS (port 443) — no additional port configuration is required
Connection Attributes
Collect the following values before you configure the connection in the Optimove platform.
| Attribute | Description | Required | Example |
|---|---|---|---|
| Host | Server hostname of your Databricks workspace | Always | dbc-abc123.cloud.databricks.com |
| HTTP Path | SQL Warehouse HTTP path | Always | /sql/1.0/warehouses/xyz789 |
| Catalog | Unity Catalog name containing your data | Always | production |
| Client ID | Service principal Application ID (UUID) | If using OAuth | 12345678-abcd-efgh-ijkl-123456789012 |
| Client Secret | Service principal client secret | If using OAuth | Generated during setup |
| Token | Personal Access Token | If using PAT | dapi_xxxxxxxxxxxxxxxx |
Authentication: OAuth M2M or PAT
Provide either OAuth service principal credentials (Client ID and Client Secret) or a Personal Access Token. OAuth M2M is the recommended approach for production use. For the PAT path, see Alternative: Personal Access Token (PAT) below.
Step 1: Create a Service Principal
- Go to your Databricks Account Console — the account-level admin console, not the workspace.
- Navigate to User management > Service principals.
- Click Add service principal and give it a descriptive name, for example
optimove-zero-copy. - Note the Application ID (UUID) — this is your
client_id.
The Unity CatalogGRANTstatements in Step 5 reference the service principal by this name. If you choose a different name, substitute it consistently throughout.
Step 2: Generate the OAuth Client Secret
- In the Account Console, select the service principal you created.
- Open the Secrets tab.
- Click Generate secret.
- Copy and securely store both the Client ID and the Client Secret.
The client secret is displayed only at creation time. Store it securely. If it is lost, generate a new one.
OAuth secrets have a maximum lifetime of 730 days (2 years). Plan to rotate the secret before it expires to avoid connection failures. Each service principal supports up to 5 active secrets, so you can generate a replacement before the current one expires.
Step 3: Assign the Service Principal to the Workspace
The service principal is created at the account level. It must be explicitly added to the target workspace and granted the required entitlements.
- In the Account Console, go to User management > Service principals.
- Select your service principal.
- Open the Workspaces tab and verify the service principal is assigned to the workspace where your data resides. If it is not, click Add to workspace and select the target workspace.
- Under Workspace entitlements, enable:
- Workspace access
- Databricks SQL access
Without Workspace access, the service principal cannot connect to the workspace at all. Without Databricks SQL access, it cannot execute SQL queries.
Step 4: Grant SQL Warehouse Access
The service principal needs CAN USE permission on the SQL Warehouse. This is configured through the workspace UI, not via SQL.
- Open your Databricks Workspace.
- Navigate to SQL Warehouses in the left sidebar.
- Select the warehouse you want Optimove to use.
- Open the Permissions tab.
- Add the service principal and grant Can Use.
SQL Warehouse typeClassic and Pro SQL warehouses may have 2–5 minutes of warm-up time when a connection is first established or when a scheduled sync begins. A Serverless SQL warehouse minimizes warm-up delays and improves query throughput, at the cost of slightly higher compute spend.
Auto-start
If the SQL Warehouse has auto-stopped due to inactivity, it starts automatically when Optimove connects. This may add a delay to the first query after an idle period. No manual intervention is required.
Step 5: Grant Unity Catalog Permissions
Run the following in the Databricks SQL Editor. Replace the placeholders with your actual catalog, schema, and service principal name.
Grant catalog access:
GRANT USE CATALOG ON CATALOG `your_catalog` TO `optimove-zero-copy`;Grant schema access:
GRANT USE SCHEMA ON SCHEMA `your_catalog`.`your_schema` TO `optimove-zero-copy`;Grant read access on tables:
-- Grant SELECT on all current tables in the schema
GRANT SELECT ON SCHEMA `your_catalog`.`your_schema` TO `optimove-zero-copy`;
GrantingSELECT ON SCHEMAcovers all current and future tables in that schema — this is the Unity Catalog equivalent of Snowflake'sFUTUREgrants. If you prefer fine-grained control, grantSELECTon individual tables instead:
GRANT SELECT ON TABLE `your_catalog`.`your_schema`.`your_table` TO `optimove-zero-copy`;Step 6: Collect the Host and HTTP Path
- In your Databricks Workspace, go to SQL Warehouses.
- Select the warehouse Optimove will connect to.
- Open the Connection Details tab.
- Copy:
- Server hostname → the
hostvalue - HTTP path → the
http_pathvalue
- Server hostname → the
Step 7: Verify the Setup
Confirm the grants are in place before testing the connection.
Check grants on catalog:
SHOW GRANTS ON CATALOG `your_catalog`;Check grants on schema:
SHOW GRANTS ON SCHEMA `your_catalog`.`your_schema`;Check grants for service principal:
SHOW GRANTS TO `optimove-zero-copy`;Step 8: Test the Connection in Optimove
Once all steps are complete, run the connection test in the Optimove platform's Zero Copy configuration. The test validates:
- Network connectivity and authentication to the workspace
- Access to the specified catalog (
USE CATALOG) - The ability to execute queries on the SQL warehouse
Step 9: Configure the IP Access List
Apply network restrictions after you have confirmed the connection works in Step 8.
If your Databricks workspace has IP Access Lists enabled, you must allowlist Optimove's IP addresses.
- In your Databricks Workspace, go to Settings > Security > IP Access Lists.
- Add all Optimove IP addresses for your region — US or EU.
- Find the full list on the IP Allow List page.
Include all Optimove IPs for your region. The service may connect from any IP in the regional pool, so a partial list causes intermittent connection failures.
IP Access Lists are a Databricks Enterprise tier feature and are disabled by default. If your workspace does not have them enabled, no action is needed for this step. Changes to IP access lists may take several minutes to propagate.
Permissions Summary
| Permission | Scope | How to grant | Purpose |
|---|---|---|---|
| Workspace access | Workspace | Account Console UI | Connect to the workspace |
| Databricks SQL access | Workspace | Account Console UI | Execute SQL queries |
CAN USE | SQL Warehouse | Workspace UI, Permissions tab | Use the compute resource |
USE CATALOG | Catalog | SQL GRANT statement | Access the catalog |
USE SCHEMA | Schema | SQL GRANT statement | Access the schema |
SELECT | Schema or table | SQL GRANT statement | Read table data |
Alternative: Personal Access Token (PAT)
If OAuth M2M is not available in your environment, you can authenticate with a Personal Access Token instead.
A PAT is tied to an identity and inherits the permissions of the user or service principal it was generated for. OAuth M2M with a service principal is recommended for production.
To generate a PAT for a user:
- In the Databricks Workspace, click your user icon in the top right.
- Go to Settings > Developer.
- Click Manage next to Access tokens.
- Click Generate new token.
- Set a description and an expiration (maximum 730 days), then copy the token.
Databricks automatically revokes PATs that have not been used for 90 days. Ensure scheduled syncs run regularly to keep the token active. Workspace admins may also enforce a maximum lifetime for new tokens.
To generate a PAT for a service principal, use the REST API:
curl -X POST "https://YOUR_HOST/api/2.0/token-management/on-behalf-of/tokens" -H "Authorization: Bearer ADMIN_PAT" -H "Content-Type: application/json" -d '{"application_id": "SERVICE_PRINCIPAL_APP_ID", "comment": "Optimove ZCL", "lifetime_seconds": 7776000}'When using a PAT, provide the token value instead of client_id and client_secret. All of the catalog, schema, and table grants in Step 5 are still required — the PAT inherits permissions from the identity it was generated for.
Troubleshooting
| Error | Cause | Fix |
|---|---|---|
403 FORBIDDEN | Service principal lacks CAN USE on the SQL Warehouse, or is missing workspace entitlements | Verify Workspace access and Databricks SQL access are enabled (Step 3) and that CAN USE is granted on the warehouse (Step 4) |
User does not have USE CATALOG on Catalog | Missing USE CATALOG grant | Run the catalog grant in Step 5 |
User does not have USE SCHEMA on Schema | Missing USE SCHEMA grant | Run the schema grant in Step 5 |
User does not have SELECT on Table | Missing SELECT grant on the table or schema | Run the SELECT grant in Step 5 |
Invalid access token | The PAT or OAuth client secret is incorrect or expired | Regenerate the token or client secret and update the connection |
CATALOG_NOT_FOUND | The specified catalog name does not exist | Verify the catalog name in Catalog Explorer |
IP is not allowed | The workspace IP access list is blocking the service | Ensure all Optimove IPs for your region are allowlisted (Step 9) |
Updated about 2 hours ago
