> For the complete documentation index, see [llms.txt](https://dataflint.gitbook.io/dataflint-for-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dataflint.gitbook.io/dataflint-for-spark/integrations/databricks.md).

# Databricks

Connect a Databricks workspace to DataFlint with a read-only service principal

Connect a Databricks workspace to DataFlint with a read-only service principal

## Summary

DataFlint connects to your Databricks workspace as a service principal that you create. It needs **Can View** access to the jobs and clusters you want monitored and read access to their cluster logs. It never runs code on your clusters, never reads tables or files, and cannot start, stop or change clusters or jobs.

Setup has four steps:

1. Install the DataFlint Spark plugin (optional, adds deeper metrics).
2. Create a service principal and grant it Can View on the jobs and clusters to monitor.
3. Enable cluster log delivery (DBFS, S3 or a Unity Catalog volume).
4. Add the workspace in the DataFlint console.

### Supported clouds

| Cloud             | Status     | Cluster logs                     |
| ----------------- | ---------- | -------------------------------- |
| Databricks on AWS | Supported  | DBFS, S3 or Unity Catalog volume |
| Azure Databricks  | Supported  | DBFS or Unity Catalog volume     |
| Databricks on GCP | Contact us | —                                |

### What DataFlint reads

* **Spark event logs** from cluster log delivery: execution plans, stage and task metrics, shuffle data and, when the plugin is installed, DataFlint's enrichment.
* **Cluster metadata**: configuration, node types, autoscaling settings, runtime versions and cluster events.
* **Job and run metadata**: job definitions, run history, durations, status and run output for failure diagnosis.
* **Optional:** billing usage from Unity Catalog system tables, for DBU-level cost attribution (see [#optional-cost-analytics-from-system-tables](#optional-cost-analytics-from-system-tables "mention")).

DataFlint does not read your business data, user tables or files stored in Delta Lake or cloud storage.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FDJIeuvBYLQ734dXDADOL%2Fimage.png?alt=media&#x26;token=4fc88815-36a5-4190-9c22-9db42a05b358" alt=""><figcaption></figcaption></figure>

## Step 1: Install the DataFlint Spark plugin (optional)

The open-source plugin enriches the Spark event log with operator-level metrics and query metadata. It is optional; DataFlint ingests standard Spark event logs without it. Installation is described on [Install on DataBricks](/dataflint-for-spark/getting-started/install-on-databricks.md).

## Step 2: Create a service principal

{% hint style="warning" %}
This step requires workspace admin permission.
{% endhint %}

{% tabs %}
{% tab title="Workspace UI" %}
**1.** In your workspace, open **Settings** (top-right menu) → **Identity and access** → **Service principals**.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2F91GW3MUZU4vwLLjWNfCv%2Fimage.png?alt=media&#x26;token=fdd7a51a-428c-488f-85ce-41be4935dc18" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FMIUOxDr46RYRBKAA4tIz%2Fimage.png?alt=media&#x26;token=aa692651-c8e5-40ec-b3cc-634e8d1de59d" alt=""><figcaption></figcaption></figure>

**2.** Click **Add service principal**, give it a descriptive name such as `DataFlintServicePrincipal`, and add it.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FzoddrnhotdU9BZPWHW2C%2Fimage.png?alt=media&#x26;token=9f64180d-7fbc-4c8c-aa07-5a64ab53b06e" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FPBxticWp2uLvUWnv1EUs%2Fimage.png?alt=media&#x26;token=23cc95ce-e50d-4ebe-a0f0-30136da544fb" alt=""><figcaption></figcaption></figure>

**3.** Open the service principal, go to **Secrets**, and click **Generate secret**.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FdzGRAkt9CkPONULMfgNz%2Fimage.png?alt=media&#x26;token=628bd7fa-33c2-453e-b016-b6c15a599bc0" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FjzIR4vpL9gQo6qCQKezm%2Fimage.png?alt=media&#x26;token=1747b8b4-d07d-41e0-8dd8-b9c235b100b0" alt=""><figcaption></figcaption></figure>

**4.** Copy the **Client ID** and the **Secret**. The secret is shown once.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2F2jwMi7fvx20k8jR9dgtf%2Fimage.png?alt=media&#x26;token=2fad7933-f32e-4ba4-8183-7dec48db9326" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Databricks CLI" %}

```bash
databricks service-principals create --display-name DataFlintServicePrincipal
databricks service-principal-secrets-proxy create <SERVICE_PRINCIPAL_ID>
```

The second command prints the client ID and the secret. The secret is shown once.
{% endtab %}
{% endtabs %}

DataFlint authenticates with OAuth machine-to-machine (client credentials) using this client ID and secret. A personal access token is also accepted if your workspace policy prefers it.

### Grant Can View

Grant the service principal **Can View** on every job and every cluster you want DataFlint to monitor. In the UI: open the job (or cluster) → **Permissions** → add the service principal with **Can View**. For many jobs, set this in your job template or through your orchestration tooling.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FfG39W2cPZqjbM1ROiCSn%2Fimage.png?alt=media&#x26;token=08b4ee6b-ec51-4776-9868-2f36593b9d26" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FxTAcTLQYvezOALYZgBO4%2Fimage.png?alt=media&#x26;token=6b2960a4-4d67-413f-9485-540282e5d328" alt=""><figcaption></figcaption></figure>

## Step 3: Enable cluster log delivery

For each cluster (or the cluster policy your jobs use), open **Compute → Advanced options → Logging** and set a log destination. Each cluster must have exactly one destination, but clusters do not have to share it: DataFlint reads every cluster's log configuration and resolves the log path from it, so some clusters can write to DBFS, others to S3 or a Unity Catalog volume, in one location or several. The only requirement is that DataFlint can read every location in use. Can View covers DBFS; three Unity Catalog grants cover a volume; the log access role below covers S3.

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FZ2GfxehE28gO2FuDfUEe%2Fimage.png?alt=media&#x26;token=d20f6804-5c63-4a8e-abe5-e2d96fe37432" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2F5pegmSqOvJ3zns9Au65v%2Fimage.png?alt=media&#x26;token=15431196-f3e6-4bc8-aa57-7de1b3c89a8c" alt=""><figcaption></figcaption></figure>

* **DBFS** — no extra setup. Can View on the cluster already lets DataFlint read its logs.
* **Unity Catalog volume (AWS and Azure)** — grant the service principal read access to the volume. No cloud role is needed; Unity Catalog governs the access. `USE CATALOG` is often already held by all account users.

  ```sql
  GRANT USE CATALOG ON CATALOG <catalog>                   TO `DataFlintServicePrincipal`;
  GRANT USE SCHEMA  ON SCHEMA  <catalog>.<schema>          TO `DataFlintServicePrincipal`;
  GRANT READ VOLUME ON VOLUME  <catalog>.<schema>.<volume> TO `DataFlintServicePrincipal`;
  ```

  Volume log delivery needs Unity Catalog compute in Standard access mode or Dedicated mode assigned to a user, and the compute owner needs `READ VOLUME` and `WRITE VOLUME` on the volume, because Databricks writes the logs as the owner.
* **S3 (Databricks on AWS)** — DataFlint reads the bucket through an IAM role in your account that you create and DataFlint assumes with your external ID. The DataFlint console generates the Terraform or CloudFormation for it (Admin Panel → Environment Management → Add New → Databricks → choose S3 → "Need to create this role?"). The role grants only `s3:GetObject` and `s3:ListBucket` on the log bucket. Enter its ARN in the **Log access role ARN** field.

{% hint style="info" %}
If your clusters write logs to more than one bucket, add each bucket to the role's policy.
{% endhint %}

### Log access role (S3 only)

The console generates this role for you. The listing below is the reference your security team can review; if the generated file and this page differ, this page is authoritative.

* **DataFlint principal:** `arn:aws:iam::975050001706:role/eks-dataflint-service-role`
* **Your external ID:** generated for your DataFlint account and shown in the console when you add the workspace. It is required in the trust policy so that only your DataFlint account can use the role.

{% hint style="warning" %}
BYOC customers: the principal is the `eks-dataflint-service-role` in your own DataFlint account, not the one above. The console shows the correct value.
{% endhint %}

**Trust policy.** Replace `YOUR_EXTERNAL_ID` with the value shown in the console.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::975050001706:role/eks-dataflint-service-role"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR_EXTERNAL_ID"
        }
      }
    }
  ]
}
```

**Permissions policy.** Replace `YOUR_LOG_BUCKET` with the bucket your clusters deliver logs to. Add a second pair of resources for every additional log bucket.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDatabricksClusterLogRead",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_LOG_BUCKET",
        "arn:aws:s3:::YOUR_LOG_BUCKET/*"
      ]
    }
  ]
}
```

| Actions                                           | What DataFlint does with it                                                                  | What it cannot do                                      |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| `s3:ListBucket`, `s3:GetObject` on the log bucket | Finds and reads the Spark event log and driver log that each cluster delivers to the bucket. | Write or delete, or read outside the buckets you list. |

If the bucket uses SSE-KMS, also allow `kms:Decrypt` on the key for the same role.

## Step 4: Add the workspace in DataFlint

In the DataFlint console go to **Admin Panel → Environment Management → Add New → Databricks** and fill in:

* **Workspace name**
* **Workspace URL** — for example `https://dbc-abc123.cloud.databricks.com` or `https://adb-1234567890.12.azuredatabricks.net`
* **Auth type** — OAuth M2M (client ID and secret) or personal access token
* **Region** and **Cloud provider**
* **Cluster log destination** — choose **S3** if any cluster writes to an `s3://` path, even when other clusters write to DBFS, and enter the **Log access role ARN**. Choose **DBFS** only when no cluster uses S3. Clusters that log to a Unity Catalog volume need no choice here; the grants in Step 3 are enough. AWS only; Azure workspaces use DBFS.

Click **Add Pipeline**. The environment shows as **Pending approval** while DataFlint verifies access; ingestion starts once it is approved, and runs appear in the console after the next completed job.

## Tagging your jobs

Add these tags on the job, on the cluster, or as Spark configuration, and DataFlint groups and filters by them:

<figure><img src="https://2982210886-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fcg8pTm3VgVaeMncRl8LP%2Fuploads%2FEbTHZeq4qmN1p8ak7Osl%2Fimage.png?alt=media&#x26;token=c0a76004-3eae-4e52-bf3f-634b015e460f" alt=""><figcaption></figcaption></figure>

| Tag                                                             | Meaning                                                                                                                           |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `DATAFLINT_ENV`                                                 | Environment, for example `prod` or `dev`                                                                                          |
| `DATAFLINT_TEAM`                                                | Team that owns the job                                                                                                            |
| `DATAFLINT_DOMAIN`                                              | Business unit or product the job belongs to                                                                                       |
| `DATAFLINT_APP_NAME`                                            | Logical application name shared by related jobs                                                                                   |
| `DATAFLINT_DAG_ID`, `DATAFLINT_DAG_RUN_ID`, `DATAFLINT_TASK_ID` | Orchestrator identifiers, for example from Airflow                                                                                |
| `DATAFLINT_TRY_NUMBER`                                          | Optional. The task's attempt number, so retries show as attempts of one step. Needs `DATAFLINT_DAG_ID` and `DATAFLINT_DAG_RUN_ID` |
| `DATAFLINT_MAP_INDEX`                                           | Optional. The index of a mapped task instance, `-1` for unmapped tasks                                                            |
| `DATAFLINT_JOB_TYPE`, `DATAFLINT_VARIANT`, `DATAFLINT_VERSION`  | Job classification and version                                                                                                    |

Set `DATAFLINT_ENABLED=false` on a cluster or job to exclude it from ingestion. If you already tag with your own names, DataFlint can map them to these on request.

### Airflow example

Set the orchestrator tags from Airflow's templates, for example on the cluster of a `DatabricksSubmitRunOperator`:

```python
DatabricksSubmitRunOperator(
    task_id="transform",
    json={
        "new_cluster": {
            "spark_version": "15.4.x-scala2.12",
            "node_type_id": "i3.xlarge",
            "num_workers": 2,
            "custom_tags": {
                "DATAFLINT_DAG_ID": "{{ dag.dag_id }}",
                "DATAFLINT_DAG_RUN_ID": "{{ run_id }}",
                "DATAFLINT_TASK_ID": "{{ task.task_id }}",
                "DATAFLINT_TRY_NUMBER": "{{ ti.try_number }}",
                "DATAFLINT_MAP_INDEX": "{{ ti.map_index }}",
            },
        },
        "spark_python_task": {"python_file": "dbfs:/jobs/transform.py"},
    },
)
```

Or ask DataFlint about the Airflow integration, which reads your DAGs, runs and retries straight from Airflow.

## Optional: cost analytics from system tables

For DBU-level cost attribution, DataFlint can query the Unity Catalog `system.billing` schema through a SQL warehouse you designate. This is enabled together with your DataFlint contact. It needs `SELECT` on `system.billing.usage` and `system.billing.list_prices`, and `CAN USE` on the warehouse. No other catalog objects are read.

## Authentication details

DataFlint uses the OAuth 2.0 client-credentials flow with the official Databricks SDK. Tokens are requested from your workspace's OIDC token endpoint with scope `all-apis`, sent as `Authorization: Bearer` on every request, and refreshed automatically before they expire. No manual token management is needed.

**Partner telemetry.** DataFlint identifies itself to Databricks on every call as product `dataflint-ingestion/<version>` with partner tag `dataflint`, as required by the Databricks Technology Partner Program.

## API endpoints used

| Purpose                   | Endpoints                                                                                                                                                                                                              |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Clusters                  | `GET /api/2.1/clusters/list`, `GET /api/2.1/clusters/get`, `POST /api/2.1/clusters/events`                                                                                                                             |
| Jobs and runs             | `GET /api/2.1/jobs/list`, `GET /api/2.1/jobs/get`, `GET /api/2.1/jobs/runs/list`, `GET /api/2.1/jobs/runs/get`, `GET /api/2.1/jobs/runs/get-output`, `GET /api/2.2/jobs/runs/get`, `GET /api/2.2/jobs/runs/get-output` |
| Serverless query history  | `GET /api/2.0/sql/history/queries`                                                                                                                                                                                     |
| Instance pools            | `GET /api/2.0/instance-pools/get`                                                                                                                                                                                      |
| Cluster logs on DBFS      | `GET /api/2.0/dbfs/list`, `GET /api/2.0/dbfs/read`                                                                                                                                                                     |
| Cluster logs on a volume  | `GET /api/2.0/fs/directories/{path}`, `GET /api/2.0/fs/files/{path}`                                                                                                                                                   |
| Identity                  | `GET /api/2.0/preview/scim/v2/Me`, `POST /oidc/v1/token`                                                                                                                                                               |
| Cost analytics (optional) | `POST /api/2.0/sql/statements`, `GET /api/2.0/sql/statements/{id}`                                                                                                                                                     |

All calls are read-only from DataFlint to your workspace over HTTPS.

## Troubleshooting

**No runs appear in DataFlint.** Check that the environment is approved, that the service principal has Can View on the jobs and clusters, that cluster log delivery is enabled with exactly one destination, and that a job has completed since the environment was added.

**Permission denied (403) reported for a cluster or job.** The service principal lacks Can View on it; for S3 logs, the log-access role cannot read the bucket; for volume logs, the service principal lacks `READ VOLUME` on the volume or `USE` on its catalog or schema. The console's ingestion status names the object and the missing permission.

**Ingestion stops after the secret is rotated.** Edit the environment in the console and enter the new client secret or token.

## Security summary

* **Read-only.** No write, delete or modify operation is ever performed in your workspace.
* **Metadata only.** Logs, cluster configuration and job metadata; no tables, files or query results.
* **Encrypted.** All traffic is TLS; credentials are stored encrypted and never logged.
* **Revocable.** Delete the service principal's secret, or the principal itself, to end access at any time.

## Support

* Email: <support@dataflint.io>
* Databricks Partner Operations: <partnerops@databricks.com>