> For the complete documentation index, see [llms.txt](https://dataflint.gitbook.io/dataflint-for-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dataflint.gitbook.io/dataflint-for-spark/saas-platform/dataproc-saas-installation.md).

# Google Cloud Dataproc

## Summary

DataFlint reads Spark event logs and cluster and job metadata from your Google Cloud Dataproc project through a service account that you create. The service account is **read-only**: DataFlint cannot create, change or delete clusters or jobs.

You will:

1. Create a dedicated service account.
2. Grant it Dataproc Viewer on the project and Storage Object Viewer on the buckets that hold Spark event logs and driver output.
3. Create a JSON key.
4. Add the project in the DataFlint console.

The entire process should take a few minutes.

{% hint style="info" %}
**Supported:** Dataproc on Compute Engine. For Dataproc Serverless, contact us.
{% endhint %}

{% hint style="warning" %}
Service account keys are sensitive credentials. Store them like passwords. The key is pasted into the DataFlint console and stored encrypted; it is never shared over email.
{% endhint %}

### How it works

Dataproc writes Spark event logs to Cloud Storage. By default this is the cluster's **temp bucket**, under `<cluster-uuid>/spark-job-history/`; clusters that set `spark:spark.eventLog.dir` write to the bucket you configured. Dataproc writes each job's driver output to the cluster's **staging bucket**. DataFlint discovers the exact locations from each cluster's configuration, so you do not need to send bucket names.

### Required IAM permissions (minimal)

Grant the DataFlint service account:

* Project-level: `roles/dataproc.viewer` — lists clusters and reads cluster and job metadata.
* Bucket-level: `roles/storage.objectViewer` on
  * every bucket your clusters write Spark event logs to (the Dataproc temp bucket by default, or the bucket in `spark:spark.eventLog.dir` if you set one);
  * the cluster staging bucket (or the output bucket you configured for the cluster), where Dataproc writes job driver output. DataFlint reads it to explain failed runs.

Both grants are read-only.

## Installation

Pick one method. All methods create the same resources.

{% tabs %}
{% tab title="Google Cloud Console (UI)" %}

### Step 1: Create a service account

1. Open **IAM & Admin → Service Accounts**.
2. Click **Create service account**.
3. Use:
   * Name: `dataflint-events-reader`
   * Description: `Read-only access to Dataproc Spark event logs`
4. Click **Done**.

### Step 2: Create a service account key (JSON)

1. Open the service account.
2. Go to **Keys**.
3. Click **Add key → Create new key**.
4. Select **JSON**.
5. Download and store the file securely.

### Step 3: Grant Dataproc Viewer

1. Open **IAM & Admin → IAM**.
2. Click **Grant access**.
3. Principal: `dataflint-events-reader@<PROJECT_ID>.iam.gserviceaccount.com`
4. Role: **Dataproc Viewer** (`roles/dataproc.viewer`)

### Step 4: Grant bucket read access

Repeat for the temp bucket (or your custom event-log bucket) and for the staging bucket:

1. Open **Cloud Storage → Buckets** and open the bucket.
   * The temp bucket usually looks like `dataproc-temp-<region>-<project-number>-...`; the staging bucket like `dataproc-staging-<region>-<project-number>-...`.
2. Open **Permissions**.
3. Click **Grant access**.
4. Add the same principal as above.
5. Role: **Storage Object Viewer** (`roles/storage.objectViewer`)
   {% endtab %}

{% tab title="gcloud / gsutil (CLI)" %}

### Prerequisites

```bash
gcloud auth login
gcloud config set project YOUR_PROJECT_ID
```

### Create service account

```bash
gcloud iam service-accounts create dataflint-events-reader \
  --display-name="DataFlint Spark Events Reader" \
  --description="Read-only access to Dataproc Spark event logs"
```

### Create key

```bash
gcloud iam service-accounts keys create dataflint-sa-key.json \
  --iam-account=dataflint-events-reader@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

### Grant Dataproc Viewer (project-level)

```bash
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:dataflint-events-reader@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/dataproc.viewer" \
  --condition=None
```

### Grant Storage Object Viewer (bucket-level)

```bash
# List candidate buckets
gsutil ls | grep -E "dataproc-(temp|staging)" || true

# Grant read-only access (repeat for every relevant bucket)
gsutil iam ch \
  serviceAccount:dataflint-events-reader@YOUR_PROJECT_ID.iam.gserviceaccount.com:objectViewer \
  gs://dataproc-temp-REGION-PROJECT_NUMBER-SUFFIX
gsutil iam ch \
  serviceAccount:dataflint-events-reader@YOUR_PROJECT_ID.iam.gserviceaccount.com:objectViewer \
  gs://dataproc-staging-REGION-PROJECT_NUMBER-SUFFIX
```

### Optional: one-shot setup script

{% code title="setup-dataflint-dataproc-reader.sh" %}

```bash
#!/bin/bash
set -euo pipefail

# EDIT THESE
PROJECT_ID="your-project-id"
REGION="us-central1"
SA_NAME="dataflint-events-reader"
KEY_OUTPUT_PATH="./dataflint-sa-key.json"

SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

echo "Creating service account (if missing)..."
gcloud iam service-accounts create "${SA_NAME}" \
  --display-name="DataFlint Spark Events Reader" \
  --description="Read-only access to Dataproc Spark event logs" \
  --project="${PROJECT_ID}" 2>/dev/null || true

echo "Creating key..."
gcloud iam service-accounts keys create "${KEY_OUTPUT_PATH}" \
  --iam-account="${SA_EMAIL}"

echo "Granting Dataproc Viewer..."
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/dataproc.viewer" \
  --condition=None \
  --quiet

echo "Granting bucket access for dataproc temp and staging buckets in region ${REGION}..."
for bucket in $(gsutil ls | grep -E "dataproc-(temp|staging)-${REGION}" || true); do
  echo "  ${bucket}"
  gsutil iam ch "serviceAccount:${SA_EMAIL}:objectViewer" "${bucket}"
done

echo "Done."
echo "Service account: ${SA_EMAIL}"
echo "Key file: ${KEY_OUTPUT_PATH}"
```

{% endcode %}
{% endtab %}

{% tab title="Terraform" %}

### Terraform example

This creates:

* `dataflint-events-reader` service account
* a service account key
* `roles/dataproc.viewer` binding on the project
* `roles/storage.objectViewer` on the temp bucket and the staging bucket

{% code title="main.tf" %}

```hcl
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

variable "project_id" {
  description = "GCP Project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "dataproc_temp_bucket" {
  description = "Dataproc temp bucket name, or your custom event-log bucket (without gs://)"
  type        = string
}

variable "dataproc_staging_bucket" {
  description = "Dataproc staging bucket name (without gs://)"
  type        = string
}

resource "google_service_account" "dataflint_events_reader" {
  account_id   = "dataflint-events-reader"
  display_name = "DataFlint Spark Events Reader"
  description  = "Read-only access to Dataproc Spark event logs"
}

resource "google_service_account_key" "dataflint_events_reader_key" {
  service_account_id = google_service_account.dataflint_events_reader.name
}

resource "google_project_iam_member" "dataproc_viewer" {
  project = var.project_id
  role    = "roles/dataproc.viewer"
  member  = "serviceAccount:${google_service_account.dataflint_events_reader.email}"
}

resource "google_storage_bucket_iam_member" "temp_bucket_viewer" {
  bucket = var.dataproc_temp_bucket
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:${google_service_account.dataflint_events_reader.email}"
}

resource "google_storage_bucket_iam_member" "staging_bucket_viewer" {
  bucket = var.dataproc_staging_bucket
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:${google_service_account.dataflint_events_reader.email}"
}

output "service_account_email" {
  value = google_service_account.dataflint_events_reader.email
}

output "service_account_key_base64" {
  value     = google_service_account_key.dataflint_events_reader_key.private_key
  sensitive = true
}
```

{% endcode %}

{% code title="terraform.tfvars" %}

```hcl
project_id              = "your-project-id"
region                  = "us-central1"
dataproc_temp_bucket    = "dataproc-temp-us-central1-123456789-abcdefg"
dataproc_staging_bucket = "dataproc-staging-us-central1-123456789-abcdefg"
```

{% endcode %}

### Apply and export the key

```bash
terraform init
terraform apply

terraform output -raw service_account_key_base64 | base64 -d > dataflint-sa-key.json
```

{% endtab %}
{% endtabs %}

## Add it in DataFlint

In the DataFlint console go to **Admin Panel → Environment Management → Add New → Dataproc** and fill in:

* **Environment name**
* **Engine type** — Compute Engine
* **Region** — one region per environment. Add another environment for each additional region.
* **GCP project ID**
* **Cluster ID** (optional) — restrict ingestion to one cluster
* **Service account key** — paste the JSON key file contents

Click **Add Environment**. The environment shows as **Pending approval** while DataFlint verifies access; ingestion starts once it is approved.

### Rotating the key

Create a new key for the service account, edit the environment in the console, paste the new key, and then delete the old key in Google Cloud.

## Validate the access (recommended)

Use the key to test that listing clusters and reading objects works.

```bash
gcloud config set project YOUR_PROJECT_ID
gcloud auth activate-service-account \
  --key-file="$PWD/dataflint-sa-key.json"

# Should succeed (needs roles/dataproc.viewer)
gcloud dataproc clusters list --region=YOUR_REGION

# Should succeed (needs roles/storage.objectViewer)
gsutil ls gs://YOUR_DATAPROC_TEMP_BUCKET/
gsutil ls gs://YOUR_DATAPROC_STAGING_BUCKET/
```

{% hint style="warning" %}
If you don't activate the service account, the test can succeed using your personal credentials.
{% endhint %}