✨Our Features

Real-time cluster status

See metrics on the current SQL query being run, including node-level granularity at real time

Run Summary

Get performance summary of your run, including the option to check which query used the most resources.

SQL queries or performance metrics with performance alerts are being marked with a warning icon, just like a warning light in your car's dashboard.

Cluster Status

Measure your spark resource usage with a graph of executors over time, and the relevant resource configuration for your allocation mode

Error Handling

In case of SQL that failed, hover on it to see the exception, click to copy the error to clipboard

SQL in-depth plan

In the summary screen you can choose a query to see a visualized plan, including alerts on specific query nodes

Heat Map

See which part of your SQL is taking the longest, hover on precentage to see how much compute time the part took

This feature is WIP, and might not identify the node's right duration. It also not accurate with very fast SQL queries Some nodes are running at the same stage (such in the example filtering and selecting) so the same percentage is shown to both of them Please raise any issue/inconsistency with this feature

SQL plan modes

You can switch to advanced mode to see repartitions, sorting and broadcasting.

Or you can switch to "Only IO" mode to see only input, output and joins

Syntax highlighting

Hover on long SQL expressing to see them with syntax highlighting

Alerts

Get alerts on performance issues, including suggestions for fixes with specific config/code changes suggestions!

Integrations

Apache Iceberg Integration

DataFlint knows to collect iceberg write metrics that does not exist in spark, and alert in cases of performance issues when using functionality like update/delete/merge operators

Require adding the iceberg metric reporter, see Apache Iceberg for more details