βͺRelease Notes
Version 0.2.7
DataFusion Comet support
bug fixes
Version 0.2.6
Nvidia RAPIDS for Spark support
bug fixes
Version 0.2.5
Bug Fixes
Version 0.2.4
New visibility & alerts - Driver's memory
Updated README
Bug Fixes
Version 0.2.3
Driver memory monitoring & alert
Updated readme
Bug fixes
Version 0.2.3
New alert - Large data Broadcast, for requesting to broadcast large data sets with the broadcast() function
New alert - Large filter conditions, for wiring long filter conditions instead of using join logic
UI Improvements
Version 0.2.2
Support spark versions 2.4 logs in history server with version later than 3.2 Limited feature-set is available due to events having less data than spark 3.0 and up
Version 0.2.1
Better Databricks stage to node support
Support spark.dataflint.runId in custom history server providers when appId is not the spark appId
Version 0.2.0
Better support for Databricks Photon plans
Input nodes shows partitions filters and push down filters
Stage Breakdown - press the blue down arrow on sql node to see stage information
Version 0.1.7
Apache Iceberg alerts improvements
Add avg file size in read/write
More information when hovering on stage
Version 0.1.6
Apache Iceberg support
Better node naming
Read metrics and reading small files alerts
Write metrics and overwriting most of table alerts
Require enabling iceberg metric reporter, can be done for you by enabling spark.dataflint.iceberg.autoCatalogDiscovery to true, or setting the iceberg metric reporter manually for each catalog, for example:
Version 0.1.5
Add support for history server with cluster-mode jobs (i.e. with attempt numbet)
Fix "wasted cores" calculation
Fix status tab SQL is flickering when there is SQL with sub queriers
Version 0.1.4
Fix scala 2.13 support
Version 0.1.3
DataFlint SaaS support
partition Skew Alert:
Version 0.1.2
Scala 2.13 support
A spark flag to disable web app mixpanel telemetries -
spark.dataflint.telemetry.enabled
(true/false)Renamed Core Activity Rate to Wasted Cores Ratio (which is 100 - Core Activity Rate), and added an alert for wasted cores too high
Version 0.1.1
Resources tab - see a graph of your cluster executors count over time, use it to tune your resource allocation settings and save cost!
Minor visual fixes
DataFlint Resource Tab:
Version 0.1.0
Small fix to platform identification
Version 0.0.8
Databricks support
Visual improvements
public release
Version 0.0.7
Heat map
Version 0.0.6
Flint Assistant, require OpenAI Key
Version 0.0.5
Syntax highlighting for SQL plan parts
Calculating container memory usage and using it for GB memory/hour calculations
Version 0.0.4
Minor fix relates to spark operator and nginx
Version 0.0.3
SQL plan modes
IO only, shows only input, joins and output:
Basic mode (default), shows also transformations like filters, aggregations and selects:
Advances, shows repartitions, broadcasts and sorts
Also there is plans informations for:
Joins
Sorts
Selects
Repartitions
Version 0.0.2
DBU calculation instead of core/hour in summary bar
Add memory config to configuration tab
Filter Nodes has condition:
Advanced mode for SQL plan, that also presents shuffle nodes
Additional changes
Support both http and https access with enabling mix-content only on https mode
Support for spark 3.5.X
Version 0.0.1
Initial version, includes:
Status page
Summary page
Configuration Page
Alerts page
Last updated