SupportEnhancedSQL Parameter (Connect-Databricks Cmdlet)
This property enhances SQL functionality beyond what can be supported through the API directly, by enabling in-memory client-side processing.
Connect-Databricks -SupportEnhancedSQL bool
When SupportEnhancedSQL = true, the cmdlet offloads as much of the SELECT statement processing as possible to Databricks and then processes the rest of the query in memory. In this way, the cmdlet can execute unsupported predicates, joins, and aggregation.
When SupportEnhancedSQL = false, the cmdlet limits SQL execution to what is supported by the Databricks API.
Execution of Predicates
The cmdlet determines which of the clauses are supported by the data source and then pushes them to the source to get the smallest superset of rows that would satisfy the query. It then filters the rest of the rows locally. The filter operation is streamed, which enables the cmdlet to filter effectively for even very large datasets.
Execution of Joins
The cmdlet uses various techniques to join in memory. The cmdlet trades off memory utilization against the requirement of reading the same table more than once.
Execution of Aggregates
The cmdlet retrieves all rows necessary to process the aggregation in memory.