Unit testing for pipelines (Beta)

BetaJul 26, 2026VerifiedAdded Sep 22, 2026

You can now write and run Python unit tests for your Spark Declarative Pipelines on Lakeflow transformation logic directly in the Lakeflow Pipelines Editor. Validate Python and SQL transformations against mock data using catalog-table redirection, including Auto CDC, streaming tables, expectations, and append flows.

SQL Streaming Governance Data integration Observability

Read Databricks's release notes

https://docs.databricks.com/aws/en/release-notes/product/2026/july#unit-testing-for-pipelines-beta

Summaries of vendors' own notes. Product names and logos belong to their owners; logos via logo.dev.

More Lakeflow releases

LakeflowJul 24, 2026

Lakeflow Designer Summer Release

The Lakeflow Designer Summer Release adds new operators, expands the Output and Join operators, and improves the canvas editing experience. The following updates are now available: Prepare operator : Use the new Prepare operator for common data-cleansing tasks, including changing text case and data types, replacing values, and applying formulas.

UpdateGovernanceBIObservabilitySource

LakeflowJul 23, 2026

C++ SDK for the Zerobus Ingest connector is now available in Beta

A C++ SDK for the Zerobus Ingest connector in Lakeflow Connect is now available in Beta. The SDK provides a high-throughput gRPC client for ingesting records directly into Unity Catalog Delta tables, with support for JSON and Protocol Buffers. See Use Zerobus Ingest .

BetaGovernanceData integrationObservabilitySource

LakeflowJul 22, 2026

Override the REPLACE WHERE predicate for a single refresh

You can now override a standalone streaming table's defined REPLACE WHERE predicate for a single refresh by adding a WHERE clause to the REFRESH STREAMING TABLE command. The override applies only to that refresh and doesn't change the table definition, which is useful for one-time backfills and corrections.

UpdateStreamingDeveloper toolsSource

LakeflowJul 22, 2026

Lakeflow Designer support for compliance security profile standards

Lakeflow Designer now supports all compliance security profile standards supported by Databricks and is available by default for workspaces with the compliance security profile enabled. For a full list of supported compliance standards, see Compliance security profile . See Lakeflow Designer .

UpdateGovernanceSecuritySource

LakeflowJul 31, 2026

OpenAI connector (Beta)

The OpenAI connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector uses the OpenAI Organization Admin API to ingest organization administration data, such as users, projects, API keys, usage, costs, and audit logs, from OpenAI into Databricks.

BetaPricingData integrationObservabilitySource

LakeflowJul 21, 2026

Reddit Ads connector is now available in Beta

The managed Reddit Ads connector in Lakeflow Connect is now available in Beta. You can use the connector to ingest advertising campaign data and performance metrics from Reddit Ads into Databricks. See Reddit Ads connector .

BetaData integrationObservabilityPerformanceSource

LakeflowAug 3, 2026

Environment variables for serverless jobs are in Beta

Environment variables for serverless Lakeflow Jobs tasks are in Beta . You can define named environment variable entries at the job level and select one for each task to pass deployment-specific configuration to your application code. A workspace admin must enable the preview from the workspace Previews page.

BetaSource

LakeflowAug 3, 2026

Automatic cost attribution for materialized views and streaming tables in Databricks SQL

Materialized view and streaming table refreshes in Databricks SQL now inherit custom tags from the enclosing SQL warehouse environment. These tags propagate to billing surfaces like system.billing.usage , enabling proper cost attribution of materialized view and streaming table updates back to the source SQL warehouse.

UpdateSQLStreamingGovernanceSource