Reference · built on requestConnectorDatabases & warehouses

Databricks

Tables, views, and change data feeds from Databricks, queried on SQL warehouses through the Statement Execution API under Unity Catalog.

Request accessPricing ↗

About

Databricks is the Data Intelligence Platform from Databricks, run on AWS, Azure, and Google Cloud. Tables, views, volumes, functions, and models are governed in Unity Catalog under a three-level namespace, catalog.schema.object, with privileges such as USE CATALOG, USE SCHEMA, SELECT, and MODIFY that inherit downward. SQL runs on SQL warehouses, serverless, pro, or classic, which the Statement Execution API reaches at /api/2.0/sql/statements: a POST names a warehouse_id and a statement, waits up to 50 seconds or returns an id to poll, and delivers results inline up to 25 MiB or as external links up to 100 GiB. Delta Lake tables with change data feed expose row-level inserts, updates, and deletes through table_changes().

An operator on Fibric runs approved statements against the catalogs you grant and proposes row changes, each with a receipt and an undo.

This is a reference listing. It documents what Fibric would read from Databricks and what it could propose, based on the vendor's published interfaces. Fibric builds it under a managed deployment when you request it; selecting it here installs nothing.

Inputs

  • Rows from tables and views the service principal holds SELECT on, through POST /api/2.0/sql/statements with a warehouse_id
  • Statement status and result chunks from GET /api/2.0/sql/statements/{id}, in JSON_ARRAY, ARROW_STREAM, or CSV
  • Row-level changes from table_changes() with _change_type, _commit_version, and _commit_timestamp on tables with change data feed enabled
  • Catalog, schema, and table metadata from Unity Catalog, with lineage and audit records in system tables
  • Statement state as PENDING, RUNNING, SUCCEEDED, FAILED, CANCELED, or CLOSED from the status endpoint

Proposed actions

  • Target capability: propose an INSERT or UPDATE against a Unity Catalog table where the principal holds MODIFY, run once after approval
  • Target capability: propose ALTER TABLE ... SET TBLPROPERTIES to enable delta.enableChangeDataFeed on a table you name
  • Target capability: propose cancelling a running statement with POST /api/2.0/sql/statements/{id}/cancel

Proposed actions are target capabilities. Every action runs propose-first and needs a validated deployment and the appropriate permissions.

What you can build

  • Explain a KPI from the rows in the lakehouse

    Run the governing query on a serverless warehouse, page the result by chunk, and show the reviewer which rows changed the figure between two runs.

    With Metric Reconciliation, Root Cause

  • Follow order changes through the change data feed

    Read table_changes() from the last _commit_version you processed and treat update_postimage rows with a new status as events worth a proposal.

    With Order Risk, Anomaly Notice

  • Catch a pipeline that stopped writing

    Compare the latest _commit_timestamp per table with the expected cadence and raise a notice when a table goes quiet.

    With Data Freshness, Integration Health

Requirements

  • A Databricks workspace with Unity Catalog enabled and a SQL warehouse the principal may use
  • USE CATALOG and USE SCHEMA on the path to each table, plus SELECT to read and MODIFY to propose writes
  • A service principal assigned to the workspace with OAuth client credentials
  • delta.enableChangeDataFeed set on any table whose row changes you want, before the changes happen
Authentication
A Databricks service principal using OAuth machine-to-machine client credentials, a client ID and secret issued once the principal is assigned to the workspace, or OAuth token federation from your identity provider.

Limits

  • Change data feed records only changes made after it was enabled and is not a permanent history; VACUUM removes change files with the table's retention
  • Inline results above 25 MiB fail; ARROW_STREAM and CSV need EXTERNAL_LINKS, whose presigned URLs expire within 15 minutes
  • A synchronous call waits between 5 and 50 seconds; longer statements must be polled by id
  • Results stay fetchable for at least 12 hours after a statement reaches a terminal state, then the statement is CLOSED

Access and pricing

Reference listing. Fibric builds the connector under a managed deployment when you request it. Your quote covers the build, capabilities, usage, and support.

Request Databricks ↗

Questions and answers

Which compute runs the queries?
A SQL warehouse. The Statement Execution API requires a warehouse_id, and Databricks recommends serverless warehouses where available. A stopped warehouse starts when a statement arrives, when a scheduled job needs it, or when a JDBC or ODBC connection opens.
How large can a result be?
Up to 25 MiB inline as JSON_ARRAY. Larger results, up to 100 GiB, use the EXTERNAL_LINKS disposition, which returns presigned URLs with an expiry of 15 minutes or less. A statement itself may be at most 16 MiB of SQL.
How do I get row-level changes from a table?
Enable change data feed with the table property delta.enableChangeDataFeed = true, then query table_changes('tableName', startVersion, endVersion). Each row carries _change_type as insert, update_preimage, update_postimage, or delete, plus _commit_version and _commit_timestamp. Only changes after enablement are recorded.
Ask about Databricks

Ask about the capabilities and requirements in this listing.

For project-specific requirements, contact Fibric.

This connector listing is developed, published, and supported by Fibric. It describes integration with Databricks through published interfaces. Third-party names and logos identify the systems an integration connects to; they are the property of their respective owners, who are not affiliated with Fibric and do not sponsor or endorse this listing. Trademark policy