Databricks metadata collector
The Databricks metadata collector provides read-only access to metadata from an external Databricks account.
The collector harvests metadata from data assets in Databricks Hive Metadata, Unity Catalog (including Delta Lake), Workflows, and Notebooks.
Metadata cataloged
The Databricks collector catalogs the following information.
| Object | Information cataloged |
|---|---|
| Columns | Name, Description, JDBC type, Column Type, Is Nullable, Default Value, Column size, Column index Extended metadata: Tags Note: Deprecated columns and any lineage related to these deprecated columns are not cataloged. |
| Table | Name, Description, Schema, Primary key, Foreign key Extended metadata: Tags, Owner, Type, Creation date, Last Modified, Location, Provider, Version, Size, File Count, Partition Columns, Properties |
| Model | Name, Owner, Description, Created By, Created At, Last Modified By, Last Modified At, Securable Kind, Securable Type |
| Views | Name, Description, SQL definition, Tags |
| Schema | Name Extended metadata: Tags |
| Database | Type, Name, Server, Port, Environment, JDBC URL Extended metadata: Tags |
| Notebook | Notebook ID, Path, Language Type \(SQL, Python, Scala, R\) |
| Function | Name, Description, Function Type |
| Job | Title, Description, Creator, Created At, Job run as, Format, Max Concurrent Runs, Notification On Start, Timeouts \(sec\), Notification On Success, Schedule, Git Source, Notification on Failure, Tags, List of tasks, List of clusters |
| Cluster | Name, Description, Node Type ID, Driver Node Type ID, Spark Version, Number of Workers, Autoscale Max Workers, Autoscale Min Workers, AWS Attributes, Tags |
| Task | Task Key, Type of Task \(Notebook, dbt, Spark jar, Python script, Python wheel, Pipeline task, SQL\), Task timeout, Retry interval, Cluster used by the task, Max retries, Depends on, Libraries, Notifications \(On start, On success, On failure\), Notebook File Path, Notebook Source, Notebook Parameters, Spark Jar Main Class Name, Spark Jar Parameters, Python Script File path, Python Script Parameters, Spark Submit Parameters, Pipeline ID, Pipeline Full Refresh, Python Wheel Package Name, Python Wheel Entry Point, Python Wheel Parameters, SQL Warehouse, SQL Query ID, SQL Dashboard ID, SQL Alert ID, Dbt Project Directory, Dbt Profiles Directory, Dbt warehouse, Dbt catalog, Dbt schema, Dbt commands |
| External Location | Name, External URL, Description, Data Source Type, Created Date, Created By, Owner |
| Storage Credential | Name, Description, Credential, Created Date, Created By, Owner |
| Volume | Name, Description, Type, Owner, Created By, Created At, Last Modified By, Last Modified At, Metastore ID |
| Materialized View | Name, SQL Definition, Created, Last Modified |
| Metric View | Name, Description, YAML Definition, Source Table, Source Table Type, Filter, Created, Last Modified |
Following additional information is cataloged when you run the collector with the Enable Governance Metadata Collection option.
| Object | Information Cataloged |
|---|---|
| Row Filter Access Control | Name |
| Column Mask Access Control | Name |
| Attribute Based Access Control | Name, Description, Created by, Created at, Modified by, Modified at, On securable type, For securable type, To principals, Except principals |
| Workspace bindings | Workspace ID, Binding type |
| Privileges | Granted to, Granted by, Privilege type, Granted on object, Inherited from |
Relationships between objects
The harvested metadata includes catalog pages for the following data asset types. Each catalog page has a relationship to the other related data asset types.
| Data asset page | Relationships |
|---|---|
| Table | - Columns contained in Table - Table Indexes - Has privileges Columns contained in Table |
| Schema | - Database that contains Schema - Table that is part of Schema - Has privileges |
| Database | Schema contained in Database |
| Columns | - Table Indexes - Table containing Column |
| Table Indexes | Columns |
| Job | - Clusters used by tasks in Job - Tasks contained within Job |
| Cluster | - Cluster contained in job - Task using Cluster |
| Task | - Job containing Task - Cluster used by Task - Tasks depending on Task |
| Notebook | - Folder containing Notebook - Task sourcing data from Notebook |
| Folder | - Folders contained in Folder - Notebooks contained in Folder |
| External Location | - Uses storage credential - Connects to datasource \(S3 bucket, S3 Object, Azure container or Azure blob\) - Has workspace bindings |
| Storage Credential | - Has workspace bindings - Used by External Location |
| Model | - Registered in schema - Stored in data assets \(S3 Bucket, S3 Object\) |
| Volume | - Contained within schema - Stored in data assets \(S3 Bucket, S3 Object\) |
| Materialized View | - Schema that contains Materialized Views - Columns that are part of Materialized Views |
| Metric View | - Schema that contains Metric Views - Columns that are part of Metric Views |
| Pipeline | - Copies data to Databricks Table/Database Schema - Ingests data from Database Table/Database Schema |
| Unity Catalog Metastore | Databases contained in metastore |
| Row Filter Access Control | - Applies to table - Uses function - Using column - Contained within schema |
| Column Mask Access Control | - Applies to column - Uses function - Contained within schema |
| Attribute Based Access Control | - Applies to catalog, schema and table - Defined on catalog, schema and table - Uses function |
| Catalog | - Has workspace bindings - Has privileges |
Lineage for Databricks
The following lineage information is collected by the Databricks collector.
Note: The collector does not support lineage for SQL statements defined via variable statements.
| Object | Lineage available |
|---|---|
| Column in view | The collector identifies the associated column in an upstream view or table for both Hive metastore and Unity Catalog: - Where the data is sourced from - That sort the rows via ORDER BY - That filter the rows via WHERE/HAVING - That aggregate the rows via GROUP BY Note: Deprecated columns and any lineage related to these deprecated columns are not cataloged. |
| Notebook | Tasks that reference Notebook. \(Only if Databricks Unity Catalog is enabled\). |
| Table | - The collector identifies the upstream and downstream tables and their external locations \(S3 and ADLS Gen2 data assets\) along with the intermediate Job. - The collector harvests the Databricks table lineage to S3 object. |
Authentication supported
The Databricks collector supports Personal access token authentication and Oauth service principal authentication.
- Prepare to run the Databricks collector
Complete authentication and permission configuration tasks before running the Databricks metadata collector. - Create a Databricks metadata collector
Create a collector to import metadata from Databricks.
Parent Topic:Configuring metadata collectors