Snowflake metadata collector
The Snowflake metadata collector provides read-only access to metadata from an external Snowflake account.
The collector harvests metadata from Snowflake Data Cloud. It collects tabular objects, Horizon objects such as policies and tags, and Snowflake data quality artifacts. It also harvests functions and stored procedures. The collector can gather column statistics and table query counts. In addition, it can harvest metadata for Streamlit applications.
Metadata cataloged
The Snowflake collector catalogs the following information.
Note: The collector harvests all versions of overloaded functions and stored procedures. Each version has its own title/name in the catalog, but a distinct identifier.
| Object | Information cataloged |
|---|---|
| Column | Name, Comments, Data Type, Is Nullable, Default Value, Key type \(Primary, foreign\) |
| Table | Name, Comments, Primary key, Schema, External URL \(Snowsight\)Extended metadata: Is Iceberg, Is dynamic, Is transient, Created Date, Last Modified Date, Last Updated By \(Snowflake LAST\_DDL\_BY\), Owner \(Snowflake role that owns table\), Definition Altered Date, Table Size |
| View | Name, Comments, Schema, External URL \(Snowsight\), Definition Altered Date, Last Modified By \(Snowflake LAST\_DDL\_BY\), Is Iceberg, Is dynamic, Is temporary, Created Date, Last Modified Date, Owner \(Snowflake role that owns table\), SQL definition |
| Materialized View | Name, Comments, Schema, External URL \(Snowsight\), Definition Altered Date, Last Modified By \(Snowflake LAST\_DDL\_BY\), Is Iceberg, Is dynamic, Is temporary, Created Date, Last Modified Date, Owner \(Snowflake role that owns table\) |
| Schema | Identifier, NameExtended metadata: Comments, Created date, Updated |
| Database | Type, Name, Identifier, Server, Port, Environment, JDBC URLExtended metadata: Comments |
| User-defined function\(this includes User-defined functions created in Snowpark\) | NameExtended metadata: Function Definition, Comments, Created Date, Owner \(Snowflake Role\), Snowflake Function Type \(Return Type\), Function Signature, Function Language, Packages, Runtime version, Updated |
| Stored Procedure\(this includes Stored Procedures created in Snowpark\) | NameExtended metadata: Function Definition, Owner \(Snowflake Role\), Comments, Return Type, Function Language, Packages, Created Date, Function signature, Snowflake Function Type, Runtime Version, Updated, Stored Procedure Type |
Relationships between objects
The harvested metadata includes catalog pages for the following data asset types. Each catalog page has a relationship to the other related data asset types.
| Data asset page | Relationships |
|---|---|
| Snowflake Row access policies | Tables |
| Snowflake Masking Policies | Columns, Tag value |
| Table | Columns, Tag value, Data Metric Function that analyzes, Views, Tables, or Functions which the Table depends on |
| Snowflake Tags | Tag value |
| Columns | Table, Tag Value, System Tag Value |
| View | Schema that contains Views, Columns that are part of Views, Data Metric Function that analyzes View, Views, Tables, or Functions which the View depends on |
| Materialized View | Schema that contains Materialized Views, Columns that are part of Materialized Views |
| Schema | Database that contains Schema, Table that is part of Schema, View that is part of Schema, Materialized View that is part of Schema, Streamlit app that is part of schema |
| Streamlit App | Schema that contains Streamlit App |
| Database | Schema contained in Database |
| Data Metric Function | Table or View analyzed by this Snowflake Data Metric Function, Table, Views, Tables, or Functions which the Function depends on |
Lineage and dependencies for Snowflake
The following lineage information is collected by the Snowflake collector. Note that any lineage for SQL Statements defined via variable statements are not supported.
Important things to note:
- Any lineage for SQL Statements defined via variable statements are not supported.
- The collector traces these relationships from a View’s columns to ultimate source Table columns across SQL expressions and subqueries.
| Object | Lineage available |
|---|---|
| Column in View | The collector identifies the associated column in an upstream view or table: - Where the data is sourced from - That sort the rows via ORDER BY - That filter the rows via WHERE/HAVING - That aggregate the rows via GROUP BY |
| User-defined function | A Function and: - Views referenced in Function - Tables referenced in Function - Functions referenced in Function A View and Function referencing View |
| Stored Procedure | The collector identifies: - The associated column in an upstream view or table - Where the data is sourced from - That sort the rows via ORDER BY - That filter the rows via WHERE/HAVING - That aggregate the rows via GROUP BY - The downstream table that has its data updated by this stored procedure. Note: The following stored procedures are not supported: - Stored procedures with multitable inserts. However, multiple separate insert statements that insert into one table *are supported*. - Stored procedure with multiple SELECT and INSERT statements not separated by a semicolon delimiter. - Stored procedures used to create tables or any transient tables created during the execution scope of a stored procedure. |
Dependencies
The Snowflake collector catalogs dependencies between tables, views, and functions using the Snowflake Account Usage view OBJECT_DEPENDENCIES. See the Snowflake documentation for more information about the OBJECT_DEPENDENCIES view.
If you include the profiling and sampling specific parameters while running the collector, the following additional information is harvested for Columns.
Note: The user/role must have read access to data to be able to harvest profiling information (column statistics).
| Object | Information cataloged |
|---|---|
| Column | - Average Length \(sample\) - Average Value \(sample\) - Data Distribution - Distinct Values - Estimated Distinct Values - Estimated Non-null Values - Maximum Length \(sample\) - Maximum Value \(sample\) sorted numerically or alphabetically \(z-a\) - Minimum Length \(sample\) - Minimum Value \(sample\) sorted numerically or alphabetically \(a-z\) - Non-null Values \(sample\) - Sample String Values \(first 5 items in a column\) |
| Table | - Row Count - Sample Count \(Target sample size\) |
Following additional information is cataloged when you run the collector with the Collect Snowflake policy information and Collect Snowflake tag information options enabled.
| Object | Information cataloged |
|---|---|
| Columns | Snowflake masking policies |
| Snowflake Tags | Name, Allowed Values, Comments |
| Snowflake Tag Values | Name, Value |
| Snowflake Masking Policies | Name, Comments , Date created, Snowflake tags, Owner, Type, Policy body, Schema, Database |
| Snowflake Row access policies | Name, Comments , Date created, Snowflake tags, Owner, Type, Policy body, Schema, Database |
Following additional information is cataloged when you run the collector with the Collect Snowflake table usage information and Table usage lookback days options enabled.
| Object | Information cataloged |
|---|---|
| Table | Query Count |
Following additional information about data quality metrics is cataloged when you run the collector with the Collect data metric function information option enabled.
| Object | Information cataloged |
|---|---|
| Data Metric Function | - Identifier: Name of the function - Title: Name of the function - Description - Body Definition of the function |
| Table | - Schedule: cron expressions - Data metric function observations: function name, column arguments, most recent metric time, value |
Following additional information is cataloged when you run the collector with theCollect Streamlit app information option enabled.
| Object | Information cataloged |
|---|---|
| Snowflake Streamlit App | - Identifier: App Name - Title: App Title - Description: App comments - Main file - Root location - Date Created - Owner |
- Prepare to run the Snowflake collector
Set up authentication and grant role permissions to enable Snowflake metadata collection. - Create a Snowflake metadata collector
Create a collector to import metadata from Snowflake.
Parent Topic:Configuring metadata collectors