Flat File
The Flat File integration is used to upload a single flat file (like csv, parquet, and feather) into the data lineage graph.
Web App
Fields
Field | Value | Example |
---|---|---|
source | The name of the source, see sources | my-source |
Namespace | Namespace for the connection, see namespaces | default |
Python Library
The dbt integration can be run as a standalone python library to extract data lineage from flat files like csv and parquet.
The library is available via pip
pip install grai_source_flat_file
More information about the API is available here.
Example
The library is split into a few distinct functions but if you only wish to extract nodes/edges you can do so as follows:
from grai_source_flat_file import FlatFileIntegration
from grai_schemas.v1.source import SourceV1
source = SourceV1(name="my-source", type="my-type")
integration = FlatFileIntegration(source=source, namespace="dbt", file_name="/path/to/file.csv")
nodes, edges = integration.get_nodes_and_edges()
Last updated on November 7, 2023