Visualizing Data Lineage
Display your DeltaStream ecosystem in adjustable flowchart form.
Last updated
Display your DeltaStream ecosystem in adjustable flowchart form.
Last updated
If you have a large number of data objects, it can be difficult to keep track of where all your data is coming from, how it’s being queried, and where it’s ending up. DeltaStream addresses this via Stream 360.
Stream 360 displays the movement of your data as a flowchart, from source to destination, including how it’s transformed along the way. You can see at a glance all your DeltaStream objects – streams, changelogs, tables, materialized views – and their relationships with queries. This gives you a clear visual representation of data lineage throughout your DeltaStream ecosystem. You can:
understand your processes
validate data at particular points in the data lineage
see all aspects of the pipeline you’re building – joins, sources, pipeline results (such as writings to a materialized view), and more
examine streams and materialized views and quickly ascertain what data you may wish to add or whether the data you need is in the stores displayed.
move or group objects to isolate and examine a specific process
The end-to-end view also helps you avoid negative or unanticipated downstream impacts resulting from any changes you make.
You can also use Stream 360 to view metrics about DeltaStream objects and queries:
Click on a particular DeltaStream object, and a side panel displays that enables you to print records in the object and view the object’s columns and metadata.
Click on a particular query, and a side panel displays with some basic metrics and metadata about the query. You can also terminate queries from this panel.
From any screen in DeltaStream, click the Org Name arrow and select the organization you wish to display.
In the lefthand navigation, click the Stream 360 icon ( ) to display the flowchart.
To see the location of the displayed objects, in the top right corner check View location. Paths display immediately above each object.
To see whether multiple objects share a topic, in the top right corner check View Topic Link. If a process involves multiple topics, checking this box highlights those topics, regardless of where they may display on the canvas, and shows how they are connected. This makes it easier yet to envision more complex workflows.
You can quickly drill down on any particular DeltaStream object in the flow to see details of that object.
Stream 360 highlights all lines connecting all the objects in this flow. It also opens a separate panel that displays additional details about the object, including queries that make use of that object.
In the simple example below, we selected the users_log changelog. Its flow from object to enriched stream is highlighted, and details display in a panel on the right.
And in this example, the final result – the enriched pageviews stream – is selected. The lines that represent the flow of all the objects involved in this enriched stream are highlighted, and details of the enriched stream display in the side panel.
Finally, you can enter an object name, in whole or in part, in the Search bar above the TOC to get details about a particular object.
You can rearrange the way your flows display in multiple ways. To do this, you can click and drag individual objects or use the buttons in the lower-left of the Stream 360 screen. Any lines connecting an object with a query or destination move with the object — that is, changing the location of objects on the Stream 360 canvas does not break the flow; you're modifying the visual display only.
Move an object around on the canvas
Click and drag the object.
Move multiple objects around as a group
Ctrl click (Windows) or Cmd click (Mac) the objects you want. Then click and drag the objects as a group.
Increase the size of the objects
Click +
Decrease the size of the objects
Click -
Center the entire flow
Lock the objects in place
To start, click the TOC panel () to display your organizational hierarchy. Then click the item you want.
Click
Click
