Create DeltaStream Objects to Structure Raw Data
In DeltaStream a Data Store provides access to the raw data in your external systems. To process that data in queries, we define DeltaStream objects that attach metadata and data format information to the underlying store data.
Understanding the Data
As an example, below is a defined Apache Kafka store that contains several entities:
demodb.public/msk_public# LIST ENTITIES;
Entity name
-----------------------
ds_syslogs
ds_pageviews
ds_shipments
ds_users Now assume all entities are in JSON format. See CREATE STORE and UPDATE ENTITY for using other serialization formats. For information around data formats -- for example, CREATE STREAM -- refer to the relation’s DDL statements.
You can inspect the entities to understand the kind of data you have -- for example the ds_pageviews entity:
demodb.public/msk# PRINT ENTITY ds_pageviews;
{"userid":"User_7"} | {"viewtime":1677196372920,"userid":"User_7","pageid":"Page_82"}
{"userid":"User_3"} | {"viewtime":1677196372962,"userid":"User_3","pageid":"Page_97"}
{"userid":"User_6"} | {"viewtime":1677196373021,"userid":"User_6","pageid":"Page_80"}
{"userid":"User_1"} | {"viewtime":1677196373081,"userid":"User_1","pageid":"Page_73"}
{"userid":"User_2"} | {"viewtime":1677196373122,"userid":"User_2","pageid":"Page_35"}
{"userid":"User_7"} | {"viewtime":1677196373182,"userid":"User_7","pageid":"Page_58"}Here is the ds_users entity:
PRINT ENTITY reads directly from the underlying entity in the store. For Kafka-type stores, that entity is a Kafka topic.
Defining DeltaStream Objects
When you know what the data looks like in your entities, you can attach a DeltaStream structure to them for use in queries. In the example below, ds_pageviews is the underlying entity and pageviews is the DeltaStream Stream relation defined on top of it:
See CREATE STREAM for more information.
Since the ds_users entity hosts user information that changes over time, you can define a Changelog relation to capture ongoing changes to each userid:
See CREATE CHANGELOG for more information.
When you have defined objects, you can list them through their database and namespace:
In addition to listing them, you can also use their database and namespace to describe them:
Using DeltaStream Objects
When you define a DeltaStream relation on top of an entity, you can query that relation using DeltaStream SQL. For example, you can use the pageviews stream relation in interactive queries:
You can also use that relation in persistent queries, where it is continuously used as a source or sink:
Last updated