search
Start Trial

Create DeltaStream Objects to Structure Raw Data

In DeltaStream a Data Store provides a layer of abstraction around the raw streaming data. To process that data in queries, we must make sense of the data. To this end, we use DeltaStream objects for defining the metadata and data format that in turn describe the structure of the data for its native format.

hashtag
Understanding the Data

As an example, below is a defined Apache Kafka store that contains several entities:

demodb.public/msk_public# LIST ENTITIES;
      Entity name       
-----------------------
  ds_syslogs      
  ds_pageviews         
  ds_shipments         
  ds_users

Now assume all entities are in JSON format. See CREATE STORE and UPDATE ENTITY for using other serialization formats. For information around data formats -- for example, CREATE STREAM -- refer to the relation’s DDL statements.

You can inspect the entities to understand the kind of data you have -- for example the ds_pageviews entity:

demodb.public/msk# PRINT ENTITY ds_pageviews;
{"userid":"User_7"} | {"viewtime":1677196372920,"userid":"User_7","pageid":"Page_82"}
{"userid":"User_3"} | {"viewtime":1677196372962,"userid":"User_3","pageid":"Page_97"}
{"userid":"User_6"} | {"viewtime":1677196373021,"userid":"User_6","pageid":"Page_80"}
{"userid":"User_1"} | {"viewtime":1677196373081,"userid":"User_1","pageid":"Page_73"}
{"userid":"User_2"} | {"viewtime":1677196373122,"userid":"User_2","pageid":"Page_35"}
{"userid":"User_7"} | {"viewtime":1677196373182,"userid":"User_7","pageid":"Page_58"}

Here is the ds_users entity:

hashtag
Defining DeltaStream Objects

When you know what the data looks like in your entities, you can attach a structure to them for reference in queries. In the example below, as ds_pageviews is a continuous stream of immutable page events from your users, you can define a Stream for it:

See CREATE STREAM for more information.

Since the ds_users entity hosts user information that changes over time, you can define a Changelog to capture ongoing changes to each userid:

See CREATE CHANGELOG for more information.

circle-info

Note For certain applications, it may be more useful to have access to a snapshot of the resulting data. See Materialized View and CREATE MATERIALIZED VIEW AS for more information on how to create a view for the data.

When you have defined objects, you can list them through their database and namespace:

In addition to listing them, you can also use their database and namespace to describe them:

hashtag
Using DeltaStream Objects

When you define objects for an entity, it becomes available to DeltaStream as a consumable entity. For example, you can use them in interactive queries:

You can also use an entity in persistent queries, where they are continuously used as a source or sink:

PreviousExample: Setting Up Custom Roles for Production and StageNextUse Namespacing for Organizing Data

Last updated