Free Trial Quick Start

DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and schemas for namespacing and organizing your data.

Using DeltaStream’s free 14-day trial? Follow this guide to build an end-to-end streaming application in minutes. We provide you with a default organization – named after the email address you used to sign on – and a default Kafka store with synthetic data. You’ll use these resources to:

1. Inspect the data in the streaming trial store.

2. Create a database.

3. Create a stream and changelog for your Kafka topics.

4. Enrich your data and query it.

1. Inspect Data in Your Trial Store

You can access a pre-defined DeltaStream trial_store when you sign in to your trial account. This store is a discrete AWS MSK (Managed Streaming for Kafka) cluster that includes several topics with synthetic data producers; the producers continuously publish messages into these topics.

The trial store displays in several places:

• The Welcome page

• The Workspace page

• The Resources page

When you log on, DeltaStream displays the Workspace page. This page provides an at-a-glance dashboard view of your overall DeltaStream organization.

To begin exploring your trial store, in the lefthand navigation click Resources (). The Resources page displays with the Stores tab active and your trial store listed beneath it.

To display the topics contained in the trial store, click anywhere in the trial store row and open the trial_store page.

Now confirm the store connectivity and inspect the data in each topic. To do this:

1. Click a topic. The topic Details page displays.

2. Click Print. This displays the live stream of data flowing to the topic.

Here is an image of the content for the pageviews topic.

2. Create a Database

DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and schemas for namespacing and organizing your data. To create a new database, go to the Catalog page from the main menu and click the + button. Enter the database name, and click SAVE to create your first database.

For this guide, we named our database TestDB. Note that you can create as many databases as you need. Once you create a database, it will also have a schema named public, but you can add more schemas if you wish.

2. Create a Database

Now it’s time to declare a database and relations and write queries on the streaming data. Databases present a logical organization layer for your streaming data. They make it possible to provide access controls and governance across all your data.

To create a new database

2. Enter the database name. In this guide we name it TestDB.

3. Click SAVE.

The newly-created database displays all the topics in the Kafka cluster to which you have access.

You can create as many databases as you wish. Each new database includes a public. You can also add more schemas if you wish.

To add a new schema

2. Enter the schema name. Then click SAVE.

3. Create Streams and Changelogs

Your goal here is to understand pageviews by users over time. You do this by joining the pageviews and users topics.

Start by creating relations backed by Kafka topics. Use DeltaStream’s DDL statements to define your streaming data in a topic as an append-only stream.

To create a stream

2. Copy the SQL DDL statement below, and paste it into the SQL pane (above the Results pane). This creates a discrete stream backed by the pageviews topic from the Kafka cluster; each pageview is an independent event. This stream reflects the view time of each page by user.

3. Click Run.

CREATE STREAM pageviews (
    viewtime BIGINT, 
    userid VARCHAR, 
    pageid VARCHAR
)WITH (
    'topic'='pageviews', 
    'value.format'='JSON'
);

Next, declare a changelog backed by the users topic and ordered by UserID. A changelog enables you to interpret events in a topic as UPSERT events. (In DeltaStream, changelogs are simply another type of relation.) Events require a primary key; DeltaStream interprets each event as an insert or update for the given primary key. In this case, the changelog relation reflects specific details by user, such as gender and interests.

Use the following statement in the SQL pane to declare the users changelog:

CREATE CHANGELOG users_log (
    registertime BIGINT, 
    userid VARCHAR, 
    regionid VARCHAR, 
    gender VARCHAR, 
    interests ARRAY<VARCHAR>, 
    contactinfo STRUCT<phone VARCHAR, city VARCHAR, "state" VARCHAR, zipcode VARCHAR>, 
    PRIMARY KEY(userid)
)WITH (
    'topic'='users', 
    'key.format'='json', 
    'key.type'='STRUCT<userid VARCHAR>', 
    'value.format'='json'
);

As with the pageviews stream, the users changelog displays in the TestDB public schema. To view the streams, in the Catalog pane click to expand the TestDB database and public schema.

4. Run Queries

Now you can write a continuous query in SQL to process this streaming data in real time.

Let’s start with an interactive query, in which the query results stream back to you. You can use such queries to:

• inspect your streams and changelogs

• build queries iteratively by inspecting the query result.

Let’s inspect the pageviews stream. To do this, enter the following interactive query and then click Run:

SELECT * FROM pageviews;

DeltaStream compiles your query into a streaming job, runs the job, and streams the result into the Results pane, as per the below image:

While interactive query results stream in, DeltaStream provides persistent queries. These are continuous queries wherein the query results are written either to a store or a materialized view.

Let’s write a persistent query that joins the pageviews stream with the users_log changelog relations. This creates a third relation called an enriched pageviews stream that provides user details for each pageview event, including view time of each page by user and detailed user information.

While we’re at it, we also convert the epoch time to the timestamp with a timezone using the TO_TIMESTAMP_LTZ function.

Start by creating a stream called enriched_pv. Then join the pageviews stream with data from the users_log changelog and write the results to the enriched_pv stream.

CREATE STREAM enriched_pv 
AS SELECT
    TO_TIMESTAMP_LTZ(viewtime, 3) AS viewtime,  
    p.userid AS userid, 
    pageid, 
    TO_TIMESTAMP_LTZ(registertime, 3) AS registertime, 
    regionid, 
    gender, 
    interests, 
    contactinfo
FROM pageviews p
JOIN users_log u ON u.userid = p.userid;

When the query finishes, you have a new Kafka topic named enriched_pv in your Kafka cluster and a new stream added to the streams in your TestDB database.

Finally, examine the contents of the new stream. Run the following simple query in the SQL pane:

SELECT * FROM enriched_pv;

The result of this interactive continuous query is an enriched pageviews stream that streams to the client as shown below:

That’s it. In just a few steps you’ve used DeltaStream to connect two different Kafka topics and persist the enriched data, either to query in real time or write out to its final destination. In so doing you avoid the extra steps and expense in a data warehouse.

4. Clean Up

When your task is completed, it’s time to clean up your environment. To do this:

2. Click Terminate. Follow the prompt, and then click Terminate again.

Modifying Your Workspace

This quickstart guide uses simple examples to get you up and running quickly. But the DeltaStream workspace is customizable. if you begin using more extensive queries or a greater number of relations, you may find it more efficient to modify the size of your workspace panes, or even toggle on or off specific sections, to focus on the parts of the workspace that matter most at any given time. You can: