Trial Quick Start

DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and schemas for namespacing and organizing your data.

Using DeltaStream’s free 14-day trial? Follow this guide to build an end-to-end streaming application in minutes. We provide you with a default organization – named after the email address you used to sign on – and a default Kafka store with synthetic data. You’ll use these resources to:

1. Inspect the data in the streaming trial store.

2. Create a database.

3. Create a stream and changelog for your Kafka topics.

4. Enrich your data and query it.

Note The trial version limits you to 3 queries. Also, user-defined functions aren’t supported, and there are no materialized views. You can add your own external store, but it must be available via the Internet. Contact DeltaStream support if you wish to set up a private store.

1. Inspect Data in Your Trial Store

You can access a pre-defined DeltaStream trial_store when you sign in to your trial account. This store is a discrete AWS MSK (Managed Streaming for Kafka) cluster that includes several topics with synthetic data producers; the producers continuously publish messages into these topics.

Note In DeltaStream you define stores to represent each Kafka cluster. DeltaStream also works with other stores, such as AWS Kinesis and Postgres.

The trial store displays in several places:

• The Welcome page

• The Workspace page

• The Resources page

When you log on, DeltaStream displays the Workspace page. This page provides an at-a-glance dashboard view of your overall DeltaStream organization.

To begin exploring your trial store, in the lefthand navigation click Resources (). The Resources page displays with the Stores tab active and your trial store listed beneath it.

To display the topics contained in the trial store, click anywhere in the trial store row and open the trial_store page.

Now confirm the store connectivity and inspect the data in each topic. To do this:

1. Click a topic. The topic Details page displays.

2. Click Print. This displays the live stream of data flowing to the topic.

Here is an image of the content for the pageviews topic.

Note After you verify that data is streaming into your trial store, you may wish to click Stop to halt the stream.

To view more details on the data streaming in, click to display the events unformatted and then click the arrow to view an event in code format.

2. Create a Database

DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and schemas for namespacing and organizing your data. To create a new database, go to the Catalog page from the main menu and click the + button. Enter the database name, and click SAVE to create your first database.

For this guide, we named our database TestDB. Note that you can create as many databases as you need. Once you create a database, it will also have a schema named public, but you can add more schemas if you wish.

2. Create a Database

Now it’s time to declare a database and relations and write queries on the streaming data. Databases present a logical organization layer for your streaming data. They make it possible to provide access controls and governance across all your data.

Note In DeltaStream, relations are the building blocks of user applications and pipelines. You must create a relation to represent each Kafka topic you wish to include in a query.

To create a new database

1. From the lefthand navigation, click Catalog () and then click +.
2. Enter the database name. In this guide we name it TestDB.

3. Click SAVE.

The newly-created database displays all the topics in the Kafka cluster to which you have access.

You can create as many databases as you wish. Each new database includes a public. You can also add more schemas if you wish.

To add a new schema

1. From the lefthand navigation, click , and towards the right click Add Schema.
2. Enter the schema name. Then click SAVE.

3. Create Streams and Changelogs

Your goal here is to understand pageviews by users over time. You do this by joining the pageviews and users topics.

Start by creating relations backed by Kafka topics. Use DeltaStream’s DDL statements to define your streaming data in a topic as an append-only stream.

Note In DeltaStream, a stream is simply one type of relation.

To create a stream

1. Navigate to the main workspace by clicking .
2. Copy the SQL DDL statement below, and paste it into the SQL pane (above the Results pane). This creates a discrete stream backed by the pageviews topic from the Kafka cluster; each pageview is an independent event. This stream reflects the view time of each page by user.

3. Click Run.

CREATE STREAM pageviews (
    viewtime BIGINT, 
    userid VARCHAR, 
    pageid VARCHAR
)WITH (
    'topic'='pageviews', 
    'value.format'='JSON'
);

Note The above stream is created in the currently-used database and schema – TestDB and public, respectively. DeltaStream uses the default store declared above as the store for the pageviews topic. To specify another store, use the WITH clause.

Next, declare a changelog backed by the users topic and ordered by UserID. A changelog enables you to interpret events in a topic as UPSERT events. (In DeltaStream, changelogs are simply another type of relation.) Events require a primary key; DeltaStream interprets each event as an insert or update for the given primary key. In this case, the changelog relation reflects specific details by user, such as gender and interests.

Use the following statement in the SQL pane to declare the users changelog:

CREATE CHANGELOG users_log (
    registertime BIGINT, 
    userid VARCHAR, 
    regionid VARCHAR, 
    gender VARCHAR, 
    interests ARRAY<VARCHAR>, 
    contactinfo STRUCT<phone VARCHAR, city VARCHAR, "state" VARCHAR, zipcode VARCHAR>, 
    PRIMARY KEY(userid)
)WITH (
    'topic'='users', 
    'key.format'='json', 
    'key.type'='STRUCT<userid VARCHAR>', 
    'value.format'='json'
);

As with the pageviews stream, the users changelog displays in the TestDB public schema. To view the streams, in the Catalog pane click to expand the TestDB database and public schema.

4. Run Queries

Now you can write a continuous query in SQL to process this streaming data in real time.

Let’s start with an interactive query, in which the query results stream back to you. You can use such queries to:

• inspect your streams and changelogs

• build queries iteratively by inspecting the query result.

Let’s inspect the pageviews stream. To do this, enter the following interactive query and then click Run:

SELECT * FROM pageviews;

DeltaStream compiles your query into a streaming job, runs the job, and streams the result into the Results pane, as per the below image:

Note After you verify that data is streaming in you may wish to click Stop Query.

While interactive query results stream in, DeltaStream provides persistent queries. These are continuous queries wherein the query results are written either to a store or a materialized view.

Let’s write a persistent query that joins the pageviews stream with the users_log changelog relations. This creates a third relation called an enriched pageviews stream that provides user details for each pageview event, including view time of each page by user and detailed user information.

While we’re at it, we also convert the epoch time to the timestamp with a timezone using the TO_TIMESTAMP_LTZ function.

Start by creating a stream called enriched_pv. Then join the pageviews stream with data from the users_log changelog and write the results to the enriched_pv stream.

CREATE STREAM enriched_pv 
AS SELECT
    TO_TIMESTAMP_LTZ(viewtime, 3) AS viewtime,  
    p.userid AS userid, 
    pageid, 
    TO_TIMESTAMP_LTZ(registertime, 3) AS registertime, 
    regionid, 
    gender, 
    interests, 
    contactinfo
FROM pageviews p
JOIN users_log u ON u.userid = p.userid;

Note The above persistent query creates a new topic in the trial store DB. When you create a new topic, DeltaStream adds a prefix name to the topic name based on your trial email and some unique random characters. For example, for the email [e-mail], DeltaStream creates a topic prefix like t_testgmailcom_4evmsyg_. Creating the topic enriched_pv in turn creates the topic t_testgmailcom_4evmsyg_enriched_pv. You can view these topics in the trial store topics list.

Important Topic name prefixes are a requirement only for the trial store we have set. Prefixes are not added if you use any other store, such as your own Apache Kafka or AWS Kinesis.

Note DeltaStream compiles and launches the query as an Apache Flink streaming job. You can view the query along with its status in the Query Management page; to do this, in the lefthand navigation click Queries ().

When the query finishes, you have a new Kafka topic named enriched_pv in your Kafka cluster and a new stream added to the streams in your TestDB database.

Finally, examine the contents of the new stream. Run the following simple query in the SQL pane:

SELECT * FROM enriched_pv;

The result of this interactive continuous query is an enriched pageviews stream that streams to the client as shown below:

That’s it. In just a few steps you’ve used DeltaStream to connect two different Kafka topics and persist the enriched data, either to query in real time or write out to its final destination. In so doing you avoid the extra steps and expense in a data warehouse.

4. Clean Up

When your task is completed, it’s time to clean up your environment. To do this:

1. Terminate the queries. To do this, click to display the Queries page. Then click . The Query Details page displays.

   
2. Click Terminate. Follow the prompt, and then click Terminate again.

3. Drop the created streams, changelogs, and materialized views. To do this, click to navigate to the corresponding database and schema, and as with the query, click and follow the prompt to drop the streams, changelogs, and materialized views.

Important If you have a query that uses a stream, changelog, or materialized view, you must terminate the query before dropping the relation.

Modifying Your Workspace

This quickstart guide uses simple examples to get you up and running quickly. But the DeltaStream workspace is customizable. if you begin using more extensive queries or a greater number of relations, you may find it more efficient to modify the size of your workspace panes, or even toggle on or off specific sections, to focus on the parts of the workspace that matter most at any given time. You can:

• Hide the Results pane. This gives you more room in the SQL pane to work with more extensive SQL. To do this, at the top of your workspace click . Click it a second time to re-display the Results pane.
• Hide the SQL pane. This gives you more room to examine query results. To do this, at the top of your workspace click . Click it a second time to re-display the SQL pane.
• Hide the lefthand (Catalog and Stores) pane. This gives you more horizontal screen real estate and creates a cleaner, more expansive workspace. To do this, at the top of your workspace click . Click it a second time to re-display the lefthand pane.

When activated, the icons display in color – for example, .

Tip You can click any two or all three of these icons at once to isolate the precise workspace you wish. For example, if you click and you have almost the entire screen to work with SQL.

Finally, you can manually re-size your panes without hiding them altogether. To resize the SQL and Results panes, or the Catalog and Stores panes, click and drag . To resize the left and right panes, click and drag .