Store

How DeltaStream works with your data stores

Stores are where your streaming data resides. Apache Kafka and Amazon Kinesis are two types of stores. DeltaStream reads data from streaming stores and performs the desired computation. Then it writes the results of the computation to that same store or another store.

You own and manage your own stores. To access the data in a store, you configure connectivity and access to it. For instance, if you have an Apache Kafka cluster provided by Confluent Cloud, you can declare a store in DeltaStream by setting up the connectivity and access. When you've defined the store, DeltaStream can read from entities in the Kafka cluster and write into entities in the Kafka cluster.

DeltaStream supports the following:

• Apache Kafka (AWS MSK, Confluent Cloud, and RedPanda)

• AWS Kinesis

• PostgresSQL (only as source for CDC streams)

• Snowflake

• Databricks (only as sink for CTAS queries).

Streaming and Non-streaming Entities

Streaming Entity

DeltaStream stores the streaming data in an entity. The DeltaStream entity is an interface around the event organization layer for the physical streaming stores. In Apache Kafka-type stores a DeltaStream entity corresponds with a Kafka topic; in AWS Kinesis-type stores a DeltaStream entity corresponds with a Kinesis data stream.

DeltaStream uses entities to store the data backing streams and changelogs.

You can create, delete, and view the content of entities.

Non-streaming Entity

DataStream also uses entity to represent the tables in non-streaming stores such as PostgreSQL, Snowflake, and Databricks. Similar to the concept of entities in streaming stores, DeltaStream also uses entities to refer to, inspect, add, or delete tables in postgreSQL, Snowflake, and Databricks.

Schema Registry

A schema registry is a centralized repository for managing and validating schemas for data in Apache Kafka topics. DeltaStream uses the schema registry to represent a schema registry service for Apache Kafka clusters.

Here's an example: If you use Confluent Cloud with a schema registry service, you can define a schema registry in DeltaStream that represents the Confluent Cloud's schema registry service. Then you can use that service to assign the schema registry to the stores that use that service. DeltaStream uses the corresponding schema registry to fetch the topic schemas to deserialize topic content.