# Use Multiple Data Stores in Queries

In DeltaStream, your streaming data resides in a [store](https://docs.deltastream.io/overview/core-concepts/store "mention"). Apache Kafka and Amazon Kinesis are two examples of such data stores. DeltaStream reads data from a streaming data store, performs the desired computation, and writes the results of the computation to the same data store (or to another, if you wish).

This article demonstrates how you can easily move data from one of your data stores to another. It also shows how you can perform joins between data in different data stores.

Before you begin,  To do so, please review the following:

* [store](https://docs.deltastream.io/how-do-i.../create-and-manage-data-stores/store "mention")
* [relation](https://docs.deltastream.io/how-do-i.../relation "mention")

### Before you begin

Before you begin, you should create data stores and DeltaStream objects in DeltaStream, if you haven't already.  Set up the following:

* Two data stores—a Kafka data store called `MSK` and a Kinesis data store called `kinesis_store`
* A stream in `kinesis_store` called `pageviews_kinesis`
* A stream in `MSK` called `users_kafka`

For details on setting these up, please see [store](https://docs.deltastream.io/how-do-i.../create-and-manage-data-stores/store "mention") and [relation](https://docs.deltastream.io/how-do-i.../relation "mention").

### Working with Multiple Data Stores

1. In the lefthand navigation, click **Resources** ( ![](https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2Fgit-blob-b8a3f3aca0b7459aa58df4a07a41134959fbe33f%2FResourcesIcon%20\(3\).png?alt=media) ) to display a list of data stores. Here you'll find your Kafka and Kinesis stores.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2FOjsgJ8pDsEBQDThezpEO%2FStoreList3Stores.png?alt=media&#x26;token=7c444454-b902-4204-a849-f0347fed9d01" alt="" width="563"><figcaption><p>Kafka and Kinesis Stores.</p></figcaption></figure>

2. The Kinesis data store has a Kinesis data stream called `pageviews`.  In DeltaStream, use a [create-stream](https://docs.deltastream.io/reference/sql-syntax/ddl/create-stream "mention") query, as shown below, to create a [#stream](https://docs.deltastream.io/overview/core-concepts/databases#stream "mention") called `pageviews_kinesis` that is backed by the `pageviews` Kinesis stream.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2Fgit-blob-1463743114636ecbc11eea5b3e1bdc5a836805d7%2FCreatPageviewsKinesisStream.png?alt=media" alt="" width="563"><figcaption><p>Creating the <code>pageviews_kinesis</code>Stream.</p></figcaption></figure>

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2Fgit-blob-06fda29b02291c927ec75c47e08b6b0b959d4e99%2FPageviewsKinesisQueryStreamingResults.png?alt=media" alt="" width="563"><figcaption><p>Kinesis Stream Results</p></figcaption></figure>

3. The Kafka data store has a Kafka topic called `ds_users`. In DeltaStream, use a [create-stream](https://docs.deltastream.io/reference/sql-syntax/ddl/create-stream "mention") query shown below to create a [#stream](https://docs.deltastream.io/overview/core-concepts/databases#stream "mention") called `users_kafka` that is backed by the `ds_users` Kafka topic.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2Fgit-blob-71f628c66aeb25d1919acb1ff41eb478b51908c7%2FCreatingUsersKafkaStream.png?alt=media" alt="" width="563"><figcaption><p>Creating the <code>users_kafka</code>Stream.</p></figcaption></figure>

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2Fgit-blob-1df7c3e772c08f4a0a7a1e8f769b43e7a49410dc%2FUsersKafkaStreamResults.png?alt=media" alt="" width="563"><figcaption><p>Kafka Stream Results</p></figcaption></figure>

With your two data stores established, each with an associated relation, you can write your queries.

### Move Data from One Data Store to Another

To migrate data out of Kinesis and into Kafka, create a new stream that selects all the columns from an existing stream. Then you can specify the data store that backs the new stream.

In the setup here, there's a stream belonging to your Kinesis data store called `pageviews_kinesis`. Use a [create-stream-as](https://docs.deltastream.io/reference/sql-syntax/query/create-stream-as "mention") query (CSAS) to create a new stream that is essentially a copy of the `pageviews_kinesis` stream in the Kafka data store. Label this new stream `pageviews_kafka`.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2F1Jv85G4yOyFxFKTBQtQx%2FCreateKafkaCopyKinesis.png?alt=media&#x26;token=a587e337-252b-4968-9ac7-61a8173ddd8d" alt="" width="563"><figcaption><p>Creating a Kafka Stream that is a copy of a Kinesis Stream.</p></figcaption></figure>

Note the CSAS query above includes the `store` property in the `WITH` clause of the sink stream, with a value of `kafka_store`. A CSAS query creates a new stream; specifying this property informs the query about which data store should back this new stream. Leave the `store` property empty, and the query defaults to using the session’s current data store. You must include the `topic.replicas` and `topic.partitions` properties; these are necessary when creating a new Kafka topic.&#x20;

{% hint style="success" %}
**Tip**   If the source in this query was a stream backed by a Kafka topic, then DeltaStream would use the source’s `topic.replicas` and `topic.partitions` values by default. But here you're reading from a Kinesis-backed stream, and DeltaStream cannot infer the value of these properties; you must set them explicitly in the query.
{% endhint %}

Navigate to **Resources** > **Kafka Store** to see that a new Kafka topic, called `pageviews_kafka`, has been created and displays in the topic list. The streaming data for the newly-created `pageviews_kafka` stream is stored in this topic. Click the topic to print and see the records flowing through. These records are a copy of the `pageviews_kinesis` records.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2FKM7fayS12WDtESZrUH6l%2FNewStreamCopy.png?alt=media&#x26;token=352ad034-2bf1-4cca-bf6a-de6793931f8a" alt="" width="375"><figcaption><p>The new stream copy.</p></figcaption></figure>

### Join Data from Sources Belonging to Different Data Stores

In DeltaStream, when you define a [#stream](https://docs.deltastream.io/overview/core-concepts/databases#stream "mention") or [#changelog](https://docs.deltastream.io/overview/core-concepts/databases#changelog "mention") you can use it as a source for queries. The example below demonstrates how you can join the `pageviews_kinesis` stream and the `users_kafka` stream. Simply use these streams as sources in your query; you do not need any additional specifications related to the data stores that back them. DeltaStream keeps this information as metadata with the stream. Behind the scenes, DeltaStream seamlessly reads from both data stores, performs the join, and then outputs the result to the sink stream. Again, use a [create-stream-as](https://docs.deltastream.io/reference/sql-syntax/query/create-stream-as "mention") query to create the output stream. Since this joins two streams, this query is an [#interval-join-stream-stream](https://docs.deltastream.io/reference/sql-syntax/query/select#interval-join-stream-stream "mention") that requires the `WITHIN` clause.

<figure><img src="https://1288764042-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdbd9e6ZJodkgF1H6AVay%2Fuploads%2FLS5RTfDJea0hFdJ8qM5l%2FCreateStreamAsSelectQuery.png?alt=media&#x26;token=7a6c09f5-7d45-44b8-9b96-678ff238a48b" alt="" width="563"><figcaption><p>Using a <code>CREATE STREAM AS SELECT</code> query.</p></figcaption></figure>

Note in the CSAS above that in the `WITH` clause you specify the output data store to be `kinesis_store`, similar to what you did in the [section above](#move-data-from-one-store-to-another). But even though you're creating a new Kinesis stream, called `pageviews_enriched`, you don’t need to provide the `topic.shards` property. DeltaStream infers the default value from the left-most source when possible. The sink stream and the left-most source are both backed by Kinesis streams, so `pageview_kinesis`’s `topic.shards` property is applied to `pageviews_enriched`.

Navigate to **Resources** > **Kinesis\_store** > **Topics** to see that there's a new Kinesis entity called `pageviews_enriched`. This is where the streaming data for the newly-created `pageviews_enriched` stream is stored. Click that entity to print and see the records flowing through. These records result from the join between `pageviews_kinesis` and `users_kafka`.
