# What is DeltaStream? How DeltaStream combines streaming and batch processing for real-time analytics using the compute engine of your choice. DeltaStream is a unified analytics platform that combines batch and stream processing and real-time analytics in one platform. Write queries in SQL, and DeltaStream routes them to the optimal execution engine – for example, Flink for streaming, Spark for batch, and ClickHouse for materialized views. Hierarchical namespacing enables you to process and query streams, similar to the way in which you’d structure data in Iceberg, Snowflake, or Delta Table. You can also use DeltaStream’s SQL-based interface to create stream processing applications such as streaming pipelines, materialized views, microservices, and more. You deploy DeltaStream within your organization’s cloud, keeping your data confined to your environment. Depending on the scale and complexity of your workloads, you can do your real-time analytical processing in DeltaStream or continue to use your preferred 3rd-party engine such as AWS Athena or Trino. For example: * Query materialized views in ClickHouse with native support for upserts * Stream from Apache Kafka or AWS Kinesis to Clickhouse for extra low-latency queries * Add upsert capability to your Snowflake or Clickhouse processing * Read from and write to Iceberg tables (with support for both the Glue catalog and the REST catalog) * Augment your Databricks Lakehouse with Apache Flink for true real-time stream processing Deltastream integrates with streaming storage services including Apache Kafka, AWS Kinesis, Confluent Cloud, AWS MSK, WarpStream, and Redpanda.
# Core Concepts Understand the basic concepts and technologies that underpin DeltaStream DeltaStream is a unified analytics platform that combines batch and stream processing and real-time analytics in one platform. Write queries in SQL, and DeltaStream routes them to the optimal execution engine – for example, Flink for streaming, Spark for batch, and ClickHouse for materialized views. You can also use DeltaStream’s SQL-based interface to build stream processing applications such as streaming pipelines, materialized views, microservices, and more. {% hint style="info" %} **Note** DeltaStream sits on top of your streaming data stores, meaning that you should already have one or more streaming stores such as Apache Kafka or AWS Kinesis. {% endhint %} This section provides more details on the core concepts in DeltaStream. Below is a brief walk-through of the most important constructs with which to familiarize yourself. {% @storylane/embed subdomain="deltastream" url="" linkValue="3tiqde8nosn2" %} # Access Control How DeltaStream handles user access ## Organization An organization is the unit of tenant isolation in DeltaStream. All objects ([databases](/overview/core-concepts/databases), data [stores](/overview/core-concepts/store), [queries](/overview/core-concepts/queries), and so on) are set up under the organization. ## User A user represents an authenticated identity within DeltaStream. A user may be a member of multiple organizations. Within each organization, a user can hold one or more [roles](#role). ## DeltaStream Object DeltaStream objects are entities you can protect with access controls. Access to an object is denied by default; it must be explicitly granted privileges to a role. Each object resides within a hierarchy, with the [organization](#_organiation) as the root. Organizations contain data [stores](/overview/core-concepts/store), [schema registries](/overview/core-concepts/store#_schema_registry), [descriptors](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors), [databases](/overview/core-concepts/databases), [function sources](/overview/core-concepts/function), and [queries](/overview/core-concepts/queries). Databases contain namespaces, which in turn contain streams, changelogs, and materialized views.

Securable object container hierarchy

Every DeltaStream object is owned solely by the role used when the object was created. The owner role has all [privileges](#_privilege) on the object and can grant or revoke privileges to other [roles](#_role). Object ownership can be transferred from one role to another. ## Privilege In DeltaStream, privileges determine which role can access and perform operations on a given object. Every object has a set of privileges that can be granted on it. Privileges are managed using the [GRANT](/reference/sql-syntax/command/grant-privileges) and [REVOKE](/reference/sql-syntax/command/revoke-privileges) commands. Use of these commands is restricted to: * The role that owns an object (that is, the role with `OWNERSHIP` privilege on the object) * Any roles that have the `MANAGE_GRANTS` global privilege for the object * By default, this is `SECURITYADMIN`. {% hint style="info" %} **Note** A role that holds the global `MANAGE_GRANTS` privilege can grant additional privileges to the current (grantor) role. {% endhint %} #### **Available privileges**
PrivilegeApplicable objectDescription
CREATE_DATABASEOrganizationAllows databases to be created
CREATE_STOREOrganizationAllows data stores to be created
CREATE_SCHEMA_REGISTRYOrganizationAllows schema registries to be created
CREATE_DESCRIPTOR_SOURCEOrganizationAllows descriptor sources to be created
CREATE_FUNCTION_SOURCEOrganizationAllows function sources to be created
CREATE_FUNCTIONOrganizationAllows functions to be created
CREATE_QUERYOrganizationAllows queries to be run
MANAGE_MEMBERSOrganizationAllows management of users and roles
MANAGE_GRANTSOrganizationAllows management of privileges
USAGEOrganization, Database, Schema, Namespace, data Store, Schema Registry, Descriptor Source, Function Source, Function, Query, RoleAllows viewing and basic usage of the object. Additional privileges may be required for some actions
CREATEDatabase, SchemaFor databases, allows new schemas to be created with in the database.

For schemas, allows defining new relations within the schema.
SELECTRelationAllows select queries to use to the relation
INSERTRelationAllows queries to inserting data into the relation
## Role A role is an entity to which privileges can be granted. You assign roles to [users](#_user) to allow them to perform actions against an [object](#_securable_object). Users can have multiple roles. This enables them to switch roles to perform actions with a different set of privileges. {% hint style="info" %} **Note** DeltaStream includes a set of[ built-in roles](#built-in-roles) under every organization. Built-in roles cannot be dropped and the [privileges](#_privilege) assigned to these roles cannot be revoked. {% endhint %} Roles also can be granted to other roles. This creates a hierarchy of roles wherein the privileges associated with one role are inherited by any role that's higher in the hierarchy. Here's an example: In the diagram below, the `SysAdmin` role automatically inherits any privilege assigned to a custom role.

Role hierarchy

### Built-in roles The following are built into DeltaStream and are available "out of the box:" * **ORGADMIN** — Root of the the role hierarchy. This role manages operations at the organization level. * **SYSADMIN** — Has privileges to create, manage, and drop objects. * **USERADMIN** — Has privileges to manage users and roles within the organization. * **SECURITYADMIN** — Manages any object grants globally. This role inherits privileges from the `USERADMIN` role. * **PUBLIC** — A pseudo role granted to all roles within the organization. This role can be granted privileges on objects that are inherited by all other roles. ### Custom roles Custom roles can be created in two ways: * by someone with the `USERADMIN` (or a higher) role * by any other role to which the `MANAGE_MEMBERS` privilege has been granted. When you create a new role it is not assigned to any user, nor granted to any other role. For roles you intend to be owners of objects, we recommend you use a custom role hierarchy wherein the top-most custom role in the hierarchy is assigned to the `SYSADMIN` system role. This enables system administrators to manage all the objects in the organization, while still restricting management of users and roles to the `USERADMIN` role. {% hint style="warning" %} If you do not assign `SYSADMIN` through a role hierarchy to a custom role, the system administrator cannot manage objects owned by the custom role. This means that only roles that have been granted the `MANAGE_GRANTS` privilege (the `SECURITYADMIN` role by default) can view the objects and modify their access grants. {% endhint %} For instructions to create custom roles, see [CREATE ROLE](/reference/sql-syntax/ddl/create-role). ## Best Practices We recommend the following best practices for access control: * Reserve the `ORGADMIN` role for administrative tasks only. **It must not be used for day-to-day access.** * Administrators should have 2 separate logins: * a login with the `ORGADMIN` role for administrative tasks * a separate role for day-to-day access. * `ORGADMIN` must not own any objects and it must never be granted to any other roles. * `SECURITYADMIN` is also a very powerful role with the `MANAGE_GRANTS` privilege. Only use this role in limited cases. * `USERADMIN` should be the default role for managing users and role grants. This role typically is the owner of all other custom roles. * `SYSADMIN` should be the default role for managing objects and should always be the eventual owner of all other objects. You can grant custom roles to `SYSADMIN.` But you must never grant it ownership of other roles. This maintains the separation of responsibility between `SYSADMIN` and `USERADMIN`. * When you build a custom role hierarchy, define functional roles and access roles. You can grant privileges on objects to access roles, and you can grant access roles to functional roles. * Use [SCIM](/enterprise-security-integrations/okta-scim-integration)-based automation to manage grants of functional roles to users. # Compute Pools Get real-time analytics from data stored in Iceberg tables. A compute pool is a set of dedicated resources to run batch queries. Compute pools are Apache Spark clusters you use to perform real-time analytics on data you read from Iceberg tables. They are similar to Databricks’ [all-purpose compute](https://docs.databricks.com/aws/en/compute/use-compute). You create a compute pool much like you’d create any new DeltaStream object. Define it at the organization level and follow the same access control rules. Specific to compute pools, however, you also must select a pool size – S, M, or L. DeltaStream auto-configures and instantiates the pool based on your selection. You never manage or interact with Spark directly. From there, you can execute queries, joins, and so on just as you would with any other data store in DeltaStream. (Joins currently apply only from Iceberg to Iceberg.) Depending on your use case, you can perform your analysis without using an external query engine such as AWS Athena or Trino. Also, as with DeltaStream data stores, you can create multiple compute pools per organization. {% hint style="info" %} **Note** You do not need a compute pool if you are only writing to Iceberg – if, for example, you’re streaming filtered Kafka data into Iceberg tables. Compute pools are necessary only if you wish to read from/query Iceberg data. {% endhint %} ### Using SQL DDL with Compute Pools in DeltaStream [Create Compute\_Pool](/reference/sql-syntax/ddl/create-compute_pool) [Update Compute\_Pool](/reference/sql-syntax/ddl/update-compute_pool) [Create Store](/reference/sql-syntax/ddl/create-store) [List Compute\_Pool](/reference/sql-syntax/command/list-compute_pools) [Drop Compute\_Pool](/reference/sql-syntax/ddl/drop-compute_pool) [Start Compute\_Pool](/reference/sql-syntax/command/start-compute_pool) [Stop Compute\_Pool](/reference/sql-syntax/command/stop-compute_pool) ###
# Data Store How DeltaStream works with your data stores In DeltaStream, data stores are where your streaming data resides. Apache Kafka and Amazon Kinesis are two types of data stores. DeltaStream reads data from streaming data stores and performs the desired computation. Then it writes the results of the computation to that same data store or another store. You own and manage your own data stores. To access the data in a data store, you configure connectivity and access to it. For instance, if you have an Apache Kafka cluster provided by Confluent Cloud, you can declare a data store in DeltaStream by setting up the connectivity and access. When you've defined the data store, DeltaStream can read from entities in the Kafka cluster and write into entities in the Kafka cluster. DeltaStream supports the following: * Apache Kafka (AWS MSK, Confluent Cloud, and RedPanda) * AWS Kinesis * PostgresSQL * Snowflake * Databricks (only as sink for [CTAS queries](/reference/sql-syntax/query/create-table-as)). * ClickHouse (for materialized views) * AWS S3 * Oracle * Apache Iceberg with the REST catalog * Apache Iceberg with the AWS Glue catalog ## Streaming and Non-streaming Entities ### Streaming Entity A DeltaStream entity is an interface around the event organization layer for the physical streaming stores. In Apache Kafka-type data stores a DeltaStream entity corresponds with a Kafka topic; in AWS Kinesis-type data stores a DeltaStream entity corresponds with a Kinesis data stream. DeltaStream uses entities to store the data backing [streams](/overview/core-concepts/databases#_stream) and [changelogs](/overview/core-concepts/databases#_changelog). Relations such as streams and changelogs are DeltaStream objects defined on top of those entities. Use `PRINT ENTITY` to inspect the underlying entity directly, and use queries such as `SELECT` to work with the DeltaStream relation defined on top of it. {% hint style="info" %} **Note** You can create, delete, and view the content of entities. {% endhint %} ### Non-streaming Entity DeltaStream uses entities to represent the tables in non-streaming data stores such as PostgreSQL, Snowflake, and Databricks. Similar to the concept of entities in streaming data stores, DeltaStream also uses entities to refer to, inspect, add, or delete tables in postgreSQL, Snowflake, and Databricks. ## Schema Registry A schema registry is a centralized repository for managing and validating schemas for data in Apache Kafka topics. DeltaStream uses the schema registry to represent a schema registry service for Apache Kafka clusters. Here's an example: If you use Confluent Cloud with a schema registry service, you can define a schema registry in DeltaStream that represents the Confluent Cloud's schema registry service. Then you can use that service to assign the schema registry to the data stores that use that service. DeltaStream uses the corresponding schema registry to fetch the topic schemas to deserialize topic content. # Database Data organization in DeltaStream Databases are the foundation for organizing data in DeltaStream. They provide the building block of its namespacing model. You create databases for logical groupings for different teams or projects. For instance, you can create one database for a logging project and another for an ads team. ## Namespace A namespace is a logical grouping of relational objects such as streams, changelogs, materialized views, and tables. Namespaces are grouped in a database. A combination of databases and namespaces enable you to organize their streams, changelogs, and other database objects in a hierarchical fashion in DeltaStream. Such hierarchies also are a basis for providing role-based access control (RBAC) in DeltaStream in the same way as with other relational databases. ## DeltaStream Object DeltaStream provides a relational model for streaming data wherein data is stored in **objects**. DeltaStream supports the following object types: * Stream * Changelog * Materialized View * Table In DeltaStream, these objects are building blocks of your applications and pipelines. You can specify object names as fully- or partially-qualified names by specifying a [database](/overview/core-concepts/databases) and/or [namespace](#schema) name in the format of `[..]`, like this: `db1.public.pageviews` Otherwise, DeltaStream uses the current database and namespace in the scope of a client to identify an object. ### Stream A stream is a sequence of immutable, partitioned, and partially-ordered events. {% hint style="success" %} **Tip** DeltaStream uses the terms "events" and "records" synonymously. {% endhint %} * A stream is a relational representation of data in streaming data stores, such as the data in a Kafka topic or a Kinesis stream. * The records in a stream are independent of each other; there is no correlation between two records in a stream. * A stream declares the schema of the records; this includes the column name, the column type, and optional constraints. ### Changelog As with a stream, a changelog is * a sequence of partitioned and partially-ordered events * a relational representation of data in the streaming data stores, such as the data in a Kafka topic or a Kinesis stream. A changelog defines a `PRIMARY KEY` used to represent the change over time for records with the same primary key. Records in a changelog correlate with each other based on the `PRIMARY KEY`. This means a record in a changelog either is an insert (if it’s the first time the record with the given `PRIMARY KEY` is appended to the changelog) or an upsert (if a previous record with the same `PRIMARY KEY` has already been inserted into the changelog). ### Materialized View A materialized view creates a snapshot of a streaming query result and continuously updates the snapshot as records arrive to the query input(s). A materialized view is queryable in DeltaStream; when you query it the results are computed using the data in the snapshot at query runtime. {% hint style="info" %} **Note** Queries on a materialized view are not streaming queries. They are the same as the queries on tables and materialized views in traditional relational databases. {% endhint %} ### Table A table is similar to a materialized view in that it stores records from a streaming source. Unlike materialized views, however, tables do not support upserts. Rather, DeltaStream stores all records from a source or an upstream query operation (such as a `JOIN` or aggregation) as a sequence of records, as they are provided, for the sink that writes to the table. When you use a table with records that have a primary key -- for example, a [changelog](#changelog) -- the resulting rows in the table represent the incremental changes to each record key. ### Row Key Each record in a [stream](#_stream) or [changelog](#_changelog) can have a row key. (Defining a row key is optional for an object.) The value of a key for a given record is extracted from its corresponding message, which is read from the source relation’s [entity](/overview/core-concepts/store#entity). For example, if you use a Kafka topic as the object’s entity, Kafka messages’ key bytes assign row key values to the object’s records, based on the object’s row key definition (if any). {% hint style="info" %} **Note** Some operations such as `GROUP BY` and `JOIN` impact the row key definition and add row keys to their results’ records. {% endhint %} When writing query results to a sink, the records’ keys are written as the messages’ keys into the sink relation’s [entity](/overview/core-concepts/store#entity). For example, when the result of a join query is written into a Kafka topic, the row keys of the resulting records are set as Kafka messages’ keys. For more details, see [Row Key Definition](/reference/sql-syntax/relation-key-definition). # Function Defining your own functions in DeltaStream to extend DeltaStream capabilities for your business In DeltaStream, **functions** refer to user-defined functions that extend DeltaStream's processing capabilities. There are multiple built-in functions available you can use in your queries. You can also create and implement more functions and add them to DeltaStream via function declaration syntax. There are three types of functions you can declare in DeltaStream. 1. **UDFs** (User-Defined Functions): These functions can have zero or more arguments and return one value. 2. **UDAFs** (User-Defined Aggregate Functions): These functions compute values across multiple rows (events) and return one value. 3. **UDTFs** (User-Defined Table Functions): These functions can have zero or more input arguments, but they return multiple values that can result in multiple rows in the output. {% hint style="info" %} **Note** As DeltaStream is powered by [Apache Flink](https://flink.apache.org/), you can implement functions as Apache Flink user-defined functions. For more details, see the corresponding [Apache Flink documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/). {% endhint %} Package the implemented functions, plus their dependencies, in a [JAR](https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html) file and upload the JAR file to DeltaStream. Then you can declare a new function and add it to the functions already available in DeltaStream. You can use the declared function in the expressions in your DeltaStream queries. For more details, see [Create a Function](/how-do-i.../creating-a-function). # Query Querying data in DeltaStream -- how you get things done There are two main types of queries: 1. [Streaming or Continuous Queries](#_streaming_or_continuous_query) 2. [Queries on Materialized Views](#query-on-materialized-views) In the DeltaStream UI **Query** tab, or when [listing queries with the CLI](/reference/sql-syntax/command/list-queries), DeltaStream displays only long-running streaming queries. Queries on materialized views are snapshot queries and therefore not long-lived. {% hint style="info" %} **Note** In DeltaStream, SQL queries that actually process data are just called queries. {% endhint %} ## Streaming or Continuous Queries In DeltaStream, a streaming or continuous query does the following: 1. reads from one or more [streams](/overview/core-concepts/databases#_stream) and/or [changelogs](/overview/core-concepts/databases#_changelog) 2. processes the data according to the query logic 3. generates one or more streams or changelogs Streaming queries are continuous. When you start them they keep running until you explicitly terminate them. Streaming queries themselves take two forms in DeltaStream: 1. **Persistent streaming (continuous) queries:** These queries persist the results in the streaming storage by creating new streams or changelogs. 2. **Interactive streaming (continuous) queries:** When you start these queries, they run continuously and stream the results of the query to you. Streaming queries start in the `NEW` state. When you run them they transition to the `RUNNING` state. At this point they process records from the source object(s) and write into the sink object. When a streaming query fails or otherwise stops operating, it displays as an `ERRORED` state. The system may automatically restart a query if it detects an underlying problem, such as a network issue. In failure scenarios, DeltaStream restarts queries from the last known state, at which point they proceed to operate normally. {% hint style="info" %} **Note** You can report back unrecoverable failures either via the [DESCRIBE QUERY](/reference/sql-syntax/command/describe-query) command or by clicking on the query state in the DeltaStream Web app. {% endhint %} If you terminate a query on your own, DeltaStream considers the query complete and performs system cleanup. See [Query](/reference/sql-syntax/query) for a list of statements that can start a query for processing records within a [Data Store](/overview/core-concepts/store). ## Query on Materialized Views When you build a [Materialized View](/overview/core-concepts/databases#_materialized_view) in DeltaStream, you can use standard SQL queries to query the content of the view. Unlike continuous queries, queries on materialized views compute and return results based on the content of the view at the time you issue the query. These queries behave like the queries on tables in traditional relational databases. # Region The impact of regional data storage in DeltaStream A **region** specifies the cloud and cloud region from where you access your streaming stores (such as Apache Kafka and AWS Kinesis), and is also where DeltaStream runs its queries. When you create a DeltaStream [store](/overview/core-concepts/store), you must specify an `access_region`. All [queries](/overview/core-concepts/queries) that use the store run in the specified `access_region`. It is possible to use an `access_region` that differs from the cloud/region hosting the database. But this incurs additional cost and latency due to cross-region traffic. Here's an example: When you create an MSK store, if you specify `us-west-1` as the access region, all the queries in DeltaStream that read and write from this store run in the `us-west-1` region. For a list of supported `access_regions,` use the [SHOW REGIONS](broken://pages/dqmTShTXTtD4hHXCMHxo) command. # SQL Do everything you need in DeltaStream using only SQL SQL is the primary interface for DeltaStream. Using a simple and familiar SQL interface, you can * create databases * run continuous queries * build materialized views * and more DeltaStream also provides SQL extensions that enable you to express streaming concepts that don’t have equivalents in traditional SQL. ## DDL SQL DDLs are queries that create, delete, or modify database structures. Unlike queries, DDLs do not process data. Examples of useful DDLs include: * [creating databases](/reference/sql-syntax/ddl/create-database) * [creating streams](/reference/sql-syntax/ddl/create-stream) * [dropping entities](/reference/sql-syntax/ddl/drop-entity) * [updating stores](/reference/sql-syntax/ddl/update-store) DDLs are useful in both the DeltaStream UI and the CLI. However, the UI also provides an interface for many of these DDLs, such as creating [stores](/overview/core-concepts/store). For a full list of SQL DDLs, go to [the DDL page](/reference/sql-syntax/ddl) of the [SQL Syntax](/reference/sql-syntax) section. ## Query There are two main types of SQL queries that do process data: 1. [Streaming or continuous queries](/overview/core-concepts/queries#_streaming_or_continuous_query) 2. [Materialized view queries](/overview/core-concepts/queries#query-on-materialized-views) See more about both in the [Query](/overview/core-concepts/queries) page under [Core Concepts](/overview/core-concepts). When you write streaming queries in the UI, successful queries display in the **Query** tab. Interactive streaming queries and queries on materialized views both print results on the page. Similarly, for the CLI you can list successful streaming queries with the [LIST QUERIES](/reference/sql-syntax/command/list-queries) command; interactive streaming queries and queries on materialized views in this case print results directly in the console. For a full list of SQL queries, go to the [Query](/reference/sql-syntax/query) section under [SQL Syntax](/reference/sql-syntax). ## Command SQL commands are actions that do not change database entities and do not process data. Examples of useful commands include: * listing and describing DeltaStream entities * inviting users to your organization * granting privileges * printing the records in [entities](/overview/core-concepts/store#entity) SQL commands are useful when using the CLI. DeltaStream provides the functionality of these commands through its UI. For a full list of SQL commands, go to the [Commands](/reference/sql-syntax/command) section under [SQL Syntax](/reference/sql-syntax). # Visualizing Data Lineage Display your DeltaStream ecosystem in adjustable flowchart form. If you have a large number of data objects, it can be difficult to keep track of where all your data is coming from, how it’s being queried, and where it’s ending up. DeltaStream addresses this via Stream 360. Stream 360 displays the movement of your data as a flowchart, from source to destination, including how it’s transformed along the way. You can see at a glance all your DeltaStream objects – streams, changelogs, tables, materialized views – and their relationships with queries. This gives you a clear visual representation of data lineage throughout your DeltaStream ecosystem. You can: * understand your processes * validate data at particular points in the data lineage * see all aspects of the pipeline you’re building – joins, sources, pipeline results (such as writings to a materialized view), and more * examine streams and materialized views and quickly ascertain what data you may wish to add or whether the data you need is in the data stores displayed. * move or group objects to isolate and examine a specific process The end-to-end view also helps you avoid negative or unanticipated downstream impacts resulting from any changes you make. You can also use Stream 360 to view metrics about DeltaStream objects and queries: * Click on a particular DeltaStream object, and a side panel displays that enables you to print records in the object and view the object’s columns and metadata. * Click on a particular query, and a side panel displays with some basic metrics and metadata about the query. You can also terminate queries from this panel. ### Using Stream 360 From any screen in DeltaStream, click the **Org Name** arrow and select the organization you wish to display. In the lefthand navigation, click the Stream 360 icon ( ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdvHVTuFqRFIvJhEywuwqYt1Qoq4pwgg4ZEpKGsjM5F5XjidwYGQ0pjqR6mJkoICXXfrGq9UX7jU-0jcfAPyasp5iUCBJxal_pXtfE5-2wSKsynCBTqeB1W6paCfAv-yL5TEm9MKA?key=I5KlA42aPCnDTSburBrQ34jD) ) to display the flowchart.
To see the location of the displayed objects, in the top right corner check **View location**. Paths display immediately above each object.
To see whether multiple objects share a topic, in the top right corner check **View Topic Link**. If a process involves multiple topics, checking this box highlights those topics, regardless of where they may display on the canvas, and shows how they are connected. This makes it easier yet to envision more complex workflows.
### Viewing DeltaStream Object Details You can quickly drill down on any particular DeltaStream object in the flow to see details of that object. To start, click the **TOC panel** (![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfURpTS5W7anawQ3edy0IDu0OfVIz-_TdO-p5TDQHj8cDz6qrbg6irrxoCwpcQ5SxrYiW1pxHfDScoJzPKpFJzMEIn1UsNzrMCWExR9SigoT7PlA1q8ENGitZADCGVsW3AW8NPaHQ?key=I5KlA42aPCnDTSburBrQ34jD)) to display your organizational hierarchy. Then click the item you want.
Stream 360 highlights all lines connecting all the objects in this flow. It also opens a separate panel that displays additional details about the object, including queries that make use of that object. In the simple example below, we selected the **users\_log** changelog. Its flow from object to enriched stream is highlighted, and details display in a panel on the right.
And in this example, the final result – the **enriched pageviews** stream – is selected. The lines that represent the flow of all the objects involved in this enriched stream are highlighted, and details of the enriched stream display in the side panel.
Finally, you can enter an object name, in whole or in part, in the **Search** bar above the TOC to get details about a particular object. ### Moving DeltaStream Objects and Reorganizing Flows You can rearrange the way your flows display in multiple ways. To do this, you can click and drag individual objects or use the buttons in the lower-left of the Stream 360 screen. Lines that connect an object with a query or destination move with the object — that is, changing the location of objects on the Stream 360 canvas does not break the flow; you're modifying the visual display only.
| To do this... | ...do this | | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Move an object around on the canvas | Click and drag the object. | | Move multiple objects around as a group | **Ctrl click** (Windows) or **Cmd click** (Mac) the objects you want. Then click and drag the objects as a group. | | Increase the size of the objects | Click **+** | | Decrease the size of the objects | Click **-** | | Center the entire flow | Click ![](/files/1lzRIizRpRVN8SFZYUxk) | | Lock the objects in place | Click ![](/files/4Wij583NxyIDodr4LoeZ) | # Free Trial Quick Start How to get started for free with DeltaStream DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and namespaces for organizing your data. Using DeltaStream’s free 14-day trial? Follow this guide to build an end-to-end streaming application in minutes. We provide you with a default organization – named after the email address you used to sign on – and a default Kafka store with synthetic data. You’ll use these resources to: 1. Inspect the data in the streaming trial store. 2. Create a database. 3. Create a stream and changelog for your Kafka topics. 4. Enrich your data and query it. {% hint style="info" %} **Note** The trial version limits you to 3 queries. Also, user-defined functions aren’t supported, and there are no materialized views. You can add your own external store, but it must be available via the Internet. Contact DeltaStream support if you wish to set up a private store. {% endhint %} ## 1. Inspect Data in Your Trial Store You receive access to a pre-defined DeltaStream `trial_store` when you sign in to your trial account. This store is a discrete AWS MSK (Managed Streaming for Kafka) cluster that includes several topics with synthetic data producers; the producers continuously publish messages into these topics. {% hint style="info" %} **Note** In DeltaStream you define stores to represent each Kafka cluster. DeltaStream [also works with other stores](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/HVBsmkx7ud3Nrlm15M5s), such as AWS Kinesis and Postgres. {% endhint %} The trial store displays in several places: * The **Welcome** page * The **Workspace** page * The **Resources** page When you log on, DeltaStream displays the **Workspace** page. This page provides an at-a-glance dashboard view of your overall DeltaStream organization. To begin exploring your trial store, in the lefthand navigation click **Resources** ( ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXf2UP1jpkEr4Eu0kSVKEQW4OoJt2uBqcKujck-pcFFCFTuvl8QNodvhTLkukTHB8nXTcrvZwHsuT0sZt3DSWRZf_VeIDPgUIirE1-VXq7J92VmTX-9QryH6uW8CMSnUVomFnE2m-VFUkKILdei-GmafdpRo?key=UefObijvgyeIVbiCnQQu0w) ). The **Resources** page displays with the **Data Stores** tab active and your trial store listed beneath it.
To display the topics contained in the trial store, click anywhere in the trial store row and open the **trial\_store** page.
Now confirm the store connectivity and inspect the data in a topic. To do this: 1. Click anywhere in the row of the topic you want. The topic **Details** pane slides open. * You can also display the topic **Details** pane by clicking ![](/files/USyioprlkiAtNigUK4TB) under the **Actions** column. 2. Click **Print**. This displays the live stream of data flowing to the topic. Here is an image of the data flowing into the pageviews topic.
{% hint style="success" %} **Tip** After you verify that data is streaming into your trial store, you may wish to click **Stop** to halt the stream. {% endhint %} There's a range of additional information you can view. For more details, please see [Explore Data Store and Topic Details](/how-do-i.../create-and-manage-data-stores/explore-data-store-and-topic-details). ## 2. Create a Database Now it’s time to declare a database and [DeltaStream objects](/how-do-i.../relation#defining-deltastream-objects) and write queries on the streaming data. Databases present a logical organization layer for your streaming data. They make it possible to provide access controls and governance across all your data. {% hint style="info" %} **Note** DeltaStream objects are the building blocks of user applications and pipelines. qYou must create an object to represent each Kafka topic you wish to include in a query. {% endhint %} **To create a new database** 1. From the lefthand navigation, click **Databases** ( ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXe8QkPzbYgBPSlrrbNduTMdyseCBstzcci4U40Ji_JvgU03ng2-KeA7jfiILEvc7U36UGwmwbOrnt_M7H5WPNN6OAxxDr4ZmGJI81gIOvp3lhdT-k6XJ48Y3nvLUUVWjhVanwRoIrGNN9sxHbWRaJaEsnkZ?key=UefObijvgyeIVbiCnQQu0w) ) and then click **+ Add Database**. 2. Enter the database name. In this guide we name it `DemoDB`. 3. Click **SAVE**. The newly-created database displays all the topics in the Kafka cluster to which you have access. For this guide, we named our database `DemoDB`. You can create as many databases as you wish. Each new database includes a namespace named `public`, but you can add more namespaces if you wish. **To add a new namespace** 1. Click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeSSM89YLa0JSnL6Y9pxMGXCBEQ3SqCRkPnUMa0GAC7PFtvofxwIyF7ZOEOfFg5tUBQrLZgPFfJjG948Er0zPPakLiB8iQ8EoNTIS-6UPy_EfOc8z3NAU77qRojAz0mhdxizrCDYaTaMcNIQQc8ZBKvL8B_?key=UefObijvgyeIVbiCnQQu0w), click the database you want, and towards the right click **+ Add Namespace**. 2. At the promt, enter the namespace name. Then click **SAVE**. ## 3. Create Streams and Changelogs Your goal here is to understand pageviews by users over time. You do this by joining the pageviews and users topics. Start by creating relations backed by Kafka topics. Use DeltaStream’s DDL statements to define your streaming data in a topic as an append-only stream. {% hint style="info" %} **Note** In DeltaStream, a stream is simply one type of object. {% endhint %} **To create a stream** 1. Navigate to the main workspace by clicking ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdu9lDc_IuaZ3yQHkSx7RlEL-5lgjB57u_bsfyJyIu2qW-g5dXj_9gjhMI4QFwsWlRfQaeCRyBm-IobyzrHL4A4n7OT6p3U_k9Ck8OTFmrTi4N5kI-qWNBmIPXT7yX8NJ5BLcRtKjKJ2S6CT6_uVUqbz5_B?key=UefObijvgyeIVbiCnQQu0w). 2. Copy the SQL DDL statement below, and paste it into the SQL pane (above the **Results** pane).\ This creates a discrete stream backed by the pageviews topic from the Kafka cluster; each pageview is an independent event. This stream reflects the view time of each page by user. 3. Click **Run**. ```sql CREATE STREAM pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR )WITH ( 'topic'='pageviews', 'value.format'='JSON' ); ``` DeltaStream displays a **Success** message in the Results pane, followed by details of the stream you just created. {% hint style="success" %} **Tip** You may need to expand the Results pane to see all of the details. To do this, click and drag the pane handle ( ![](/files/Kds3fMjgxn5nGXQppxqe) ). See below for more tips on [modifying your DeltaStream workspace](#modifying-your-workspace). {% endhint %}
{% hint style="info" %} **Note** The above stream is created in the currently-used database and namespace – `DemoDB` and `public`, respectively. DeltaStream uses the default store declared above as the store for the pageviews topic. To specify another store, use the WITH clause. {% endhint %} Next, declare a changelog backed by the users topic and ordered by UserID. A changelog enables you to interpret events in a topic as UPSERT events. (In DeltaStream, changelogs are simply another type of object.) Events require a primary key; DeltaStream interprets each event as an insert or update for the given primary key. In this case, the changelog relation reflects specific details by user, such as gender and interests. To declare the users changelog, paste the following statement in the SQL pane and then click **Run**: ```sql CREATE CHANGELOG users_log ( registertime BIGINT, userid VARCHAR, regionid VARCHAR, gender VARCHAR, interests ARRAY, contactinfo STRUCT, PRIMARY KEY(userid) )WITH ( 'topic'='users', 'key.format'='json', 'key.type'='STRUCT', 'value.format'='json' ); ``` As with the pageviews stream, the users changelog displays in the `DemoDB` public schema. To view the streams, in the lefthand navigation click **Databases** ( ![](/files/VRjWh8bZPBOtjSIGsW0k) ), and in the **Databases** pane click to expand the `DemoDB` database and public namespace.
## 4. Run Queries Now you can write a continuous query in SQL to process this streaming data in real time. Let’s start with an **interactive query**, in which the query results stream back to you. You can use such queries to: * inspect your streams and changelogs * build queries iteratively by inspecting the query result. Let’s inspect the pageviews stream. To do this, enter the following interactive query and then click **Run**: ```sql SELECT * FROM pageviews; ``` DeltaStream compiles your query into a streaming job, runs the job, and streams the result into the **Results** pane, as per the below image:
{% hint style="success" %} **Tip** After you verify that data is streaming in you may wish to click **Stop Query**. {% endhint %} While interactive query results stream in, DeltaStream provides persistent queries. These are continuous queries wherein the query results are written continuously either to a store or a materialized view. Let’s write a persistent query that joins the pageviews stream with the `users_log` changelog relations. This creates a third object called an enriched pageviews stream that provides user details for each pageview event, including view time of each page by user and detailed user information. While we’re at it, we also convert the epoch time to the timestamp with a timezone using the `TO_TIMESTAMP_LTZ` function. Start by creating a stream called **enriched\_pv**. Then join the **pageviews** stream with data from the **users\_log** changelog and write the results to the **enriched\_pv** stream. ```sql CREATE STREAM enriched_pv AS SELECT TO_TIMESTAMP_LTZ(viewtime, 3) AS viewtime, p.userid AS userid, pageid, TO_TIMESTAMP_LTZ(registertime, 3) AS registertime, regionid, gender, interests, contactinfo FROM pageviews p WITH ( 'starting.position'='latest') JOIN users_log u WITH ( 'starting.position'='latest') ON u.userid = p.userid; ``` {% hint style="info" %} **Note** The above **persistent query** creates a new topic in the trial store DB. When you create a new topic, DeltaStream adds a prefix name to the topic name based on your trial email and some unique random characters. For example, for the email , DeltaStream creates a topic prefix like `t_testgmailcom_4evmsyg_`. Creating the topic `enriched_pv` in turn creates the topic `t_testgmailcom_4evmsyg_enriched_pv`. You can view these topics in the trial store topics list. {% endhint %} {% hint style="warning" %} **Important** Topic name prefixes are a requirement only for the trial store we have set. Prefixes are not added if you use any other store, such as your own Apache Kafka or AWS Kinesis. {% endhint %}
{% hint style="info" %} **Note** DeltaStream compiles and launches the query as an Apache Flink streaming job. You can view the query along with its status in the **Query Management** page; to do this, in the lefthand navigation click **Queries** ( ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXe7LqKj37iaUsNTigPBcPp1WaVUWRQwI_DmOoYo97zHkQcmIF3ZmWX3JDBkHtny35cYm-f9Xr_MPCDwQnEyo0kbpa9sl5SlXa2qh0gMV1zTl20E906dQ-NQTYG3p1D5R6yb98SVp-R9ZjmKgT0mZnhEbWM?key=UefObijvgyeIVbiCnQQu0w) ). {% endhint %} When the query finishes, you have a new Kafka topic named `enriched_pv` in your Kafka cluster and a new stream added to the streams in your TestDB database. Finally, examine the contents of the new stream. Run the following simple query in the SQL pane: ```sql SELECT * FROM enriched_pv; ``` The result of this interactive continuous query is an enriched pageviews stream that streams to the client as shown below:
That’s it. In just a few steps you’ve used DeltaStream to connect two different Kafka topics and persist the enriched data, either to query in real time or write out to its final destination. In so doing you avoid the extra steps and expense that might be the case in a data warehouse. ## 4. Clean Up When your task is completed, it’s time to clean up your environment. To do this: 1. Terminate the queries. To do this, click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdY0NpT5uh6gOM_G8IOm46JSV_oYILBhP5AVbwXng7f9knY81JgB_2Ok1aPpMXYqgqIfvvc7pSAi32Ye_jpRtHs5lSOmH5U5GIc8-U0OdaP7gEZr7miLSKbYit7CPaJhz0J3g4yyJ9yTj7-WVIoGlO-sDbK?key=UefObijvgyeIVbiCnQQu0w) to display the **Queries** page. Then, next to the query you wish to terminate, click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdHMsC5oLWP8_qP49KcrLjXrTYXcWDKM6Oc9RIgMM5f7pjXEyYkQ3WEORJaE-50Aqf8skN6_QnH98GWPl_Xq02jBryASEZikU8-qBqsgERXe70yhSv6TnSTObgJkqwOS7Au2rqr2qkLoqsybqxadPWo0ty2?key=UefObijvgyeIVbiCnQQu0w). 2. Follow the instructions in the prompt, and then click **Terminate**. The system displays a message indicating you've marked that query for termination.
3. Drop the created streams, changelogs, and materialized views. To do this, click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdJ70vylMMRRB_i1ZRA_XWlZMKsDyV2tW0d7VOMkQECcE_RJTD5_-aZ8DbEsE1b52neN9z2v5n_vF67tW1f8UvcOeHznhzXN4H5fvbPND5LRDhZ-I-2Xsw5-bLucoBqdsKT_4VcRsY9PLLy6vvUj_4jGjRz?key=UefObijvgyeIVbiCnQQu0w) to navigate to the corresponding database and schema, and as with the query, click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdkqXP6cmnBAqoeXaHOAQeoSuXVuATDP1rGnFP9DgFv5NPJ6xDxzC0EaslnJIQAQVMDxcE635zc_lNGg5CcM-uhpBbpRJ0h3doJUMLpGsGLZiBH0oWNKEoTBMp35bQkgvNfuTykPCiG9ClQQ693VFjMsiz1?key=UefObijvgyeIVbiCnQQu0w) and follow the prompt to drop the streams, changelogs, and materialized views.
{% hint style="warning" %} **Important** If you have a query that uses a stream, changelog, or materialized view, you must terminate the query before dropping the relation. {% endhint %} ## Modifying Your Workspace This quickstart guide used simple examples to get you up and running quickly. But the DeltaStream workspace is customizable. If you begin using more extensive queries or a greater number of objects, you may find it more efficient to modify the size of your workspace panes, or even toggle on or off specific sections, to focus on the parts of the workspace that matter most at any given time. You can: * Hide the **Results** pane. This gives you more room in the SQL pane to work with more extensive SQL. To do this, at the top of your workspace click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfPBgA2_hmGp69AJkKx2pw4JKYSNM6dUEr6HgWs52rxWX7kWUaUsi7nhgqFnS2IiPuoQO1k4G6OKF4GCpbssRqvgDhPI-NTfEmA_p6dDbXCnZEXJ2VVZhwrumXIDQLlxt1K6gESF_tdx9XZgTA3L-qr25_b?key=UefObijvgyeIVbiCnQQu0w). Click it a second time to re-display the **Results** pane. * Hide the **SQL** pane. This gives you more room to examine query results. To do this, at the top of your workspace click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXf4eNReXlAzN9o41XvRmlPNSZkloouwSE51WDbYJqR_y1dWjDe5icShJaMOcVkLfEMMV1Im-RF8HyN9YJkiOwgzINfqTjmES23HPBNC_dbTYh0qsSsW-GwFYbVhaI-jDieZ-Vl2ubzsZGRVjpsQzIyDda8w?key=UefObijvgyeIVbiCnQQu0w). Click it a second time to re-display the **SQL** pane. * Hide the lefthand (**Database** and **Stores**) panes. This gives you more horizontal screen real estate and creates a cleaner, more expansive workspace. To do this, at the top of your workspace click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfy3XC9xCjCMffm-r8UK0XAd83ePllWY5yWuFnh5N-brhSa_cFXPpSmW8yJwB8Pyc5W6EcW5zBjU5QsBr_7REiPNjaTvuwne_NJHwpn38AJaEVizRLO84ioySAfYykdYmGGKjcZmKdz5ytzMyJEzyuZ541C?key=UefObijvgyeIVbiCnQQu0w). Click it a second time to re-display the lefthand pane. When activated, the icons display in color – for example, ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdBN0dbydR2wD7lhmaN1Z4OWrxR_BpkluIWZW7VN_0NKvVMe721etzaosjf5k_-aTuTcvBz0SNEqnHcesd8Kdb8JxOLKFA1qNOUbw7oKH_pKcqN5G3GT1YwygBfjPz6PoSM-UT8EUaKRmuf63DLJjrHpBA?key=UefObijvgyeIVbiCnQQu0w). {% hint style="success" %} **Tip** You can click any two or all three of these icons at once to isolate the precise workspace you wish. For example, if you click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfxBw_PuxHge41OBsyT5pbig17DGJqhM8IVQj-cdoPr1VWzmmB3tajzwwY8h2wh20pdeI-hEt6w78IUVCo2azWBnIcexqSAm7k8xBtEp5etbk20f5wqKiCevSoVjqM0tkyl7Jp4wvy9wvXOEdO8oH7fP165?key=UefObijvgyeIVbiCnQQu0w) and ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfqGhlAMGE8z8SKezb1hRnY4UVkiWpJRkYDaBwPVIZk6PybwWm9H8Kj5jPILjlPovxmibdETxFmWLV3uIVBmxD7LlICxs9ofgCw3TPpiWEexTik69nom_4qVlzuYAT3nO3VtqRA8hOF3lMpifkFH4kBSOfo?key=UefObijvgyeIVbiCnQQu0w) you have almost the entire screen to work with SQL. {% endhint %} Finally, you can manually re-size your panes without hiding them altogether. To resize the SQL and Results panes, or the **Database** and **Stores** panes, click and drag ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXcFcIfcwLxVRDJf_thq4YryVSX3Jonlc34X_N_ZN8m2o-VM-ANHkXTVEk6KLdarNcA-FiT0kWgTAa46FW69_sTLdKa2QwD3svLy2_rg32yPJhamQB-a9uvpTNOycZiC_6Q66cjJU62d23a_6VK0wxa4j8PQ?key=UefObijvgyeIVbiCnQQu0w). To resize the left and right panes, click and drag ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXd1v835g-KnSiHvXj-asnawI5B37NhtxA1-LAjGmq6BE7I77TfcNybtTMJwug3mlo32b3WiSXmwDigrEMZ22OcCBUQFRS4hb3w7kCIv6YIxdfalbuVpFyGMSrsq2j4aKZ-lG_bkjs0D4u2d8hYwBfl9V2Ty?key=UefObijvgyeIVbiCnQQu0w). # Starting with the Web App How to get started with streaming analytics using the DeltaStream user interface This guide steps you through the process of building an end-to-end streaming application with DeltaStream’s Web application. By the end of this guide, you will have hands-on experience with foundational concepts in DeltaStream and be able to build your own similar applications. Here's what you'll do: 1. Connect to your streaming store (in this case, Apache Kafka) by creating a store in DeltaStream. 2. Create your first database. 3. Create streams and changelogs for your Kafka topics. 4. Create new streams, changelogs, and materialized views using DeltaStream’s continuous queries. 5. (Optional) Share your streaming data with other members of your organization. Before you begin: * Create a DeltaStream account and sign in to the Web app. * Created a DeltaStream organization or join an existing organization. {% hint style="info" %} **Note** The examples in this guide use topics in Apache Kafka. But the steps should be the same if you have your data in other streaming stores such as Amazon Kinesis or Redpanda. {% endhint %} ## 1. Create a Data Store In DeltaStream, a data store is a streaming service, such as Apache Kafka or Amazon Kinesis, where your streaming data resides. Before you write any queries, you must configure DeltaStream to connect to the data store where data resides. This is the first step to take before you process any data using DeltaStream SQL statements. **To create and configure a new data store:** 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display the **Resources** page. 2. Click **+ Add Data Store**, and from the menu that displays click the store type you want (in this case, Kafka). {% hint style="info" %} **Notes** Be sure to match the data store type with the streaming data store where streaming data resides. Also, in addition to self-hosted services, for Kafka data store types you can also configure [Confluent Cloud](https://www.confluent.io/confluent-cloud/) and [Amazon MSK](https://aws.amazon.com/msk/) in DeltaStream. {% endhint %}

Adding a store

3. Choose a unique **Name** for the data store. Use only alphanumeric characters, dashes, and underscores, and limit the name to a maximum of 255 characters.\ To follow along here, name the store mskconsumer. This guide uses an Amazon MSK cluster. {% hint style="success" %} **Tip** In practice, it's best to select the same region as the one where your data is stored. That helps minimize data transfer costs. {% endhint %} 4. Add at least one **URI** port to which to connect, then click **+**.\ Separate multiple entries with “,” 5. Select a **Schema Registry**. 6. Optionally, complete the authentication options as appropriate for the store. For details, see [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store#auth-options).
7. Click **Add**. Your new data store displays in the list of existing stores on the **Resources** page
To ensure your new data store is set up correctly, click on it to expand it and display the **Topics** section. From here you can view the list of data store entities.
In the list of topics, find the topic you want and click on it. Then click **Print** to see the messages coming in to this topic in real time, as in the example below.
## 2. Create a Database DeltaStream provides a relational model on top of your streaming data. Similar to other relational systems, DeltaStream uses databases and namespaces for organizing your data. **To create a new Database** 1. In the lefthand navigation click **Databases** ( ![](/files/VRjWh8bZPBOtjSIGsW0k) ). The **Databases** pane displays. 2. At the top of the **Database** pane click **+** and then click **Database**.
3. At the prompt enter a unique name for the database and then click **Add**. In this guide the database is labeled `DemoDB`. {% hint style="info" %} **Note** You can create as many databases as you wish. Any database you create includes a namespace labeled `public`. But you can add more namespaces if you wish. {% endhint %} ## 3. Create Streams and Changelogs Now use DeltaStream’s DDL statements to create relations on top of your Kafka topics . To work with your streaming data in an entity as an append-only stream, in which each event is an independent event in your stream, you define it as a ***Stream***. In this guide you declare a stream on the `pageviews` topic, since each pageview is an independent event. To define a stream, in the lefthand navigation click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ), and in the SQL pane write the DDL statement. Here is the statement to create a pageviews stream; you can copy it and paste it into your work: ```sql CREATE STREAM pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR )WITH ( 'topic'='pageviews', 'value.format'='JSON' ); ``` This stream is created in the currently-used database and namespace -- in this case, `DemoDB` and `public`, respectively. Also, there is no store specified in the `WITH` clause; instead, DeltaStream uses the default store you declared above as the store that contains the `pageviews` topic. Next, declare a changelog for the `users` topic. A changelog indicates you want to interpret events in an entity as UPSERT events. In this case the events must have a primary key; each event is interpreted as an insert or update for the given primary key. Use the following statement in the **Workspace** SQL pane to declare the `users` changelog: ```sql CREATE CHANGELOG users_log( registertime BIGINT, userid VARCHAR, regionid VARCHAR, gender VARCHAR, interests ARRAY, contactinfo STRUCT, PRIMARY KEY(userid) )WITH ( 'topic'='users', 'key.format'='json', 'key.type'='STRUCT', 'value.format'='json' ); ``` When you declare the `pageviews` stream and `users` changelog, the events display in the public namespace of the `DemoDB` database. To view them, click **Databases** ( ![](/files/VRjWh8bZPBOtjSIGsW0k) ) in the lefthand navigation, then go to **DemoDB** > **Public.**
## 4. Run Queries After you declare streams and changelogs, you can write continuous queries in SQL to process this streaming data in real time. You can run 2 types of queries: 1. Interactive queries 2. Streaming or continuous queries In interactive queries, the query results stream back to the you. Use these types of queries to inspect your streams and changelogs or build queries iteratively by inspecting the query result. In a [Query](/overview/core-concepts/queries#_streaming_or_continuous_query), the query results are stored back in a store or materialized view instead of streaming back to you. Let's start with an interactive query. Here's an example: inspect the `pageviews` stream using the following interactive query: ```sql SELECT * FROM pageviews; ``` When you run this query, DeltaStream compiles it into a streaming job, then runs the query and streams the result into the Web app. The results resemble this:
Now try a persistent query that joins the `pageviews` stream with the `users` changelog to create an enriched pageviews stream that includes user details for each pageview event. While you're at it, use the `TO_TIMESTAMP_LTZ` function to convert the epoch time to a timestamp with a time zone: ```sql CREATE STREAM csas_enriched_pv AS SELECT TO_TIMESTAMP_LTZ(viewtime, 3) AS viewtime, p.userid AS userid, pageid, TO_TIMESTAMP_LTZ(registertime, 3) AS registertime, regionid, gender, interests, contactinfo FROM pageviews p JOIN users_log u ON u.userid = p.userid; ``` Click **RUN**. In the background DeltaStream compiles and launches your query as an Apache Flink streaming job, and displays a confirmation similar to the below when the query completes.
To view the query, along with its status, in the lefthand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ). After the query successfully runs there's a new Kafka topic named `csas_enriched_pv` in your Kafka cluster, and a new stream added to the streams in your database, `TestDB`. To examine the content of the new stream, run the following interactive continuous query from the **SQL** page in the Web app: ```sql SELECT * FROM csas_enriched_pv; ``` The following image shows the result of running the above query. The result streams to the client as shown below:
With the pageviews stream enriched, you can build a materialized view to compute the number of pageviews per user. To do this, enter the following statement in the **SQL** pane of your workspace to stream this materialized view: ```sql CREATE MATERIALIZED VIEW user_view_count AS SELECT userid, COUNT(*) AS view_count FROM csas_enriched_pv GROUP BY userid; ``` {% hint style="info" %} **Note** Materialized views are not available to free trial users. {% endhint %} When you run this query, DeltaStream launches a streaming job that runs the `SELECT` statement and materializes the query result. You can query this materialized view the same way you would query a materialized view in a traditional relational database. But the difference here is that DeltaStream leverages the streaming job to always keep the data in the materialized view fresh. The following is a simple query to get the current view count for a user with the userid of `User_2`. ```sql SELECT * FROM user_view_count WHERE userid = 'User_2'; ``` The result of this query displays in one row, as shown below:
Note that at the time of running the above query the number of pageviews for `User_2` is `3`. Now run the query again. This time you should see an updated result for the pageview count for the user. This demonstrates that every time you run a query on a materialized view, you receive the most up-to-date result. DeltaStream ensures the data in the view is continuously updated, using the continuous query that declared the materialized view. Here is an image of the same query run on the materialized view just a few seconds later:
The result is updated again -- in this case, to `11` from the previous value of 3. ## 5. Clean Up It's important to practice good hygiene when you're done! To clean up your DeltaStream environment: 1. In the lefthand navigation, click Queries ( ![](/files/HOEvY09XthGMf2h6wEx6) ) to display the **Queries** page. 2. Click the query or queries you wish to terminate. 3. When the Query Details page displays, towards the top right click **Terminate**. Follow the prompt in the window that displays, and then click **Terminate**. 4. Navigate to the corresponding database and schema and drop the created streams, changelogs, and materialized views. {% hint style="success" %} **Tip** If there's is a query that uses an object (stream, changelog, or materialized view), be sure to terminate the query before you drop the object. {% endhint %} # Starting with the CLI How to get started using the DeltaStream command line interface DeltaStream provides a Command Line Interface (CLI) you can use to interact with the platform from a terminal. This guide walks you through the steps to build a streaming application with DeltaStream’s CLI. Follow these steps to: * Get hands-on experience with foundational concepts in DeltaStream. * Gain the knowledge to build applications similar to the one in this guide. While this guide uses topics in Apache Kafka, the steps should be the same regardless of whether you have data in other streaming data stores such as Amazon Kinesis or Redpanda. {% hint style="info" %} **Note** If you prefer to use DeltaStream’s Web application, see [Starting with the Web App](/getting-started/starting-with-web-app) for those details and procedures. {% endhint %} We assume you already have created your account and signed into DeltaStream. Also, we assume you already have created a DeltaStream organization or have joined an existing organization. You accomplish the following steps in this guide for the CLI: * Download the DeltaStream CLI. * Connect to your streaming data store (in this case, Apache Kafka) by creating a data store in DeltaStream. * Create your first database. * Create streams and changelogs for your Kafka topics. * Create new streams, changelogs, and materialized views using DeltaStream’s continuous queries. * (Optional) Share your streaming data with other users in your organization. ## Download the DeltaStream CLI 1. Log into DeltaStream. 2. At the bottom of the lefthand navigation, click **Support Center** ( ![](/files/DFU410tHebNZsod77mzs) ). The **Support** menu displays.

Where to download the DeltaStream CLI

3. Click the OS you want. The DeltaStream CLI begins downloading automatically into a `dscli` folder. 4. Unzip and deploy the files as you would with any compressed application. ## Launch the CLI Get the `API endpoint` when you launch the CLI from the help section of the DeltaStream UI. ```bash ~ % /path/to/dscli --server [API endpoint] ```
{% hint style="warning" %} **Note** When you use the CLI for the first time, DeltaStream stores the `--server` flag in your user configuration, and you will not need it for future requests. {% endhint %} ## Create a Data Store The first step is to create a data store in DeltaStream. A data store is a streaming service such as Apache Kafka or Amazon Kinesis where your streaming data resides. {% hint style="warning" %} **Note** The CLI refers to data stores simply as "stores." {% endhint %} Use the [CREATE STORE](/reference/sql-syntax/ddl/create-store) statement to create a data store in the DeltaStream CLI: ```sh /# CREATE STORE MSK WITH ( 'type' = KAFKA , 'kafka.sasl.hash_function' = SHA512, 'kafka.sasl.password' = '**********', 'kafka.sasl.username' = 'mskconsumer', 'uris'='b-1-public.cmnuseast1datagenmsk.89yho3.c26.kafka.us-east-1.amazonaws.com:9196' ); /msk# SHOW STORES; Name | Kind | Metadata | Owner | Created at | Updated at -------+-------+---------------+----------+----------+----------------------+----------------------- msk | Kafka | {} | sysadmin | 2023-02-20T11:03:18Z | 2023-02-20T11:03:18Z /msk# ``` Your new data store displays as the default data store in the prompt. Use the [LIST STORES](/reference/sql-syntax/command/list-stores) command to view the available data stores you have created in DeltaStream. {% hint style="info" %} **Note** No default data store indicates in the prompt until you create your first data store. {% endhint %} As is indicated below, in declaring the data store, DeltaStream provides the required configurations to connect and use the streaming data store from DeltaStream. You can now inspect the data store and print to view the content of its topics: ```sh /msk# SHOW ENTITIES; Entity name ----------------------- ds_pageviews ds_pageviews_pb ds_shipments ds_users /msk# PRINT ENTITY ds_pageviews; {"userid":"User_9"} | {"viewtime":1676891565084,"userid":"User_9","pageid":"Page_28"} {"userid":"User_7"} | {"viewtime":1676891565264,"userid":"User_7","pageid":"Page_16"} {"userid":"User_7"} | {"viewtime":1676891565404,"userid":"User_7","pageid":"Page_84"} {"userid":"User_4"} | {"viewtime":1676891565524,"userid":"User_4","pageid":"Page_17"} {"userid":"User_3"} | {"viewtime":1676891565664,"userid":"User_3","pageid":"Page_47"} {"userid":"User_5"} | {"viewtime":1676891565784,"userid":"User_5","pageid":"Page_72"} ``` ## Create a Database In DeltaStream, you use databases to organize your streaming data in an intuitive namespace. Use the [CREATE DATABASE](/reference/sql-syntax/ddl/create-database) statement to create a database. When you create a database, DeltaStream in turn creates a default namespace, named `public`, in the database. The following shows the statement that creates a new database labelled `TestDB`: ```sh /msk# CREATE DATABASE TestDB; testdb.public/msk# SHOW DATABASES; Name | Default | Owner | Created at | Updated at ---------+---------+----------+----------------------+----------------------- testdb | ✓ | sysadmin | 2023-02-20T11:18:46Z | 2023-02-20T11:18:46Z testdb.public/msk# SHOW NAMESPACES; Name | Default | Owner | Created at | Updated at ---------+---------+----------+----------------------+----------------------- public | ✓ | sysadmin | 2023-02-20T11:18:46Z | 2023-02-20T11:18:46Z ``` The prompt displays the current database and namespace. ## Create Streams and Changelogs Now create DeltaStream objects on top of your Kafka topics. You do this using DeltaStream’s DDL statements. {% hint style="info" %} **Note** In the CLI, you can use the `SHOW` command and the `LIST` command interchangeably. {% endhint %} To manage streaming data in an entity as an append-only stream, in which each streaming event is independent, define it as a [stream](/overview/core-concepts/databases#_stream). In the example below, you declare a stream on the `ds_pageviews` topic, as each pageview event is an independent event: ```sh testdb.public/msk# CREATE STREAM pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ( 'topic'='ds_pageviews', 'value.format'='JSON' ); testdb.public/msk# SHOW RELATIONS; Name | Type | Owner | Created at | Updated at ------------+--------+----------+----------------------+----------------------- pageviews | Stream | sysadmin | 2023-02-20T11:21:58Z | 2023-02-20T11:21:58Z ``` {% hint style="info" %} **Note** The DeltaStream UI uses the term `OBJECTS` instead of `RELATIONS`. {% endhint %} Next, declare a changelog for the `ds_users` topic. A [changelog](/overview/core-concepts/databases#_changelog) indicates that you wish to interpret events in an entity as UPSERT events. The events should have a primary key, and each event is interpreted as an insert or update for the given primary key. Use the [CREATE CHANGELOG](/reference/sql-syntax/ddl/create-changelog) command to declare the `users` changelog: ```sh testdb.public/msk# CREATE CHANGELOG users( registertime BIGINT, userid VARCHAR, regionid VARCHAR, gender VARCHAR, interests ARRAY, contactinfo STRUCT, PRIMARY KEY(userid) ) WITH ( 'topic'='ds_users', 'key.format'='json', 'key.type'='STRUCT', 'value.format'='json' ); testdb.public/msk# SHOW RELATIONS; Name | Type | Owner | Created at | Updated at ------------+-----------+----------+----------------------+----------------------- pageviews | Stream | sysadmin | 2023-02-20T11:21:58Z | 2023-02-20T11:21:58Z users | Changelog | sysadmin | 2023-02-20T11:29:58Z | 2023-02-20T11:29:58Z ``` ## Run Queries Now that you have declared streams and changelogs, you can write continuous queries in SQL to process this streaming data in real time. Start with a [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query), wherein the query results stream back to you. You can use such queries to inspect your streams and changelogs, or to build queries iteratively by inspecting the query’s result. Here's an example: Use the following interactive query to inspect the `pageviews` stream: ```sh testdb.public/msk# SELECT * FROM pageviews; ^C to exit Waiting for sandbox to be in running state 'defined'................ Waiting for interactive query('defined') to be in running state... Interactive query is running | {"viewtime":1676893147888,"userid":"User_8","pageid":"Page_39"} | {"viewtime":1676893148008,"userid":"User_4","pageid":"Page_72"} | {"viewtime":1676893148148,"userid":"User_7","pageid":"Page_41"} | {"viewtime":1676893148268,"userid":"User_5","pageid":"Page_98"} | {"viewtime":1676893148243,"userid":"User_7","pageid":"Page_12"} | {"viewtime":1676893148369,"userid":"User_1","pageid":"Page_10"} | {"viewtime":1676893148423,"userid":"User_9","pageid":"Page_12"} | {"viewtime":1676893148488,"userid":"User_8","pageid":"Page_75"} ``` While interactive query results display, DeltaStream provides a [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query), wherein the query results are saved back in a store or a [materialized view](/overview/core-concepts/databases#_materialized_view). To proceed, write a persistent query that joins the `pageviews` stream with the `changelog` to create an enriched pageviews stream that includes user details for each pageview event. While you're at it, also convert the epoch time to a timestamp with a time zone via the `TO_TIMESTAMP_LTZ` function. ```sql CREATE STREAM csas_enriched_pv AS SELECT TO_TIMESTAMP_LTZ(viewtime, 3) AS viewtime, p.userid AS userid, pageid, TO_TIMESTAMP_LTZ(registertime, 3) AS registertime, regionid, gender, interests, contactinfo FROM pageviews p JOIN "users" u ON u.userid = p.userid; ``` During this time, DeltaStream compiles and launches the query as an Apache Flink streaming job. You should be able to use the [LIST QUERIES](/reference/sql-syntax/command/list-queries) command to view the query along with its status: ```sh testdb.public/msk# testdb.public/msk# SHOW QUERIES; ID | State | DSQL | Owner | Created at | Updated at ---------------------------------------+---------+--------------------------------------------------------------+----------+----------------------+----------------------- a913595a-ad09-452e-81e1-3f440b56fae2 | RUNNING | CREATE STREAM | sysadmin | 2023-02-20T11:42:53Z | 2023-02-20T11:42:53Z | | csas_enriched_pv AS SELECT | | | | | TO_TIMESTAMP_LTZ(viewtime, | | | | | 3) AS viewtime, p.userid | | | | | AS userid, pageid, | | | | | TO_TIMESTAMP_LTZ(registertime, | | | | | 3) AS registertime, regionid, | | | | | gender, interests, contactinfo | | | | | FROM pageviews p JOIN users u | | | | | ON u.userid = p.userid; | | | testdb.public/msk# ``` When the query runs successfully, you have a new Kafka topic named `csas_enriched_pv` in your Kafka cluster, plus a new stream added to the streams in your `TestDB` database. To examine the contents of the new stream, run the following query: ```sh testdb.public/msk# testdb.public/msk# SELECT * FROM csas_enriched_pv; ^C to exit Interactive query is running | {"viewtime":"2023-02-20T11:47:05.717Z","userid":"User_6","pageid":"Page_99","registertime":"2023-02-20T11:47:05.676Z","regionid":"Region_6","gender":"FEMALE","interests":["News","Movies"],"contactinfo":{"phone":"9492229999","city":"Irvine","state":"CA","zipcode":"92617"}} | {"viewtime":"2023-02-20T11:47:05.856Z","userid":"User_5","pageid":"Page_77","registertime":"2023-02-20T11:46:44.951Z","regionid":"Region_4","gender":"OTHER","interests":["Game","Sport"],"contactinfo":{"phone":"6503889999","city":"Palo Alto","state":"CA","zipcode":"94301"}} | {"viewtime":"2023-02-20T11:47:06.056Z","userid":"User_9","pageid":"Page_24","registertime":"2023-02-20T11:46:47.991Z","regionid":"Region_9","gender":"OTHER","interests":["News","Travel"],"contactinfo":{"phone":"6503349999","city":"San Mateo","state":"CA","zipcode":"94403"}} | {"viewtime":"2023-02-20T11:47:06.196Z","userid":"User_4","pageid":"Page_45","registertime":"2023-02-20T11:46:36.748Z","regionid":"Region_5","gender":"FEMALE","interests":["Game","Sport"],"contactinfo":{"phone":"4083366881","city":"San Jose","state":"CA","zipcode":"95112"}} testdb.public/msk# ``` Now that you have the enriched pageviews stream, you can build a materialized view in which you compute the number of pageviews per user. To create this materialized view, type the following statement: ```sql CREATE MATERIALIZED VIEW user_view_count AS SELECT userid, COUNT(*) AS view_count FROM csas_enriched_pv GROUP BY userid; ``` When you run the above query, DeltaStream launches a streaming job that runs the [SELECT](/reference/sql-syntax/query/select) statement and materializes the result of the query. To view this query, use the [LIST QUERIES](/reference/sql-syntax/command/list-queries) command: ```sh testdb.public/msk# testdb.public/msk# SHOW QUERIES; ID | State | DSQL | Owner | Created at | Updated at ---------------------------------------+---------+--------------------------------------------------------------+----------+----------------------+----------------------- cf6d2092-ff25-461f-801d-5fbb0d5ceb58 | RUNNING | CREATE MATERIALIZED VIEW | sysadmin | 2023-02-20T12:56:53Z | 2023-02-20T12:56:53Z | | user_view_count AS SELECT | | | | | userid, COUNT(*) AS view_count | | | | | FROM csas_enriched_pv GROUP BY | | | | | userid; | | | a913595a-ad09-452e-81e1-3f440b56fae2 | RUNNING | CREATE STREAM | sysadmin | 2023-02-20T11:42:53Z | 2023-02-20T11:42:53Z | | csas_enriched_pv AS SELECT | | | | | TO_TIMESTAMP_LTZ(viewtime, | | | | | 3) AS viewtime, p.userid | | | | | AS userid, pageid, | | | | | TO_TIMESTAMP_LTZ(registertime, | | | | | 3) AS registertime, regionid, | | | | | gender, interests, contactinfo | | | | | FROM pageviews p JOIN users u | | | | | ON u.userid = p.userid; | | | testdb.public/msk# ``` You can query the resulting materialized view the same way you'd query a materialized view in a traditional relational database -- except that in DeltaStream, the streaming job always keeps the data in the materialized view fresh. Below is a simple query to get the current view count for a user with the userid of `User_2`: ```sh testdb.public/msk# testdb.public/msk# SELECT * FROM user_view_count WHERE userid = 'User_2'; userid | view_count ---------+------------- User_2 | 580 testdb.public/msk# he ``` As you see, the number of pageviews for `User_2` is `580` at the time you run the above query. Run the query again, and you see an updated result for the pageview count for the user. This demonstrates that every time you run a query on a materialized view, you receive the most up-to-date result. DeltaStream ensures the data in the view is continuously updated using the continuous query that declared the materialized view. Now wait a few more seconds. Then run the same query on the materialized view. You should see something similar to the below: ```sh testdb.public/msk# testdb.public/msk# SELECT * FROM user_view_count WHERE userid = 'User_2'; userid | view_count ---------+------------- User_2 | 580 testdb.public/msk# testdb.public/msk# testdb.public/msk# SELECT * FROM user_view_count WHERE userid = 'User_2'; userid | view_count ---------+------------- User_2 | 818 testdb.public/msk# ``` As you see, the result is updated to `818` from the previous value of `580`. In summary, this guide has demonstrated how DeltaStream makes it easy to build a stream processing applications using the CLI. You've created a query that joins `pageviews` and `users` and creates a new stream called `csas_enriched_pv`. You also ran another query that creates a materialized view named `user_view_count` from `csas_enriched_pv`. ## Clean Up Now it's time to clean up your environment. To do this: 1. Terminate the queries. 2. Drop the created streams, changelogs, and materialized views. 3. Go to the corresponding database and namespace to drop the streams, changelogs, and materialized views. {% hint style="warning" %} **Important** If there is a query that uses a stream, changelog, or materialized view, terminate the query before you drop the relation. {% endhint %} # Create and Manage Data Stores {% content-ref url="/pages/s02rywww8yPDpHST2afD" %} [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store) {% endcontent-ref %} {% content-ref url="/pages/89nnl9WTTMz7SNXjz4lE" %} [Explore Data Store and Topic Details](/how-do-i.../create-and-manage-data-stores/explore-data-store-and-topic-details) {% endcontent-ref %} {% content-ref url="/pages/9IIknbAlPUsBfM6zfKMu" %} [Use Multiple Data Stores in Queries](/how-do-i.../create-and-manage-data-stores/using-multiple-stores-in-queries) {% endcontent-ref %} # Create Data Stores for Streaming Data Before you write any queries, you must configure DeltaStream to connect to the data store where data resides. This article shows how to use the Web application to create a data store. This is the first step to take before you process any data using DeltaStream SQL statements. {% hint style="info" %} **Note** In addition to self-hosted services, for Kafka data store types you can also configure [Confluent Cloud](https://www.confluent.io/confluent-cloud/) and [Amazon MSK](https://aws.amazon.com/msk/) in DeltaStream. {% endhint %} ## Creating a Data Store 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display the **Resources** page. 2. Click **+ Add Store**, and from the menu that displays click the data store type you want (in this case, Kafka).
{% hint style="info" %} **Note** Be sure to match the data store type with the streaming store where streaming data resides. {% endhint %} 3. Choose a unique **Name** for the data store. Use only alphanumeric characters, dashes, and underscores, and limit the name to a maximum of 255 characters. 4. If indicated, add at least one **URI** port to which to connect, then click **+**.\ Separate multiple entries with a comma. 5. Complete the authentication options as appropriate for the data store. See below for more information. 6. Click **Add**. ## Authentication Parameters For more detailed information about common and store-specific authentication parameters, see [CREATE STORE](/reference/sql-syntax/ddl/create-store#store_parameters) in the **Reference** section. ### Kafka Authentication Parameters

Kafka Store Parameters

### Confluent Authentication Parameters

Confluent Kafka Store Parameters

### Kinesis Authentication Parameters We recommend you use an assumed IAM role to authenticate with Kinesis stores (below). We do not recommend you use AWS credentials. To authenticate with Kinesis stores we recommend using an assumed IAM role; you can use AWS credentials to authenticate but we do not recommend that method.

AWS Kinesis Store Parameters

### Databricks Authentication Parameters

Databricks Store Parameters

### Snowflake Authentication Parameters

Snowflake Store Parameters

### Postgres Authentication Parameters

Postgres Store Parameters

### ClickHouse Authentication Parameters

ClickHouse Store Parameters

### Iceberg AWS Glue Authentication Parameters

Iceberg AWS Glue Parameters

### Iceberg REST Authentication Parameters

Iceberg REST Parameters

### AWS S3 Authentication Parameters

AWS S3 Parameters

# Explore Data Store and Topic Details DeltaStream provides multiple ways you can view details about your data stores and the topics they contain. You can: * View data store metadata * View details on data streaming in * Search for specific files ### Viewing Store Metadata View a range of information about your data stores from the individual data store's details page. To reach that page, navigate to the **Resources** page and click the data store you want. Then click **Details**. DeltaStream displays details such as the data store status and encryption type; who created the data store and when; and the store URI. In addition, you can: **View who can access a data store**. To do this, scroll down to the **RBAC** row, and on the right click **Show**. DeltaStream displays the store owner and access permissions.
**View the history of actions on a data store**. To do this, scroll down to the **Audit Log** row. To see all actions from the moment of store creation, click **Show**. To view a time-based subset of these actions, click the **From beginning** down arrow, select the time interval you want, and then click **Show**.
### Viewing details on the data streaming in: To display the unformatted events in list form, click ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeByihgOSYbWdhavWoPiExEH0UD2enGQ0T8pg7_X4dkYLrc3n7fuzr0dVFIqUX5KODloVceUcy7sHt8fzppTzrAZ35gZKsVMx2KcG5BPThWyWdjWGVN6kGeEADK3-d1nWqgHlkpDxBhHdbs_GXu2V3ITE_7?key=UefObijvgyeIVbiCnQQu0w).
To display the code in an event, click the desired event in the list.
### Searching for Specific Files To search for files within a specific time window, click ![](/files/VUny44NrkYQp8vGSjS8J). When the **time search** box opens, enter the desired parameters.
# Use Multiple Data Stores in Queries In DeltaStream, your streaming data resides in a [Data Store](/overview/core-concepts/store). Apache Kafka and Amazon Kinesis are two examples of such data stores. DeltaStream reads data from a streaming data store, performs the desired computation, and writes the results of the computation to the same data store (or to another, if you wish). This article demonstrates how you can easily move data from one of your data stores to another. It also shows how you can perform joins between data in different data stores. Before you begin, To do so, please review the following: * [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store) * [Create DeltaStream Objects to Structure Raw Data](/how-do-i.../relation) ### Before you begin Before you begin, you should create data stores and DeltaStream objects in DeltaStream, if you haven't already. Set up the following: * Two data stores—a Kafka data store called `MSK` and a Kinesis data store called `kinesis_store` * A stream in `kinesis_store` called `pageviews_kinesis` * A stream in `MSK` called `users_kafka` For details on setting these up, please see [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store) and [Create DeltaStream Objects to Structure Raw Data](/how-do-i.../relation). ### Working with Multiple Data Stores 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display a list of data stores. Here you'll find your Kafka and Kinesis stores.

Kafka and Kinesis Stores.

2. The Kinesis data store has a Kinesis data stream called `pageviews`. In DeltaStream, use a [CREATE STREAM](/reference/sql-syntax/ddl/create-stream) query, as shown below, to create a [Database](/overview/core-concepts/databases#stream) called `pageviews_kinesis` that is backed by the `pageviews` Kinesis stream.

Creating the pageviews_kinesisStream.

Kinesis Stream Results

3. The Kafka data store has a Kafka topic called `ds_users`. In DeltaStream, use a [CREATE STREAM](/reference/sql-syntax/ddl/create-stream) query shown below to create a [Database](/overview/core-concepts/databases#stream) called `users_kafka` that is backed by the `ds_users` Kafka topic.

Creating the users_kafkaStream.

Kafka Stream Results

With your two data stores established, each with an associated relation, you can write your queries. ### Move Data from One Data Store to Another To migrate data out of Kinesis and into Kafka, create a new stream that selects all the columns from an existing stream. Then you can specify the data store that backs the new stream. In the setup here, there's a stream belonging to your Kinesis data store called `pageviews_kinesis`. Use a [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as) query (CSAS) to create a new stream that is essentially a copy of the `pageviews_kinesis` stream in the Kafka data store. Label this new stream `pageviews_kafka`.

Creating a Kafka Stream that is a copy of a Kinesis Stream.

Note the CSAS query above includes the `store` property in the `WITH` clause of the sink stream, with a value of `kafka_store`. A CSAS query creates a new stream; specifying this property informs the query about which data store should back this new stream. Leave the `store` property empty, and the query defaults to using the session’s current data store. You must include the `topic.replicas` and `topic.partitions` properties; these are necessary when creating a new Kafka topic. {% hint style="success" %} **Tip** If the source in this query was a stream backed by a Kafka topic, then DeltaStream would use the source’s `topic.replicas` and `topic.partitions` values by default. But here you're reading from a Kinesis-backed stream, and DeltaStream cannot infer the value of these properties; you must set them explicitly in the query. {% endhint %} Navigate to **Resources** > **Kafka Store** to see that a new Kafka topic, called `pageviews_kafka`, has been created and displays in the topic list. The streaming data for the newly-created `pageviews_kafka` stream is stored in this topic. Click the topic to print and see the records flowing through. These records are a copy of the `pageviews_kinesis` records.

The new stream copy.

### Join Data from Sources Belonging to Different Data Stores In DeltaStream, when you define a [Database](/overview/core-concepts/databases#stream) or [Database](/overview/core-concepts/databases#changelog) you can use it as a source for queries. The example below demonstrates how you can join the `pageviews_kinesis` stream and the `users_kafka` stream. Simply use these streams as sources in your query; you do not need any additional specifications related to the data stores that back them. DeltaStream keeps this information as metadata with the stream. Behind the scenes, DeltaStream seamlessly reads from both data stores, performs the join, and then outputs the result to the sink stream. Again, use a [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as) query to create the output stream. Since this joins two streams, this query is an [SELECT](/reference/sql-syntax/query/select#interval-join-stream-stream) that requires the `WITHIN` clause.

Using a CREATE STREAM AS SELECT query.

Note in the CSAS above that in the `WITH` clause you specify the output data store to be `kinesis_store`, similar to what you did in the [section above](#move-data-from-one-store-to-another). But even though you're creating a new Kinesis stream, called `pageviews_enriched`, you don’t need to provide the `topic.shards` property. DeltaStream infers the default value from the left-most source when possible. The sink stream and the left-most source are both backed by Kinesis streams, so `pageview_kinesis`’s `topic.shards` property is applied to `pageviews_enriched`. Navigate to **Resources** > **Kinesis\_store** > **Topics** to see that there's a new Kinesis entity called `pageviews_enriched`. This is where the streaming data for the newly-created `pageviews_enriched` stream is stored. Click that entity to print and see the records flowing through. These records result from the join between `pageviews_kinesis` and `users_kafka`. # Manage Users and User Roles {% content-ref url="/pages/fZHqCDVt7pLkdS3nIBLo" %} [Inviting Users to an Organization](/how-do-i.../roles/invite-user) {% endcontent-ref %} {% content-ref url="/pages/j3yR6PdRcSnGQYROHpNT" %} [Administering Users in your Organization](/how-do-i.../roles/user-management-for-admins) {% endcontent-ref %} {% content-ref url="/pages/wdb2Wn6htvbPGFryz7gk" %} [Using the CLI to Manage User Roles](/how-do-i.../roles/managing-user-roles) {% endcontent-ref %} {% content-ref url="/pages/kmMIVVUP4O5essLeN99a" %} [Example: Setting Up Custom Roles for Production and Stage](/how-do-i.../roles/example-setting-up-custom-roles-for-prod-and-stage) {% endcontent-ref %} # Inviting Users to an Organization If you are an `orgadmin`, `securityadmin`, or `useradmin`, you can invite people to join your organization. This tutorial demonstrates how to use the DeltaStream Web application to send invitations. ### Invite a User 1. Towards the top of the screen, use the **Org** and **Role** dropdown menus to select the organization and role you want.

Selecting an organization and a role

2. Verify that your role has been granted the **MANAGE\_MEMBERS** privilege. The built-in `useradmin` and `orgadmin` roles have this privilege by default. In the left-hand navigation, toward the bottom, hover over the **Profile** icon, and from the menu that displays click **Access Control**.

Accessing the Access Control Page

3. When the **Access Control** page displays, click **Users**, and then click **Invite User**. The **Invite User** box displays. 4. Complete the requested information, including any additional roles you wish to grant them. 1. In the **Granted Roles** drop-down menu, click each role you wish to assign them. When you’re done, click outside the menu to close it. If you select a role in error, simply click it again and DeltaStream removes it from the list. 5. Click to select the role you wish this person to have when they first log on to DeltaStream 6. When you’re done, click **Add**. The person’s email address now displays in the I**nvited Users** section. {% hint style="success" %} **Tip** If you wish to invite an additional `orgadmin`, set the **Default Login Role** to `sysadmin` and **Granted Roles** to `orgadmin`. {% endhint %} ### Cancelling an Invitation If you've invited someone in error, it's easy to change your mind and revoke an invitation. To do this: 1. Under **Invited Users**, find the person you wish to uninvite. 2. Click **Cancel Invite**. 3. At the **Revoke Invitation** prompt, enter the person’s email address and click **Delete**.\ The system displays a message alerting you that the person’s invitation was revoked, and removes their name from the **Invited Users** list.

Revoking an invitation

{% hint style="info" %} **Note** You can also delete a pending invitation via the DeltaStream CLI. To do this, use the [`REVOKE INVITATION` command](/reference/sql-syntax/command/revoke-invitation): ```sql REVOKE INVITATION 8f7a4504-ce64-4ee3-a9b5-227925e9dq44; ``` {% endhint %} # Administering Users in your Organization This article is helpful if you've just been given access to a new DeltaStream [organization](/overview/core-concepts/access-control#organiation). It discusses the responsibilities of different administrator roles. {% hint style="info" %} **Note** For additional background, see this [high-level overview of access control concepts](/overview/core-concepts/access-control) in DeltaStream. {% endhint %} One of the first things to do when you begin to assemble a new organization in DeltaStream is to invite users to add other administrators to administrative roles, including: * sysadmin * securityadmin * useradmin * orgadmin Note the role hierarchy illustrated in the diagram below:

DeltaStream Role Hierarchy

Admin roles are built into DeltaStream. There are 4 types of admin roles: 1. **ORGADMIN** — Root of the the role hierarchy. This role manages operations at the organization level. 2. **SYSADMIN** — Role that has privileges to create, manage, and drop objects. 3. **USERADMIN** — Role that has privileges to manage users and roles within the organization. 4. **SECURITYADMIN** — Role that manages any object grants globally. This role inherits privileges from the `useradmin` role. {% hint style="warning" %} **Important** Admin roles are the most powerful roles in DeltaStream. Only select individuals should have them. {% endhint %} ## Using the OrgAdmin Role The `orgadmin` role is the single most powerful role in DeltaStream. Use it only for routine administrative tasks, such as inviting and removing users. Do not use it to grant access to resources for activities such creating and querying streams. For example, when you create objects such as databases, namespaces, or DeltaStream objects, use the `sysadmin` or lower roles in the hierarchy. Similarly, when you invite new users or create new roles, use the `useradmin` role. {% hint style="success" %} **Tip** As a best practice, use a less powerful role (that is, one with fewer privileges) whenever possible. {% endhint %} When someone first gets access to DeltaStream, they're granted the `orgadmin` role. In turn they inherit the `sysadmin` and `securityadmin` roles, and their default role will be `sysadmin`. ### 1. Adding OrgAdmin Users To add people to your organization, use the `orgadmin` or `useradmin` role. To invite others to be an `orgadmin`, use the [`INVITE USER` command](/reference/sql-syntax/command/invite-user). ```sql INVITE USER 'user@demo.org' WITH ( 'roles' = (orgadmin, sysadmin), 'default' = sysadmin ); ``` {% hint style="warning" %} **Important** Do not set OrgAdmin as the default role for people you invite into your organization. {% endhint %} For more details, see [Switching Roles](/how-do-i.../roles/managing-user-roles#switch-roles) and [Inviting Users to an Organization](/how-do-i.../roles/invite-user). ### 2. Switching to a different role After you invite others to be `orgadmin`, switch to use a different role. ```sql USE ROLE useradmin; ``` ## Using the UserAdmin Role The `useradmin` role should be the default for managing users and creating custom roles. The `useradmin` role is also powerful, as it includes the privilege to [grant roles](/reference/sql-syntax/command/grant-role). As with `orgadmin` and `securityadmin`, be judicious when assigning the role of `useradmin`. ### 1. Managing Invitations #### Inviting other people to the organization The below example shows how to use the [`INVITE USER`](/reference/sql-syntax/command/invite-user) command to invite someone who has the `useradmin` and `public` roles. The default role is `public`: ```sql INVITE USER 'user@demo.org' WITH ( 'roles' = (useradmin, "public"), 'default' = "public" ); ``` #### Listing Invitations The below example shows how to use the [`LIST INVITATIONS` ](/reference/sql-syntax/command/list-invitations)command; to see invitations you've only just sent, use `LIST PENDING INVITATIONS`: ``` /# LIST PENDING INVITATIONS; Invitation ID | Org name | Org ID | Email | Invited by | User roles | Default role ---------------------------------------+----------+--------------------------------------------------------------+---------------+-------------+------------------+--------------- 8f7a4504-ce64-4ee3-a9b5-227925e9dq44 | doc_org | 830e26fe-de4g-4996-839f-bccb258f8f91 | user@demo.org | useradmin | useradmin,public | public ``` #### Revoking Invitations To delete a pending invitation, use the [`REVOKE INVITATION` ](/reference/sql-syntax/command/revoke-invitation)command: ```sql REVOKE INVITATION 8f7a4504-ce64-4ee3-a9b5-227925e9dq44; ``` #### Describing a User To print (view) information about an existing user, use the [`DESCRIBE USER` ](/reference/sql-syntax/command/describe-user)command: ``` /# DESCRIBE USER 'user@demo.org' ; Given name | Family name | Email | Locale -------------+-------------+---------------+--------- user | Demo | user@demo.org | en GrantedRoles | Inherited ---------------+------------ useradmin | public | ✓ ``` ### 2. Managing Roles #### Granting a specific role to an individual or to another role The below example shows how to use the [`GRANT ROLE`](/reference/sql-syntax/command/grant-role) command to grant the `sysadmin` role to a user: ```sql GRANT ROLE sysadmin TO USER 'user@demo.org'; ``` The below example shows how to grant the custom role `my_role` to the `sysadmin` role: ```sql GRANT ROLE my_role TO ROLE sysadmin; ``` #### Revoking a role from either a user or another role The below example shows how to use the [`REVOKE ROLE` ](/reference/sql-syntax/command/revoke-role)command to revoke the `sysadmin` role from a user: ```sql REVOKE ROLE sysadmin FROM USER 'user@demo.org'; ``` The below example shows how to revoke the custom role `my_role` from the `sysadmin` role: ```sql REVOKE ROLE my_role FROM ROLE sysadmin; ``` #### Creating a custom role The below example shows how to use the [`CREATE ROLE`](/reference/sql-syntax/ddl/create-role) command to create the `production_role` role: ```sql CREATE ROLE production_role; ``` #### Dropping a custom role The below example shows how to use the [`DROP ROLE` ](/reference/sql-syntax/ddl/drop-role)command to drop the `production_role` role: ```sql DROP ROLE production_role; ``` ## Using the `SecurityAdmin` Role The `securityadmin` role should be the default role for managing object grants. As the `securityadmin` role inherits privileges from the `useradmin` role, it's also a powerful role. And as with the `orgadmin` and `useradmin` roles, take care to give the role of `securityadmin` only to people who will need it. ### 1. Granting Privileges to Roles DeltaStream allows a finite number of privileges. To view them, see [Access Control Core Concepts](/overview/core-concepts/access-control#_privilege). The below example shows how to use the [`GRANT `](/reference/sql-syntax/command/grant-privileges) command to grant the `CREATE_DATABASE` privilege to the `my_role` role: ```sql GRANT CREATE_DATABASE ON ORGANIZATION TO ROLE my_role; ``` #### Revoking privileges from existing roles The below example shows how to use the [`REVOKE `](/reference/sql-syntax/command/revoke-privileges) command to revoke the `CREATE_DATABASE` privilege from the `my_role` role: ```sql REVOKE CREATE_DATABASE ON ORGANIZATION FROM ROLE my_role; ``` ### 2. Granting Ownership of Objects to Different Roles You can only transfer ownership of an object when either: * the current role is the owner of the object and has been granted the destination role\ — or — * the current role is `securityadmin` Custom roles should be owned by the `useradmin`. You can grant the `Sysadmin` or other custom roles but not grant ownership. {% hint style="success" %} **Tip** Avoid granting ownership of roles to the `sysadmin` role or to any roles that `sysadmin` inherits. {% endhint %} The below example shows how to use the [`GRANT OWNERSHIP`](/reference/sql-syntax/command/grant-ownership) command to grant ownership of the `db` [Database](/overview/core-concepts/databases) to the `my_role` role: ```sql GRANT OWNERSHIP ON DATABASE db TO my_role; ``` ## Using the SysAdmin Role The `sysadmin` role has the privileges to create, manage, and drop objects. Most day-to-day tasks are done in the `sysadmin` role or in a custom role that has been granted to the`sysadmin`. Some of the main actions a `sysadmin` can perform include: * [`CREATE STORE`](/reference/sql-syntax/ddl/create-store) / [`DROP STORE`](/reference/sql-syntax/ddl/drop-store) * [`CREATE ENTITY`](/reference/sql-syntax/ddl/create-entity) / [`DROP ENTITY`](/reference/sql-syntax/ddl/drop-entity) * [`CREATE DATABASE`](/reference/sql-syntax/ddl/create-database) / [`DROP DATABASE`](/reference/sql-syntax/ddl/drop-database) * [`CREATE SCHEMA`](/reference/sql-syntax/ddl/create-schema_registry) / [`DROP SCHEMA`](/reference/sql-syntax/ddl/drop-schema) * [`CREATE STREAM`](/reference/sql-syntax/ddl/create-stream) / [`DROP STREAM`](/reference/sql-syntax/ddl/drop-stream) * [`CREATE CHANGELOG`](/reference/sql-syntax/ddl/create-changelog) / [`DROP CHANGELOG`](/reference/sql-syntax/ddl/drop-changelog) * [`CREATE MATERIALIZED VIEW AS`](/reference/sql-syntax/query/materialized-view/create-materialized-view-as) * [`SELECT`](/reference/sql-syntax/query/select) * [`TERMINATE QUERY`](/reference/sql-syntax/query/terminate-query) {% hint style="success" %} **Tip** This role typically is at the root of the hierarchy for all custom roles that also create, manage, or drop objects. In this way the `sysadmin` role is in charge of all objects in the organization. {% endhint %} See also: * [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store) * [Use Multiple Data Stores in Queries](/how-do-i.../create-and-manage-data-stores/using-multiple-stores-in-queries) * [Create DeltaStream Objects to Structure Raw Data](/how-do-i.../relation) * [Use Namespacing for Organizing Data](/how-do-i.../database) * [Create and Query Materialized Views](/how-do-i.../creating-and-querying-materialized-views) # Using the CLI to Manage User Roles Every DeltaStream organization member belongs to one or more roles. As each role may allow a unique and specific subset of capabilities, in most cases people are assigned multiple roles. Review the basic commands below for details on how to manage someone's roles. ## Listing Roles Individuals can view their current set of roles via the [`LIST ROLES` ](/reference/sql-syntax/command/list-roles)command. In the example below, you can see that while the current role is `sysadmin`, it also has access to other roles including `public`, `orgadmin`, `securityadmin`, and `useradmin`. ``` /# LIST ROLES; Name | Current | Created at ----------------+---------+----------------------- public | | 2023-08-09T22:49:46Z orgadmin | | 2023-08-09T22:49:46Z securityadmin | | 2023-08-09T22:49:46Z useradmin | | 2023-08-09T22:49:46Z sysadmin | ✓ | 2023-08-09T22:49:46Z ``` ## Switching Roles To switch to a different role, you can can use the [`USE` command](/reference/sql-syntax/command/use). The example below shows how to switch from the `sysadmin` role to the `useradmin` role. ``` /# USE ROLE useradmin; ``` Now, if you `LIST ROLES` again, you can see that the current role has changed to `useradmin`. ``` /# LIST ROLES; Name | Current | Created at ----------------+---------+----------------------- public | | 2023-08-09T22:49:46Z orgadmin | | 2023-08-09T22:49:46Z securityadmin | | 2023-08-09T22:49:46Z useradmin | ✓ | 2023-08-09T22:49:46Z sysadmin | | 2023-08-09T22:49:46Z ``` ## Describing a Role You can also use the [CREATE ROLE](/reference/sql-syntax/ddl/create-role) command to learn more information about a specific role. {% hint style="info" %} **Note** Your current role must have access to whichever role you wish to describe. {% endhint %} The example below shows that the `useradmin` role is also granted the `public` role, and below that you can view the privileges of the role. Since the `useradmin` role is for managing other users, the `ManageMembers` privilege is granted to this role. Other roles may have other privileges. ``` /# DESCRIBE ROLE useradmin; Name | Created at ------------+----------------------- useradmin | 2023-08-09T22:49:46Z Granted Roles Name ---------- public Granted Privileges Type | Target | ID/Name | Grant option ----------------+--------------+---------+--------------- ManageMembers | Organization | | ✓ ``` # Example: Setting Up Custom Roles for Production and Stage This article takes you through the steps of setting up an organization with custom roles for production (prod) and stage. Follow along, and by the end of the article you should have: * 2 databases – `prod_db` and `stage_db` * 2 custom roles – `prod_manager` and `stage_manager` * `prod_manager` should have have full access to the `prod_db` database * `stage_manager` should have have full access to the `stage_db` database * 1 admin user * the admin user is the first user, granted the `ORGADMIN` role and inheriting all other admin roles * 2 non-admin users * prod user is granted `prod_manager` and `stage_manager` * stage user is granted only `stage_manager` ## Logging in as Admin User When you're the first to log into an organization, by default you log in as the `sysadmin` built-in role. But you have access to all of the built-in admin roles, including `orgadmin`. In this example, as the admin user, you have 4 small tasks: 1. Create databases 2. Create custom roles 3. Grant the appropriate privileges for these newly-created custom roles 4. Invite non-admin users and assign them these custom roles ### 1. Creating Databases When you create objects in an organization, use either the `sysadmin` role or a role that's lower in the hierarchy. Let's switch to the `sysadmin` role: 
USE ROLE sysadmin;
Use the [CREATE DATABASE](/reference/sql-syntax/ddl/create-database) to create the `prod_db` and `stage_db` databases: ```sql CREATE DATABASE prod_db; CREATE DATABASE stage_db; ``` Now list your databases to view them: ``` prod_db.public/# LIST DATABASES; Name | Default | Owner | Created at | Updated at -----------+---------+----------+----------------------+----------------------- stage_db | | sysadmin | 2023-12-13T22:28:04Z | 2023-12-13T22:28:04Z prod_db | ✓ | sysadmin | 2023-12-13T22:27:57Z | 2023-12-13T22:27:57Z ``` ### 2. Creating Custom Roles Switch to the `useradmin` role. This role has the privilege for creating roles and granting roles to other members of your organization: 
USE ROLE USERADMIN;
Use the [CREATE ROLE](/reference/sql-syntax/ddl/create-role) to create the `prod_manager` and `stage_manager` roles: ```sql CREATE ROLE prod_manager; CREATE ROLE stage_manager; ``` When you first create these roles, they are standalone roles detached from any other roles. However, you should grant these custom roles to the `sysadmin` role . When you do this they become lower members of the `sysadmin` role's hierarchy. Use the [GRANT ROLE](/reference/sql-syntax/command/grant-role) command to grant these custom roles to the `sysadmin`: ```sql GRANT ROLE prod_manager TO ROLE sysadmin; GRANT ROLE stage_manager TO ROLE sysadmin; ``` Now use [LIST ROLES](/reference/sql-syntax/command/list-roles) to view the new roles you just created: ``` /# LIST ROLES; Name | Current | Created at ----------------+---------+----------------------- public | | 2023-12-13T22:16:12Z securityadmin | | 2023-12-13T22:16:12Z useradmin | ✓ | 2023-12-13T22:16:12Z prod_manager | | 2023-12-13T22:32:41Z orgadmin | | 2023-12-13T22:16:12Z sysadmin | | 2023-12-13T22:16:12Z stage_manager | | 2023-12-13T22:32:44Z ``` ### 3. Granting the Appropriate Privileges for Each New Role With the new roles in place, it's time to grant these roles the usage privilege each role needs to work with its respective databases. As the `sysadmin` role is the owner of these databases, you must use that role to grant the usage privilege to the custom roles. Start by switching to the `sysadmin` role: 
USE ROLE sysadmin;
Now use [GRANT PRIVILEGES](/reference/sql-syntax/command/grant-privileges) to grant privileges to your custom roles. For this example, grant the `USAGE` privilege for each database to the matching role: ```sql GRANT USAGE ON DATABASE prod_db TO ROLE prod_manager; GRANT USAGE ON DATABASE stage_db TO ROLE stage_manager; ``` To see details and make sure you're on the right track, use[DESCRIBE ROLE](/reference/sql-syntax/command/describe-role) to view the new granted privileges for your `prod_manager` role: ``` /# DESCRIBE ROLE prod_manager; Name | Created at ---------------+----------------------- prod_manager | 2023-12-13T22:32:41Z Granted Roles Name ---------- public Granted Privileges Type | Target | ID/Name | Grant option --------+----------+---------+--------------- Usage | Database | prod_db | ``` {% hint style="info" %} **Note** At any time you can[ view the full list of privileges](/overview/core-concepts/access-control#available-privileges). {% endhint %} ### 4. Inviting Users and Assigning them Custom Roles Great -- you've set up your databases and the custom roles to use them. Now it's time to invite users and assign them to a set of roles. Inviting users is the responsibility of the `useradmin`, so let's switch to that role: 
USE ROLE useradmin;
Use [INVITE USER](/reference/sql-syntax/command/invite-user) to invite new users: ```sql INVITE USER 'prod_user@demo.org' WITH ('roles'=(prod_manager, stage_manager), 'default'=stage_manager); INVITE USER 'stage_user@demo.org' WITH ('roles'=(stage_manager), 'default'=stage_manager); ``` DeltaStream uses a person's email address as their unique identifier. For example, in the above code block you're inviting **** to your organization and granting to them the `prod_manager` and `stage_manager` roles. For , you're granting only the `stage_manager` role. {% hint style="info" %} **Note** The `useradmin`can either grant additional roles to people (via the [GRANT ROLE](/reference/sql-syntax/command/grant-role) command) or revoke roles from people (via the [REVOKE ROLE](/reference/sql-syntax/command/revoke-role)command). {% endhint %} Now gain visibility into how interacting with the organization looks from each user's perspective. ## Interacting with the Organization as Prod User As the prod user, after accepting the invitation into this organization (see [ACCEPT INVITATION](/reference/sql-syntax/command/accept-invitation)), you see that your current role is the `stage_manager` role (because the `useradmin` set it as the default role for the prod user). Use [LIST ROLES](/reference/sql-syntax/command/list-roles) to verify the role you're using: ``` /# LIST ROLES; Name | Current | Created at ----------------+---------+----------------------- public | | 2023-12-13T22:16:12Z securityadmin | | 2023-12-13T22:16:12Z useradmin | | 2023-12-13T22:16:12Z prod_manager | | 2023-12-13T22:32:41Z orgadmin | | 2023-12-13T22:16:12Z sysadmin | | 2023-12-13T22:16:12Z stage_manager | ✓ | 2023-12-13T22:32:44Z ``` If you list the databases you only see `stage_db`. That's because `stage_db` is the only database available to the `stage_manager` role you are assuming: ``` /# LIST DATABASES; Name | Default | Owner | Created at | Updated at -----------+---------+----------+----------------------+----------------------- stage_db | | sysadmin | 2023-12-13T22:28:04Z | 2023-12-13T22:28:04Z ``` Since the prod user has access to both the `stage_manager` and `prod_manager` roles, you can switch to the `prod_manager` role: ```sql USE ROLE prod_manager; ``` Now using the `prod_manager` role, if you list your databases again you can see that only the `prod_db` database displays: ``` /# list DATABASES; Name | Default | Owner | Created at | Updated at ----------+---------+----------+----------------------+----------------------- prod_db | | sysadmin | 2023-12-13T22:27:57Z | 2023-12-13T22:27:57Z ``` ## Interacting with the Organization as Stage User As the stage user, after you accept the invitation into this organization (see [ACCEPT INVITATION](/reference/sql-syntax/command/accept-invitation)), you see that your current role is the `stage_manager` role (because the `useradmin` set it as the default role for the prod user). Use [LIST ROLES](/reference/sql-syntax/command/list-roles) to verify the role you're using: ``` /# LIST ROLES; Name | Current | Created at ----------------+---------+----------------------- public | | 2023-12-13T22:16:12Z securityadmin | | 2023-12-13T22:16:12Z useradmin | | 2023-12-13T22:16:12Z prod_manager | | 2023-12-13T22:32:41Z orgadmin | | 2023-12-13T22:16:12Z sysadmin | | 2023-12-13T22:16:12Z stage_manager | ✓ | 2023-12-13T22:32:44Z ``` If you list the databases you only see `stage_db`. That's because `stage_db` is the only database available to the `stage_manager` role you're assuming: ``` /# LIST DATABASES; Name | Default | Owner | Created at | Updated at -----------+---------+----------+----------------------+----------------------- stage_db | | sysadmin | 2023-12-13T22:28:04Z | 2023-12-13T22:28:04Z ``` Notably, the `stage_manager` doesn't have access to the `prod_manager` role. The system displays an error it you try to use the `prod_manager` role: ``` /# USE ROLE prod_manager; ERROR: role prod_manager not granted to user ``` # Create DeltaStream Objects to Structure Raw Data In DeltaStream a [Data Store](/overview/core-concepts/store) provides access to the raw data in your external systems. To process that data in queries, we define **DeltaStream objects** that attach metadata and data format information to the underlying store data. ## Understanding the Data As an example, below is a defined Apache Kafka store that contains several entities: ```sh demodb.public/msk_public# LIST ENTITIES; Entity name ----------------------- ds_syslogs ds_pageviews ds_shipments ds_users ``` Now assume all entities are in `JSON` format. See [CREATE STORE](/reference/sql-syntax/ddl/create-store) and [UPDATE ENTITY](/reference/sql-syntax/ddl/update-entity) for using other serialization formats. For information around data formats -- for example, [CREATE STREAM](/reference/sql-syntax/ddl/create-stream) -- refer to the relation’s DDL statements. You can inspect the entities to understand the kind of data you have -- for example the `ds_pageviews` entity: ```sh demodb.public/msk# PRINT ENTITY ds_pageviews; {"userid":"User_7"} | {"viewtime":1677196372920,"userid":"User_7","pageid":"Page_82"} {"userid":"User_3"} | {"viewtime":1677196372962,"userid":"User_3","pageid":"Page_97"} {"userid":"User_6"} | {"viewtime":1677196373021,"userid":"User_6","pageid":"Page_80"} {"userid":"User_1"} | {"viewtime":1677196373081,"userid":"User_1","pageid":"Page_73"} {"userid":"User_2"} | {"viewtime":1677196373122,"userid":"User_2","pageid":"Page_35"} {"userid":"User_7"} | {"viewtime":1677196373182,"userid":"User_7","pageid":"Page_58"} ``` Here is the `ds_users` entity: ```sh demodb.public/msk# PRINT ENTITY ds_users; {"userid":"User_6"} | {"registertime":1677196517022,"userid":"User_6","regionid":"Region_9","gender":"OTHER","interests":["News","Movies"],"contactinfo":{"phone":"6503889999","city":"Palo Alto","state":"CA","zipcode":"94301"}} {"userid":"User_8"} | {"registertime":1677196517619,"userid":"User_8","regionid":"Region_5","gender":"FEMALE","interests":["News","Movies"],"contactinfo":{"phone":"6502215368","city":"San Carlos","state":"CA","zipcode":"94070"}} {"userid":"User_1"} | {"registertime":1677196518042,"userid":"User_1","regionid":"Region_3","gender":"FEMALE","interests":["News","Movies"],"contactinfo":{"phone":"9492229999","city":"Irvine","state":"CA","zipcode":"92617"}} {"userid":"User_4"} | {"registertime":1677196518620,"userid":"User_4","regionid":"Region_6","gender":"OTHER","interests":["News","Movies"],"contactinfo":{"phone":"6503889999","city":"Palo Alto","state":"CA","zipcode":"94301"}} ``` `PRINT ENTITY` reads directly from the underlying entity in the store. For Kafka-type stores, that entity is a Kafka topic. ## Defining DeltaStream Objects When you know what the data looks like in your entities, you can attach a DeltaStream structure to them for use in queries. In the example below, `ds_pageviews` is the underlying entity and `pageviews` is the DeltaStream [Database](/overview/core-concepts/databases#stream) relation defined on top of it: ```sql CREATE STREAM pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ('topic'='ds_pageviews', 'value.format'='JSON'); ``` See [CREATE STREAM](/reference/sql-syntax/ddl/create-stream) for more information. Since the `ds_users` entity hosts user information that changes over time, you can define a [Database](/overview/core-concepts/databases#changelog) relation to capture ongoing changes to each `userid`: ```sql CREATE CHANGELOG users_log ( registertime BIGINT, userid VARCHAR, regionid VARCHAR, gender VARCHAR, interests ARRAY, contactinfo STRUCT, PRIMARY KEY(userid) ) WITH ('topic'='ds_users', 'key.format'='JSON', 'key.type'='STRUCT', 'value.format'='JSON'); ``` See [CREATE CHANGELOG](/reference/sql-syntax/ddl/create-changelog) for more information. {% hint style="info" %} **Note** For certain applications, it may be more useful to have access to a snapshot of the resulting data. See [Database](/overview/core-concepts/databases#materialized_view) and [CREATE MATERIALIZED VIEW AS](/reference/sql-syntax/query/materialized-view/create-materialized-view-as) for more information on how to create a view for the data. {% endhint %} When you have defined objects, you can list them through their database and namespace: ```sh demodb.public/msk# LIST OBJECTS; Name | Type | Owner | Created at | Updated at -----------------------+------------------+----------+----------------------+----------------------- users_log | Changelog | sysadmin | 2023-01-12T20:41:00Z | 2023-01-12T20:41:00Z pageviews | Stream | sysadmin | 2023-01-12T20:39:02Z | 2023-01-12T20:39:02Z ``` In addition to listing them, you can also use their database and namespace to describe them: ```sh demodb.public/msk# DESCRIBE OBJECT pageviews; Name | Type | Metadata | Columns | Details | Primary key | Owner | Created at | Updated at ------------+--------+--------------------------------------------------------------+------------------------------------------+--------------------+-------------+----------+----------------------+----------------------- pageviews | Stream | {value.format : json,store : msk,topic : pageviews} | viewtime BIGINT | store=msk | | sysadmin | 2023-01-12T20:39:02Z | 2023-01-12T20:39:02Z | | | userid VARCHAR | topic=pageviews | | | | | | | pageid VARCHAR | | | | | ``` ## Using DeltaStream Objects When you define a DeltaStream relation on top of an entity, you can query that relation using DeltaStream SQL. For example, you can use the `pageviews` stream relation in interactive queries: ```sh demodb.public/msk# SELECT * FROM pageviews; {"userid":"User_5"} | {"viewtime":1677274911334,"userid":"User_5","pageid":"Page_14"} {"userid":"User_8"} | {"viewtime":1677274911528,"userid":"User_8","pageid":"Page_65"} {"userid":"User_9"} | {"viewtime":1677274911766,"userid":"User_9","pageid":"Page_49"} {"userid":"User_3"} | {"viewtime":1677274911812,"userid":"User_3","pageid":"Page_21"} {"userid":"User_3"} | {"viewtime":1677274912412,"userid":"User_3","pageid":"Page_25"} {"userid":"User_1"} | {"viewtime":1677274912569,"userid":"User_1","pageid":"Page_56"} {"userid":"User_6"} | {"viewtime":1677274912819,"userid":"User_6","pageid":"Page_20"} ``` You can also use that relation in persistent queries, where it is continuously used as a source or sink: ```sql CREATE STREAM user2_views AS SELECT userid, pageid FROM pageviews WHERE userid = 'User_2'; ``` # Use Namespacing for Organizing Data Much like relational databases, DeltaStream uses [Database](/overview/core-concepts/databases)s and namespaces for logically organizing data, regardless of the [Data Store](/overview/core-concepts/store) in which the data resides. You can use this logical separation to namespace [Query](/overview/core-concepts/queries) statements based on the scope of the query and what stores are involved. This is of particular value when your stores are accessible by more than one team and you require collaboration on the data. All DeltaStream objects are hosted in a database and namespace. Here's how to use this framework to organize your data: ## Create a Database In addition to [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store), you must also create a database to process data. To do this: 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). Then, to the right of **Database**, click **+**, and then **Database**.

Adding a Database

2. At the prompt, enter a unique name for the database and then click **Add**. The new database displays in the **Database** list of available databases. ## Using Database Namespaces You can use namespaces to further organize objects within a database. This makes it possible to manage complex projects. Here's an example: refining your `pageviews` events before you share them with your business analytics team for further analysis. 1. Aggregate the page visits for each user and write that into a `my-db.analytics.user_visits` stream:

Aggregating Page Visits as a Stream

2. Use the new `page_visits` stream and enrich it with the user’s latest location for a full picture on how your users are visiting pages on your website.\ Here, you're using your users’ [Database](/overview/core-concepts/databases#_changelog) object to get the location information. 3. Write this into the analytics team’s public namespace in the `analytics-db`, `analytics-ds.public.user_visit_location`:

Writing User Location Results

## Reducing Clutter and Taming the Chaos Using databases and namespaces, you can reduce clutter in the `analytics-db` . To do this: 1. Refine the necessary data in the `my-db` database. 2. Use the refinement to push an enriched set of records to the `public` namespace of the `analytics-db`. {% hint style="success" %} **Tip** This namespacing approach enables you to organize where to write data into and where it resides, while also ensuring the right consumer can access it across your company. {% endhint %} # Create and Query Materialized Views A [Materialized View](/overview/core-concepts/databases#materialized_view) is a DeltaStream object that captures the result of a query and is continuously updated as new records are ingested in the query’s source object(s). This article demonstrates how to create materialized views and run queries on them. ## Create a Materialized View To begin, use the [CREATE MATERIALIZED VIEW AS](/reference/sql-syntax/query/materialized-view/create-materialized-view-as) statement to create a new materialized view. Note that with this statement, you can only generate columns with [Data Types](/reference/sql-syntax/data-types#primitive-data-types) when you use the [SELECT](/reference/sql-syntax/query/select) query. ### Example 1: Top-K query #### Create a materialized view using window functions In this first example, create a [Database](/overview/core-concepts/databases#_stream) called `mv_pageviews` on the data stored in the `ds_pageviews` topic in your default data store. Use this DDL statement to create this stream: ```sql CREATE STREAM mv_pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ('topic'='pageviews', 'value.format'='JSON'); ``` Now, if you wish to track how many pages each user visits every 30 seconds, you can use a query with a [SELECT](/reference/sql-syntax/query/select#window-function) to create a materialized view. Call this materialized view `visits_count`. DeltaStream keeps its content up to date in a real-time manner. ```sql CREATE MATERIALIZED VIEW visits_count AS SELECT window_start, window_end, userid, COUNT(pageid) AS pgcnt FROM TUMBLE(pageviews, size 30 SECONDS) GROUP BY window_start, window_end, userid; ``` To review the details on the above two objects, in the lefthand navigation click **Databases** ( ![](/files/VRjWh8bZPBOtjSIGsW0k) ), and drill down on the database to display your streams**.**

Drilling down from Database to display streams

#### Top-k query When you have created the `visits_count` materialized view, you can run a top-k query against it. This query retrieves the top 3 users who have visited the most number of pages so far. It uses `GROUP BY` and the aggregate function `SUM` to calculate the total number of pages each user has visited across all time windows. The system orders results based on the total page counts and the top 3 records that are returned: ```sql SELECT userid, SUM(pgcnt) AS total FROM visits_count GROUP BY userid ORDER BY total DESC LIMIT 3; ``` Here is a sample result for the query:

Result of the query

For more details on query capabilities on materialized views, please see [SELECT (FROM MATERIALIZED VIEW)](/reference/sql-syntax/query/materialized-view/select-from-materialized-view) ### Example 2. Combine two streams’ records In this example, you use a materialized view to run different queries on `JOIN` results. Assume you have a new stream, called `users`, created using the DDL below. This stream captures some information about each user, such as the time a given user has registered and the user’s interests. ```sql CREATE STREAM users_info ( registertime BIGINT, userid VARCHAR, regionid VARCHAR, gender VARCHAR, interests ARRAY, contactinfo STRUCT ) WITH ('topic'='ds_users', 'value.format'='json'); ``` #### Create a materialized view using `JOIN` Now suppose you wish to enrich the information already captured in the `pageviews` stream by combining it with each user’s information from `users`. To do this, you can run an [SELECT](/reference/sql-syntax/query/select#interval-join-stream-stream) query and store the result in a materialized view called `visits_info`. ```sql CREATE MATERIALIZED VIEW visits_info AS SELECT p.userid AS uid, p.pageid, u.registertime, u.interests[1] AS top_interest FROM pageviews p JOIN users u WITHIN 1 minute ON u.userid = p.userid; ``` #### Run queries on the `JOIN` results Now that the `visits_info` materialized view is created, DeltaStream continually updates its content in real-time as new records are ingested in either the `pageviews` or the `users`. You can retrieve the latest records in the materialized view by running queries directly on it. For example, the below query finds info on pages visited by users whose top interest is gaming. ```sql SELECT * FROM visits_info WHERE top_interest = 'Game'; ``` The output looks like this:

Output of a sample query

As another example, you can find top interests of all the users who have visited a specific page (such as `Page_15`) so far and sort them according to the total user count per interest. ```sql SELECT top_interest, count(uid) AS cnt FROM visits_info WHERE pageid = 'Page_15' GROUP BY top_interest ORDER BY cnt DESC; ``` The query results look like this:

Sample query results

# Create a Compute Pool to Work with Iceberg A compute pool is a set of dedicated resources to run batch queries. Compute pools are Apache Spark clusters you use to perform real-time analytics on data you read from Iceberg tables. They are similar to Databricks’ [all-purpose compute](https://docs.databricks.com/aws/en/compute/use-compute). However, you never have to manage or interact with Spark directly. DeltaStream auto-configures and instantiates the pool based on a pool size you select. [More on Compute Pools](/overview/core-concepts/compute-pools). ## Creating a Compute Pool You create a compute pool much like you’d create any new DeltaStream object. Define it at the organization level and follow the same access control rules. Specific to compute pools, however, you also must select a pool size – S, M, or L. {% hint style="info" %} **Note** You do not need a compute pool if you are only writing to Iceberg – if, for example, you’re streaming filtered Kafka data into Iceberg tables. Compute pools are necessary only if you wish to read from/query Iceberg data. {% endhint %} ### Working with Compute Pools via the DeltaStream UI #### To create a compute pool 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display the **Resources** page. 2. Click to activate the **Compute Pools** tab. Then click **+ Add Compute Pool**.
3. When the **Add Compute Pool** window displays, enter the required information. 1. To specify a pool size, click the **Pool Size** down arrow and select from **Small**, **Medium**, or **Large**. 4. Compute pools by default shut off automatically after 60 minutes of inactivity. To change this value, click to activate the **Enable Auto-Stop** slider, then use the arrows to specify a different auto-shutoff length. 5. Click **Add**. The **Resources** page redisplays, with your new pool added to the list of pools under the **Compute Pools** tab. After you create a compute pool, you can perform multiple actions: * Edit its pool size and auto-shutoff interval * Stop it * Delete it * View a full history of its activity. You can access these actions in either of 2 ways: 1. Right-click on the compute pool you want and select from the context menu.
2. On the right, under the **Action** column, click the desired icon.
We'll use the right-click context menu to navigate for the next few procedures, but you can instead click an icon, if you prefer. **To edit a compute pool size and auto-shutoff interval** 1. Right-click the compute pool, and from the menu that displays click **Edit ``** 2. Make the changes you wish, and then click **Update**. **To stop or start a compute pool** * Right-click the compute pool, and from the menu that displays click **Stop.** The compute pool redisplays with a Status of **Stopped**. Repeat this step to re-start the compute pool. **To delete a compute pool** 1. Right-click the compute pool, and from the menu that displays click **Delete.** 2. When the Delete Compute Pool box displays, follow the instructions and then click **Delete**. **To view compute pool activity** 1. Right-click the compute pool, and from the menu that displays click **View Details.**\ The **Details** pane displays on the right, with the **Overview** tab open. This tab only displays basic information (owner, status, and created/modified dates).displays, which 2. To view the complete history of this compute pool (created time, started/stopped time, ownership, and so on), click the **Details** tab.
{% hint style="warning" %} **Important** Creating a compute pool essentially is a prerequisite for reading from/querying Iceberg tables. Please see [Iceberg REST Catalog](/integrations/setting-up-data-store-integrations/iceberg-rest-catalog) or [Iceberg AWS Glue Catalog](/integrations/setting-up-data-store-integrations/iceberg-aws-glue-catalog) for detailed instructions on setting up Iceberg tables to work with DeltaStream. {% endhint %} ### Working with Compute Pools via the DeltaStream CLI Follow the links below for instructions on using the DeltaStream SQL DDL to create and manage compute pools: [Create Compute\_Pool](/reference/sql-syntax/ddl/create-compute_pool) [Update Compute\_Pool](/reference/sql-syntax/ddl/update-compute_pool) [Create Store](/reference/sql-syntax/ddl/create-store) [List Compute\_Pool](/reference/sql-syntax/command/list-compute_pools) [Drop Compute\_Pool](/reference/sql-syntax/ddl/drop-compute_pool) [Start Compute\_Pool](/reference/sql-syntax/command/start-compute_pool) [Stop Compute\_Pool](/reference/sql-syntax/command/stop-compute_pool) # Create a Function A [function](/overview/core-concepts/function) is custom code used to extend DeltaStream's processing capabilities. There are multiple [built-in functions](/reference/sql-syntax/query/functions/built-in-functions) available for you to use in your queries, but you can create and implement more functions and add them to DeltaStream. This article demonstrates how to write a function, add it to DeltaStream, and use it in DeltaStream queries. {% hint style="info" %} **Note** Functions are only valid for approved organizations. Please contact us if you wish to enable functions in your organization. {% endhint %} ## Writing a Function DeltaStream queries are powered by Apache Flink. To write functions, you leverage Flink's APIs (please see [Flink documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/#user-defined-functions) for more details). DeltaStream supports Flink's [Scalar Functions](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/#scalar-functions) and [Aggregate Functions](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/#aggregate-functions). To help you write these functions, we've provided this repository that includes [examples](https://github.com/deltastreaminc/deltastream-examples/tree/main/udf/java/examples) and [templates](https://github.com/deltastreaminc/deltastream-examples/tree/main/udf/java/templates). You can fork or copy the template to get started. To write functions, you need: * Java (preferably Java 17) * Maven This article assumes a simple use case wherein you wish to reverse a string such that you have the following behavior: * Query: `SELECT reverse(col1) AS res FROM my_source_stream;` * Input: `{ "col1": "ReverseMe" }` * Output: `{ "res": "eMesreveR" }` Under the `src/main/java` directory, you can add a new class called `MyStringReverser`. This class can also exist in other packages under `src/main/java`, such as `src/main/java/x/y/z/MyStringReverser.java`. The `MyStringReverser` implementation looks as follows: ```java import org.apache.flink.table.functions.FunctionContext; import org.apache.flink.table.functions.ScalarFunction; public class MyStringReverser extends ScalarFunction { @Override public void open(FunctionContext context) { // Initialization logic } public String eval(String s) { // Processing logic return new StringBuilder(s).reverse().toString(); } } ``` In this example, your `MyStringReverser` class `extends ScalarFunction`, the API used to define a scalar function-typed UDF. As there's a one-to-one mapping of inputs to outputs (that is, for every input string you output a single reversed version of that string), a scalar function is appropriate. To define an aggregate function instead, your class would extend Flink's `AggregateFunction` class ([find an example here](https://github.com/deltastreaminc/deltastream-examples/blob/main/udf/java/examples/src/main/java/examples/aggr/WeightedAvg.java)). ### Important Components for Functions that `extends ScalarFunction`: * **Function `open(FunctionContext context)`:**\ Any initialization logic exists in the `open()` method. You can use FunctionContext to add metrics to this function. [More on adding custom metrics](/reference/metrics/custom-metrics-in-functions). * **Function `eval(T val)`:**\ Functions that extend `ScalarFunction` must have an `eval()` method. This method is where the processing logic lies. The method signature should match the input and output types of your function. In this case, the input and output values are both strings. However, functions can have multiple input values with different types, and the output type does not need to match the input type. Furthermore, the same function can have multiple `eval()` methods defined. ### Important Components for Functions that `extends AggregateFunction`: * **Function `open(FunctionContext context)`:**\ Any initialization logic exists in the `open()` method. You can use FunctionContext to add metrics to this function. [More on adding custom metrics](/reference/metrics/custom-metrics-in-functions). * **Function `createAccumulator()`:**\ This method creates and initializes the accumulator for this function. The accumulator is an intermediate data structure that stores the aggregated values until a final aggregation result is computed. * **Function `getValue(MyAccumulator acc)`:**\ This method is called every time an aggregation result is materialized. The returned value could be either an early and incomplete result (periodically emitted as data arrives) or the final result of the aggregation. * **Function `accumulate(MyAccumulator acc, T val)`:**\ This method is called on each input row. It contains the logic for updating the accumulator. The method signature should match the input types of your function, with the accumulator as the first parameter. The function can have multiple `accumulate()` methods defined with different custom types and arguments. * **(Optional) Function `retract(MyAccumulator acc, T val)`:**\ You must implement this method when you use it in queries with `OVER` windows. The logic in this method should retract the input values from the accumulator instance. The function can have multiple `retract()` methods defined with different custom types and arguments. * **(Optional) Function `merge(MyAccumulator acc, Iterable it)`:**\ You must implement this method when you use it in queries with `SESSION` windows and bounded aggregations. The logic in this method merges a group of accumulator instances into a single accumulator instance. After you write your function, build a `.jar` file. If you used the DeltaStream [template](https://github.com/deltastreaminc/deltastream-examples/tree/main/udf/java/templates), you can review instructions on how to build the `.jar` file using Maven. ## Creating a Function in a DeltaStream Organization When you have written and built a function, you can add it in DeltaStream. This consists of 2 steps: 1. Uploading the `.jar` file as a Function Source 2. Defining a Function from the Function Source ### Adding a Function Source Using a role with the [`CREATE_FUNCTION_SOURCE` privilege](https://docs.deltastream.io/overview/core-concepts/access-control#available-privileges), create a new function source using the [`CREATE FUNCTION SOURCE`](/reference/sql-syntax/ddl/create-function_source) statement. ```sql CREATE FUNCTION_SOURCE my_function_source WITH ( 'file' = '/path/to/my/jar/file/my-udf-1.0-SNAPSHOT.jar', 'description' = 'Function source with method to reverse Strings' ); ``` Now you can [list your function sources](/reference/sql-syntax/command/list-function_sources) to verify it was created: ``` my_db.public/my_store# LIST FUNCTION_SOURCES; Name | Description | Language | Owner | Created at | Updated at | Status | Messages ---------------------+--------------------------------+----------+----------+----------------------+----------------------+--------+----------- my_function_source | Function source with method to | java | sysadmin | 2024-05-15T15:10:36Z | 2024-05-15T15:11:10Z | ready | | reverse Strings | | | | | | ``` ### Adding a Function from a Function Source Using a role with the [`CREATE_FUNCTION`](https://docs.deltastream.io/overview/core-concepts/access-control#available-privileges) privilege, create a new function using the [`CREATE FUNCTION`](/reference/sql-syntax/ddl/create-function) statement. Note that in this statement's `WITH` clause you specify the function source added in the previous step, plus the class name for the function you wrote in the previous section. ```sql CREATE FUNCTION reverse(s VARCHAR) RETURNS VARCHAR LANGUAGE JAVA WITH( 'source.name'='my_function_source', 'class.name'='x.y.z.MyStringReverser' -- Assuming MyStringReverser is in package x.y.z ); ``` Now [list your Function](/reference/sql-syntax/command/list-functions) to verify it was created: ``` my_db.public/my_store# LIST FUNCTIONS; Source | Class | Signature | Owner | Properties | Created at | Updated at ---------------------+------------------+-----------------------------+----------+------------+----------------------+----------------------- my_function_source | MyStringReverser | reverse(s VARCHAR) VARCHAR | sysadmin | {} | 2024-05-15T15:12:56Z | 2024-05-15T15:13:21Z ``` ## Writing SQL Queries with the New Function Now that you have added your function to DeltaStream, the final step is to actually use the function in a SQL query. To do this, simply call the function in using the method signature defined when you ran `CREATE FUNCTION`. ```sql SELECT viewtime, userid, pageid, reverse(pageid) AS pid_reversed FROM pageviews; | {"viewtime":1715786783663,"userid":"User_4","pageid":"Page_24","pid_reversed":"42_egaP"} | {"viewtime":1715786784667,"userid":"User_6","pageid":"Page_54","pid_reversed":"45_egaP"} | {"viewtime":1715786785673,"userid":"User_5","pageid":"Page_98","pid_reversed":"89_egaP"} ... ``` # Secure my Connection to a Data Store Connect and securely stream data from your data stores without ever exposing your sensitive data to the public Internet. In DeltaStream, these secure connections are called private links. {% content-ref url="/pages/EOHzlG1XOPTMF0roFig5" %} [Introducing DeltaStream Private Links](/how-do-i.../creating-private-links/introducing-deltastream-private-links) {% endcontent-ref %} {% content-ref url="/pages/RLl2ulbXuiT2UkKphkvH" %} [Creating an AWS Private Link from DeltaStream to your Confluent Kafka Dedicated Cluster](/how-do-i.../creating-private-links/creating-an-aws-private-link-from-deltastream-to-your-confluent-kafka-dedicated-cluster) {% endcontent-ref %} {% content-ref url="/pages/F71DT9aO2lj8YPEFs2Mi" %} [Enabling Private Link Connectivity to Confluent Enterprise Cluster and Schema Registry](/how-do-i.../creating-private-links/enabling-private-link-connectivity-to-confluent-enterprise-cluster-and-schema-registry) {% endcontent-ref %} {% content-ref url="/pages/NmXzFVk12lUuqashXZVQ" %} [Creating a Private Link from DeltaStream to Amazon MSK](/how-do-i.../creating-private-links/creating-a-private-link-from-deltastream-to-amazon-msk) {% endcontent-ref %} {% content-ref url="/pages/gcw2rcFfRbTjLhRIOl8G" %} [Creating a Private Link for RDS Databases](/how-do-i.../creating-private-links/creating-a-private-link-for-rds-databases) {% endcontent-ref %} {% content-ref url="/pages/r3cRilC5wCfHb4crYJEq" %} [Deleting a Private Link](/how-do-i.../creating-private-links/deleting-a-private-link) {% endcontent-ref %} # Introducing DeltaStream Private Links Private links enable you to securely connect DeltaStream to a data store or schema registry in your private VPC without exposing your data to the public internet. Sensitive data never leaves your private network perimeter. You can securely access a storage platform, then query your data however you wish. There are separate setup instructions for each of 4 storage platforms: * [Confluent Cloud Kafka](/how-do-i.../creating-private-links/creating-an-aws-private-link-from-deltastream-to-your-confluent-kafka-dedicated-cluster) * [Enterprise Kafka Cluster and private schema registry](/how-do-i.../creating-private-links/enabling-private-link-connectivity-to-confluent-enterprise-cluster-and-schema-registry) * [AWS Managed Kafka (MSK)](/how-do-i.../creating-private-links/creating-a-private-link-from-deltastream-to-amazon-msk) * [PostgreSQL database hosted within AWS](/how-do-i.../creating-private-links/creating-a-private-link-for-rds-databases) In addition to more robust security, using private links support: * faster performance due to lower latency and higher bandwidth (since network traffic does not leave the AWS internal network) * regulatory compliance by effectively isolating sensitive data within a private network environment. This lowers the risk of a data breach; decreases the potential attack surface for malicious actors; and makes possible fine-grained access controls on private endpoints. * streamlined network management * lower network data transfer cost within AWS Private links are specific both to your DeltaStream organization and to your data store’s AWS region. Setup is important; the steps are exacting and involve using the DeltaStream command line interface (CLI) in tandem with your Confluent Cloud or AWS management console. In broad steps, you: 1. Create a test cluster (typical but not always necessary). 2. Use the DeltaStream CLI to build the SQL that instantiates the private link request from the DeltaStream platform. This involves providing details such as region, endpoint service name, resource ARNs, and DNS information, in your Confluent Cloud or AWS account console that you then copy and paste into the SQL you’re writing. 3. Establish connectivity between the two systems; when you establish an MSK or RDS PostgreSQL private link you must accept private link requests made by DeltaStream to your AWS account. Similarly, from within the Confluent Cloud dashboard you must also accept private link connections made to Confluent Cloud dedicated clusters. Here are the instructions for creating private links: * [Creating an AWS private link to Confluent Cloud Kafka](/how-do-i.../creating-private-links/creating-an-aws-private-link-from-deltastream-to-your-confluent-kafka-dedicated-cluster) * [Creating a private link to Confluent Cloud Enterprise](/how-do-i.../creating-private-links/enabling-private-link-connectivity-to-confluent-enterprise-cluster-and-schema-registry) * [Creating a private link to RDS Postgres](/how-do-i.../creating-private-links/creating-a-private-link-for-rds-databases) * [Creating a private link to AWS MSK](/how-do-i.../creating-private-links/creating-a-private-link-from-deltastream-to-amazon-msk) # Creating an AWS Private Link from DeltaStream to your Confluent Kafka Dedicated Cluster Below is the procedure for creating private links for the data you’re streaming via DeltaStream. It includes the configuration needed within your dedicated Confluent Kafka cluster. {% hint style="info" %} **Note** Currently DeltaStream supports private links only in AWS. {% endhint %} ### Before You Begin * Review [Introducing Private Links](/how-do-i.../creating-private-links/introducing-deltastream-private-links). * You must have signed up with DeltaStream and created at least one organization. Private links function within the context of an organization; from a logical perspective, you enable your private links within a specific DeltaStream organization. * [Download the DeltaStream CLI](/getting-started/starting-with-cli) if you don’t already use it. Currently you cannot create private links via the DeltaStream UI – only the CLI. * Request DeltaStream Ops to enable private link connectivity for your account. * You must have the DeltaStream platform AWS account number that sends private link connectivity requests to your Kafka data stores. [Contact DeltaStream support](https://console.deltastream.io/support-center) to obtain this number. * Optionally, work with DeltaStream Operations if you wish to run all your queries (that is, stream processing) within a dedicated AWS dataplane. By default all DeltaStream customer queries run in a shared multi-tenant dataplane; network policies isolate all traffic among multiple customers. This dedicated data plane completely isolates your workload from other DeltaStream customers by using fully-isolated compute and VPC networking resources. ### Creating a Private Link This procedure involves building a SQL statement. When you run the statement, DeltaStream processes the link request automatically. However, the private link is not fully established until it is accepted or approved by an administrator from your organization responsible for maintaining Kafka stores. The entire statement resembles the below; we’ll go through it segment by segment, showing how and where to find the information you need to complete the statement. ```sql CREATE AWS PRIVATE LINK myprivatelinktoconfluent WITH ('private_link.target_type' = CONFLUENT_KAFKA, 'private_link.service_name' = 'com.amazonaws.vpce.us-east-1.vpce-svc-0ac5b449a5268bdf2', 'private_link.hosts' ( '*.use1-az2.dom4g23k71p2.us-east-1.aws.confluent.cloud:9092' USING PORT 9092 IN 'use1-az2', '*.use1-az4.dom4g23k71p2.us-east-1.aws.confluent.cloud:9092' USING PORT 9092 IN 'use1-az4', '*.use1-az6.dom4g23k71p2.us-east-1.aws.confluent.cloud:9092' USING PORT 9092 IN 'use1-az6')); ``` The private link created using the above SQL command is an inbound private link connection to your dedicated Kafka cluster within your Confluent cloud account. {% hint style="info" %} **Note** There are separate but related instructions for creating private links for [AWS Managed Kafka (MSK)](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/ynSRO8WFpfVu7FI9Dmj5), [RDS Postgres](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/8fQ6A91V8j4OoD40nn8Y), and [Confluent Enterprise edition](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/mO5KZFCyDFxORWoS7AbA). {% endhint %} ### Section 1: Creating a Dedicated Test Cluster in Confluent {% hint style="warning" %} **Important** The below steps guide you to create a test dedicated Kafka cluster. It assumes you do not already have a dedicated cluster or just wish to review private link options available in DeltaStream using a test cluster. If you already have a dedicated test cluster, skip this section and proceed now to Section 2. {% endhint %} {% hint style="info" %} **Note** While we attempted to make the below procedure all-inclusive – that is, to relieve you of having to discover on your own where to go and what to look for – if you would like more details, see [Use AWS PrivateLink with Confluent Cloud](https://docs.confluent.io/cloud/current/networking/private-links/aws-privatelink.html#use-aws-privatelink-with-ccloud) for in-bound dedicated clusters. {% endhint %} To create a dedicated test cluster to run inside a dedicated VPC: 1. From your Confluent Cloud console home screen, under **Resource Overview**, click **Environments**. 2. Click **Default**. 3. When the default screen displays, click **Create cluster**.
4. From the **Create cluster** screen, click **Dedicated**. When the cost displays in the **Cluster cost** pane, click **Launch Dedicated provisioning flow**.
5. The **Create cluster** screen redisplays. Click **Begin configuration**. 1. For the cluster type click **AWS**. 2. For **Region/zones**, verify the region is the one you intend; otherwise click the down arrow and select another region. 3. For **Availability**, click either **Single zone** or **Multi zone**.\ Your choice depends on your availability needs. For example, for a simple test cluster you can choose single zone to save on total dedicated cluster cost. 6. Towards the bottom of the screen, click **Continue**. The **Create Cluster** screen redisplays. 7. Click **PrivateLink**. This is the link you must configure for Confluent.
11. In the **Network** section, click either **Create New** or **Use Existing**, depending on your pre-existing setup. * If you select **Create New**, enter a network name. * If you select **Use Existing**, you must also enter a network. To do this, in your Confluent Cloud Console navigate to **Environments** > **Default**. In the **Default** screen, click **For Dedicated Clusters**, and from the list that displays select the cluster you want. * Return to the **Create Cluster** screen, click the **Choose Network** down arrow, and from the list that displays select the same cluster. 12. Under **Network zone placement**, verify the boxes for three zones are the ones you have already configured for your computing environment. 13. Retain any pre-existing default values and click **Continue**. The next few steps lead you to precisely where in your configuration you can find the information and settings you need to copy and then paste into the DeltaStream CLI and establish the private link. 14. Retain the default value of **Automatic**, and click **Continue**. The **Create cluster** screen redisplays.
15. When the screen refreshes, enter the cluster name. Then, towards the bottom of the screen, click **Launch Cluster**.
16. The default screen redisplays. Confluent begins provisioning your new cluster. It may take a few minutes for it to become available. 17. Verify the cluster creation. To do this: 1. Towards the top navigation click **Network Management**. 2. Click **For dedicated clusters** and click the cluster you just created. The cluster screen displays, detailing its configuration. Your cluster is now ready. This is the cluster to connect privately, and this cluster detail screen is where you’ll find and copy information you need to create the private link in the DeltaStream CLI. In your environment for a dedicated cluster, click to activate the **network** tab to display the relevant information about this cluster. {% hint style="warning" %} **Important** This cluster details screen is the screen to which you’ll need to return. Bookmark this screen, or keep it open. You'll need it to copy additional information to enter into the DeltaStream CLI when you create your private link. If you end your session prematurely you can return to this screen by navigating to **Environments** > **Network Management** > **For Dedicated Clusters**, and then clicking the name of the cluster you just created. {% endhint %} 18. Toward the lower right, click **Private Link Access**. The **Add PrivateLink access** screen displays, including information to enter in sections labeled Step 1 and Step 2. 19. Enter a name for this private link. {% hint style="success" %} **Tip** For simplicity’s sake you may wish to name this link *DeltaStream*. {% endhint %} 20. In the Step 1 that displays on this screen, enter the **AWS Account Number** you received from DeltaStream support before you began this procedure. Entering the AWS account number populates the VPC service endpoint service name field in Step 2. This VPC endpoint service name is the endpoint you must enter when you begin working with the DeltaStream CLI. 21. Click **Add**. The cluster detail screen redisplays, now including details of the dedicated link.

Cluster details

To verify that the cluster is being provisioned, check to activate the **Ingress connections** tab, and in the DeltaStream section look at the cluster status. It should be ready. {% hint style="info" %} **Note** If the cluster status is still provisioning, wait a few more minutes and then refresh the screen. {% endhint %} Now switch to DeltaStream and launch the DeltaStream CLI. ### Section 2: Writing the SQL Statement in DeltaStream to Create the Private Link Reminder: The private link you’re creating looks similar to this: ```sql CREATE AWS PRIVATE LINK myprivatelink WITH ('private_link.target_type' = CONFLUENT_KAFKA, 'private_link.service_name' = '$$YourEndpointServiceName', 'private_link.hosts' ( '*.Hostname1:9092' USING PORT 9092 IN 'Host1-az1', '*.Hostname2:9092' USING PORT 9092 IN 'Host2-az2', '*.Hostname3:9092' USING PORT 9092 IN 'Host3-az3')); ``` 1. Start by specifying which DeltaStream organization you wish to use: `Use $$yourorgname` The private link connection you’re creating will be in this specific DeltaStream organization. No other DeltaStream organization will have access to this private link. 2. (Optional) Verify how many private links you already have. (If you’re setting up a private link for the first time you can omit this step.) To do this, enter `LIST AWS PRIVATE LINKS;` 3. Create a private link. To do this, enter `CREATE AWS PRIVATE LINK $$yourconfluentcloud` 5. Enter the **VPC Endpoint Service Name** created by Confluent for the dedicated cluster: `'private_link.service_name' =` 6. Now return to the Confluent console cluster detail page you bookmarked earlier. Copy the **VPC Endpoint Service Name**, and in the DeltaStream UI append the service name to the SQL statement.
Now provide the Kafka broker DNS sub-domains. To do this: 7. Return to the Confluent console cluster details page. Note the DNS subdomain section, which consists of three different hostnames, each in a subnet within the VPC.
8. Now convert Confluent SQL syntax to the DeltaStream SQL syntax. To do this, begin by copying the first endpoint service name, beginning after the colon. 9. In the DeltaStream CLI, preface the host name with `*`. Then paste in the hostname. 10. Append the host name with the port used by Kafka, and then provide the proper availability zone (typically 9092). * Essentially, what you’re doing here is modifying the format of the information on the network overview page of your Confluent cloud, in the DNS subdomain section, into the format specified by DeltaStream.
{% hint style="info" %} **Note** You can also think of the DeltaStream private link as similar to a generic port feature. This enables DeltaStream to support MSK, Postgres, or other technologies in the future. {% endhint %} 11. For the `IN` port, from the **Cluster** page in Confluent, copy the prefix, which in this case is `use1-aws`, and append it to the statement in the DeltaStream CLI. This is the DNS sub-domain information in front of the colon. 12. Repeat these steps for the next two endpoints. Your SQL statement in the DeltaStream CLI is now ready. Run the statement. DeltaStream displays a confirmation and begins creating the private link:
To find out the status of this process, at the prompt type `LIST AWS COMMAND PRIVATE LINKS;` This command reviews the current state of the private links. It verifies the information entered and indicates the private link is in progress. The private link is ready to be used. You can proceed to write queries to the cluster in DeltaStream as if it were a cluster over the Internet. But it is a dedicated cluster. It is not available on the Internet. {% hint style="info" %} **Note** You can also return to the Confluent page and see if there’s a status change listed there, as well. Refresh the page; this may take a few moments. {% endhint %} To begin using the private link, in DeltaStream [create a Confluent Kafka Store for your dedicated Kafka cluster](/integrations/setting-up-data-store-integrations/confluent-cloud) that’s accessible via the newly-created private link. You can use either the CLI or the UI to do this. {% hint style="warning" %} **Important** The dedicated Kafka cluster is only accessible to the organization you used to create a private link to the cluster. If you use the UI, verify that the organization you select when creating the store is the same one for which you enabled this private link. {% endhint %} ### Verifying Private Link Connectivity To verify connectivity you can use the DeltaStream console UI to add a Kafka topic. To do this you must set the number of partitions to 1 and the number of replicas to 3.
From DeltaStream you should get an alert that the operation was successful. You should also be able to see the newly-created topic in DeltaStream. Optionally, verify the connection in Confluent. To do this: 1. Switch to the Confluent console and navigate to the **Cluster Overview** page (**Environments** > **Default** > **$$yourclustername**) 2. In the lefthand navigation click **Topics**. 3. Click **Enable metadata read access**. If the **Topics** page displays without the **Enable metadata read access** option, go to the lefthand navigation and click **Cluster settings**. 4. When the **Cluster settings** page displays, click to activate the **Security** tab. Then, in the lefthand navigation, click **Stream Lineage**.
5. The **Resource metadata access** section displays. Click the toggle to **On**, and at the prompt click **Ok**. 6. In the lefthand navigation pane, click **Topics**. The Topics list page now displays any topics you’ve created
# Enabling Private Link Connectivity to Confluent Enterprise Cluster and Schema Registry Below is the procedure for creating private links for the data you’re streaming via DeltaStream. It includes the configuration needed within your Kafka cluster (in this case, Confluent Enterprise). {% hint style="info" %} **Note** Currently DeltaStream supports private links only in AWS. {% endhint %} ### Before You Begin * Review [Introducing Private Links](/how-do-i.../creating-private-links/introducing-deltastream-private-links). * You must have signed up with DeltaStream and created at least one organization. Private links function within the context of an organization; from a logical perspective, you enable your private links within a specific DeltaStream organization. * [Download the DeltaStream CLI](/getting-started/starting-with-cli) if you don’t already use it. Currently you cannot create private links via the DeltaStream UI. * Request DeltaStream Operations to enable private link connectivity for your account. * You must have the DeltaStream platform AWS account number that sends private link connectivity requests to your Kafka data stores. [Contact DeltaStream support](https://console.deltastream.io/support-center) to obtain this number. * Optionally, work with DeltaStream Operations if you wish to run all your queries (that is, stream processing) within a dedicated AWS dataplane. By default all DeltaStream customer queries run in a shared multi-tenant dataplane; network policies isolate all traffic among multiple customers. This dedicated data plane separates your workload from other DeltaStream customers by using fully-isolated compute and VPC networking resources. {% hint style="info" %} **Note** There are separate but related instructions for creating private links for a [Confluent Cloud dedicated cluster](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/T5OiaOccNryRnCeMLyYw); [AWS Managed Kafka (MSK)](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/ynSRO8WFpfVu7FI9Dmj5); and [PRDS Postgres](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/8fQ6A91V8j4OoD40nn8Y). {% endhint %} ### Creating an Environment in Confluent 1. From your Confluent Console, navigate to **Environments** and click **Create Environment**. The **Create Environment** window opens. Click **Advanced**.
2. The **Create Cluster** screen displays. Enter a cluster name. Then: 1. For **Cluster Type**, click **Enterprise**. 2. For **Provider and region**, click **AWS**. Then click the **Region** down arrow and select the region you need.
3. Scroll down the page for more choices: 1. In the **Uptime SLA** section, click **99.9%** (if you’re testing; you may prefer **99.99%** for production instances). 2. In the **Networking** section, click **Private**. 3. Leave the network configuration as is for now. 4. Check to turn on the **Resource metadata access** slider. This setting enables you to verify your connectivity after you’re done by checking topics coming into your cluster.
4. Click **Launch Cluster**. The **Cluster** details screen displays, indicating you have not yet completed your setup. 5. In the righthand column, click **Create a PrivateLink configuration**.\ To create a Privatelink, access the **navigation menu** within your environment, and then click **Network Management**. Then click **Create gateway configuration** to continue the process.
6. The **Add Network Configuration** screen displays. Enter the provider and region once again, and enter a network name. Then click **Continue**. The **Enterprise cluster** details page displays again. 7. Click to activate the **Network management** tab.
8. Click the network name link. The **network** details page displays. 9. Note down the **PrivateLink Service ID** as `$$YOUR_ENDPOINT_SERVICE`. You use this variable a few steps later, when you create a private link in the DeltaStream CLI.
10. Click **+ Create access point**. The **Create access point** screen displays, overlaid over the network details page.
{% hint style="info" %} **Note** In the **Create Access Point** screen above, you must enter the **step 4 VPC Interface Endpoint ID**. To get this ID, in the DeltaStream CLI follow the first two steps of the procedure below, then copy the ID and return to the **Create Access Point** screen to paste in the ID. {% endhint %} ### Creating the Private Link in the DeltaStream CLI This procedure involves building a SQL statement. When you complete and run the statement, DeltaStream processes the link request automatically. Note that the private link is not established until it is accepted or approved by administrators from your organization who are responsible for maintaining Kafka stores. 1. From the DeltaStream CLI, issue the following SQL command to create a private link for both the enterprise cluster and access to the schema registry. Paste in the endpoint service ID you just copied. {% code fullWidth="false" %} ```sql CREATE AWS PRIVATE LINK confluentent WITH ('private_link.target_type' = CONFLUENT_KAFKA, 'private_link.service_name' = '$$YOURENDPOINTSERVICE', 'private_link.hosts' ( '*.useast-1.aws.private.confluent.cloud:9092' USING PORT 9092 IN '*', '*.useast-[c]1.aws.private.confluent.cloud:443' USING PORT 443 IN '*')); ``` {% endcode %} 2. Next, verify the status of this private link. To do this, type `LIST AWS PRIVATE LINKS`.\ \ `| ID | Name | Target Type | Service Name | Status | Messages | Vpc Endpoint Id | Discovery Iam Role Arn | Created At | Updated At | Deleted At | Path |` `+--------------------------------------+-------------------+-----------------+---------------------------------------------------------+-------------+----------------------+------------------------+--------------------------------------------------------------+-----------------------------------+-----------------------------------+-------------+-----------------------+` `| 6de573d2-9635-42c8-895a-f934d7a57ecb | confluententent | confluent_kafka | com.amazonaws.vpce.us-east-2.vpce-svc-013e133da40f09f35 | in-progress | Private link pending | vpce-0c6d676494043f269 | arn:aws:iam::145624980286:role/pl-nxsxhuuwgvbmrck27e2npjl6zm | 2025-05-07 17:48:58.844 +0000 UTC | 2025-05-07 17:49:58.154 +0000 UTC | | ["confluententent"] |` 3. Go back to the Confluent console and return to the **Create access point** screen. Paste the endpoint service ID you copied earlier into the box in Step 4 of this screen.
4. Enter a name for this access point. 5. Click **Create access point**. The status of the access point displays as provisioning.
6. Return to the DeltaStream CLI and again type `list AWS PRIVATE LINKS`. The link should display as `READY` within 1-3 minutes. ```sql | ID | Name | Target Type | Service Name | Status | Messages | Vpc Endpoint Id | Discovery Iam Role Arn | Created At | Updated At | Deleted At | +--------------------------------------+-------------- +----------------- +---------------------------------------------------------+--------- +-----------+------------------------ +-------------------------------------------------------------- +----------------------------------- +-----------------------------------+-------------+ | b9bb786e-2dac-4275-8194-4f72b424414c | confluentent | confluent_kafka | com.amazonaws.vpce.us-east-1.vpce- svc-0b01079b35b08bb30 | ready | Current | vpce-087b9c93bee5343b0 | arn:aws:iam::792739327446:role/pl- xg5xq3rnvrbhlamuj5zlijcbjq | 2025-01-21 21:20:24.037 +0000 UTC | 2025-01-21 21:23:59.322 +0000 UTC | ``` The private link is now ready for you to test the connection. ### Testing the Private Link and Schema Registry Connection 1. Return to the Confluent Cloud environment. Navigate to the **enterprise cluster** details page, and verify the newly-created cluster is running.
2. Navigate to the **API Keys** section of the cluster. Then click **Create Key**.
3. In the **Select account for API Key** screen, click **My account**.
4. Click **Next** and then download and store the newly-created access key file 5. Open the download key-secret file. It should resemble the following: ``` === Confluent Cloud API key === API key: KEY.... API secret: SECRET.... Resource: lkc-p3nm1m Bootstrap server: lkc-myenterprise-east-1.aws.private.confluent.cloud:9092 ``` 6. Return to the **enterprise cluster** details page and create a schema registry credential. To do this:\ a. At the bottom right of the cluster details screen, click **+ Add Key**.\ \ \ \ b. Next, navigate to the environment's menu and click **Schema Registry**.\ c. Click **API Keys**.\ d. Click **Add API Key**. * This is separate from the Confluent cluster API key you downloaded earlier. 8. Download the **API key** and the **Secret**. #### Creating a schema registry endpoint and a new data store to connect to the Confluent enterprise cluster You do this from the DeltaStream UI. 1. Create a schema registry from the DeltaStream Web console To do this, open DeltaStream and navigate to the **Resources** page. 2. Click to activate the **Schema Registries** tab. Then click **+ Add Schema Registry**.
3. When the **Add Schema Registry** window opens, enter the desired information. 4. In the **Add One Or More URIs To Connect** box, paste in the **Schema Registry endpoint service ID**. 5. Paste in the schema registry **API Key** and **Secret**. 6. Click **Add**.
7\. Return to the **Resources** page and verify the schema registry is in the ready status.
8\. Click to activate the **Stores** tab and create a new data store. To do this, click **+ Add Store**. The **Add Store** window opens. Enter the required information: * Type in a name for the store. * In the **Add One Or More URLs To Connect** box, paste the **Bootstrap Servers**. * In the **Schema Registry** box, paste in the Schema Registry you created in Confluent.
9\. Enter the **API Key** and **Secret**. 10\. Click **Add**. The new store transitions to the `Ready` state in 1-2 minutes. #### Verifying the connection to the enterprise cluster To do this, start by adding topics to your new store. 1. When the **Resources** page redisplays, click the name of the store you just created. The **Store** details page displays. 2. Click **Add Topic** to create a new topic. The **Add Topic** window opens. In here: 1. Enter a name for the topic. 2. In **Number of Partitions** box, type **1**. 3. In **Number of Replicas** box, type **3**. 3. Click **Add**.
The new topic displays.
4. Return to the Confluent Cloud dashboard to review the enterprise cluster metadata. To do this: * Navigate to the **cluster** details page, and then click **Topics**.
The newly-created topic displays on your Confluent Cloud console. This completes the verification process for your enterprise cluster. Now verify the schema registry. To do this, you create a changelog in the AVRO file format using any existing relations or streams within DeltaStream. # Creating a Private Link from DeltaStream to Amazon MSK Below is the procedure for creating private links for the data you’re streaming via DeltaStream. It includes the configuration needed within AWS MSK. {% hint style="info" %} **Note** Currently DeltaStream supports private links only in AWS. {% endhint %} ### Before You Begin * Review [Introducing Private Links](/how-do-i.../creating-private-links/introducing-deltastream-private-links). * You must have signed up with DeltaStream and created at least one organization. Private links function within the context of an organization; from a logical perspective, you enable your private links within a specific DeltaStream organization. * [Download the DeltaStream CLI](/getting-started/starting-with-cli) if you don’t already use it. Currently you cannot create private links via the DeltaStream UI. * [Request that DeltaStream Operations enable private link](https://console.deltastream.io/support-center) connectivity for your account. * Optionally, work with DeltaStream Operations if you wish to run all your queries (that is, stream processing) within a dedicated AWS dataplane. By default all DeltaStream customer queries run in a shared multi-tenant dataplane; network policies isolate all traffic among multiple customers. This dedicated data plane completely isolates your workload from other DeltaStream customers by using fully-isolated compute and VPC networking resources. ### Creating a Private Link This procedure involves building a SQL statement. When you run the statement, DeltaStream processes the link request automatically. Note that the private link is not established until it is accepted or approved by administrators from your organization who are responsible for maintaining Kafka stores. The entire statement resembles the below; we’ll go through it segment by segment, showing how and where to find the information you need to complete the statement. {% code overflow="wrap" %} ```sql CREATE AWS PRIVATE LINK msktest WITH ('private_link.target_type' = MSK_PRIVATE_LINK, 'private_link.msk_cluster_arn' = 'arn:aws:kafka:us-east-1:123456789123:cluster/ds-cc118-stage-he17-0/349e3343-1547-458a-86ab-7d4b424288f9-8', 'private_link.msk_auth_type' = MSK_AUTH_SASL_IAM, 'private_link.hosts' ( 'b-1.iam.dscc118stagehe170.f4tzmf.c8.kafka.us-east-1.amazonaws.com:14001' USING PORT 14001 IN 'use1-az6')); ``` {% endcode %} The private link created using the above SQL command is an inbound private link connection to your dedicated Kafka cluster within your Confluent cloud account. {% hint style="info" %} **Note** There are separate but related instructions for creating private links for [Confluent Kafka](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/T5OiaOccNryRnCeMLyYw), [RDS Postgres](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/8fQ6A91V8j4OoD40nn8Y), and [Confluent Enterprise edition](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/mO5KZFCyDFxORWoS7AbA). {% endhint %} ### Section 1: Enable Multi-VPC Configuration (private link) for AWS MSK Follow the below guide to enable multi-VPC for your AWS cluster. [More details from AWS on MSK private link setup](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html). When you have enabled the private link for the MSK cluster, follow the below procedure to capture the details you need to configure the private link to MSK in the DeltaStream CLI. 1. Go to your Amazon MSK console page and navigate to **Clusters** > **$$yourclustername** to open the cluster details page. 2. In the **Cluster** summary section, locate and copy the **MSK Cluster ARN**. 3. Save the Cluster ARN to a Notepad or other text-based application. You will need to refer back to it again. 4. Note down the cluster ARN as **$$yourclusterarn**
3. Indicate the MSK auth type you must use for the connection. This varies, depending on your AWS MSK cluster setup and configuration; it could be SASL/SCRAM authentication or IAM role-based authentication. For the below example we use IAM-based authentication. 4. Capture the broker host URIs. To do this, return to the Amazon MSK **Cluster** details page, and toward the top right of the page click **View client information**. 5. In the **Private endpoint (multi-VPC)** column, multiple broker IAM endpoints display. Capture the **bootstrap URIs** next to the private endpoint column. * You must copy each of these endpoints into the SQL command you’re building in the DeltaStream UI. 6. Capture the **availability zone** associated with each broker in the MSK cluster. To do this, return to your **Cluster** details page. Below the **Cluster summary** section click **Properties** and then scroll down to the **Broker details** section. 7. Although there is an **Availability Zone** tab in this section, that is the virtual zone name, and it is not what you need. Instead, you must locate and copy the physical zone ID. To get the physical zone ID, click the corresponding link in the **Client subnets** column. (Each broker endpoint has an associated subnet.) 8. When the **Client subnet** details page displays, locate and copy the **Availability Zone ID** value for each broker subnet. ### Section 2: Writing the SQL Statement in DeltaStream to Create the Private Link 1. Launch the DeltaStream CLI. The prompt opens to your default organization. * If you’re uncertain which organization this is, type `list ORGANIZATIONS`, and in the list that displays, scan down the column labeled `is Current` and find the organization with a value of `true`. 2. Return to the prompt and specify the organization:\ `Use ORGANIZATION [$$yourorganization]` The private link connection you’re creating will be in this specific DeltaStream organization. No other DeltaStream organization has access to this private link. 3. Create the private link. To do this, update the SQL with the variables captured in **Section 1** above. 
CREATE AWS PRIVATE LINK msktest
WITH ('private_link.target_type' = MSK_PRIVATE_LINK,
      'private_link.msk.cluster.arn' = '$$Yourclusterarn',
      'private_link.msk_auth_type' = MSK_AUTH_SASL_IAM,
      'private_link.hosts' (
            '$$brokerhost1:$$brokerhost1PORT' USING PORT $$brokerhost1port IN 'brokerhost1az',
            '$$brokerhost2:$$brokerhost2PORT' USING PORT $$brokerhost2port IN 'brokerhost2az',
            '$$brokerhost3:$$brokerhost3PORT' USING PORT $$brokerhost3port IN 'brokerhost3az'));
{% code overflow="wrap" %} ```sql CREATE AWS PRIVATE LINK msktest WITH ('private_link.target_type' = MSK_PRIVATE_LINK, 'private_link.msk_cluster_arn' = 'arn:aws:kafka:us-east-1:123456789012:cluster/ds-test-express-broker/11112222-ec9e-4ffe-3333-e1750154c038-8', 'private_link.msk_auth_type' = MSK_AUTH_SASL_IAM, 'private_link.hosts' ( 'boot-qui.iam.cluster.gpzcm2.c8.kafka.us-east-1.amazonaws.com:14100' USING PORT 14100 IN '*', 'boot-st9.iam.cluster.gpzcm2.c8.kafka.us-east-1.amazonaws.com:14099' USING PORT 14099 IN '*', 'boot-32r.iam.cluster.gpzcm2.c8.kafka.us-east-1.amazonaws.com:14098' USING PORT 14098 IN '*')); ``` {% endcode %} Your SQL statement in the DeltaStream CLI is now ready. When you run the statement, DeltaStream displays a confirmation and begins creating the private link. ### Section 3 – Establishing Private Link Connectivity Between the DeltaStream Platform and the MSK Cluster To complete the private link connection you must edit your MSK cluster policy to add the Discovery IAM Role ARN that the DeltaStream platform uses. To do this: 1. In the DeltaStream CLI, list your available private links by typing\ `LIST AWS PRIVATE LINKS;` 2. Locate your newly-created `msktest` private link entry, and copy the **Discovery IAM role ARN**. 3. Navigate back to your MSK **Cluster** details page and under the **Properties** tab scroll down to the **Security settings** section and click **Edit cluster policy**. The **Edit cluster policy** page displays. 4. Replace the code following “AWS”: with the ARN you just copied. 5. Click **Save changes**.
6. Return to the DeltaStream CLI and again list the status of the private link by typing `list AWS private links`. In a few moments you should see the private link transition from in progress to ready. The cluster is ready to use as a Kafka store from within the DeltaStream platform. ### Verifying Private Link Connectivity To verify connectivity you can use the DeltaStream console UI to add a Kafka topic. 1. Launch the DeltaStream UI and create a new data store. [Follow the instructions to create a store](/how-do-i.../create-and-manage-data-stores/store), if you need. Within about a minute, topics begin to populate your data store. This indicates the private connection is successful and working. To do this you must set the number of partitions to 1 and the number of replicas to 3.
From DeltaStream you should get an alert that the operation was successful. You should also be able to see the newly-created topic in DeltaStream. # Creating a Private Link for RDS Databases RDS is used primarily for writing to a table or a changelog. Creating an RDS private link makes it possible to access a postgres store via a DeltaStream private link without exposing RDS endpoints on the public Internet. Private links are specific both to your DeltaStream organization and to your data store’s AWS region. Setup is important; the steps are exacting and involve using the DeltaStream command line interface (CLI) in tandem with AWS management console. In the DeltaStream CLI you build the SQL that instantiates the private link request from the DeltaStream platform. It supports a single RDS instance, or an RDS proxy name to which you wish to connect. Currently DeltaStream does not support multi-replica RDS clusters.. When you establish an RDS PostgreSQL private link you must accept private link requests made by DeltaStream to your AWS account. Below is the procedure for creating private links for the data you’re streaming via DeltaStream. It includes the necessary cluster configuration. {% hint style="info" %} **Note** Currently DeltaStream supports private links only in AWS. {% endhint %} ### Before You Begin * Review [Introducing Private Links](/how-do-i.../creating-private-links/introducing-deltastream-private-links). * You must have signed up with DeltaStream and created at least one organization. Private links function within the context of an organization; from a logical perspective, you enable your private links within a specific DeltaStream organization. * [Download the DeltaStream CLI](/getting-started/starting-with-cli) if you don’t already use it. Currently you cannot create private links via the DeltaStream UI. * Request DeltaStream Ops to enable private link connectivity feature for your account. * You must have the DeltaStream platform AWS account number that sends private link connectivity requests to your RDS data stores. [Contact DeltaStream support to obtain this number](https://console.deltastream.io/support-center). * Optionally, work with DeltaStream Operations if you wish to run all your queries (that is, stream processing) within a dedicated AWS dataplane. By default all DeltaStream customer queries run in a shared multi-tenant dataplane; network policies isolate all traffic among multiple customers. This dedicated data plane completely isolates your workload from other DeltaStream customers by using fully-isolated compute and VPC networking resources. ### Creating a Private Link To begin, you first establish an endpoint service for RDS. Then you build a SQL statement that includes the RDS endpoint . When you run the statement, DeltaStream processes the link request automatically. Note that the private link is not established until it is accepted or approved by administrators from your organization who are responsible for maintaining data stores. The SQL statement resembles the below: {% code overflow="wrap" %} ```sql CREATE AWS PRIVATE LINK rdsprivatelinktest WITH ('private_link.target_type' = POSTGRES, 'private_link.service_name' = '$$YOUR_ENDPOINT_SERVICE_NAME', 'private_link_hosts' ('$$RDS_HOSTNAME: $$RDS_PORT' USING PORT $$RDS_PORT IN '*')); ``` {% endcode %} {% hint style="info" %} **Note** There are separate but related instructions for creating private links for [AWS Managed Kafka (MSK)](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/ynSRO8WFpfVu7FI9Dmj5), [Confluent Kafka](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/T5OiaOccNryRnCeMLyYw), and [Confluent Enterprise edition](broken://spaces/utNVXZAEzOM5Pa3UQ5Ax/pages/mO5KZFCyDFxORWoS7AbA). {% endhint %} ### Setting up an Endpoint Service for RDS. RDS does not by default support private links. So to establish this connection with DeltaStream, you must instead set up an endpoint service on top of the RDS endpoint. See [Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more information and details on how to do this yourself. ### Writing the SQL Statement in DeltaStream to Create the Private Link to RDS After you have set up the endpoint service, gather the RDS hostname and RDS Endpoint Service name and insert the variables: * if your `RDS_ENDPOINT_SERVICE_NAME` is `‘com.amazonaws.vpce.us-east-1.vpce-svc-123456789aa0112'` * and your `RDS_HOST_NAME` is `ds-myrds-stage-xyz1-db-0.abcdefgh1239.us-east-1.rds.amazonaws.com` * and your `RDS_PORT` is `5432` then your statement will resemble the following (sections in bold are those you get from your RDS console): 
CREATE AWS PRIVATE LINK rdsprivatelinktest 
WITH ('private_link.target_type' = POSTGRES, 
      'private_link.service_name' = 'com.amazonaws.vpce.us-east-1.vpce-svc-123456789aa0112', 
      'private_link.hosts' (
            'ds-myrds-stage-xyz1-db-0.abcdefgh1239.us-east-1.rds.amazonaws.com:5432' 
    USING PORT 5432 IN '*'));
After you complete and run the statement, verify the status of the link. To do this, wait a few moments, then type ```sql LIST AWS PRIVATE LINKS; ``` The link you just created displays a list of private links with their current status. ### Accepting the Private Link Request to the RDS Endpoint Service When your link changes to the following status: `“Warning: Private link service not found, was the principal accepted?` you must manually accept the principal that the DeltaStream platform uses to initiate the private link connection. Here’s how to do this: 1. From the DeltaStream CLI, copy the **Discovery IAM Role ARN** that displays next to the private link status when you issue issue the list aws private links command. 2. Switch to the AWS console session in which you manage RDS and VPC for your AWS account. 3. Navigate to the **Endpoint services** page. 4. Click to activate the **Allow principals** tab.
5. In the **Allow principles** section, above the righthand column click **Allow principals**.
6. When the **Allow principals** screen displays, paste in the **Discovery ARN** you just copied. 7. Click **Allow Principals**. 8. Return to the DeltaStream CLI and verify the status of the RDS link is **pending acceptance**. To do this, type `list aws private links.`
9. Return to the AWS console for the RDS Endpoint Services and click to activate the **Endpoint Connection** tab. 10. Click **Accept Endpoint Connection Request**.
11. Wait until the link is ready. To do this, return to the DeltaStream CLI and again type `list aws private links`. The status should change to READY within 4-5 minutes. ### Verifying Private Link Connectivity To verify connectivity, use the DeltaStream console UI to create a new Postgres data store, then drill down to a table and verify that data is streaming into it. To do this: 1. In DeltaStream navigate to **Resources** > **Stores**. 2. Click **+ Add Store**, and from the list that displays click **PostgreSQL**. 3. When the **Add Store** window displays, enter the required information. 4. Enter a valid username and password for testing RDS connectivity. Your RDS Postgres administrator can provide this. {% hint style="warning" %} **Important** In the **Add One Or More URIs To Connect** box, preface the RDS hostname with `postgres//`. Then at the end of the hostname append the port number. Do not include horizontal spaces in either case. For example: `postgres://ds-myrds-stage-xyz1-db-0.1234567891234.us-east-1.res.amazonaws.com:5432`. {% endhint %}
5. Click **Add**. The **Resources** page redisplays, listing all of your DeltaStream data stores. Locate the store you just created. * If you have already many stores you may need to search for it using the search box toward the top right of the screen. Then click your store to display the store details page. * If the store status is **Verifying**, wait a minute or two and refresh the page. The store should display with a status of **Ready**. 6. Click the store to display a schema list.
7. Click a schema to display a list of tables.
8. Click a table to display the table details page. 9. Click **Print** to display data coming into the table.
Incoming data verifies that you are connected. # Deleting a Private Link ### Deleting a Private Link Any time your private link is going to be non-operational – for example, if you’ve created a test cluster and private link before creating one in production – it’s best to delete the link. **To delete a private link** From the DeltaStream CLI, at the prompt type ```sql DROP AWS PRIVATE LINK $$yourprivatelinkname; ``` It may take a few moments for DeltaStream to remove the related infrastructure. {% hint style="success" %} **Tip** If you drop a private link, the data stores that used it remain listed in DeltaStream, but they do not load any topics and are not functional. It’s best to delete any data stores that use a private link that you have deleted. {% endhint %} # Serialize my Data The below articles provide background and information on setting up and configuring what you need to convert and consume complex data. {% content-ref url="/pages/S2pwtmAMA2TymCOBVQ6M" %} [Working with ProtoBuf Serialized Data and DeltaStream Descriptors](/how-do-i.../serialization/working-with-protobuf-serialized-data-and-deltastream-descriptors) {% endcontent-ref %} {% content-ref url="/pages/w8ZMRzX95ncwuhIVk97B" %} [Working with Avro Serialized Data and Schema Registries](/how-do-i.../serialization/working-with-avro-serialized-data-and-schema-registries) {% endcontent-ref %} {% content-ref url="/pages/dAaxWLYxuFYAaz5kgy6C" %} [Configuring Deserialization Error Handling](/how-do-i.../serialization/configuring-deserialization-error-handling) {% endcontent-ref %} {% content-ref url="/pages/qYbMKliMKIuoFWLEoLAW" %} [Data Formats (Serialization)](/reference/sql-syntax/data-format-serialization) {% endcontent-ref %} # Working with ProtoBuf Serialized Data and DeltaStream Descriptors In streaming data stores such as [Apache Kafka](https://kafka.apache.org/) and [Amazon Kinesis](https://aws.amazon.com/kinesis/), producers send data events as bytes that consumers of the data must interpret. The most popular formats for data serialization include [JSON](https://www.json.org/json-en.html), [ProtoBuf](https://protobuf.dev/), and [Apache Avro](https://avro.apache.org/docs/), and DeltaStream supports all of these. This article focuses on [ProtoBuf](https://protobuf.dev/) and how to create and use a [Descriptor](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) for data serialization/deserialization. Begin with a [Data Store](/overview/core-concepts/store) with entities whose data records are serialized with ProtoBuf. This means you also have ProtoBuf messages and file descriptors to serialize and deserialize these data events. In DeltaStream, you can create a DeltaStream [Descriptor](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) -- which is a wrapper around your ProtoBuf file descriptor -- and associate it with any [Data Store](/overview/core-concepts/store#entity) that requires the ProtoBuf file descriptor for serialization/deserialization. ## Create a descriptor When working with ProtoBuf, you first define a ProtoBuf message and then generate a ProtoBuf file descriptor from that message. DeltaStream then uses this ProtoBuf file descriptor to generate any code necessary for serializing and deserializing data that conforms to the ProtoBuf message structure. In this example, the ProtoBuf message, which lives in the file `p.proto`, resembles the following: ```protobuf message Pageviews { int64 viewtime = 1; string userid = 2; string pageid = 3; } ``` You can generate a ProtoBuf descriptor in the file `pageviews_value.desc` from this ProtoBuf message in the file `p.proto` (see [ProtoBuf documentation](https://protobuf.dev/programming-guides/techniques/#self-description) for more details): ```sh $ protoc --descriptor_set_out pageviews_value.desc p.proto ``` Now create a DeltaStream [Descriptor](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) from this ProtoBuf file descriptor. In the CLI you can do this using the [CREATE DESCRIPTOR\_SOURCE](/reference/sql-syntax/ddl/create-descriptor_source) DDL. In the UI, follow these steps to add a descriptor: 1. In the lefthand navigation click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ). When the **Resources** page displays, click **Descriptor Sources** and then click **+ Add Descriptor Source**.
2. Choose the file containing your ProtoBuf file descriptor (`pageviews-descriptor` in this example). When prompted, name your descriptor, and then click **UPLOAD**.
3. Now you can click on the descriptor to view the message names it contains (in this example, `Pageviews`).
## Update an Entity with the Descriptor Now you can associate your descriptor with any relevant [Data Store](/overview/core-concepts/store#entity) that needs it for serialization/deserialization. To do this in the CLI, see [UPDATE ENTITY](/reference/sql-syntax/ddl/update-entity). For the UI, follow these steps: 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display the **Resources** page 2. Click the data store you want, and when the data store page displays, click the topic you want. In the example below, we selected the `KafkaStore` containing the ProtoBuf entity `pageviews_key_descriptor`:
3. As this is a Kafka data store, it allows for keys and enables you to assign a **Key Descriptor** and/or a **Value Descriptor.** For data stores that don’t allow for keys, including Kinesis data stores, you can only add a value descriptor.
4. Click **+ Add Descriptors**, and from the menus that display click the relevant descriptors to assign to this entity. In this example the `Pageviews` value descriptor is assigned, and the `Key Descriptor` is empty.
5. That's it. You've assigned your descriptor to the relevant entity; now you can use this entity to run commands (such as [PRINT ENTITY](/reference/sql-syntax/command/print-entity)) and queries with DeltaStream objects. For more information, see [Serializing with Protobuf](/reference/sql-syntax/data-format-serialization/serializing-with-protobuf). ## Queries with Descriptors and ProtoBuf With descriptors added, you can now create a DeltaStream object that specifies a `key.format` or `value.format` of `PROTOBUF` as shown in the below DDL example. See [CREATE STREAM](/reference/sql-syntax/ddl/create-stream) for more details. ```sql CREATE STREAM "pageviewsPB" (viewtime BIGINT, userid VARCHAR, pageid VARCHAR) WITH ('topic'='pageviews_pb', 'value.format'='PROTOBUF'); ``` You can also create new objects using [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as) or [CREATE CHANGELOG AS SELECT](/reference/sql-syntax/query/create-changelog-as), specifying `PROTOBUF` as the data format for the sink object. The below example shows how you can easily convert the JSON stream `pageviews_json` to a stream called `pageviews_converted_to_proto` with a ProtoBuf key and value format. ```sql CREATE STREAM pageviews_converted_to_proto WITH ( 'value.format' = 'protobuf', 'key.format' = 'PROTOBUF' ) AS SELECT * FROM pageviews_json; ``` When the sink object has a key or value format of `PROTOBUF`, the descriptor for the sink object is automatically created and assigned to the entity. You can easily view your descriptors in the **Descriptors** tab or use the [LIST DESCRIPTORS](/reference/sql-syntax/command/list-descriptors) command in the CLI. To use the descriptor outside of DeltaStream, you can download the ProtoBuf descriptor via the [COPY DESCRIPTOR\_SOURCE](/reference/sql-syntax/command/copy-descriptor_source) command. Finally, with regard to the `PRINT ENTITY` command: * If an entity in a data store has a descriptor, the descriptor is for deserialization even if the data store has a schema registry. * If the entity does not have a descriptor, the data store checks whether the schema registry contains a schema for the entity, and uses it for deserialization. * If the entity doesn’t have a descriptor and the data store doesn’t have a schema registry—or it has a schema registry, but there is no corresponding schema in the registry—DeltaStream attempts to deserialize the data in the entity as JSON. # Working with Avro Serialized Data and Schema Registries In streaming data stores such as [Apache Kafka](https://kafka.apache.org/) and [Amazon Kinesis](https://aws.amazon.com/kinesis/), producers send data events as bytes that must be interpreted by the data consumers. The most popular formats for data serialization include [JSON](https://www.json.org/json-en.html), [ProtoBuf](https://protobuf.dev/), and [Apache Avro](https://avro.apache.org/docs/). DeltaStream supports all of these. This article focuses on [Apache Avro](https://avro.apache.org/docs/). It shows how to create and use a [Data Store](/overview/core-concepts/store#schema-registry) to access [Avro schemas](https://avro.apache.org/docs/#schemas) that are necessary for data serialization/deserialization. If you have a [Data Store](/overview/core-concepts/store) with **entities** whose data records are serialized with Avro, it's likely you also have a schema registry to manage the Avro schemas that are used to serialize and deserialize these data events. To do this in DeltaStream, you create a DeltaStream [Data Store](/overview/core-concepts/store#schema-registry) entity -- which is a wrapper around your schema registry -- and associate it with one or many data stores. ## Create a Schema Registry DeltaStream supports 2 types of schema registries (more are in development): 1. Confluent Cloud 2. Confluent Platform To begin, create a schema registry in either the CLI or the UI SQL page. In the CLI, use the [CREATE SCHEMA\_REGISTRY](/reference/sql-syntax/ddl/create-schema_registry) command to create a DeltaStream [Data Store](/overview/core-concepts/store#schema-registry): ```sql CREATE SCHEMA_REGISTRY "ConfluentCloudSR" WITH ( 'type' = CONFLUENT_CLOUD, 'uris' = 'https://abcd-efghi.us-east-2.aws.confluent.cloud', 'confluent_cloud.key' = 'fake_key', 'confluent_cloud.secret' = 'fake_secret' ); ``` In the above example, you're creating a `CONFLUENT_CLOUD` type schema registry named `ConfluentCloudSR` in `AWS us-east-2`. The `uris` value provided here is the URI from the Confluent Cloud dashboard, corresponding with the schema registry. Optionally, this schema registry can also have one of the following: * a key pair for credentials, which can be supplied with the `confluent_cloud.key` and `confluent_cloud.secret` properties * the `properties.file` property (see [CREATE SCHEMA\_REGISTRY](/reference/sql-syntax/ddl/create-schema_registry) for more details) When you have defined schema registries, you can list them: ```sh demoDB.public/kafka_store# LIST SCHEMA_REGISTRIES; +------------------+----------------+--------+--------------------------------------------------+----------+-----------------------------------+-----------------------------------+----------------------+ | Name | Type | State | Uris | Owner | Created At | Updated At | Path | +------------------+----------------+--------+--------------------------------------------------+----------+-----------------------------------+-----------------------------------+----------------------+ | ConfluentCloudSR | ConfluentCloud | ready | https://abcd-efghi.us-east-2.aws.confluent.cloud | sysadmin | 2025-05-02 16:25:27.771 +0000 UTC | 2025-05-02 16:25:27.771 +0000 UTC | ["ConfluentCloudSR"] | +------------------+----------------+--------+--------------------------------------------------+----------+-----------------------------------+-----------------------------------+----------------------+ ``` ## Update Data Store with the Schema Registry Next, associate that schema registry with the relevant data store. {% hint style="info" %} **Note** You can attach one schema registry to a data store, but any number of data stores can use a particular schema registry. {% endhint %} You can describe a data store to determine whether it has a schema registry attached: ```sh demoDB.public/kafka_store# DESCRIBE STORE kafka_store; +-------------+-------------------------------------------------+----------+--------------+------------------+------------------+-----------------+ | Properties | Uri | Details | Tls Enabled | Verify Hostname | Schema Registry | Path | +-------------+-------------------------------------------------+----------+--------------+------------------+------------------+-----------------+ | {} | abcd.edghijk.kafka.us-east-1.amazonaws.com:9196 | {} | true | false | | ["kafka_store"] | +-------------+-------------------------------------------------+----------+--------------+------------------+------------------+-----------------+ ``` When you determine there is no schema registry attached to your data store, (`kafka_store`, above) you can use the [UPDATE STORE](/reference/sql-syntax/ddl/update-store) DDL to point your data store to the schema registry. ```sql UPDATE STORE kafka_store WITH ( 'schema_registry.name' = ConfluentCloudSR ); ``` When you attach a schema registry to a data store, then each time a DeltaStream command or query requires serializing or deserializing data from the data store’s entity, that data store uses the registry to look up schemas. When you work with Avro serialized data, DeltaStream requires that you attach the schema registry with the relevant Avro schemas to the data store containing that data. ## Avro-Enabled Printing and Queries Now you can successfully run commands such as [PRINT ENTITY](/reference/sql-syntax/command/print-entity) or write queries with DeltaStream objects that have Avro data formats. Below we have printed the `pageviews_avro` entity. ```sh demoDB.public/kafka_store# PRINT ENTITY pageviews_avro; +----------------------+----------------------------------------------------------------------+ | key | value | +======================+======================================================================+ | {"userid": "User_1"} | {"viewtime": 1746203009403, "userid": "User_1", "pageid": "Page_64"} | +----------------------+----------------------------------------------------------------------+ | {"userid": "User_4"} | {"viewtime": 1746203010404, "userid": "User_4", "pageid": "Page_45"} | +----------------------+----------------------------------------------------------------------+ | {"userid": "User_2"} | {"viewtime": 1746203011404, "userid": "User_2", "pageid": "Page_23"} | +----------------------+----------------------------------------------------------------------+ | {"userid": "User_9"} | {"viewtime": 1746203012404, "userid": "User_9", "pageid": "Page_48"} | +----------------------+----------------------------------------------------------------------+ | {"userid": "User_7"} | {"viewtime": 1746203013405, "userid": "User_7", "pageid": "Page_60"} | ``` When you have set up a schema registry, you can easily read or write Avro-formatted data. The query below shows how to easily convert the JSON stream `pageviews_json` to a stream with an Avro key and value format called `pageviews_converted_to_avro`. See [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as) for more details. ```sql CREATE STREAM pageviews_converted_to_avro WITH ( 'key.format' = 'avro', 'value.format' = 'avro' ) AS SELECT * FROM pageviews_json; ``` When you create DeltaSTream objects using [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as) or [CREATE CHANGELOG AS SELECT](/reference/sql-syntax/query/create-changelog-as), and you specify a key or value format such as Avro for the sink object, DeltaStream automatically generates an Avro schema and adds it to the schema registry attached to the data store. In the example above, two schemas are generated for the entity `pageviews_converted_to_avro` -- one each for the key and the value. In this way these schemas are available if you ever need to consume from these entities outside of DeltaStream. In the case of the `PRINT ENTITY` command, if an entity in a data store has a [Data Formats (Serialization)](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors), the descriptor is used for deserialization even if the data store has a schema registry. If the entity does not have a descriptor, the data store verifies that the schema registry contains a schema for the entity and then uses it for deserialization. Finally, if the entity doesn’t have a descriptor, and the data store doesn’t have a schema registry (or it has a schema registry, but there is no corresponding schema in the registry), DeltaStream tries to deserialize the data in the entity as JSON. # Configuring Deserialization Error Handling Data quality is critical with streaming and stream processing applications. A single bad record can cause multiple issues. Ideally, all records in a message broker are well formed. However, issues can occur, such as the accidental publishing of unrecognized payloads into a Kafka topic, or a memory error causing corruption of a record. Streaming applications that attempt to consume these records fail to deserialize them; in turn this usually causes the application to fail. This article shows different ways you can configure your queries to handle such failures in DeltaStream. To configure deserialization error handling, provide the `source.deserialization.error.handling` property in the `WITH` clause for a query's sources. Choose from 3 modes (documentation for these: [FROM](/reference/sql-syntax/query/select/from#_source_properties)): * `TERMINATE` * `IGNORE` * `IGNORE_AND_LOG` The following sections detail each of these modes for a scenario in which a malformed record made its way into a Kafka topic, which will be the source for one of your DSQL queries. Assume the following `pageviews` stream defined by the DDL below: ```sql CREATE STREAM pageviews ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ('store'='kafka_store', 'topic'='pageviews', 'value.format'='JSON'); ``` The `pageviews` topic contains the following records: ```json { "viewtime":1660931394412, "userid":"User_1", "pageid":"Page_22" } { "viewtime":"malformed_viewtime", "userid":"User_6", "pageid":"Page_32" } { "viewtime":1660931396413, "userid":"User_2", "pageid":"Page_96" } ``` ## TERMINATE Mode The `TERMINATE` mode is the default option, and the strictest. Any records the query fails to deserialize cause the query to terminate. Here's a query using the `source.deserialization.error.handling` property default: ```sql CREATE STREAM pageviews_copy AS SELECT * FROM pageviews; ``` Here's a query explicitly setting the `source.deserialization.error.handling` property to `TERMINATE`: ```sql CREATE STREAM pageviews_copy AS SELECT * FROM pageviews WITH ('source.deserialization.error.handling'='TERMINATE'); ``` The output of these queries contains the following records: ```json { "viewtime":1660931394412, "userid":"User_1", "pageid":"Page_22" } ``` Note that only one of the three input records shows up in the output. As the first record is well-formed JSON and matches the value types specified in the `pageviews` DDL, the query successfully deserializes that record and produces the output. The second record, however, has an unexpected value for the `viewtime` column: the string `"malformed_viewtime"` when the query is expected to deserialize that field as `BIGINT`. This causes a deserialization error for the source. As per the `TERMINATE` error handling mode, this error causes the entire query to fail and halts any further progress. ## IGNORE Mode Use the `IGNORE` mode to instruct queries to continue if they encounter deserialization errors. With the `IGNORE` mode, the system skips any records that fail to be deserialized from a query's source. Query setting the `source.deserialization.error.handling` property to `IGNORE`: ```sql CREATE STREAM pageviews_copy AS SELECT * FROM pageviews WITH ('source.deserialization.error.handling'='IGNORE'); ``` The output of this query contains the following records: ```json { "viewtime":1660931394412, "userid":"User_1", "pageid":"Page_22" } { "viewtime":1660931396413, "userid":"User_2", "pageid":"Page_96" } ``` Two of the three input records display in the output. As discussed in the previous section, the first and third records are well-formed JSON records that the query successfully deserialized. The second record failed deserialization. As the source uses the `IGNORE` error handling mode, the second record is dropped and does not display in the output. The query meanwhile continues processing records, so the third record displays in the output. Unlike with the `TERMINATE` mode, the query doesn't fail; it continues to process any additional records in the source. ## IGNORE\_AND\_LOG Mode The `IGNORE_AND_LOG` mode is similar to the `IGNORE` mode in that it skips any records the query fails to deserialize. However, in this mode the skipped records are also logged. It's useful to log these records if you wish to know which payloads were problematic, so that you might replay or act upon any skipped records. The error log for this mode is specified by the required `source.deserialization.error.log.topic` and optional `source.deserialization.error.log.store` properties, plus any store-specific properties. DeltaStream uses the error log topic, if that topic already exists. If the topic does not exist, DeltaStream creates it according to the additional store-specific properties listed below. If you do not specify the error log store, then by default DeltaStream uses the store in which the source DeltaStream object belongs. {% hint style="info" %} **Note** "Store" and "data store" are the same. In the DeltaStream UI it's "data store;" in the CLI it's "store." {% endhint %} The additional properties for Kafka: * `source.deserialization.error.log.topic.partitions` * `source.deserialization.error.log.topic.replicas` The additional properties for Kinesis: * `source.deserialization.error.log.topic.shards` ### Kafka source object This subsection focuses on how `IGNORE_AND_LOG` behaves for a query with the source object backed by a Kafka topic. You can reuse your `pageviews` stream in this example, as it is associated with the Kafka data store named `kafka_store`. Here's a query setting the `source.deserialization.error.handling` property to `IGNORE_AND_LOG`: ```sql CREATE STREAM pageviews_copy AS SELECT * FROM pageviews WITH ( 'source.deserialization.error.handling'='IGNORE_AND_LOG', 'source.deserialization.error.log.topic'='pageviews_copy_errlog', 'source.deserialization.error.log.topic.partitions'=1, 'source.deserialization.error.log.topic.replicas'=3, 'source.deserialization.error.log.store'='kafka_store' ); ``` If in setting the `source.deserialization.error.log.store` property you don't specify a particular store for the error log topic to use , the default is to use the same store as the `pageviews` stream -- in this case, `kafka_store`. The output of this query contains the following records: ```json { "viewtime":1660931394412, "userid":"User_1", "pageid":"Page_22" } { "viewtime":1660931396413, "userid":"User_2", "pageid":"Page_96" } ``` The error log topic `pageviews_copy_errlog` contains the following records: ```json { "partition": 0, "offset": 1, "topic": "pageviews", "store": "store_test", // "key" field would also be here if the source record contained a key "value": "eyAidmlld3RpbWUiOiJtYWxmb3JtZWRfdmlld3RpbWUiLCAidXNlcmlkIjoiVXNlcl82IiwgInBhZ2VpZCI6IlBhZ2VfMzIiIH0=", "timestamp": 1693341135576 } ``` Similarly to the `IGNORE` case, only the first and third input records display in the output. The second record failed deserialization and is skipped in the output, but information about this record displays in the error log. This information includes common metadata such as the original record's partition, offset, topic, store, and timestamp. Note that the `value` field has been encoded with [base64](https://developer.mozilla.org/en-US/docs/Glossary/Base64). Since the query could not deserialize the original record, the Kafka record's entire value byte array is encoded using base64 and that string is included in the JSON error log payload. In this example, the original Kafka record doesn't contain a key; if it did, it would also be encoded using base64 and there would be a similar `key` field in the JSON payload. To validate that the `value` in the error log is what you expect, manually decode the field: ```sh $ echo "eyAidmlld3RpbWUiOiJtYWxmb3JtZWRfdmlld3RpbWUiLCAidXNlcmlkIjoiVXNlcl82IiwgInBhZ2VpZCI6IlBhZ2VfMzIiIH0=" | base64 -d | jq . { "viewtime": "malformed_viewtime", "userid": "User_6", "pageid": "Page_32" } ``` ### Kinesis source object This subsection focuses on how `IGNORE_AND_LOG` behaves for a query with the source object backed by a Kinesis stream. Consider a new `pageviews_kinesis` stream created by the following DDL: ```sql CREATE STREAM pageviews_kinesis ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ('store'='kinesis_store', 'topic'='pageviews', 'value.format'='JSON'); ``` Here's a query setting the `source.deserialization.error.handling` property to `IGNORE_AND_LOG`: ```sql CREATE STREAM pageviews_copy AS SELECT * FROM pageviews_kinesis WITH ( 'source.deserialization.error.handling'='IGNORE_ANG_LOG', 'source.deserialization.error.log.store'='alt_kinesis_store', 'source.deserialization.error.log.topic'='pageviews_copy_errlog', 'source.deserialization.error.log.topic.shards'=1 ); ``` Here you're setting the `source.deserialization.error.log.store` to `alt_kinesis_store` to specify a particular store for the error log topic. The output of this query contains the following records: ```json { "viewtime":1660931394412, "userid":"User_1", "pageid":"Page_22" } { "viewtime":1660931396413, "userid":"User_2", "pageid":"Page_96" } ``` The error log topic `pageviews_copy_errlog` contains the following records: ```json { "partitionKey": "some_partition_key", "seqNum": "49644062644797676009557574763794547223012948382276648962", "shardId": "shardId-000000000000", "store": "store_test", "stream": "pageviews_kinesis", "data": "eyAidmlld3RpbWUiOiJtYWxmb3JtZWRfdmlld3RpbWUiLCAidXNlcmlkIjoiVXNlcl82IiwgInBhZ2VpZCI6IlBhZ2VfMzIiIH0=", "timestamp": 1693341135576 } ``` As with the `IGNORE` case, only the first and third input records display in the output. The second record failed deserialization and is skipped in the output, but information about that record appears in the error log. This includes common metadata such as the original record's partition key, sequence number, shard ID, store, and timestamp. Note that the `data` field has been encoded with [base64](https://developer.mozilla.org/en-US/docs/Glossary/Base64). Since the query could not deserialize the original record, the Kinesis stream's entire data byte array is encoded using base64 and that string is included in the JSON error log payload. To validate that the `data` in the error log is what you expect, manually decode the field: ```sh $ echo "eyAidmlld3RpbWUiOiJtYWxmb3JtZWRfdmlld3RpbWUiLCAidXNlcmlkIjoiVXNlcl82IiwgInBhZ2VpZCI6IlBhZ2VfMzIiIH0=" | base64 -d | jq . { "viewtime": "malformed_viewtime", "userid": "User_6", "pageid": "Page_32" } ``` # CDC Pipeline Using a Single Slot Debezium can capture change data (CDC) from multiple tables within the same PostgreSQL database using just **one replication slot**. This means it doesn’t need to create a separate slot for each table - it streams all table changes through a single slot, which reduces overhead and makes replication more efficient. Instead of maintaining one replication slot per table, all relevant changes are multiplexed through a single slot, minimizing replication slot overhead, simplifying resource management, and ensuring consistent log consumption across the subscribed tables. ## Pipeline creation in DeltaStream ### Example Scenario Create a CDC pipeline that captures changes from two PostgreSQL tables—`public.customers` and `test.users`—and writes each table’s changes into a corresponding Snowflake table. ### Step 1 Run DDL to define a single CDC source to read changes that Debezium captured for source tables. When capturing changes from multiple tables with different schemas, the `before` and `after` fields, representing the state of a row before and after a change, are defined as `BYTES`. This choice allows flexibility, as using a fixed schema would not accommodate the variety in column names and data types across different source tables. The `BYTES` data type enables a generic representation of these changes until they are transformed and written into a destination with defined schemas. ```sql CREATE STREAM cdc_raw_source ( op VARCHAR, ts_ms BIGINT, `before` BYTES, `after` BYTES, `source` STRUCT<`schema` VARCHAR, `table` VARCHAR> ) WITH ( 'store'='postgres_store', 'value.format'='json', 'postgresql.cdc.table.list'='public.customers,test.users'); ``` When capturing changes from multiple tables, set the `postgresql.cdc.table.list` source property to a comma-separated list of fully qualified table names in Postgres in the form `..` ### Step 2 Run query to write CDC records for all tables into a shared Kafka topic. ```sql CREATE STREAM cdc_raw_sink WITH ('store'='kafka_store', 'topic.partitions' = 1, 'topic.replicas' = 3) AS SELECT op, source->`schema` AS src_schema, source->`table` AS src_table, ts_ms, `before`, `after` FROM cdc_raw_source WITH ('postgresql.slot.name' = 'cdc') WHERE after IS NOT NULL; ``` ### Step 3 Run one query per table to write its captured changes into its own dedicated Kafka topic. 
CREATE STREAM cdc_raw_customers WITH ('store'='kafka_store', 'topic.partitions' = 1, 'topic.replicas' = 3) AS
SELECT * FROM cdc_raw_sink WITH ('starting.position'='earliest')
WHERE src_schema = 'public' AND src_table = 'customers';
{% code overflow="wrap" %} ```sql CREATE STREAM cdc_raw_users WITH ('store'='kafka_store', 'topic.partitions' = 1, 'topic.replicas' = 3) AS SELECT * FROM cdc_raw_sink WITH ('starting.position'='earliest') WHERE src_schema = 'test' AND src_table = 'users'; ``` {% endcode %} ### Step 4 you can use "Generate Stream DDL" command to obtain the tables' schema, if the input tables' schema is complex. {% code overflow="wrap" %} ```sql GENERATE STREAM DDL customers_tb with ('store'='postgres_store', 'value.format'='json','postgresql.table.name' = 'customers', 'postgresql.schema.name' = 'public', 'postgresql.db.name' = 'dbname'); ``` {% endcode %} 
GENERATE STREAM DDL users_tb with ( 'store'='postgres_store', 
  'value.format'='json','postgresql.table.name' = 'users', 'postgresql.schema.name' = 'test', 'postgresql.db.name' = 'dbname');
### Step 5 Run a DDL per topic to define table specific CDC source for the table. ```sql CREATE STREAM customers_cdc ( op STRING, ts_ms BIGINT, "before" STRUCT< `id` BIGINT, first_name STRING, last_name STRING, `email` STRING, biography STRING>, "after" STRUCT< `id` BIGINT, first_name STRING, last_name STRING, `email` STRING, biography STRING> ) WITH ( 'value.format' = 'json', 'store' = 'kafka_store', 'topic' = 'cdc_raw_customers' ); ``` ```sql CREATE STREAM users_cdc ( op STRING, ts_ms BIGINT, "before" STRUCT< uid VARCHAR, `name` VARCHAR, city VARCHAR, balance BIGINT>, "after" STRUCT< uid VARCHAR, `name` VARCHAR, city VARCHAR, balance BIGINT> ) WITH ( 'value.format' = 'json', 'store' = 'kafka_store', 'topic' = 'cdc_raw_users' ); ``` ### Step 6 Write a query for each source to sink its changes into a Snowflake table ```sql CREATE TABLE customers_cdc_sflk WITH ( 'store' ='snowflake_store', 'snowflake.db.name'='sflk_db', 'snowflake.schema.name'='PUBLIC' ) AS SELECT after->id AS id, after->first_name AS first_name, after->last_name AS last_name, after->`email` AS `email`, after->biography AS biography, ts_ms AS event_write_time, op FROM customers_cdc WITH ('starting.position'='earliest'); ``` ```sql CREATE TABLE users_cdc_sflk WITH ( 'store' ='snowflake_store', 'snowflake.db.name'='sflk_db', 'snowflake.schema.name'='PUBLIC' ) AS SELECT after->uid AS uid, after->`name` AS `name`, after->city AS city, after->balance AS balance, ts_ms AS event_write_time, op FROM users_cdc WITH ('starting.position'='earliest'); ``` # Call DeltaStream’s REST API This page describes how you can use DeltaStream's REST API from your applications to submit statements and run queries. DeltaStream exposes a single **POST** endpoint for executing SQL statements: ```bash POST https:///v2/statements Content-Type: application/json Authorization: Bearer Body: {"statement": ""} ``` Some statements return **rows** (e.g., `SELECT`), others return only **acknowledgements** (e.g., `CREATE MATERIALIZED VIEW`). The response is JSON and easy to parse. *** ### 1) Create an API token 1. In the DeltaStream web UI, go to **Integration → API Tokens**. 2. Click **Create**, give the token a name, choose a **Role** (this controls permissions), and click **Create** again. 3. **Copy the token once** and store it securely (you won’t be able to view it again). > Tip: Use the least-privileged role that still lets your statements succeed.
*** ### 2) Find your API endpoint In the DeltaStream UI, click the **? (Help)** icon (bottom-left). Copy the **API ENDPOINT** for your org. See the screenshot below for details.
*** ### 3) Make a request (recommended CURL patterns) #### Use environment variables (safer) ```bash export DS_ENDPOINT="https://api-.deltastream.io/v2/statements" export DS_TOKEN="" # Example: query a materialized view for a specific passenger curl -s "$DS_ENDPOINT" \ -H "Authorization: Bearer $DS_TOKEN" \ -H "Content-Type: application/json" \ -d '{"statement":"SELECT * FROM flightdb.public.disrupted_premium_passengers WHERE passenger_id = '\''PASS-2451'\'';"}' | jq ``` **Why the funny quoting?**\ Inside a single-quoted bash string, use `'\''` to embed a single quote in SQL. If you don’t like that, use a heredoc: ```bash read -r -d '' SQL <<'SQL' SELECT * FROM flightdb.public.disrupted_premium_passengers WHERE passenger_id = 'PASS-2451'; SQL curl -s "$DS_ENDPOINT" \ -H "Authorization: Bearer $DS_TOKEN" \ -H "Content-Type: application/json" \ -d "{\"statement\":\"$SQL\"}" | jq ``` **Windows PowerShell** ```bash $DS_ENDPOINT = "https://api-.deltastream.io/v2/statements" $DS_TOKEN = "" $SQL = @" SELECT * FROM flightdb.public.disrupted_premium_passengers WHERE passenger_id = 'PASS-2451'; "@ $Body = @{ statement = $SQL } | ConvertTo-Json -Compress Invoke-RestMethod -Method Post -Uri $DS_ENDPOINT -Headers @{ Authorization = "Bearer $DS_TOKEN" } -ContentType "application/json" -Body $Body ``` *** ### Sample JSON response (trimmed) ```json { "createdOn": 1760293498, "data": [ [ "PASS-2451", "Mary Smith", "GOLD", "UA387", "DELAYED", "JFK", "ATL", "2025-10-12 02:41:12.828", "AISLE" ] ], "metadata": { "columns": [ {"name":"passenger_id","nullable":false,"type":"VARCHAR"}, {"name":"full_name","nullable":false,"type":"VARCHAR"}, {"name":"loyalty_tier","nullable":false,"type":"VARCHAR"}, {"name":"flight_id","nullable":false,"type":"VARCHAR"}, {"name":"flight_status","nullable":false,"type":"VARCHAR"}, {"name":"origin","nullable":false,"type":"VARCHAR"}, {"name":"destination","nullable":false,"type":"VARCHAR"}, {"name":"disruption_timestamp","nullable":false,"type":"TIMESTAMP"}, {"name":"seating_preference","nullable":false,"type":"VARCHAR"} ], "context": { "databaseName":"Proactivecsa", "organizationID":"019718d7-1860-7f76-833d-1b0237282b6a", "roleName":"sysadmin", "schemaName":"public", "storeName":"KafkaCluster1" }, "encoding":"json", "partitionInfo":[{"rowCount":1}] }, "sqlState": "00000", "statementID": "0199d9ab-2d72-7841-8d02-c0ca23a27c04" } ``` #### How to read this * **data**: array of rows (each row is an array aligned to **metadata.columns**). * **metadata.columns**: names, types, and nullability for each column in order. * **sqlState**: SQL status code (`00000` indicates success). * **statementID**: server-side identifier for auditing/debugging. * **createdOn**: request processing timestamp (epoch seconds). If your statement doesn’t return rows (e.g., `CREATE MATERIALIZED VIEW`, `INSERT`, `REFRESH`), **data** may be empty while `sqlState` still indicates success. *** ### Common pitfalls & fixes * **401/403**: invalid/missing token or insufficient role permissions. Regenerate or change role. * **400**: malformed JSON (check quoting) or invalid SQL. * **Wrong endpoint**: ensure you’re hitting `.../v2/statements` on your org’s API domain. * **Quoting issues**: prefer heredocs or use the environment-variable pattern above. *** ### Quick JS example (for agents/actions) ```javascript async function runStatement(sql) { const res = await fetch("https://api-.deltastream.io/v2/statements", { method: "POST", headers: { "Authorization": `Bearer ${process.env.DS_TOKEN}`, "Content-Type": "application/json" }, body: JSON.stringify({ statement: sql }) }); if (!res.ok) { const err = await res.text(); throw new Error(`DeltaStream ${res.status}: ${err}`); } const json = await res.json(); // map rows to objects using metadata.columns const cols = json.metadata.columns.map(c => c.name); const rows = (json.data || []).map(r => Object.fromEntries(r.map((v,i) => [cols[i], v]))); return { rows, meta: json.metadata, sqlState: json.sqlState, statementID: json.statementID }; } ``` *** ### Security notes * **Never hard-code tokens** in code or docs. Use env vars / secret managers. * **Rotate** any token that was exposed publicly. * Always use **HTTPS**. * Scope tokens to the **minimal role** needed. *** #### TL;DR 1. Create a token → 2) copy your org’s API endpoint → 3) `POST /v2/statements` with `{"statement":""}` and a Bearer token. That’s it. # Expose a Materialized View over MCP > **Beta Notice:** MCP support in DeltaStream is currently in **beta**. Functionality may evolve as we continue to refine the integration. The **Model Context Protocol (MCP)** is an open standard that lets AI agents securely connect to external data sources, tools, and APIs at runtime. It defines a simple set of endpoints for discovering, describing, and invoking capabilities exposed by an MCP server—enabling agents to safely access and use organizational data and systems without custom integrations. For more details, see the [MCP specification](https://modelcontextprotocol.io/specification/2025-06-18). ### Overview DeltaStream offers native MCP support, allowing any materialized view to be easily exposed through its MCP endpoint. This enables AI agents to query and interact with live, materialized data using standardized MCP interfaces. Access is governed by DeltaStream’s role-based access control (RBAC) system, ensuring secure and fine-grained authorization for all requests. ### Configuration There is no configuration to enable MCP support within DeltaStream. It is a REST endpoint where access is governed by RBAC. To grant an agent access: * *Optional -* **Create a Role that has limited permissions -** Create a role that only has access to materialized views to be exposed over the MCP endpoint. * **Materialized view exposure** — Any materialized view that the token's role has `SELECT` privilege on will be exposed over the MCP endpoint as a tool the agent can discover and use. * **Create an API token** — Create an `API_TOKEN` for a role via the DeltaStream console or Admin API, and assign the intended role to that token. * **Provide the token to the agent** — Configure the agent to call DeltaStream's MCP endpoint and include the token in the `Authorization: Bearer ` header. Security recommendations: * Grant least privilege (only `SELECT` on required views). * Rotate tokens regularly. With a properly scoped API token, agents can immediately discover and interact with permitted materialized views using standard MCP endpoints. ### Example Setup 1. **Create a materialized view** — Follow the guide in the [DeltaStream documentation](https://docs.deltastream.io/how-do-i.../creating-and-querying-materialized-views) to create the materialized view you want to expose. 2. **Create a role for the AI agent** — Switch to the Org Admin role, create a new role and grant permissions. In this example, we'll call it `ai_agent` and the view will be called `myview` in the database `mydb` and the schema `public`. ```sql USE ROLE orgadmin; CREATE ROLE ai_agent; GRANT USAGE ON DATABASE mydb TO ROLE ai_agent; GRANT USAGE ON SCHEMA public TO ROLE ai_agent; GRANT SELECT ON mydb.public.myview TO ROLE ai_agent; ``` 3. **Create an API token** — Generate a new API token and set `token.role_name` to the role you created above (`ai_agent`). ```sql CREATE API_TOKEN myagent_token WITH ('token.role_name' = ai_agent); ``` 4. **Test the MCP endpoint** — Verify access and discover exposed tools using `curl`. Refer to the [DeltaStream REST API documentation](https://docs.deltastream.io/how-do-i.../call-deltastreams-rest-api) to determine your API endpoint: ```bash curl -X POST https:///mcp/v1 \ -H "Content-Type: application/json" \ -H "Accept: application/json,text/event-stream" \ -H "Authorization: Bearer " \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {} }' ``` This command will list all materialized views granted to the role associated with your API token, confirming that the agent can successfully interact with DeltaStream via MCP. ### Limitations DeltaStream MCP does not support resources, prompts, roots, notifications, version negotiations, life cycle phases, or sampling, which are part of the broader MCP protocol. # Configure DeltaStream as Private SaaS using Bring your own cloud (BYOC) model With the private SaaS model, DeltaStream provides the same serverless user experience that the customers love but with the guarantee that data remains within the customer cloud private network. ## Supported Clouds * Amazon Web Services (AWS) * Azure * Oracle Cloud Infrastructure (OCI) Customers will need to provide details about their cloud provider accounts and the regions they plan to use. These requirements vary by cloud provider. In the following sections, you’ll find the specific information the DeltaStream Support team needs for each cloud before you can deploy DeltaStream as a BYOC solution. ### Amazon Web Services for Private SaaS Provide following details to DeltaStream if you use AWS as your cloud provider: | AWS Account Info | Description | Example | | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AWS Region | AWS Region where deltastream private dataplane will be deployed | us-west-2 | | AWS Account ID | 12 digit AWS Account | | | VPC CIDR |

DeltaStream Platform will create a new VPC in the AWS Account using provided CIDR.
CIDR range support using prefix length /16 or /20

|

Choose a CIDR that does not conflict with any future peering requirements, Example CIDR:10.20.0.0/16

DeltaStream platform will create an isolated new VPC in the provided AWS Account and region.

| | Base Domain |

Domain name suffix used to host DeltaStream Platform URLS for
\* Console
\* Web API
\* Downloads for CLI

|

Example: deltastream.acme.com

Using this base domain customer will be able to connect to DeltaStream web console as console-{unique-deltastream-id}.deltastream.acme.com
where unique-deltastream-id is a random 6 alpha-numeric character (e.g. njehv2)

| | User Authentication Option |

DeltaStream platform supports following options
\* User Name + Password
\* OKTA SAML
\* Google Single sign-on

| For example, if customer has OKTA as IDP they will select OKTA SAML. | | Public or Private Access To DeltaStream Platform over Internet |

Options:
\* public - DeltaStream console and API endpoints are exposed on public internet
\* private - DeltaStream console and API endpoints are only available as a private (VPC) endpoints

| When using private option customer will need to configure access via their own VPN solutions, e.g. AWS VPN or Zscaler. DeltaStream platform supports both options. | | Customer's Administrator Workstation Platform that will be used to setup DeltaStream Private SaaS |

Supported Platform:
\* MacOS (Apple Silicon)
\* Linux/Ubuntu

| | ### Azure for Private SaaS Provide following details to DeltaStream if you use Azure as your cloud provider: | Azure Account Info | Description | Example | | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Location | Azure Location/Region | eastus2 | | Azure Tenant ID | Tenant ID | aabbcc9a-1234-4163-9fad-c288fc3b1234 | | Azure Subscription ID | Subscription ID | 12346667-cc90-66e5-8ef7-25e28cad3100 | | Resource Group | Azure Resource Group name under which all DeltaStream infrastructure will be provisioned, this resource group will be created by customer as pre-requisite | deltastream-private-stack | | VNet CIDR | Virtual Network CIDR, supported Range /16 |

Choose a CIDR that does not conflict with any future peering requirements, Example CIDR: 10.22.0.0/16

Note that DeltaStream platform will create an isolated new VNet.

| | Base Domain |

Domain name suffix used to host DeltaStream Platform URLS for
\* Console
\* Web API
\* Downloads for CLI

|

Example: deltastream.acme.com

Using this base domain customer will be able to connect to DeltaStream web console as console-{unique-deltastream-id}.deltastream.acme.com
where unique-deltastream-id is a random 6 alpha-numeric character (e.g. njehv2)

| | Console and API web endpoint Certificate | Customer will need to provide Certificate and private key that they will upload to a vault secret prior to starting DeltaStream Setup |

Customer will need both certificate chain and certificate private key, the certificate should be using Subject/SAN that uses either wildcard \*.deltastream.acme.com
or following individual endpoints:
api-{unique-deltastream-id}.deltastream.acme.com
console-{unique-deltastream-id}.deltastream.acme.com
downloads-{unique-deltastream-id}.deltastream.acme.com

| | User Authentication Option |

DeltaStream platform supports following options
\* User Name + Password
\* OKTA SAML
\* Google Single sign-on

| For example, if customer has OKTA as IDP they will select OKTA SAML. | | Public or Private Access To DeltaStream Platform over Internet |

Options:
\* public - DeltaStream console and API endpoints are exposed on public internet
\* private - DeltaStream console and API endpoints are only available as a private (VPC) endpoints

| When using private option customer will need to configure access via their own VPN solutions, e.g. AWS VPN or Zscaler. DeltaStream platform supports both options. | | Customer's Administrator Workstation Platform that will be used to setup DeltaStream Private SaaS |

Supported Platform:
\* MacOS (Apple Silicon)
\* Linux/Ubuntu

| | ### Oracle Cloud Infrastructure (OCI) for Private SaaS Provide following details to DeltaStream if you use Oracle Cloud as your cloud provider: | Oracle Cloud Info | Description | Example | | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Region | Oracle Cloud Region | us-phoenix-1 (We support regions with at least three availability domains, e.g. us-ashburn-1, us-phoenix-1) | | tenancy OCID | Oracle Tenancy OCID | We recommend creating a separate child tenancy for isolated DeltaStream Private SaaS deployment, Example: `ocid1.tenancy.oc1..aaaacccdfdf343on3yfvkxbopzhrudyadf34ofeswsbwk67gracd2342izwt3432a` | | tenancy Namespace | Oracle Tenancy namespace | axz0f8c0sn2, this can be retrieved using OCI CLI `oci os ns get` | | parent Compartment OCID | parent compartment OCID where DeltaStream stack will create child compartments hosting all DeltaStream Infrastructure |

Create a separate compartment, e.g. at root level.
Example OCID: ocid1.tenancy.oc1..aaaaaa342afdasfon3yfvkxbadf23423udygadmofeswsbwk67asdf345hyuizwt7fhjkdsa

| | VCN CIDR | Virtual Cloud Network CIDR, supported Range /16 |

Choose a CIDR that does not conflict with any future peering requirements, Example CIDR: 10.170.0.0/16

Note that DeltaStream platform will create an isolated new VCN.

| | Base Domain |

Domain name suffix used to host DeltaStream Platform URLS for
\* Console
\* Web API
\* Downloads for CLI

|

Example: deltastream.acme.com

Using this base domain customer will be able to connect to DeltaStream web console as console-{unique-deltastream-id}.deltastream.acme.com
where unique-deltastream-id is a random 6 alpha-numeric character (e.g. njehv2)

| | Console and API Web Endpoint Certificate | Customer will need to have access to Certificate and private key that they will upload to a vault secret during DeltaStream Platform Setup |

Customer will need both certificate chain and certificate private key, the certificate should be using Subject/SAN that covers either wildcard \*.deltastream.acme.com or following individual endpoints:
api-{unique-deltastream-id}.deltastream.acme.com
console-{unique-deltastream-id}.deltastream.acme.com
downloads-{unique-deltastream-id}.deltastream.acme.com

| | User Authentication Option |

DeltaStream platform supports following options
\* User Name + Password
\* OKTA SAML
\* Google Single sign-on

| For example, if customer has OKTA as IDP they will select OKTA SAML. | | Public or Private Access To DeltaStream Platform over Internet |

Options:
\* public - DeltaStream console and API endpoints are exposed on public internet
\* private - DeltaStream console and API endpoints are only available as a private (VPC) endpoints

| When using private option customer will need to configure access via their own VPN solutions, e.g. AWS VPN or Zscaler. DeltaStream platform supports both options. | | Customer's Administrator Workstation Platform that will be used to setup DeltaStream Private SaaS |

Supported Platform:
\* MacOS (Apple Silicon)
\* Linux/Ubuntu

| | \ \ Once these details have been provided to DeltaStream, the customer will receive an installation tool along with prerequisites and instructions for running it. Note that each cloud requires its own pre-requisites, e.g. for AWS we require AWS cli to be present.\ \ Additionally, the installation tool requires Administrator‑level privileges within the respective cloud environment. For instance, in an AWS account the tool needs Administrator permissions in order to create IAM roles, policies, VPCs, and other infrastructure components required by the DeltaStream platform. Here is an example command-line that customer will use for installing DeltaStream platform as a BYOC: ``` ./mgmt-client setup-dataplane \ --server https://mgmt-api.prod.deltastream-internal.name/v1 \ ${MGMT_PLANE_ACCOUNT_ID} \ ${DATAPLANE_INFRA_ID} \ --token ${DATAPLANE_TOKEN} ``` Here environment variables `${MGMT_PLANE_ACCOUNT_ID}, ${DATAPLANE_INFRA_ID}, and ${DATAPLANE_TOKEN}` are specific to a customer deployment and provided by DeltaStream at the time customer is ready to configure DeltaStream Private SaaS. The installation tool execution time depends on the cloud, it typically takes about \~30-60 minutes. Following is an example screenshot of the client tool execution:
Once the client tool execution is completed, additional platform configuration continues in the background, this include provisioning a Kubernetes cluster and deploying necessary micro-services required to operate the DeltaStream platform. This phase can take an additional 1–2 hours to finish. # Setting up Data Store Integrations DeltaStream Fusion works with the following: {% content-ref url="/pages/PerHHGuzxrTMiGYkr7vl" %} [AWS S3](/integrations/setting-up-data-store-integrations/aws-s3) {% endcontent-ref %} {% content-ref url="/pages/6ZzIaDLb2qxFA8zjOGiI" %} [ClickHouse](/integrations/setting-up-data-store-integrations/clickhouse) {% endcontent-ref %} {% content-ref url="/pages/MHCogAiB42tQe3kbd0bs" %} [Confluent Cloud](/integrations/setting-up-data-store-integrations/confluent-cloud) {% endcontent-ref %} {% content-ref url="/pages/BgfmmPg9nEhNPX6TI1xQ" %} [Databricks](/integrations/setting-up-data-store-integrations/setting-up-and-integrating-databricks-with-your-organization) {% endcontent-ref %} {% content-ref url="/pages/3eoMt7gdLqY9v6aOzWow" %} [Iceberg AWS Glue Catalog](/integrations/setting-up-data-store-integrations/iceberg-aws-glue-catalog) {% endcontent-ref %} {% content-ref url="/pages/4nGavNimK8gDv8zzQt0l" %} [Iceberg REST Catalog](/integrations/setting-up-data-store-integrations/iceberg-rest-catalog) {% endcontent-ref %} {% content-ref url="/pages/yZuaoQl9eAJpqpCDA7zx" %} [PostgreSQL](/integrations/setting-up-data-store-integrations/setting-up-and-integrating-postgresql-with-your-organization) {% endcontent-ref %} {% content-ref url="/pages/WZfqSiVaGEV2Z3P9PpZ8" %} [Snowflake](/integrations/setting-up-data-store-integrations/setting-up-and-integrating-snowflake-with-your-organization) {% endcontent-ref %} {% content-ref url="/pages/2tRvmWmBkFfxPAdxGjAD" %} [WarpStream](/integrations/setting-up-data-store-integrations/warpstream) {% endcontent-ref %} # AWS S3 Amazon Web Services Simple Storage Service (AWS S3) is a highy-scalable object storage service commonly used for business use cases involving very large amounts of data, such as data lakes, backup and restore, archival, and more.. This document walks you through setting up AWS S3 to use as a source data store in DeltaStream. ### Before You Begin Work with your internal engineering team to set up an AWS S3 account. You can start with [the AWS S3 documentation.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html) For this setup guide you must also have created a stream defined in DeltaStream labeled pageviews, which is backed by a topic in an Apache Kafka Store. More [details on creating a stream in DeltaStream](/reference/sql-syntax/ddl/create-stream). ### Adding S3 as a DeltaStream Data Store 1. Open DeltaStream. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ). The **Resources** page displays, with the **Data Stores** tab active. ![](/files/CMPH5Xad2RgJWBtFQeCt) 2. Click **+ Add Data Store**, and when the **Choose a Data Store** window opens, click **S3**. ![](/files/FFxLovUsZy6xp8AKZU3P) 3. Click **Next.** The **Add Data Store** window opens, displaying S3 fields you must complete:\
* **Store Type** – S3 * **Name** – A name to identify your DeltaStream data store * **Add One or More URIs To Connect** – the URI for a bucket or folder * **AWS Region** – The region where the bucket containing your data resides. * **Assume IAM Role ARN** * **IAM Role External ID** * **Static AWS Credentials** (Not Recommended) * **AWS Access Key** * **AWS Secret Access Key** 4. Click **Add** to create and save the data store. Your S3 store displays on the **Resources** page in your list of data stores. {% hint style="info" %} **Note** You can also use the DeltaStream CLI to create an S3 data store (just called "store" in the CLI): ```sql CREATE STORE s3_store WITH ( 'type' = S3, 'aws.iam_role_arn' = 'arn:aws:iam::123456789012:role/S3StoreDeltaStreamRole', 'aws.iam_external_id' = '12342f59e61ef7a34f9822a6226e5857', 'uris' = 's3://s3-data-bucket/', 'aws.region' = 'AWS us-east-2' ); ``` {% endhint %} ### IAM Role Requirements for S3 Stores To allow the DeltaStream platform to access your S3 bucket using an IAM Role, you must add a **trust relationship** to a DeltaStream IAM Role to the IAM Role you’ve configured for S3 access. This trust relationship allows DeltaStream to assume the role securely. Please contact DeltaStream Support to obtain a DeltaStream platform specific IAM Role ARN for your organization. Once you have it, add the following **trust policy statement** to your IAM Role: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "TrustPolicy", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::{DELTASTREAM_PLATFORM_ACCOUNT_ID}:role/{DELTASTREAM_ACCOUNT_ROLE}", ] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "12342f59e61ef7a34f9822a6226e5857" } } } ] } ``` **Note:** Be sure to replace `{DELTASTREAM_PLATFORM_ACCOUNT_ID}` and `{DELTASTREAM_ACCOUNT_ROLE}` with the actual values provided by DeltaStream Support. Also, ensure that value provided for "sts:ExternalId" condition matches the one you will pass as `'aws.iam_external_id'` when creating DeltaStream S3 STORE. ### Recommended IAM Permissions for S3 Store In addition to the trust policy you will need to ensure that the IAM role specified in the `CREATE STORE` statement for an S3-type store has sufficient permissions to **read from and write to** the designated S3 bucket. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::your-bucket-name", "arn:aws:s3:::your-bucket-name/*" ] } ] } ``` ### Define a DeltaStream Stream In this step, you create a stream called `pageviews_cdc` that is backed by data on S3. {% hint style="info" %} The S3 source periodically discovers file splits under the configured paths and emits them for processing. Discovery interval controls how often the connector looks for new S3 objects. On each run it lists keys in the service’s server-side order (for general-purpose buckets, that’s **lexicographic by key**) and emits files to the job, enforcing a configurable **maximum splits per interval** cap. The connector tracks an internal watermark: the last accepted key for each configured prefix. Therefore, any key **less than or equal to** that watermark is considered already processed and is skipped on later runs (and after restarts). To ensure new data is picked up reliably in long-running jobs, write files under key prefixes that **increase lexicographically over time** (e.g., `YYYY/MM/DD/HH/`) and use zero-padded sequence parts (e.g., `part-000001.jsonl`). Avoid backfilling into older, lexicographically smaller prefixes; if needed, place backfills under a **new, higher-sorting** path so they appear after the currently existing content. Check [CREATE STREAM](/reference/sql-syntax/ddl/create-stream#s3-specific-parameters) to see how you can configure the discovery interval and max cap on file splits. {% endhint %} In the DeltaStream workspace, run the following SQL statement: ```sql CREATE STREAM pageviews_s3 ( viewtime BIGINT, userid VARCHAR, pageid VARCHAR ) WITH ( 'store' = 's3_store', 's3.uri'='s3://your_bucket_uri/jsonl/', 's3.discovery.interval.seconds'=15, 'value.format'='jsonl' ); ``` **Notes** `value.format` can be `jsonl` or `json`. `s3.discovery.interval.seconds` is optional. The default is 10 seconds. ### Process Streaming Data and Sink to S3 For the steps below, assume you already have a stream called **pageviews** defined, which is backed by a topic in Kafka. Assume also there is an S3 store labelled **S3\_Test\_Store**. Now perform a simple filter on the pageviews stream and sink the results into S3: ```sql SELECT * FROM pageviews_s3 WITH ('s3.discovery.interval.seconds'=5); ``` **Notes**: * Files are processed in alphanumeric order * New files are discovered based on '`s3.discovery.interval.seconds`' property ### Inspect the S3 Data Store 1. In the lefthand navigation, click **Resources** ( ![](/files/S9tjDACodjCuDchbgdlZ) ). This displays a list of the existing stores. 2. Click your S3 store to open the **Data Store** page and view its contents. # ClickHouse [ClickHouse](https://clickhouse.com/) is a fast and resource-efficient real-time data warehouse and open-source database. This document walks through setting up ClickHouse to be used as a data store in DeltaStream. Step through the following demo to get an overview. Detailed instructions are below the demo. {% @storylane/embed subdomain="deltastream" url="" linkValue="qhiujgcjlwmq" %} ## Setting up a ClickHouse Account To set up your ClickHouse Cloud account, please review the [ClickHouse Cloud Quick Start](https://clickhouse.com/docs/cloud/get-started/cloud-quick-start) guide. ## Adding ClickHouse as a DeltaStream Data Store 1. Open DeltaStream. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ) and towards the top right click + **Add Data Store**. 2. When the **Choose a Data Store** window opens, click **ClickHouse** and then click **Next**.
3\. The **Add Data Store** window displays.
4. Enter the authentication and connection parameters. These include: * **Name** – A name that identifies your ClickHouse store. * **URL** – JDBC URL from the ClickHouse Cloud console. Be sure to preface this URL with a `clickhouse://` scheme:
* **Username** – User login name for the ClickHouse account. * **Password** – Password for the login name of the ClickHouse account. 4\. Click **Add** to save and create the store. *For instructions on creating the store using DSQL, see* [CREATE STORE](/reference/sql-syntax/ddl/create-store). ## Process Streaming Data and Sink it to ClickHouse {% hint style="warning" %} **Important** Before you proceed, you must already have: * A [stream](/overview/core-concepts/databases#stream) defined in DeltaStream labeled `pageviews,` which is backed by a topic in an Apache Kafka Store. * A ClickHouse store labeled `clickhouse_store` (see [#adding-clickhouse-as-a-deltastream-store](#adding-clickhouse-as-a-deltastream-store "mention")). With these two items in place you can perform a simple filter on the `pageviews` stream and then sink the results into ClickHouse. {% endhint %} ### Inspect the ClickHouse Store 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores.
2. Click `clickhouse_store.` The store page opens, displaying a list of the existing databases in your ClickHouse account:
3. (Optional) Create a new database. To do this: * Click **+ Add Database**. When prompted, enter a name for the new database and click **Add**. The new database displays in the list. 4. To see the tables that exist under a particular database, click the database name. ### Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into ClickHouse 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the [CREATE TABLE AS SELECT](/reference/sql-syntax/query/create-table-as) (CTAS) query to ingest from **pageviews** and output to a new table titled **ch\_pv\_table**.
```sql CREATE TABLE ch_pv_table WITH ( 'store' = 'clickhouse_store', 'clickhouse.db.name' = 'abc' ) AS SELECT viewtime, pageid, userid FROM pageviews WHERE pageid != 'Page_3'; ``` 3. Click **Run**. 4. View the existing queries, including the query from the step immediately prior. To do this, in the lefthand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ). It may take a few moments for the query to transition into the **Running** state. 5. To see more details about the status of the query, click the query row :
### View the results 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores. 2. To view the new table created by the above CTAS, navigate to `clickhouse_store` → `Databases` → `abc` → `ch_pv_table`. 3. To view a sample of the data in your ClickHouse table, click **Print**.
### Data Type Handling At DeltaStream, all data types are nullable by default. But in ClickHouse, data types are not-null by default. For a smooth conversion between SQL and ClickHouse data types, DeltaStream provides a reasonable default for nullable columns to prevent `Nullable()` ClickHouse data types. Data type defaults are as followed:
SQL Data TypeClickHouse Data TypeNullable Column Default
BOOLEANBooleanfalse
CHARString''
VARCHARString''
VARBINARYString''
TINYINTInt80
SMALLINTInt160
INTEGERInt320
BIGINTInt640
DECIMALDecimal(P, S)0.00x
FLOATFloat320.0
DOUBLEFloat640.0
DATEDate32toDate('1970-01-01')
TIMESTAMPDateTime64(P)toDateTime64('1970-01-01 00:00:00', P)
TIMESTAMP_WITH_LOCAL_TIME_ZONEDateTime64(P)toDateTime64('1970-01-01 00:00:00', P)
ARRAYArray(T)[]
MAPMap(K, V)map()
STRUCTTuple(T1, T2, ...)[]
If you must use a specific default that is different than what's provided in the table above, you can use a `CASE` function to alter the value at query execution: ```sql SELECT viewtime, userid, CASE WHEN pageid IS NULL THEN 'Page_0' END AS pageid_my_default FROM pageviews; ``` In the above query, whenever the original `pageviews` column `pageid` is `null`, the value `Page_0` is projected to the results. This effectively eliminates all `null` values for the column and also allows you to define the resulting data type for the column as `pageid VARCHAR NOT NULL`. # Confluent Cloud [Confluent Cloud](https://www.confluent.io/) is a fully-managed cloud service based on Apache Kafka. This article shows you how to create Confluent Cloud-type data stores directly in DeltaStream. {% hint style="info" %} **Note** For more information on creating data stores, see [Create Data Stores for Streaming Data](/how-do-i.../create-and-manage-data-stores/store). {% endhint %} ### To create a Confluent Cloud Data Store 1. In the lefthand navigation, click **Resources**, and then click **+ Add Data Store**. 2. From the menu that displays, click **ConfluentKafka.** The **Add Data Store** window opens.
3. Enter the authentication and connection parameters. This includes: 1. a unique **Name** for the data store.\ Store names are limited to a maximum of 255 characters. Only alphanumeric characters, dashes, and underscores are allowed. 2. the URIs to connect to Confluent Cloud.\ You can find your bootstrap server on Confluent's **Cluster settings** page.
3. the **API Key** and **API Secret** created on Confluent Cloud. 4. Click **Add**. Your Confluent Cloud Cluster displays on the **Resources** page in your list of data stores.
To inspect the topics in the data store, and to print their records, simply click on the data store.
With your Confluent Cloud data store set up, view the following tutorials to learn how to manage and process your streaming data with DeltaStream. * [Use Namespacing for Organizing Data](/how-do-i.../database) * [Create DeltaStream Objects to Structure Raw Data](/how-do-i.../relation) * [Create and Query Materialized Views](/how-do-i.../creating-and-querying-materialized-views) # Databricks [Databricks](https://www.databricks.com/) is a lakehouse platform in the cloud. This article walks you through setting up Databricks to be used as a data [Data Store](/overview/core-concepts/store) in DeltaStream. ## Setting up the Databricks Workspace ### Prerequisites 1. [Sign up for a Databricks account using AWS and complete the workspace setup](https://docs.databricks.com/en/getting-started/index.html) (steps 1 and 2) or use an existing Databricks workspace. 2. Have an AWS account whose S3 hosts your Delta Lake data.\ If you don't have an account, you can sign up for a [free trial of AWS](https://aws.amazon.com/free). ### Create a Databricks App Token 1. Navigate to your Databricks workspace. 2. In the top right of the screen, click down on your account name and select **User Settings**.
3. In the menu bar that displays, click **Developer,** and under **Access Tokens**, click **Manage**.
4. Click **Generate new token**. Add an optional comment for the token and then choose a lifetime for the token. Then click **Generate** to create the token.
4. Verify the save or download the newly-generated token value. You will need this when [creating the store later on](#adding-databricks-as-a-deltastream-store). For more details on generating access tokens for a workspace, see the [*Databricks documentation*](https://docs.databricks.com/en/dev-tools/auth.html#databricks-personal-access-token-authentication)*.* ### Add Databricks SQL Warehouse 1. Navigate to your Databricks workspace. 2. In the lefthand navigation click the **SQL Warehouses**. A list displays of the existing SQL warehouses in your workspace. Databricks creates a starter warehouse for you. 3. To create a new SQL warehouse, click **Create SQL warehouse**. To edit an existing SQL warehouse, to the right of the warehouse you want, click the 3 vertical dots. Then click **Edit**.
4. Configure your SQL warehouse with your preferred specifications. (To learn more about configuring your SQL warehouse, please review the [Databricks documentation](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html).) For a more optimal experience, we recommend choosing **serverless** as the SQL warehouse type. [More information about Databricks serverless SQL warehouse](https://docs.databricks.com/en/admin/sql/serverless.html).
5. Click **Save** to create the SQL warehouse. Record the warehouse ID on the overview page; you will need this ID when [you create the store later on](#adding-databricks-as-a-deltastream-store).\ You can also access the warehouse overview by clicking the name of the SQL warehouse from the **SQL Warehouses** initial landing page from step 1.
### Add an S3 Bucket as External Location for Data #### Use an existing S3 bucket or create a new one. 1. **To create a new AWS S3 bucket**: 1. In the [AWS console](https://console.aws.amazon.com), navigate to the S3 page.
2. Click **Create bucket**.
3. Enter a name for your S3 bucket and then at the bottom click **Create bucket** to create your new S3 bucket. For more details, see the Databricks [*documentation for creating, configuring, and working with Amazon S3 buckets*](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-buckets-s3.html)*.* #### Add Databricks connection to the newly-created S3 bucket 1. Navigate to your Databricks workspace. 2. In the lefthand navigation, click **Catalog**. This displays a view of your [Unity Catalog](https://www.databricks.com/product/unity-catalog). 3. At the top of the page, click **+ Add**, and from the list that displays click **Add an external location**.
4. Click **AWS Quickstart** to set up the Databricks and S3 connection, and then click **Next**. Advanced users can opt to set up their external location manually instead, but this article continues with the AWS Quickstart option.
5. Enter the name of an existing S3 bucket to link to your Databricks workspace. Then click **Generate new token**. Copy that token, then click **Launch in Quickstart**. This brings you back to the AWS console and displays a page called **Quick create stack**.
6. On the the AWS **Quick create stack** page, in the **Databricks Personal Access Token** field, enter the access token you copied in step 5. Then at the bottom of the page, click to acknowledge that AWS CloudFormation might create IAM resources with custom names. Then click **Create stack** to launch stack initialization.
7. In a few minutes, you'll see the stack creation complete.
For more information on external locations, see the [*Databricks documentation*](https://docs.databricks.com/en/sql/language-manual/sql-ref-external-locations.html)*.* #### (Optional) Create a Unity Catalog Metastore This step is relevant if you receive an error message such as `Metastore Storage Root URL Does Not Exist`. In this case: 1. Ensure you have an S3 bucket to use for metastore-level managed storage in AWS (follow the steps above to create a new S3 bucket). In this case you can use the bucket created in the previous step. 2. Navigate to the [Databricks account settings Catalog page](https://accounts.cloud.databricks.com/data). From here, either create a new metastore or edit existing metastores.
3. If you're creating a new metastore, click **Create metastore** and follow the prompts to set the name, region, S3 path, and workspaces for the metastore. 4. If you're editing an existing metastore, click on the name of the metastore you wish to edit. From this page you can assign new workspaces, set an S3 path, edit the metastore admin, and take other actions. For more information on creating a Unity Catalog metastore, see the [*Databricks documentation*](https://docs.databricks.com/en/data-governance/unity-catalog/create-metastore.html)*.* ## Adding Databricks as a DeltaStream Store 1. Open DeltaStream. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ) and then click **Add Store +**.
2. From the menu that displays, click **Databricks.** The **Add Store** window opens.
3. Enter the authentication and connection parameters. These include: * **Store Name** – A unique name to identify your DeltaStream store. (For more details see [Data Store](/overview/core-concepts/store)).\ Store names are limited to a maximum of 255 characters. Only alphanumeric characters, dashes, and underscores are allowed. * **Store Type** – Databricks * **URL** – URL for Databricks workspace. Find this by navigating to the [Databricks accounts page](https://accounts.cloud.databricks.com/workspaces) and clicking the workspace you wish to use. * **Warehouse ID** – The ID for a Databricks SQL warehouse in your Databricks workspace. (For more details see [#add-databricks-sql-warehouse](#add-databricks-sql-warehouse "mention")). * **Databricks Cloud Region** – The AWS region in which the **Cloud S3 Bucket** exists. * **Cloud S3 Bucket** – An AWS S3 bucket that is connected as an external location in your Databricks workspace (see [#add-s3-bucket-as-external-location-for-data](#add-s3-bucket-as-external-location-for-data "mention")). * **App Token** – The Databricks access token for your user in your Databricks workspace. (For more details see [#create-databricks-app-token](#create-databricks-app-token "mention").) * **Access Key ID** – Access key associated with the AWS account in which the **Cloud S3 Bucket** exists. * **Secret Access Key** – Secret access key associated with the AWS account in which the **Cloud S3 Bucket** exists. 4. Click **Add**. Your Databricks store displays on the **Resources** page in your list of stores.
{% hint style="info" %} **Note** For instructions on creating the store using DSQL, *see* [CREATE STORE](/reference/sql-syntax/ddl/create-store). {% endhint %} ## Process Streaming Data and Sink to Databricks For the steps below, assume you already have a [stream](/overview/core-concepts/databases#stream) defined called **pageviews**, which is backed by a topic in Kafka. Assume also there is a Databricks store labelled **Databricks\_Test\_Store**. (For more details see [#adding-databricks-as-a-deltastream-store](#adding-databricks-as-a-deltastream-store "mention").) Now perform a simple filter on the pageviews stream and sink the results into Databricks. {% hint style="info" %} **Note** For more information on setting up a stream or a Kafka store, see [Starting with the Web App](/getting-started/starting-with-web-app) or [Starting with the CLI](/getting-started/starting-with-cli). {% endhint %} ### Inspect the Databricks store 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing stores.
2. Click the **Databricks\_Test\_Store**. The store page displays, with the **Databases** tab active. Here you can view a list of the existing catalogs in your Databricks workspace.
3. (Optional) Create a new database. To do this, click **+ Add Database**. When prompted, enter a name for the new database and click **Add**. The new database displays in the list.\ \ **Important** If you receive this error message -- `Metastore Storage Root URL Does Not Exist` -- verify that you've properly [set up your Databricks Unity Catalog metastore](#optional-create-a-unity-catalog-metastore). 4. To see the namespaces that exist in a particular database, click the database you want. 5. (Optional) Create a new namespace. To do this: 1. Select **+ Add Namespace.** In the window that displays, enter a name for the new namespace and then click **Add**. The new namespace now displays in the list. 6. To see the tables that exist under a particular namespace, click the namespace you want. ### Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into Databricks 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the [CREATE TABLE AS SELECT (CTAS)](/reference/sql-syntax/query/create-table-as) query to ingest from **pageviews** and output to a new table titled **pv\_table**. ```sql CREATE TABLE pv_table WITH ( 'store' = 'databricks_store', 'databricks.catalog.name' = 'new_catalog', 'databricks.schema.name' = 'new_schema', 'databricks.table.name' = 'pageviews', 'table.data.file.location' = 's3://deltastream-databricks-bucket2/test' ) AS SELECT viewtime, pageid, userid FROM pageviews WHERE pageid != 'Page_3'; ``` 3. Click **Run**. 4. In the lefthand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ) to see the existing queries, including the query from the step immediately prior.\ It may take a few moments for the query to transition into the **Running** state. Keep refreshing your screen until the query transitions.
### View the results 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing stores. 2. To view the new table created by the above CTAS, navigate to **databricks\_store** --> **Databases** --> **+ Add Database** --> **Add Namespace** --> **pageviews**.\ Of course, if you wrote your CTAS such that the store/database/namespace/table names are different, navigate accordingly. 3. To view a sample of the data in your Databricks table, click **Print**.
# Iceberg AWS Glue Catalog ### **Iceberg AWS GLUE** Apache Iceberg is a high-performance table format that supports large analytic tables. This document walks through setting up Iceberg in DeltaStream using the AWS Glue catalog. {% hint style="info" %} **Note** Iceberg is unique in DeltaStream in that, if you plan on reading from or querying Iceberg data, it requires you also define an object called a **compute pool**. A compute pool is a set of dedicated resources for running batch queries. You do not need a compute pool if you are only writing to Iceberg – if, for example, you’re streaming filtered Kafka data into Iceberg tables. [More information on compute pools.](/overview/core-concepts/compute-pools) {% endhint %} ### Before You Begin Work with your internal engineering team to set up an AWS Glue account. You can start with the A[WS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/setting-up.html). For this setup guide you must also have created a stream defined in DeltaStream labeled pageviews, which is backed by a topic in an Apache Kafka data Store. More [details on creating a stream in DeltaStream](/reference/sql-syntax/ddl/create-stream). ### Adding an Iceberg AWS GLUE data store **To set up Iceberg AWS Glue** 1\. Log onto DeltaStream. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ) and, when the list of data stores displays, click **+ Add Data Store**.
2. When the **Choose a Data Store** window opens, click **Iceberg AWS Glue**. 3. Click **Next**. The **Add Data Store** window opens. ![](/files/vYhFRQbZdfmxb4NVMDhv) 3. Enter the requested authentication and connection values. With the data store set up and the Kafka stream created, you can perform a simple filter on the `pageviews` stream and then sink the results back into Iceberg. ### Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into Iceberg Here we’re reading data from Kafka and writing to AWS Glue. This ensures we’re using the correct data store. 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the CREATE TABLE AS SELECT (CTAS) query to ingest from **pageviews** and output to a new table titled **pageviews\_iceberg\_rest**. ```sql CREATE TABLE pageviews_iceberg_glue WITH ( 'store' = 'iceberg_glue_store', 'iceberg.aws.glue.db.name' = 'gradient', 'iceberg.aws.glue.table.name' = 'pageviews_iceberg' AS SELECT * FROM pageviews; ``` {% hint style="info" %} **Notes** * `iceberg.aws.glue.db.name` is required. It creates the sink table in your DB. * `iceberg.aws.glue.table.name` is optional. If you do not specify a table name, DeltaStream uses the object name on the first line – in this case, pageviews\_iceberg\_glue. {% endhint %} 3. Click **Run**. View the existing queries, including the query from the step immediately prior. To do this, in the left-hand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ). {% hint style="info" %} **Note** It may take a few moments for the query to transition into the **Running** state. Keep refreshing your screen until the query transitions. {% endhint %} To see more details about the status of the query, click the query row:
#### View the results 1. In the left-hand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores. 2. To view the new table created by the above CTAS, navigate to the `pageviews_iceberg` table. To view a sample of the data in your Iceberg table, click **Print**. ![](/files/bMBqVB6VocuuNOuhclpp) ### Process Streaming Data From Your Iceberg Data Store Now it’s time to query the data stored in Iceberg. 1. Define a `compute_pool` to be able to query the iceberg table from above. ```sql CREATE COMPUTE_POOL mypool WITH ( 'compute_pool.size' = 'small', 'compute_pool.timeout_min' = 3600'); ``` The above statement creates and starts the `compute_pool`. If this is the first `compute_pool` in the organization, DeltaStream sets it as your default pool. 2. Run a batch query. ```sql SELECT * FROM pv_table limit 10; ``` ### Inspect the Iceberg Data Store 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores.
2. Click the Iceberg Glue data store. The store page opens, displaying a list of any existing databases in your account. 3. (Optional) Create a new database. To do this: * Click **+ Add Database.** When prompted, enter a name for the new database and click **Add**. The new database displays in the list. 4. To view the tables that exist under a particular database, click the database name. ### Clean up resources 
STOP COMPUTE_POOL mypool;
TERMINATE QUERY <QUERY-ID>;
# Iceberg REST Catalog ### **Iceberg REST Catalog** Apache Iceberg is a high-performance table format that supports large analytic tables. An Apache Iceberg REST catalog is a service for managing and accessing Iceberg tables in a consistent way. It allows clients to interact with Iceberg table metadata without requiring direct access to the underlying storage. This enables multiple clients to safely use the same Iceberg tables. This document walks through setting up an Iceberg catalog in DeltaStream. {% hint style="info" %} **Note** Iceberg is unique in DeltaStream in that, if you plan on reading from or querying Iceberg data, it requires you also define an object called a **compute pool**. A compute pool is a set of dedicated resources for running batch queries. {% endhint %} You do not need a compute pool if you are only writing to Iceberg – if, for example, you’re streaming filtered Kafka data into Iceberg tables. [More information on compute pools](/overview/core-concepts/compute-pools). For the purposes of this tutorial we will use a REST catalog provided by Snowflake, but any compliant implementation will work. ### Before you Begin 1. Work with your internal engineering team to set up a Snowflake environment. You can start with the [Snowflake Open Catalog tutorial](https://other-docs.snowflake.com/opencatalog/tutorials/open-catalog-gs). Go through the overview and complete the Snowflake environment setup instructions. At that point you will have the following values: 1. \`client\_id\` 2. \`client\_secret\` 3. \`principal\_role\_name\` 4. \`catalog\_name\` 5. \`open\_catalog\_account\_identifier\` 6. S3 region that your storage bucket is located 2\. For this setup guide you must also have created a stream defined in DeltaStream named pageviews, which is backed by a topic in an Apache Kafka data store. [More details on creating a stream in DeltaStream.](/reference/sql-syntax/ddl/create-stream) ### Adding an Iceberg REST data store **To set up Iceberg REST** 1\. Log onto DeltaStream. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ) to display a list of data stores in your organization.
2\. Click **+ Add Data Store**. When the **Choose a Data Store** window opens, click **Iceberg Rest**. The **Add Data Store** window opens for Iceberg REST. ![](/files/AeMjelYQpUlm0ZPXXEEe) 3\. Enter the required authentication and connection values. These include: * **Name**. We suggest a self-describing name, such as `iceberg_rest`. * **S3 Region**. The region where your AWS S3 bucket resides. * **Catalog ID**. * **URIs**. * **Scope**. * **Client ID**. {% hint style="info" %} **Note** You can also use the DeltaStream CLI to create an Iceberg\_REST data store. To do this, run the below statement: ```sql CREATE STORE opencatalog WITH ( 'type'=iceberg_rest, 'uris' = 'https://.snowflakecomputing.com/polaris/api/catalog', 'iceberg.catalog.id' = '', 'iceberg.rest.client_id' = '', 'iceberg.rest.client_secret' = '', 'iceberg.rest.scope' = 'PRINCIPAL_ROLE:', 'iceberg.rest.s3.region'=''); ``` {% endhint %} 4\. Inspect the data store to see the namespaces available within your REST catalog. To do this, navigate to **Workspace** and then examine the newly-created data store. {% hint style="success" %} **Tip** When you view entities under a REST catalog data store, DeltaStream displays namespaces and tables, as shown below: {% endhint %}
5. Create a namespace in opencatalog for the namespace to live in. To do this, return to the workspace to verify you can use your REST catalog. Run `` `CREATE ENTITY mynamespace;` `` -This command creates a namespace called `mynamespace` under your REST catalog.
### Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into Iceberg 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the CREATE TABLE AS SELECT (CTAS) query to ingest from **pageviews** and output to a new table titled `pageviews_iceberg_rest`. ```sql CREATE TABLE pageviews_iceberg_rest WITH ( 'store' = 'opencatalog', 'iceberg.rest.catalog.namespace.name' = 'mynamespace', 'iceberg.rest.catalog.table.name' = 'pageviews_iceberg') AS SELECT * FROM pageviews; ``` 3. Click **Run**. The above statement performs several functions: * Creates a DeltaStream relation called `pageviews_iceberg_rest` .This relation can be used by other queries * Creates a table in the underlying REST catalog in the namespace called `mynamespace`. * Creates a long running query that reads data from Kafka and sinks to an Iceberg table. 4. Now view the existing queries, including the query from the step immediately prior. To do this, in the left-hand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ). {% hint style="info" %} **Note** It may take a few moments for the query to transition into the **Running** state. Keep refreshing your screen until the query transitions. {% endhint %} To see more details about the status of the query, click the query row:
#### View the results 1. In the left-hand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores. 2. To view the new table created by the above CTAS, navigate to **opencatalog** → **mynamespace** → **pageviews\_iceberg\_rest**. To view a sample of the data in your Iceberg table, click **Print**. ![](/files/bMBqVB6VocuuNOuhclpp) ### Process Streaming Data From Your Iceberg Data Store Now it’s time to query the data stored in Iceberg. To do this: 1. Define a `compute_pool` to be able to query the iceberg table from above. Navigate to **Resources > Compute Pools**, and then click **+ Add Compute Pool.** ![](/files/wBEpEK3tvdWdISLMEq3d) If this is the first `compute_pool` in the organization, DeltaStream sets it as your default pool. 2. Navigate to your DeltaStream workspace and run the following command: ```sql `SELECT * FROM pageviews_iceberg_rest LIMIT 10;` ``` ### Inspect the Iceberg Data Store 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores.
2. Click **opencatalog**. The store page opens, displaying a list of namespaces and tables. ### Clean up resources ```sql STOP COMPUTE_POOL my pool; TERMINATE QUERY
2. Click **+ Add Data Store**, and from the list that displays click **PostgresSQL**. The **Add Data Store** window displays, with Postgres-specific fields you must complete.

Postgres Store Details

3. Enter the following information: * **Store Name** – A name to identify your DeltaStream data store (See [Data Store](/overview/core-concepts/store)). * **Store Type** – POSTGRESQL. * **URI** – URI for the PostgreSQL database with `/` appended.\ For example, given a postgres URI of `my.postgresql.uri` and an open port on the database of `5432`, to connect DeltaStream to the **demo** database the URI would display as:\ `postgresql://my.postgresql.uri:5432/demo` * **Username** – Username associated with the PostgreSQL database user DeltaStream should assume. * **Password** – The password associated with the username. 4. Click **Add** to create and save the data store. {% hint style="info" %} **Note** For instructions on creating the store using DSQL, *see* [CREATE STORE](/reference/sql-syntax/ddl/create-store). {% endhint %} ### Inspect the PostgreSQL Data Store 1. In the lefthand navigation, click **Resources** ( ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores.
2. Click your PostgresSQL data store (in this case, **Postgres\_Test\_Store**). The Postgres data store page opens with the **Schemas** tab active. A list displays of the existing schemas in your PostgreSQL database.
3. (Optional) Create a new schema. To do this: * Click **+ Add Schema.** When the **Add Schema** window opens, enter a name for the new schema and then click **Add**. Your new schema displays in the entities list.

Adding a Postgres data Store Schema

4. To view the tables in a schema, click a schema name. 5. To view a sample of rows from that table, click a table in a schema and then click **Print**.

Postgres Schema Details

## Process PostgreSQL CDC Data and Sink to Kafka To follow the next few steps, you must already have a PostgreSQL data store labeled `psql_store`. You also must have a Kafka data store labeled `kafka_store.` Define a DeltaStream [stream](/overview/core-concepts/databases#stream) as your source data from PostgreSQL. Then write a query to process this data and sink it to a Kafka topic. {% hint style="info" %} **Note** For more details, see [#adding-postgresql-as-a-deltastream-store](#adding-postgresql-as-a-deltastream-store "mention"). {% endhint %} ### Defining a DeltaStream Stream on a PostgreSQL Table In this step, you create a stream called `pageviews_cdc` that is backed by data in a PostgreSQL table. This stream represents change data capture (CDC) events from the PostgreSQL table. {% hint style="info" %} **Note** DeltaStream uses [Debezium](https://debezium.io/) to capture changes in a source relation table. To learn more about how CDC works with DeltaStream, see [PostgreSQL](/reference/sql-syntax/query/change-data-capture-cdc/postgresql). {% endhint %} First, print the data for your source, which is the `pageviews` PostgreSQL table. To print sample rows from the table in DeltaStream, inspect your data store and navigate to the table you wish to print. (For more details, see [#inspect-the-postgresql-store](#inspect-the-postgresql-store "mention")). Below is an example of how to create a stream on your `pageviews` data. The fields match the Debezium standard; any insert, delete, or update to the `pageviews` table becomes an event for your `pageviews_cdc` stream. ```sql CREATE STREAM pageviews_cdc( op VARCHAR, ts_ms BIGINT, `before` STRUCT, `after` STRUCT, `source` STRUCT) WITH ( 'store'='psql_store', 'value.format'='json', 'postgresql.db.name'='demo', 'postgresql.schema.name'='public', 'postgresql.table.name'='pageviews'); ``` ### Write a CSAS (CREATE STREAM AS SELECT) Query to Sink Data into Kafka 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the [CREATE STREAM AS SELECT (CSAS)](/reference/sql-syntax/query/create-stream-as) query to ingest from `pageviews_cdc` and output to a new stream labeled `pageviews_cdc_sink`. To represent a feed of upsert events, this query filters for records whose `op` field is `CREATE` or `UPDATE`. ```sql CREATE STREAM pageviews_cdc_sink WITH ( 'store' = 'kafka_store', 'topic' = 'pageviews_cdc_sink', 'topic.partitions' = 1, 'topic.replicas' = 3) AS SELECT * FROM pageviews_cdc WITH ('postgresql.slot.name'='ds_cdc_demo') WHERE op = 'c' OR op = 'u'; ``` 3. Click **Run**. 4. In the lefthand navigation, click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ) to view existing queries, including the query from step 2, above.\ \ **Important** It can take a small amount of time for the query to transition into the **Running** state. Refresh you screen occasionally until you see the query transition into the **Running** state. 5. Verify that the query is properly working. To do this, write an interactive [SELECT](/reference/sql-syntax/query/select) query.

Verifying a Query in Postgres

# Snowflake [Snowflake](https://www.snowflake.com/) is a fully-managed cloud-based data platform for storing and processing data. Snowflake uses its own SQL query engine to provide its customers all of the functionality of an enterprise analytic database, along with many additional special features and unique capabilities.. This document walks through setting up Snowflake to be used as a data [Data Store](/overview/core-concepts/store) in DeltaStream. ## Setting up Snowflake Account ### Prerequisites 1. [Sign up for a Snowflake account using AWS](https://signup.snowflake.com/). 2. Unix-like environment (Linux, Mac OS X). 3. [Download SnowSQL](https://docs.snowflake.com/en/user-guide/snowsql-install-config) ### Add Snowflake Warehouse 1. In Snowflake, in the lefthand panel under `Admin`, select `Warehouses` to view a list of existing warehouses in your Snowflake account. Snowflake creates a starter warehouse for you called `COMPUTE_WH`.

Listing Snowflake Warehouses

2. To create a new warehouse, in the top right of the screen click **+ Warehouse**. To edit an existing warehouse, to the right of the warehouse you want click the 3 horizontal dots. Then click **Edit**. 3. Configure your warehouse with your preferred specifications. To learn more about configuring your warehouse, review the [Snowflake warehouse documentation](https://docs.snowflake.com/en/user-guide/warehouses-overview). ### Create Snowflake Key Pair Authentication 1. In a terminal window, generate a new private key: 1. Without encryption: ``` $ openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt ``` \ A new file called `rsa_key.p8` should be generated which contains the private key in PEM format. 2. With encryption: ``` $ openssl genrsa 2048 | openssl pkcs8 -topk8 -traditional -inform PEM -out rsa_key.p8 ``` \ When prompted, enter a password for this new private key. It's important to remember this password, as subsequent steps require it. \ A new file called `rsa_key.p8` is generated that contains the encrypted private key in PEM format. 2. Generate a public key ``` $ openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub ``` \ A new file called `rsa_key.pub` is generated that contains the public key in PEM format. 3. Ensure these new private and public keys are stored securely. If these files are kept on local disk, then consider restricting read and write access to these files or securing them behind a password. 4. Assign the public key to a Snowflake user. To do this: 1. Start the snowsql REPL in your terminal. Here's an example of starting snowsql for account `ABCDEF-ABC12345` with the user `jsmith` and warehouse `COMPUTE_WH`. For more information about using snowsql, see [Snowflake's documentation](https://docs.snowflake.com/en/user-guide/snowsql-use). ``` $ snowSQL -a ABCDEF-ABC12345 -u jsmith -w COMPUTE_WH ``` 2. Alter a user to set the `RSA_PUBLIC_KEY` property.\ \ If your public key looks like this... ``` $ cat rsa_key.pub -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwViztpm3l7IfLN/prlBb KJi/yTMngmTBKFp5YOyWqswlXwVlH6zehbCgFYxjo36RxxDtwqTSzaEalraalvjE aRKKb+esi8TYvbd1eiJGJqoq5WjTIHs0lwRMwLkCn6mHX9I4J0Wp6mR2DhjWvey6 IE4aKT0kXrKChs/GCOGFi3lAIRIXy9FQusYW2OyNoG1iogFLoWPH1Od5bmVl6TJv xXXKeUj0omTq+3CKZxELxgr5nYU44MQgq/NH1X+nEjIiA0W2RiS6heKNgwRUg2EQ 8UXbrHCjhgnONRnnHGukh9CpH4KTTmQc2hifTbH7g39f31hrEWBfVTECESUAY+qz kwIDAQAB -----END PUBLIC KEY----- ``` \ ...then the command to set the `RSA_PUBLIC_KEY` for user `jsmith` is the following: ``` ALTER USER jsmith SET RSA_PUBLIC_KEY='MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwViztpm3l7IfLN/prlBb KJi/yTMngmTBKFp5YOyWqswlXwVlH6zehbCgFYxjo36RxxDtwqTSzaEalraalvjE aRKKb+esi8TYvbd1eiJGJqoq5WjTIHs0lwRMwLkCn6mHX9I4J0Wp6mR2DhjWvey6 IE4aKT0kXrKChs/GCOGFi3lAIRIXy9FQusYW2OyNoG1iogFLoWPH1Od5bmVl6TJv xXXKeUj0omTq+3CKZxELxgr5nYU44MQgq/NH1X+nEjIiA0W2RiS6heKNgwRUg2EQ 8UXbrHCjhgnONRnnHGukh9CpH4KTTmQc2hifTbH7g39f31hrEWBfVTECESUAY+qz kwIDAQAB'; ``` **Note** Only security administrators (users with the `SECURITYADMIN` role) or higher can alter a user. For more information on setting up key pair authentication, see [*Snowflake documentation*](https://docs.snowflake.com/en/user-guide/key-pair-auth)*.* ## Adding Snowflake as a DeltaStream Data Store 1. Open DeltaStream. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ) and then click **+ Add Data Store**.

DeltaStream Data Store List

2. From the menu that displays, click **Snowflake.** The **Add Data Store** window opens.

Adding a Snowflake Data Store

3. Enter the authentication and connection parameters. These include: * **Store Type** – `Snowflake.` * **Name** – A name to identify your Snowflake data store (See [Data Store](/overview/core-concepts/store)). * **Uris to connect** – URI for Snowflake account in the format `https://.snowflakecomputing.com` (see [Snowflake Account Identifiers documentation](https://docs.snowflake.com/en/user-guide/admin-account-identifier#format-1-preferred-account-name-in-your-organization)). * **Snowflake Cloud Region** – Location of your Snowflake data store (See [Region](/overview/core-concepts/region)) * **Account ID** – Account identifier in the form `-` (see [Snowflake Account Identifiers documentation](https://docs.snowflake.com/en/user-guide/admin-account-identifier#format-1-preferred-account-name-in-your-organization)). * **Role Name** – The name of the access control role to use for the data store operations after connecting to Snowflake (for more information see [Snowflake documentation on Roles](https://docs.snowflake.com/en/user-guide/security-access-control-overview#roles)). To find roles in Snowflake, click the top user profile icon and select from the dropdown menu.

Selecting a Snowflake Role

* **Warehouse Name** – The name for a Snowflake compute warehouse to use for queries and other data store operations that require compute resources (see [#add-snowflake-warehouse](#add-snowflake-warehouse "mention")). * **Username** – User login name for the Snowflake account. * **Client Key File** – Snowflake account's private key in PEM format (see [#create-snowflake-key-pair-authentication](#create-snowflake-key-pair-authentication "mention")). * **Key Passphrase** – If applicable, passphrase for decrypting the Snowflake account's private key (see [#create-snowflake-key-pair-authentication](#create-snowflake-key-pair-authentication "mention")). 4. Click **Add** to save and create the data store. *For instructions on creating the* data *store using DSQL, see* [CREATE STORE](/reference/sql-syntax/ddl/create-store). ## Process Streaming Data and Sink to Snowflake To follow the next few steps, you must already have a [stream](/overview/core-concepts/databases#stream) defined in DeltaStream labeled `pageviews,` which is backed by a topic in an Apache Kafka data store. You also must have a Snowflake store labeled `snowflake_store` (see [#adding-snowflake-as-a-deltastream-store](#adding-snowflake-as-a-deltastream-store "mention")). When you have those, you can perform a simple filter on the `pageviews` stream and sink the results into Snowflake. {% hint style="info" %} **Note** For details on how to set up a stream or a Kafka store, see [Starting with the Web App](/getting-started/starting-with-web-app) or [Starting with the CLI](/getting-started/starting-with-cli). {% endhint %} ### Inspect the Snowflake data store 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing data stores.

DeltaStream Store List

2. Click `snowflake_store`. The store page opens, displaying a list of the existing databases in your Snowflake account for the role you specified as part of the store's creation:
3. (Optional) Create a new database. To do this: * Click **+ Add Database**. When prompted, enter a name for the new database and click **Add**. The new database displays in the list.
4. Click a database name to see the schemas that exist under that database. 5. (Optional) Create a new schema. To do this: 1. Click **+ Add Schema**. 2. In the window that opens, enter a name for the new schema and click **Add**. The new schema displays in the schema list. 6. To see the tables that exist under a particular schema, click the schema name. ### Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into Snowflake 1. In the lefthand navigation, click **Workspace** ( ![](/files/ZXcAkgugP7AuG9QFRXKO) ). 2. In the SQL pane of your workspace, write the [CREATE TABLE AS SELECT (CTAS)](/reference/sql-syntax/query/create-table-as) query to ingest from **pageviews** and output to a new table titled **pv\_table**. ```sql CREATE TABLE pv_table WITH ( 'store' = 'snowflake_store', 'snowflake.db.name' = 'new_db', 'snowflake.schema.name' = 'new_schema' ) AS SELECT viewtime, pageid, userid FROM pageviews WHERE pageid != 'Page_3'; ``` 3. Click **Run**. 4. In the lefthand navigation click **Queries** ( ![](/files/HOEvY09XthGMf2h6wEx6) ) to view the existing queries, including the query from the step immediately prior.\ It may take a few moments for the query to transition into the **Running** state. Keep refreshing your screen until the query transitions.
### View the results 1. In the lefthand navigation, click **Resources (** ![](/files/Zwq1BBdRyaRsv55N3KNm) ). This displays a list of the existing stores. 2. To view the new table created by the above CTAS, navigate to `snowflake_store` --> `Databases` --> `new_database` --> `new_schema` --> `pv_table`.\ Of course, if you wrote your CTAS such that the store/catalog/schema/table names are different, navigate accordingly. 3. To view a sample of the data in your Snowflake table, click **Print**.
# WarpStream **WarpStream** is an [Apache Kafka®](https://kafka.apache.org/) compatible data streaming platform built directly on object storage. It’s delivered as a single, stateless Go binary that eliminates much of the manual, error-prone Kafka housekeeping. WarpStream, automatically handles tasks such as * managing local disks * rebalancing brokers * operating ZooKeeper WarpStream clusters can scale out, and in, instantaneously, without rebalancing. This document walks you through setting up WarpStream to be used as a data [Data Store](/overview/core-concepts/store) in DeltaStream. ## Step 1: Setting up the WarpStream credentials ### Prerequisites 1. DeltaStream account - get access to DeltaStream by registering [here](https://console.deltastream.io/). 2. WarpStream account - get access to WarpStream by registering [here](https://console.warpstream.com/signup). 3. A WarpStream cluster up and running with active, populated topics. ### Create a WarpStream credential 1. Navigate to your WarpStream console and select the appropriate cluster. 2. In the top row navigation, click **Credentials**.
3. Name the credential and then click **+ Create Credentials**. Below we created the **dsCred** credential. 4. Follow the instructions, and save the credentials in a safe place. ## Step 2: Connect DeltaStream to WarpStream 1. Back in DeltaStream, navigate to the **Resources** page. 2. Click **+ Add Data Store,** and when the **Choose a Data Store** window displays, click **Kafka**.
2. Complete the requested information as follows: 1. **Name** - In this example, **WarpStream.** 2. **URLs to connect -** Applicable bootstrap server and port — here, **serverless.warpstream.com:9092** 3. **SASL Hash Function** - select **SHA512** 4. **Username** - The username from the credential you created in WarpStream 5. **Password** - The password from the credential you created in WarpStream
When you're done, click **Add**. ### Next Steps Everything is now configured for you to define [**DeltaStream Objects**](https://docs.deltastream.io/tutorials/relation) and start executing [**Queries**](/reference/sql-syntax/query) against your Apache Kafka® compatible WarpStream data stores. # Setting up Enterprise Security Integrations {% content-ref url="/pages/KhO5Mwh9rsAOicaIN0u6" %} [Okta SAML Integration](/enterprise-security-integrations/okta-saml-integration) {% endcontent-ref %} {% content-ref url="/pages/PM4VJXS8cvHEPjNBLgO5" %} [Okta SCIM Integration](/enterprise-security-integrations/okta-scim-integration) {% endcontent-ref %} # Okta SAML Integration SAML (Security Assertion Markup Language) is an open standard for exchanging security information and providing single sign-on (SSO) between 2 parties: an identity provider (IdP) and a service provider (SP). This document walks you through setting up SAML-based authentication between DeltaStream (SP) and [Okta (IdP)](https://developer.okta.com/docs/concepts/saml/). Find the Okta documentation for configuring the SAML integration at ### Initial Okta Setup 1. Log into your Okta dashboard. Then from the applications menu click **Create App Integration**.

New app integration

2. For the sign-in method, click **SAML 2.0** and then click **Next**.

SAML 2.0

3. Update the **General Settings** for the DeltaStream app integration. Optionally, also set up the [DeltaStream Logo](http://deltastream-static-assets.s3-website-us-west-2.amazonaws.com/LogoVertical.png) for your application.
4. Set up the SAML integration with the following values: {% hint style="warning" %} **Important** You must retain the default values for any setting not mentioned below. {% endhint %} #### General settings
FieldValue
Single sign-on URLhttps://auth.deltastream.io/login/callback?connection=ds-okta-saml-placeholder
Audience URIurn:auth0:deltastream:ds-okta-saml-placeholder
Name ID formatEmailAddress
Application UsernameEmail
Update application username onCreate and update
#### Attribute Statements Click **Add another** and add the following mappings:
NameName formatValue
emailBasicuser.email
firstNameBasicuser.firstName
lastNameBasicuser.lastName
The screenshot below provides a visual reference. Enter the required values and then click **Next**.
5. Contact DeltaStream at [ops@deltastream.io](mailto:http://deltastream-static-assets.s3-website-us-west-2.amazonaws.com/LogoVertical.png) and provide the following information: * Your company's name * Your company's email domain * First Org admin's email address (you can add more later) * Saml 2.0 Metadata URL Navigate to the SAML Application `Sign On` tab for the values for Metadata URL. Do not use values from the `General` tab. ### Complete Okta Setup After the integration is complete, DeltaStream provides you with the final **Audience URI,** which you must update in Okta. To do this: 1. Log into your OKTA dashboard. Then, from the **applications** menu, click **DeltaStream**.
2. Click to activate the **General** tab. Then, for **SAML Settings**, click **Edit**.
3. On the **General Settings** page, click **Next**.
4. Update the **Single sign-on URL** with the value DeltaStream provides you. 5. Update the **Audience URI (SP Entity ID)** with the value DeltaStream provides you. 6. Click **Next** and **Finish** to complete editing the Okta integration. ## References * * # Okta SCIM Integration SCIM (System for Cross-domain Identity Management) is an open standard for automating the management of user and group membership. This document walks you through setting up SCIM-based users and groups with Okta. Find Okta's documentation for these steps at ## DeltaStream and Okta SCIM Concepts DeltaStream's SCIM integration automates adding/removing [users](/overview/core-concepts/access-control#_user) from your [organization](/overview/core-concepts/access-control#_organiation) and leveraging Okta groups to grant access to DeltaStream [roles](/overview/core-concepts/access-control#_role). There are two types of groups within Okta: 1. **Assignment groups** manage [access to Okta App Integrations](https://help.okta.com/en-us/content/topics/apps/apps-assign-applications.htm). Users added to the assignment group are automatically added to DeltaStream but are not assigned any custom roles. 2. **Push Groups** enable Okta to [push existing Okta group memberships](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-about-group-push.htm) to DeltaStream. These groups are mapped to [custom roles](/overview/core-concepts/access-control#custom-roles) within DeltaStream. You cannot use Push groups to control access to built-in roles. Instead, use a custom user attribute and a dedicated Assignment group to achieve `orgadmin` role membership. Custom roles created using Push groups are not automatically granted any privileges. You must grant privileges to the roles with `MANAGE_GRANTS` privilege (by default, `orgadmin` or `securityadmin`). {% hint style="info" %} **Note** You must add a user to an Assignment group before adding them to a Push group. {% endhint %} ## SCIM Setup ### Prerequisites * Set up Okta [SAML app integration](/enterprise-security-integrations/okta-saml-integration). ### Enable SCIM Provisioning This section describes how to enable SCIM provisioning on an app integration. Additional configuration is added in subsequent sections. 1. Click to activate the **General** tab, and in the **App Settings** box click **Edit**.
2. Check **Enable SCIM provisioning** and then click **Save**.
### Set up the DeltaStream OrgAdmin Attribute This section describes how to create a new custom attribute you use to specify who should have access to the OrgAdmin built-in role. 1. In the lefthand navigation, go to **Directory** > **Profile Editor**. 2. Click **DeltaStream Users**.
3. Click **Add Attribute**.
4. Create a new boolean attribute with the following values:
KeyValue
Data typeboolean
Display nameDeltaStream OrgAdmin
Variable namedeltastreamOrgadmin
External namedeltastreamOrgadmin
External namespaceurn:ietf:params:scim:schemas:core:2.0:User
Verify that the **Attribute Type** is set to **Group**.
### Set up the DeltaStream OrgAdmin User Assignment Group This section describes how to create a new assignment group for OrgAdmins. Any users assigned to this group are granted the OrgAdmin role in DeltaStream. 1. In the lefthand navigation go to **Directory** -> **Groups** and then click **Add group**.
2. Name the group **DeltaStream OrgAdmins** and then click **Save**.
3. Click the newly-created group to configure it. 4. Click the **Applications** tab to activate it and then click **Assign applications**.
4. Assign the DeltaStream application.
5. Verify the **DeltaStream OrgAdmin** attribute is set to **true**. Then click **Save and Go Back**.
### Set up the DeltaStream User Assignment Group This section describes how to create a new assignment group for non-privileged users. 1. In the lefthand navigation, go to **Directory** > **Groups** and then click **Add group**.
2. Name the group **DeltaStream Users** and click **Save**.
3. Click the newly-created group to configure it. 4. Click the **Applications** tab to activate it, and then click **Assign applications**.
4. Assign the DeltaStream application and then click **Save**. ## SCIM user provisioning ### Assign someone to the OrgAdmin assignment group 1. In the lefthand navigation, go to **Application** > **Applications** > **Deltastream**. 1. Click to activate the **Assignments** tab. 2. Filter by **Groups.** 3. Click the **DeltaStream OrgAdmins** group.
2. Click **Assign people** and then select the individuals you wish to assign as OrgAdmins. {% hint style="info" %} **Note** Ensure the person provided as the OrgAdmin for SAML application setup is also added to this group. {% endhint %}
### Assign a user to the Users assignment group 1. From the lefthand navigation to **Application** > **Applications** > **Deltastream**. 1. Click to activate the **Assignments** tab. 2. Filter by **Groups**. 3. Click the **DeltaStream Users** group.
2. Click **Assign people** and select the individuals you wish to have access to DeltaStream. {% hint style="info" %} **Note** Assigning users does not grant them any additional roles. Configure a Push group to assign roles. {% endhint %}
## Configure security integration This section describes how to configure the SCIM integration URI and token so that Okta can push information to DeltaStream. 1. In the top toolbar, change the role to orgadmin. 2. In the lefthand navigation click Integration ( ![](/files/ihbqQ2Hpfut9j4xuNIac) ). Then click **Security Integration**.
2. Save the URI and token for later use. This security integration expires after 1 year. 3. In your Okta dashboard, in the lefthand navigation click **Applications** and then click the **DeltaStream** application:
4. Click the **Provisioning** tab to activate it, then click **edit**:
4. Copy the URI from the security integration setup in step 1 to the SCIM connector base URL. 5. Enter `email` for unique identifier field for users. 6. Check the following: 1. **Import New Users and Profile Updates** 2. **Push New Users** 3. **Push Profile Updates** 4. **Push Groups**
7. For **Authentication Mode**, click **HTTP Header**. Then copy the token from the security integration setup in step (1) and paste it into the **Authorization Bearer** box. Then click **Save**.
8. Click the **Provisioning** tab to activate it, and in the **Provisioning to App** settings, click **Edit**.
9. Check the following: 1. **Create Users** 2. **Update User Attributes** 3. **Deactivate Users** 10. Click **Save**.
## Configure Push groups Before you begin, ensure that everyone who needs access to DeltaStream has been added either to the **DeltaStream Users** or the **DeltaStream OrgAdmins** assignment groups. 1. From the lefthand navigation, go to **Application** > **Applications** > **Deltastream**. 2. Click the **Push Groups** tab to activate it. 3. Click **+ Push Groups** to search for and select a group to push.
2\. Click **Find groups by name**.
3. Enter the name of the group you wish to push -- for example, **development.** 4. Click **Push group memberships immediately**. 5. Click **Save**.
This creates a role with the same name as the group in DeltaStream. Anyone who is part of the group is also assigned this role. ## References * * * # Using an AWS S3 Store as a Source to Feed an MSK Topic This article examines how you can load any data from AWS S3 into DeltaStream to enrich other DeltaStream objects, before writing the final data into any other supported data store. ### Before you begin * You must already have an [Amazon Web Services](https://aws.amazon.com/) account ### Creating the stream 1. Create a stream from the S3 file. ```sql CREATE STREAM bronze_taxi_json ( "VendorID" BIGINT, tpep_pickup_datetime BIGINT, -- epoch-seconds tpep_dropoff_datetime BIGINT, passenger_count DOUBLE, trip_distance DOUBLE, "RatecodeID" DOUBLE, store_and_fwd_flag STRING, "PULocationID" BIGINT, "DOLocationID" BIGINT, payment_type BIGINT, fare_amount DOUBLE, extra DOUBLE, mta_tax DOUBLE, tip_amount DOUBLE, tolls_amount DOUBLE, improvement_surcharge DOUBLE, total_amount DOUBLE, congestion_surcharge DOUBLE, airport_fee DOUBLE) WITH ( 'store' = 'yellow-taxi-s3', 'timestamp'= 'tpep_pickup_datetime', 'value.format'='JSONL', 's3.uri' = 's3://s3-demo-bucket/yellow-taxi'); ``` {% hint style="success" %} **Tip** You do not need to create a new stream for each s3 file. Instead, define the stream for a folder. The stream reads all the existing files and waits for new files to read as they arrive. Simply have the URI point to the folder; pointing to the folder watches all existing and future files. This is the default behavior. {% endhint %} From this point forward, you can treat this stream as you would any other stream in DeltaStream. {% hint style="warning" %} **Important** The default query size is 2 GB. If the files in your S3 bucket are larger, you may experience memory errors. To avoid this you can modify file size to enlarge it so the system reads from your bucket. To do this, append the following clause to your query: `WITH (`\ `'query.memory.size' = '3Gi'`\ `);` {% endhint %} # Building a Proactive GenAI Agent with Real-Time Context In this tutorial, we will build a sophisticated, end-to-end solution featuring a Generative AI agent that can proactively manage airline flight disruptions for high-value customers. This agent's effectiveness hinges on its ability to access and act upon real-time context—a capability we will power using DeltaStream. We will walk through every step, from generating simulated real-time data to building the data pipeline and exposing it to an agent through a Model Context Protocol (MCP) server. **The Use Case:** An airline wants to create a "digital concierge" agent. When a flight is canceled, this agent should immediately identify affected "Platinum" or "Gold" tier passengers and rebook them on the next best flight before the customer is even aware of the disruption. Acting within the "golden seconds" after a cancellation is critical, as the best alternative seats disappear almost instantly. #### Architecture Overview Our system will have five main components: 1. **Data Generator:** A Java application that simulates real-time flight events, bookings, and customer profile updates, publishing them to Kafka. 2. **Apache Kafka:** The message bus that will transport our live event streams. 3. **DeltaStream:** The real-time context engine. It will ingest the raw Kafka streams, use SQL to join and transform them into an actionable, materialized view of "at-risk" passengers. 4. **MCP Server:** A lightweight Java server that exposes the materialized view from DeltaStream via a simple API, acting as the bridge to our GenAI agent. 5. **GenAI Agent (OpenAI):** The agent configured in a platform like OpenAI's Agent Builder, which will call our MCP server to get the real-time context needed to make intelligent decisions. #### Prerequisites Before you begin, ensure you have the following: * Java (JDK 17 or later) and Maven installed. * Access to an Apache Kafka cluster. * A DeltaStream account. * `ngrok` or a similar tool to expose your local server to the internet. #### Step 1: Generate the Real-Time Data First, we'll create a Java application to simulate our airline's operations. This generator will produce three streams of data into three distinct Kafka topics. See the data generator Java class at DeltaStream examples repository [here](https://github.com/deltastreaminc/examples/blob/main/FlightDisruptionManagerAgent/AirlineDataGenerator.java). Before running the code, create the three required topics in your Kafka cluster: * `flight_events` * `booking_events` * `customer_profiles` Now, compile and run the `AirlineDataGenerator`. You should see log messages indicating that data is being produced to your Kafka topics. #### Step 2: Build the Real-Time Pipeline in DeltaStream Next, we will define the logic in DeltaStream to process these raw Kafka streams into a clean, actionable materialized view. Log into your DeltaStream environment and execute the following SQL statements. ```sql -- Step 1: Create STREAMS to represent the raw Kafka topics. -- DeltaStream will read from these topics continuously. CREATE STREAM flight_events ( flight_id VARCHAR, flight_status VARCHAR, origin VARCHAR, destination VARCHAR, scheduled_departure TIMESTAMP_LTZ, event_timestamp TIMESTAMP_LTZ ) WITH ( 'topic' = 'flight_events', 'value.format' = 'json', 'timestamp' = 'event_timestamp' ); CREATE STREAM booking_events ( booking_id VARCHAR, flight_id VARCHAR, passenger_id VARCHAR, seat VARCHAR, booking_timestamp TIMESTAMP_LTZ ) WITH ( 'topic' = 'booking_events', 'value.format' = 'json', 'timestamp' = 'booking_timestamp' ); -- For customer profiles, which is dimensional data (slowly changing), -- we create a changelog stream CREATE CHANGELOG customer_profiles_changelog ( passenger_id VARCHAR, first_name VARCHAR, last_name VARCHAR, loyalty_tier VARCHAR, preferences STRUCT, update_timestamp TIMESTAMP_LTZ, PRIMARY KEY(passenger_id) ) WITH ( 'topic' = 'customer_profiles', 'value.format' = 'json', 'timestamp' = 'update_timestamp' ); -- Step 2: Create the final Materialized View. -- This is the core of our real-time context generation. It joins the streams -- and filters for high-value customers on disrupted flights. CREATE STREAM enriched_booking_events AS SELECT booking_id, flight_id, be.passenger_id, seat, booking_timestamp, first_name, last_name, loyalty_tier, preferences FROM booking_events be LEFT JOIN customer_profiles_changelog cpc WITH ('source.idle.timeout.millis' = 1000) ON be.passenger_id = cpc.passenger_id; CREATE MATERIALIZED VIEW disrupted_premium_passengers AS SELECT ebe.passenger_id, ebe.first_name || ' ' || ebe.last_name AS full_name, ebe.loyalty_tier, f.flight_id, f.flight_status AS flight_status, f.origin, f.destination, f.event_timestamp AS disruption_timestamp, ebe.preferences->seating AS seating_preference FROM flight_events AS f -- Join flights with their bookings. -- We use a temporal window to limit the state we need to keep. -- This assumes a booking happens reasonably close to a flight event. INNER JOIN enriched_booking_events AS ebe WITHIN 1 DAY ON f.flight_id = ebe.flight_id WHERE -- We only care about disruptions f.flight_status IN ('CANCELED', 'DELAYED') AND -- We only care about our premium customers ebe.loyalty_tier IN ('PLATINUM', 'GOLD'); ``` Once these statements are executed, DeltaStream is actively processing your Kafka data. As soon as a `CANCELED` or `DELAYED` event for a flight with a "GOLD" or "PLATINUM" passenger occurs, an entry will instantly appear in the `disrupted_premium_passengers` Materialized View. #### Step 3: Expose the Real-Time Context via MCP Server Now we need a bridge between DeltaStream and our GenAI agent. The following MCP server code provides a standard interface that OpenAI's Agent Builder can understand. **3.1 MCP Server Code** Use the Java code from the examples repository [here](https://github.com/deltastreaminc/examples/blob/main/FlightDisruptionManagerAgent/FlightConciergeMCPServerV2.java). It creates a simple web server that listens for `POST` requests, queries DeltaStream's API, and returns the result. **3.2 Configure and Run the MCP Server** 1. **Edit the Code:** Open the `FlightConciergeMCPServer.java` file and replace the placeholder values for `DS_STATEMENTS_URL` and `DS_AUTH_TOKEN` with your own DeltaStream organization URL and a valid bearer token. 2. **Compile and Run:** Build and run the Java application. It will start a server, typically on port `8080`. #### Step 4: Connect the Agent The final step is to connect our real-time context to a GenAI agent. 1. **Expose Your Server:** Your MCP server is running locally. To make it accessible to a cloud service like OpenAI, use `ngrok`. ```bash ngrok http 8080 ``` `ngrok` will give you a public HTTPS URL (e.g., `https://.ngrok-free.app`). 2. **Configure the Agent:** In OpenAI's Agent Builder (or a similar platform), create a new agent. * From the Tools section add a new **MCP Server** and provide the `ngrok` URL, making sure to append the `/mcp` path (e.g., `https://.ngrok-free.app/mcp`). The builder will automatically discover the `get_passenger_context` tool defined in our MCP server. * Give the agent its instructions. This is the crucial prompt that defines its personality and goal. **Example Agent Instructions:** > You are an elite airline concierge agent. Your primary goal is to manage flight disruptions for Platinum and Gold tier passengers with extreme urgency. When alerted, your first and only action should be to use the `get_passenger_context` tool to fetch real-time data for the affected passenger. Based on the context, your task is to find a new flight and communicate the solution to the customer clearly and calmly. Do not ask the user for information; use your tools. #### Step 5: See It in Action! With everything running, watch the magic happen: 1. The `AirlineDataGenerator` produces a `CANCELED` event for a flight carrying a "PLATINUM" passenger. 2. The event flows through Kafka to DeltaStream in milliseconds. 3. The `disrupted_premium_passengers` Materialized View is instantly updated with the passenger's details. 4. Trigger your agent (e.g., in the Agent Builder playground) with a simple prompt like "Handle disruption for PASS-1042". 5. The agent will call your MCP server, which queries DeltaStream and returns the fresh, actionable context. 6. The agent, now fully aware of the situation, can proceed to complete its mission. Congratulations! You have successfully built a sophisticated, real-time AI agent that solves a critical business problem by leveraging the power of fresh, streaming data. # Metrics # Prometheus Integration Add DeltaStream metrics endpoint as a Prometheus Metrics Endpoint [Prometheus](https://github.com/prometheus) is an open-source systems monitoring and alerting toolkit that collects and stores its metrics as time series data. In other words, it stores metrics information with the timestamp at which the data was recorded, alongside optional key-value pairs called labels. Many organizations have adopted Prometheus as part of their observability stack. This document walks you through how to set up a Prometheus integration with DeltaStream's metrics. It includes the DeltaStream metrics that are exposed. We use Grafana screenshots as the example of how to add the Prometheus integration. {% hint style="info" %} **Note** You can only scrape the DeltaStream metrics endpoint every 60 seconds. {% endhint %} ## Step 1: Create DeltaStream Metrics Token ## Step 2: Add new Prometheus Endpoint to Scrape DeltaStream Metrics 1. Add DeltaStream's metric endpoint generated from Step 1 as an additional scrape metric config in your Prometheus deployment. ```yaml - job_name: deltastream-metrics metrics_path: /metrics scheme: https scrape_interval: 60s scrape_protocols: [ "PrometheusText0.0.4" ] authorization: type: Bearer credentials: static_configs: - targets: - "api-atyo2.deltastream.io" # [ your URI from Step 1 ] labels: : "" ``` # Built-In Metrics Default deltastream query metrics provided by Prometheus integration Queries in DeltaStream are long-lived and continuous. While DeltaStream provides its own tools to notify and alert you if your queries are misbehaving, you may want also to monitor queries from their own observability stacks. This document lists the built-in metrics that DeltaStream exposes so you can monitor your queries. See [`CREATE METRICS INTEGRATION`](/reference/sql-syntax/ddl/create-metrics-integration) for consuming the metrics in this document.
Metric NameValue TypeDescription
deltastream_query_consumerRecordsLagMaxIntegerReports the Kafka Consumer lag in number of records that are currently behind.
deltastream_query_numRecordsInPerSecondIntegerReports current number of records being processed in per second.
deltastream_query_numRecordsOutPerSecondIntegerReports current number of records being processed out per second.
deltastream_query_numBytesInPerSecondIntegerReports current size of records being processed in per second. Values returned as binary bytes unit.
deltastream_query_numBytesOutPerSecondIntegerReports current size of records being processed out per second. Values returned as binary bytes unit.
# Custom Metrics in Functions [Functions](/overview/core-concepts/function) enable you to define your own processing logic and make it available in SQL. This document walks you through how to add custom metrics to functions. DeltaStream queries are powered by Apache Flink; you can implement functions as Apache Flink user-defined functions as described in the [Apache Flink documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/). See [`CREATE METRICS INTEGRATION`](/reference/sql-syntax/ddl/create-metrics-integration) for consuming the metrics in this document. You can add 3 types of custom metrics (see [Flink docs on Metrics](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/metrics/#registering-metrics) for more information): 1. Counter: an integer value that can be incremented or decremented. 2. Meter: a value that measures the average throughput of events over a time period. 3. Gauge: a value of any type that can be retrieved on demand. This value is set in the function's logic. The code block below demonstrates how you can add a custom metric (counter) to a Java function: ```java public class UdfWithMetrics extends ScalarFunction { // IMPORANT NOTE: For UDF metrics to be available, metric group name must be "deltastream_udf" private final String METRIC_GROUP = "deltastream_udf"; private transient Counter counter; private transient Meter meter; private transient int gaugeValue = 0; @Override public void open(FunctionContext context) { this.counter = context.getMetricGroup() .addGroup(METRIC_GROUP) .counter("myCounter"); this.meter = context.getMetricGroup() .addGroup(METRIC_GROUP) .meter("myMeter", new MeterView(60)); context.getMetricGroup().addGroup(METRIC_GROUP) .gauge("myGauge", (Gauge) () -> gaugeValue); } public String eval(String s) { if (s == null) { meter.markEvent(); return null; } if (s.startsWith("Value_")) { try { int val = Integer.parseInt(s.substring(6)); gaugeValue = val; } catch(NumberFormatException e){ counter.inc(); } } return s.replaceAll("_", " "); } } ``` {% hint style="warning" %} **Important** In the `context.getMetricGroup().addGroup(METRIC_GROUP)` method, the `METRIC_GROUP` value is `deltastream_udf`. You **must** set the metric group name to this value for the DeltaStream platform to properly scrape and make this metric available. {% endhint %} For the full code example, [see this example](https://github.com/deltastreaminc/deltastream-examples/blob/main/udf/java/examples/src/main/java/examples/UdfWithMetrics.java). After [setting up a metrics integration](/reference/metrics/prometheus-integration), you can find your metrics with prefix `deltastream_udf`. For instance, the `myCounter` metric in the code block above has the metric name `deltastream_udf_myCounter`. # SQL Syntax # Data Formats (Serialization) ### Protocol Buffers and Descriptors A **Descriptor** defines the data serialization format for a record’s native data format. Descriptors are defined for an entity within a [Store](/overview/core-concepts/store). This provides different serialization formats for records within a store, while defining a clear schema definition for a stream of records through the store. See [Working with ProtoBuf Serialized Data and DeltaStream Descriptors](/how-do-i.../serialization/working-with-protobuf-serialized-data-and-deltastream-descriptors) and [CREATE DESCRIPTOR\_SOURCE](/reference/sql-syntax/ddl/create-descriptor_source) for more information on how to import and create descriptors in DeltaStream. Currently, DeltaStream uses descriptors to support data in ProtoBuf format. ### JSON A JSON serialization format is assumed for an [entity](/overview/core-concepts/store#entity), if no [schema registry](/overview/core-concepts/store#schema-registry) is defined for the corresponding store and no descriptor is defined for the entity. If a schema registry is present in the store and a descriptor is defined for an entity, DeltaStream uses the descriptor to serialize that entity. ### Avro and Schema Registry A [schema registry](/overview/core-concepts/store#schema-registry) is a service to manage message schemas in streaming stores such as Apache Kafka. Message schemas are used to serialize and deserialize messages stored in entities. In DeltaStream, a schema registry is a representation of a schema registry service that can be used to fetch and store schemas for entities in the service. A store can use one schema registry at a time, but a schema registry can be used by multiple stores. When a schema registry is attached to a store, any entity in that store using data serialization formats requiring a schema registry will use the store’s associated schema registry to fetch the schemas/metadata necessary for marshalling and unmarshalling data events. The schema for an event, fetched by the schema registry, should not be confused with the [schema](/overview/core-concepts/databases#_schema) belonging to a [database](/overview/core-concepts/databases) for organizing relations. # Serializing with JSON ## Setup The following describes through examples how a DeltaStream query converts JSON payloads to DeltaStream’s [Data Types](/reference/sql-syntax/data-types) when reading from a [Database](/overview/core-concepts/databases#_stream) or [Database](/overview/core-concepts/databases#_changelog). The following examples use the stream defined below: ```sql CREATE STREAM jsonExample ( "booleanValue" BOOLEAN, "stringValue" VARCHAR, "tinyIntValue" TINYINT, "smallIntValue" SMALLINT, "intValue" INTEGER, "bigIntValue" BIGINT, "floatValue" FLOAT, "doubleValue" DOUBLE, "decimalValue" DECIMAL(4, 3), "dateValue" DATE, "timeValue" TIME, "timestampValue" TIMESTAMP(3), "timestampLtzValue" TIMESTAMP_LTZ, "bytesValue" VARBINARY, "arrayValue" ARRAY, "mapValue" MAP, "structValue" STRUCT ) WITH ( 'topic' = 'jsonExample', 'value.format' = 'JSON' ); ``` ## Simple Example With the query: ```sql SELECT * FROM jsonExample; ``` 
// input record
{
  "booleanValue": true,
  "stringValue": "howdy",
  "tinyIntValue": 1,
  "smallIntValue": 12,
  "intValue": 1234,
  "bigIntValue": 123456789,
  "floatValue": 12.34,
  "doubleValue": 1234.5678,
  "decimalValue": 1.123,
  "dateValue": "2019-12-26",
  "timeValue": "16:15:14",
  "timestampValue": "2011-12-03 10:15:30",
  "timestampLtzValue": "2021-05-31 16:15:14.528Z",
  "bytesValue": "aG93ZHk=",
  "arrayValue": [
    "News",
    "Travel"
  ],
  "mapValue": {
    "count": 17
  },
  "structValue": {
    "col1": 1234
  }
}
```json // output record { "booleanValue": true, "stringValue": "howdy", "tinyIntValue": 1, "smallIntValue": 12, "intValue": 1234, "bigIntValue": 123456789, "floatValue": 12.34, "doubleValue": 1234.5678, "decimalValue": 1.123, "dateValue": "2019-12-26", "timeValue": "16:15:14", "timestampValue": "2011-12-03 10:15:30", "timestampLtzValue": "2021-05-31 16:15:14.528Z", "bytesValue": "aG93ZHk=", "arrayValue": [ "News", "Travel" ], "mapValue": { "count": 17 }, "structValue": { "col1": 1234 } } ``` ## Partial Record Example When JSON records are missing fields specified by the `CREATE STREAM DDL` statement, those fields are given the value `NULL` in the output record. In the opposite case — when JSON records have fields that aren’t specified by the `CREATE STREAM DDL` statement — those fields are ignored. With the query: ```sql SELECT "booleanValue", "stringValue", "intValue" FROM jsonExample; ``` ```json // input record { "booleanValue": true, "stringValue": "howdy", "someOtherValue": 123 } // output record { "booleanValue": true, "stringValue": "howdy", "intValue": null } ``` ## Mismatched Types Example ### Boolean With the query: ```sql SELECT "booleanValue" FROM jsonExample; ``` ```json // input record { "booleanValue": true } //output record { "booleanValue": true } ``` ```json // input record { "booleanValue": false } //output record { "booleanValue": false } ``` ```json // input record { "booleanValue": "true" } //output record { "booleanValue": true } ``` ```json // input record { "booleanValue": "false" } //output record { "booleanValue": false } ``` ```json // input record { "booleanValue": "abc" } //output record { "booleanValue": false } ``` ```json // input record { "booleanValue": 123 } //output record { "booleanValue": false } ``` ### Character String With the query: ```sql SELECT "stringValue" FROM jsonExample; ``` ```json // input record { "stringValue": "abc" } //output record { "stringValue": "abc" } ``` ```json // input record { "stringValue": true } //output record { "stringValue": "true" } ``` ```json // input record { "stringValue": 123 } //output record { "stringValue": "123" } ``` ```json // input record { "stringValue": 123.456 } //output record { "stringValue": "123.456" } ``` ### Numeric With the query: ```sql SELECT "tinyIntValue", "doubleValue", "decimalValue" FROM jsonExample; ``` ```json // input record { "tinyIntValue": 1, "doubleValue": 123.1, "decimalValue": 1.123 } // output record { "tinyIntValue": 1, "doubleValue": 123.1, "decimalValue": 1.123 } ``` ```json // input record { "tinyIntValue": "1", "doubleValue": "123.1", "decimalValue": "1.123" } // output record { "tinyIntValue": 1, "doubleValue": 123.1, "decimalValue": 1.123 } ``` ```json // input record ("decimalValue" value is larger than defined precision) { "tinyIntValue": 1, "doubleValue": 123.1, "decimalValue": 12.123 } // output record { "tinyIntValue": 1, "doubleValue": 123.1, "decimalValue": null } ``` ```json // input record { "tinyIntValue": "abc", "doubleValue": 123.1, "decimalValue": 1.123 } // deserialization error because String can't be cast to numeric value ``` ```json // input record { "tinyIntValue": 130, "doubleValue": 123.1, "decimalValue": 1.123 } // deserialization error because 130 is out of range for TINYINT values ``` ```json // input record { "tinyIntValue": 1.1, "doubleValue": 123.1, "decimalValue": 1.123 } // deserialization error because TINYINT cannot be a floating point value ``` ### Date and Time ```sql SELECT "dateValue", "timeValue", "timestampValue", "timestampLtzValue" FROM jsonExample; ``` ```json // input record { "dateValue": "2019-12-26", "timeValue": "16:15:14", "timestampValue": "2011-12-03 10:15:30", "timestampLtzValue": "2021-05-31 16:15:14.528Z" } // output record { "dateValue": "2019-12-26", "timeValue": "16:15:14", "timestampValue": "2011-12-03 10:15:30", "timestampLtzValue": "2021-05-31 16:15:14.528Z" } ``` ```json // input record { "dateValue": 1234 } // deserialization error because date and time values must be parsed from Strings ``` ```json // input record { "dateValue": "2019-04-31" } // deserialization error due to invalid date (April only has 30 days) ``` ### Binary String With the query: ```sql SELECT "bytesValue" FROM jsonExample; ``` ```json // input record { "bytesValue": "aG93ZHk=" } // output record { "bytesValue": "aG93ZHk=" } ``` ```json // input record { "bytesValue": 1 } // deserialization error because we expect a String value ``` ### Constructed Data Types With the query: ```sql SELECT "arrayValue", "mapValue", "structValue" FROM jsonExample; ``` ```json // input record { "arrayValue": [ "News", "Travel" ], "mapValue": { "count": 17 }, "structValue": { "col1": 1234 } } // output record { "arrayValue": [ "News", "Travel" ], "mapValue": { "count": 17 }, "structValue": { "col1": 1234 } } ``` #### Array ```json // input record { "arrayValue": [] } // output record { "arrayValue": [], "mapValue": null, "structValue": null } ``` ```json // input record { "arrayValue": [ 17 ] } // output record { "arrayValue": [ "17" ], "mapValue": null, "structValue": null } ``` 
// input record
{
  "arrayValue": []
}

// output record
{
  "arrayValue": [],
  "mapValue": null,
  "structValue": null
}
#### Map ```json // input record { "mapValue": { "count": 17, "index": 102 } } // output record { "arrayValue": null, "mapValue": { "count": 17, "index": 102 }, "structValue": null } ``` ```json // input record { "mapValue": { "count": "howdy" } } // deserialization error because String can't be cast to numeric value ``` ```json // input record { "mapValue": {} } // output record { "arrayValue": null, "mapValue": {}, "structValue": null } ``` #### Struct ```json // input record { "structValue": { "col1": 1234, "col2": 5678 } } // output record { "arrayValue": null, "mapValue": null, "structValue": { "col1": 1234 } } ``` ```json // input record { "structValue": { "col2": 5678 } } // output record { "arrayValue": null, "mapValue": null, "structValue": { "col1": null } } ``` ```json // input record { "structValue": {} } // output record { "arrayValue": null, "mapValue": null, "structValue": { "col1": null } } ``` # Serializing with Primitive Data Types Serializing with primitive data types supports character strings, binary strings, and numeric data types. Below are a few (non-exhaustive) examples of serialization with primitive data types. See [the full list of supported primitive types](https://docs.deltastream.io/reference/sql-syntax/data-types#primitive-data-types). ### Character String With the following DDL and query: ```sql CREATE STREAM primitiveExample ( "stringValue" VARCHAR ) WITH ( 'topic' = 'primitiveExample', 'value.format' = 'Primitive' ); ``` ```sql SELECT * FROM primitiveExample; ``` For the given input records: ```json // input record bytes as String abc // output will read bytes as String { "stringValue": "abc" } ``` ```json // input record bytes as Integer 123 // output record will read bytes as String { "stringValue": "\u0000\u0000\u0000{" } ``` ### Numeric With the following DDL and query: ```sql CREATE STREAM primitiveExample ( "intValue" INTEGER ) WITH ( 'topic' = 'primitiveExample', 'value.format' = 'Primitive' ); ``` ```sql SELECT * FROM primitiveExample; ``` For the given input records: ```json // input record bytes as Integer 123 // output will read bytes as Integer { "intValue": 123 } ``` ```json // input record bytes as String abcd // output will be read bytes as String // It's also likely for other string bytes that there will be a // deserialization error while attempting to cast String to Integer { "intValue": 1633837924 } ``` ### Binary String With the following DDL and query: ```sql CREATE STREAM primitiveExample ( "bytesValue" VARBINARY ) WITH ( 'topic' = 'primitiveExample', 'value.format' = 'Primitive' ); ``` ```sql SELECT * FROM primitiveExample; ``` For the given input records: ```json // input record bytes as Bytes aG93ZHk= // output will read bytes as Bytes { "bytesValue": "aG93ZHk=" } ``` # Serializing with Protobuf The following table lists the data type mapping from Protobuf type to DeltaStream's [Data Types](/reference/sql-syntax/data-types). | Protobuf Type | DeltaStream Data Type | | ---------------------------------------- | --------------------- | | INT64, SINT64, SFIXED64, UINT64, FIXED64 | BIGINT | | BYTES | VARBINARY | | BOOL | BOOLEAN | | DOUBLE | DOUBLE | | FLOAT | FLOAT | | INT32, SINT32, SFIXED32, UINT32, FIXED32 | INTEGER | | STRING, ENUM | VARCHAR | | MAP | MAP | | MESSAGE | STRUCT | | repeated | ARRAY | {% hint style="info" %} **Note** Currently, there is no support for Protobuf types `ANY` and `OneOf` in serialization. Columns in a [Database](/overview/core-concepts/databases#relation) or fields in a [Struct](/reference/sql-syntax/data-types#constructed-data-types) are strictly typed at the creation time. {% endhint %} There is support for Protobuf's wrapped message types that capture primitive values, in serialization. If the message instance has a valid value set when deserializing, that value is unwrapped to its corresponding primitive type. If there is no value set in the message, that column or Struct field is set to null. The following table lists the mapping for these types: | Protobuf Message Type | DeltaStream Data Type | | ----------------------------- | ------------------------- | | `google.protobuf.BoolValue` | BOOLEAN | | `google.protobuf.Int32Value` | INTEGER | | `google.protobuf.UInt32Value` | INTEGER | | `google.protobuf.Int64Value` | BIGINT | | `google.protobuf.UInt64Value` | BIGINT | | `google.protobuf.FloatValue` | FLOAT | | `google.protobuf.DoubleValue` | DOUBLE | | `google.protobuf.StringValue` | VARCHAR | | `google.protobuf.BytesValue` | VARBINARY | | `google.protobuf.Timestamp` | TIMESTAMP, TIMESTAMP\_LTZ | # Data Types DeltaStream's data types are similar to SQL standard's data types. They are used to declare the value type for columns in relations or input/output parameters in functions. ## Primitive Data Types ### Boolean | `BOOLEAN` | Boolean value representing `true` or `false.` | | --------- | --------------------------------------------- | ### Character string |

VARCHAR
VARCHAR(n)

|

Variable-length character string.
Type parameter n represents maximum length of string (number of code points) and has to be an integer between 1and 2,147,483,647.
If no length specified, maximum length is picked.

| | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Numeric | `TINYINT` |

1-byte signed integer.
It can hold an integer value between -128 and 127.

| | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SMALLINT` |

2-byte signed integer.
It can hold an integer value between -32,768 and 32,767.

| | `INTEGER` |

4-byte signed integer.
It can hold an integer value between -2^31 and 2^31-1.

| | `BIGINT` |

8-byte signed integer.
It can hold an integer value between -2^63 and 2^63-1

| | `FLOAT` | 4-byte variable precision floating point number. | | `DOUBLE` | 8-byte variable precision floating point number. | |

DECIMAL
DECIMAL(p, s)

|

Decimal number with fixed precision and scale.
Decimal type accepts two type parameters: p: for precision and s for scale.
Precision is the maximum total number of digits to be stored in the number (an integer value between 1 and 38).
Scale is the number of decimal digits to the right of the decimal point (an integer value between 0 and p).
If no type parameters specified, default precision of 10 and default scale of 0 are picked.

| ### Date and Time | `DATE` | Calendar date consisting of `year-month-day` with values between `0000-01-01` and `9999-12-31`. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

TIME
TIME(p)

|

Time of day, without time zone, consisting of hour:minute:second\[.fractional] (up to nanosecond precision) with values between 00:00:00.000000000 and 23:59:59.999999999.
Type parameter p represents precision which is the number of digits of fractional seconds (an integer between 0 and 9).
If no precision specified, default precision of 0 is picked.

| |

TIMESTAMP
TIMESTAMP(p)

|

Timestamp, without time zone, consisting of year-month-day hour:minute:second\[.fractional] (up to nanosecond precision) with values between 0000-01-01 00:00:00.000000000 and 9999-12-31 23:59:59.999999999.
Type parameter p represents precision which is the number of digits of fractional seconds (an integer between 0 and 9).
If no precision specified, default precision of 6 is picked.
There are two types of formats for timestamps that are supported – sql and iso8601.

| |

TIMESTAMP WITH LOCAL TIME ZONE
TIMESTAMP(p) WITH LOCAL TIME ZONE
TIMESTAMP\_LTZ
TIMESTAMP\_LTZ(p)

|

Timestamp with local time zone consisting of year-month-day hour:minute:second\[.fractional] zone (up to nanosecond precision) with values between 0000-01-01 00:00:00.000000000Z to 9999-12-31 23:59:59.999999999Z.
Values for this type assume Instant semantics in the UTC time zone.
Type parameter p represents precision which is the number of digits of fractional seconds (an integer between 0 and 9).
If no precision specified, default precision of 6 is picked. TIMESTAMP\_LTZ is a synonym for TIMESTAMP WITH LOCAL TIME ZONE.
There are two types of formats for timestamps that are supported – sql and iso8601.

| ### Binary string |

VARBINARY
VARBINARY(n)
BYTES

|

Variable-length sequence of bytes.
Type parameter n represents maximum length of sequence (number of bytes) and has to be an integer between 1and 2,147,483,647.
If no length specified, default length of 1 is picked.
BYTES is a synonym for VARBINARY with maximum length (VARBINARY(2147483647)).

| | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Constructed Data Types | `ARRAY` |

Array of elements with same data type.
Type parameter t is required and represents data type of array elements.
The maximum number of elements of an array can not be specified and is fixed at 2,147,483,647.
Similar to SQL, arrays are one-indexed (first element is at index 1) and a given element can be accessed using the \[] operator with the index.

Example. In a relation DDL statement, the expression employees ARRAY\ declares a column named employees with values of type array holding character strings. employees\[6] is used to access the 6th element in a given value from this column.

| | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `MAP` |

Associative array that maps keys to values.
No duplicate keys can exist in a map and each key can map to at most one value.
Type parameters k and v are required and they represent data type of the key elements and the value elements, respectively, throughout the map.

Order of keys within a MAP is not guaranteed, and may use one of ARRAY or STRUCT to guaranteed ordering within a set of queries.
For a given key, the associated value can be accessed using the \[] operator with the key.

Example. In a relation DDL statement, the expression manager MAP\ declares a column named manager with values of type map holding key value pairs of character string types. manager\['HR'] is used to access the value associated with the key HR in a given value instance from this column.

| | `STRUCT` |

Strongly typed structured data type that represents an ordered collection of one or more fields.
A field consists of a name and a data type and captures a specific value in a given instance of struct.
Sequence of type parameter pair(s) fn ft represent name and data type for each and every field, in order.
A given field of a struct instance is accessed using the -> operator with the field name.

Example. In a relation DDL statement, the expression address STRUCT\ declares a column named address with values of type struct holding two fields of character string types, named city and zipcode. address->city is used to access the city field in a given value instance from this column.

| # Identifiers and Keywords DeltaStream SQL uses a lexical policy for identifiers (such as tables, columns, and function names) similar to SQL: * The case of identifiers is not preserved unless they are quoted. * Double-quotes `"` and back-ticks `` ` `` allow identifiers to contain non-alphanumeric characters, such as ``SELECT a AS `my field` FROM t``. String literals must be enclosed in single quotes `'`, such as `'Joe Smith'`. Duplicate a single quote for escaping, such as `SELECT 'It''s me'`. ## Reserved Keywords The following is a list of reserved keywords in DeltaStream SQL: | A | ABS | ABSOLUTE | | ----------------------------- | ------------------------------------ | ---------------------------- | | ACCEPT | ACTION | ADA | | ADD | ADMIN | ADVANCE | | AFTER | ALL | ALLOCATE | | ALLOW | ALTER | ALWAYS | | ANALYZE | AND | ANY | | API\_TOKEN | API\_TOKENS | APPLICATION | | APPROXIMATE | ARE | ARRAY | | AS | ASC | ASENSITIVE | | ASSERTION | ASSIGNMENT | ASYMMETRIC | | AT | ATOMIC | ATTRIBUTE | | ATTRIBUTES | AUTHORIZATION | AVG | | AWS | BEFORE | BEGIN | | BEGINNING | BERNOULLI | BETWEEN | | BIGINT | BINARY | BIT | | BLOB | BOOLEAN | BOTH | | BREADTH | BY | BYTES | | C | CALL | CALLED | | CAN | CARDINALITY | CASCADE | | CASCADED | CASE | CAST | | CATALOG | CATALOGS | CATALOG\_NAME | | CEIL | CEILING | CENTURY | | CHAIN | CHANGELOG | CHANGELOGS | | CHAR | CHARACTER | CHARACTERISTICS | | CHARACTERS | CHARACTER\_LENGTH | CHARACTER\_SET\_CATALOG | | CHARACTER\_SET\_NAME | CHARACTER\_SET\_SCHEMA | CHAR\_LENGTH | | CHECK | CLASS\_ORIGIN | CLICKHOUSE | | CLOB | CLOSE | COALESCE | | COBOL | COLLATE | COLLATION | | COLLATION\_CATALOG | COLLATION\_NAME | COLLATION\_SCHEMA | | COLLECT | COLUMN | COLUMNS | | COLUMN\_NAME | COMMAND\_FUNCTION | COMMAND\_FUNCTION\_CODE | | COMMIT | COMMITTED | COMPUTE\_POOL | | COMPUTE\_POOLS | CONDITION | CONDITIONAL | | CONDITION\_NUMBER | CONFIDENCE | CONFLUENT | | CONNECT | CONNECTION | CONNECTION\_NAME | | CONNECTOR | CONNECTORS | CONSTRAINT | | CONSTRAINTS | CONSTRAINT\_CATALOG | CONSTRAINT\_NAME | | CONSTRAINT\_SCHEMA | CONSTRUCTOR | CONTACT | | CONTAINS | CONTINUE | CONVERT | | COPY | CORR | CORRESPONDING | | COUNT | COVAR\_POP | COVAR\_SAMP | | CREATE | CREDENTIAL | CROSS | | CSTREAM | CUBE | CUME\_DIST | | CUMULATE | CURRENT | CURRENT\_CATALOG | | CURRENT\_DATE | CURRENT\_DEFAULT\_TRANSFORM\_GROUP | CURRENT\_PATH | | CURRENT\_ROLE | CURRENT\_SCHEMA | CURRENT\_TIME | | CURRENT\_TIMESTAMP | CURRENT\_TRANSFORM\_GROUP\_FOR\_TYPE | CURRENT\_USER | | CURSOR | CURSOR\_NAME | CYCLE | | DATA | DATABASE | DATABASES | | DATABRICKS | DATE | DATETIME\_INTERVAL\_CODE | | DATETIME\_INTERVAL\_PRECISION | DAY | DAYS | | DDL | DEALLOCATE | DEC | | DECADE | DECIMAL | DECLARE | | DEFAULT | DEFAULTS | DEFERRABLE | | DEFERRED | DEFINE | DEFINED | | DEFINER | DEGREE | DELETE | | DENSE\_RANK | DEPTH | DEREF | | DERIVED | DESC | DESCRIBE | | DESCRIPTION | DESCRIPTOR | DESCRIPTORS | | DESCRIPTOR\_SOURCE | DESCRIPTOR\_SOURCES | DETERMINISTIC | | DIAGNOSTICS | DISALLOW | DISCONNECT | | DISPATCH | DISTINCT | DISTRIBUTED | | DOMAIN | DOUBLE | DOW | | DOY | DROP | DYNAMIC | | DYNAMIC\_FUNCTION | DYNAMIC\_FUNCTION\_CODE | EACH | | ELEMENT | ELSE | EMPTY | | ENCODING | END | END-EXEC | | ENTITIES | ENTITY | EPOCH | | EQUALS | ERROR | ESCAPE | | EVENTS | EVERY | EXCEPT | | EXCEPTION | EXCLUDE | EXCLUDING | | EXEC | EXECUTE | EXISTS | | EXIT | EXP | EXPLAIN | | EXPORT | EXTEND | EXTENDED | | EXTERNAL | EXTRACT | FAILED | | FALSE | FETCH | FILE | | FILTER | FINAL | FIRST | | FIRST\_VALUE | FLOAT | FLOOR | | FOLLOWING | FOR | FOREIGN | | FORMAT | FORTRAN | FOUND | | FRAC\_SECOND | FREE | FROM | | FULL | FUNCTION | FUNCTIONS | | FUNCTION\_SOURCE | FUNCTION\_SOURCES | FUSION | | G | GAP | GENERAL | | GENERATE | GENERATED | GET | | GLOBAL | GO | GOTO | | GRANT | GRANTED | GRAPHVIZ | | GROUP | GROUPING | HAVING | | HIERARCHY | HISTORY | HOLD | | HOP | HOPPING | HOUR | | HOURS | ICEBERG\_GLUE | ICEBERG\_REST | | IDENTITY | IF | IMMEDIATE | | IMPLEMENTATION | IMPORT | IN | | INCLUDING | INCREMENT | INDEX | | INDEXES | INDICATOR | INITIALLY | | INNER | INOUT | INPUT | | INSENSITIVE | INSERT | INSTANCE | | INSTANTIABLE | INT | INTEGER | | INTEGRATION | INTEGRATIONS | INTERACTIVE | | INTERSECT | INTERSECTION | INTERVAL | | INTO | INVITATION | INVITATIONS | | INVITE | INVOKER | IS | | ISOLATION | JAVA | JOIN | | JSON | JSON\_QUERY | JSON\_VALUE | | K | KAFKA | KEEP | | KEY | KEY\_MEMBER | KEY\_TYPE | | KINESIS | LABEL | LANGUAGE | | LARGE | LAST | LAST\_VALUE | | LATERAL | LEADING | LEFT | | LENGTH | LEVEL | LIBRARY | | LIKE | LIMIT | LINK | | LINKS | LIST | LN | | LOAD | LOCAL | LOCALTIME | | LOCALTIMESTAMP | LOCATOR | LOGICAL | | LOGOUT | LOWER | M | | MAP | MATCH | MATCHED | | MATCH\_RECOGNIZE | MATERIALIZED | MAX | | MAXVALUE | MEASURES | MEDIUM | | MEMBER | MERGE | MESSAGE\_LENGTH | | MESSAGE\_OCTET\_LENGTH | MESSAGE\_TEXT | METADATA | | METHOD | METRICS | MICROSECOND | | MILLENNIUM | MILLISECOND | MILLISECONDS | | MIN | MINUTE | MINUTES | | MINVALUE | MOD | MODIFIES | | MODULE | MODULES | MONGODB | | MONTH | MONTHS | MORE | | MULTISET | MUMPS | NAME | | NAMES | NATIONAL | NATURAL | | NCHAR | NCLOB | NESTING | | NEW | NEXT | NFC | | NFD | NFKC | NFKD | | NO | NONE | NORMALIZE | | NORMALIZED | NOT | NOTIFICATION | | NULL | NULLABLE | NULLIF | | NULLS | NUMBER | NUMERIC | | OBJECT | OCTETS | OCTET\_LENGTH | | OF | OFFSET | OKTA | | OLD | OMIT | ON | | ONLY | OPEN | OPTION | | OPTIONS | OR | ORACLE | | ORDER | ORDERING | ORDINALITY | | ORGANIZATION | ORGANIZATIONS | OTHERS | | OUT | OUTER | OUTPUT | | OVER | OVERLAPS | OVERLAY | | OVERRIDING | OWNER | OWNERSHIP | | PAD | PARAMETER | PARAMETER\_MODE | | PARAMETER\_NAME | PARAMETER\_ORDINAL\_POSITION | PARAMETER\_SPECIFIC\_CATALOG | | PARAMETER\_SPECIFIC\_NAME | PARAMETER\_SPECIFIC\_SCHEMA | PARTIAL | | PARTITION | PARTITIONS | PASCAL | | PASSING | PASSTHROUGH | PASSWORD | | PATH | PATTERN | PENDING | | PERCENTILE\_CONT | PERCENTILE\_DISC | PERCENT\_RANK | | PERMUTE | PLACING | PLAIN | | PLAN | PLI | POISSONIZED | | PORT | POSITION | POSTGRES | | POSTGRESQL | POWER | PRECEDING | | PRECISION | PREPARE | PRESERVE | | PRIMARY | PRINT | PRIOR | | PRIVATE | PRIVILEGES | PROCEDURE | | PROPERTIES | PUBLIC | PYTHON | | QUARTER | QUERIES | QUERY | | QUIT | QUOTES | RANGE | | RANK | RAW | READ | | READS | REAL | RECURSIVE | | REF | REFERENCES | REFERENCING | | REGION | REGISTER | REGISTERED | | REGR\_AVGX | REGR\_AVGY | REGR\_COUNT | | REGR\_INTERCEPT | REGR\_R2 | REGR\_SLOPE | | REGR\_SXX | REGR\_SXY | REGR\_SYY | | REJECT | RELATION | RELATIONS | | RELATIVE | RELEASE | RENAME | | RENEW | REPEATABLE | REPLACE | | REPLICAS | RESCALED | RESET | | RESTART | RESTRICT | RESULT | | RETURN | RETURNED\_CARDINALITY | RETURNED\_LENGTH | | RETURNED\_OCTET\_LENGTH | RETURNED\_SQLSTATE | RETURNING | | RETURNS | REVOKE | RIGHT | | ROLE | ROLES | ROLLBACK | | ROLLUP | ROUTINE | ROUTINE\_CATALOG | | ROUTINE\_NAME | ROUTINE\_SCHEMA | ROW | | ROWS | ROW\_COUNT | ROW\_NUMBER | | RUN | SALES | SAMPLE | | SANDBOX | SAVEPOINT | SCALAR | | SCALE | SCHEMA | SCHEMAS | | SCHEMA\_NAME | SCHEMA\_REGISTRIES | SCHEMA\_REGISTRY | | SCIM | SCOPE | SCOPE\_CATALOGS | | SCOPE\_NAME | SCOPE\_SCHEMA | SCRIPT | | SCROLL | SEARCH | SECOND | | SECONDS | SECRET | SECRETS | | SECTION | SECURITY | SELECT | | SELF | SENSITIVE | SEQUENCE | | SERIALIZABLE | SERVER | SERVER\_NAME | | SESSION | SESSION\_USER | SET | | SETS | SHOW | SIMILAR | | SIMPLE | SIZE | SMALL | | SMALLINT | SNOWFLAKE | SOME | | SOURCE | SPACE | SPECIFIC | | SPECIFICTYPE | SPECIFIC\_NAME | SQL | | SQLEXCEPTION | SQLSTATE | SQLWARNING | | SQL\_TSI\_DAY | SQL\_TSI\_FRAC\_SECOND | SQL\_TSI\_HOUR | | SQL\_TSI\_MICROSECOND | SQL\_TSI\_MINUTE | SQL\_TSI\_MONTH | | SQL\_TSI\_QUARTER | SQL\_TSI\_SECOND | SQL\_TSI\_WEEK | | SQL\_TSI\_YEAR | SQRT | START | | STATE | STATEMENT | STATIC | | STDDEV\_POP | STDDEV\_SAMP | STEP | | STOP | STORE | STORES | | STRATIFY | STREAM | STREAMS | | STRING | STRUCT | STRUCTURE | | STYLE | SUBCLASS\_ORIGIN | SUBMULTISET | | SUBSET | SUBSTITUTE | SUBSTRING | | SUM | SYMMETRIC | SYSTEM | | SYSTEM\_USER | TABLE | TABLES | | TABLESAMPLE | TABLE\_NAME | TAG | | TEMPLATE | TEMPORARY | TERMINATE | | TEXT | THEN | TIES | | TIME | TIMESTAMP | TIMESTAMPADD | | TIMESTAMPDIFF | TIMESTAMP\_LTZ | TIMEZONE\_HOUR | | TIMEZONE\_MINUTE | TINYINT | TO | | TOKEN | TOP\_LEVEL\_COUNT | TRAILING | | TRANSACTION | TRANSACTIONS\_ACTIVE | TRANSACTIONS\_COMMITTED | | TRANSACTIONS\_ROLLED\_BACK | TRANSFORM | TRANSFORMS | | TRANSLATE | TRANSLATION | TREAT | | TRIGGER | TRIGGER\_CATALOG | TRIGGER\_NAME | | TRIGGER\_SCHEMA | TRIM | TRUE | | TRY | TRY\_CAST | TUMBLE | | TUMBLING | TYPE | UESCAPE | | UNBOUNDED | UNCOMMITTED | UNCONDITIONAL | | UNDER | UNION | UNIQUE | | UNKNOWN | UNNAMED | UNNEST | | UNSET | UPDATE | UPPER | | UPSERT | USAGE | USE | | USER | USERDATA | USERNAME | | USERS | USER\_DEFINED\_TYPE\_CATALOG | USER\_DEFINED\_TYPE\_CODE | | USER\_DEFINED\_TYPE\_NAME | USER\_DEFINED\_TYPE\_SCHEMA | USING | | VALUE | VALUES | VARBINARY | | VARCHAR | VARYING | VAR\_POP | | VAR\_SAMP | VERSION | VIEW | | VIEWS | VIRTUAL | WEEK | | WHEN | WHENEVER | WHERE | | WIDTH\_BUCKET | WINDOW | WITH | | WITHIN | WITHOUT | WORK | | WRAPPER | WRITE | XML | | YEAR | YEARS | ZONE | # Command # ACCEPT INVITATION ## Syntax ```sql ACCEPT INVITATION invitation_id; ``` ## Description This command accepts an invitation to join an organization. To retrieve a list of pending invitations for the authenticated user, use [LIST INVITATIONS](/reference/sql-syntax/command/list-invitations). ### Arguments #### invitation\_id Specifies the `id` of the invitation to accept. ## Examples #### Accept the first organization invite This example shows that when accepting an incoming invitation from someone else's organization, the current session joins that organization and can access the objects granted to the user with the appropriate privileges: ```sh # LIST MY INVITATIONS; +--------------------------------------+------------+------------------------------------------+ | ID | Org Name | Invited By | +======================================+============+==========================================+ | bd5b2db3-2f36-49a9-8192-ab6e76fa9290 | demo2 | emailff7192cf-524e-44f5-ae14-17b71ed40ba | | | | a@deltastream.io | +--------------------------------------+------------+------------------------------------------+ # ACCEPT INVITATION bd5b2db3-2f36-49a9-8192-ab6e76fa9290; +------------+--------------------------------------+------------+------------------------------------------+ | Type | Name | Command | Summary | +============+======================================+============+==========================================+ | invitation | bd5b2db3-2f36-49a9-8192-ab6e76fa9290 | UPDATE | invitation to join organization | | | | | "ecdc3aa3-6b61-49b4-93b6-d855795a1552" | | | | | was accepted | +------------+--------------------------------------+------------+------------------------------------------+ /# LIST ORGANIZATIONS; +--------------------------------------+------------+-------------+-------------+--------------+--------------------+-------------------------------+ | ID | Name | Is Current | Is Default | Description | Profile Image Uri | Created At | +======================================+============+=============+=============+==============+====================+===============================+ | ecdc3aa3-6b61-49b4-93b6-d855795a1552 | demo2 | true | true | | | 2024-05-09 04:53:21 +0000 UTC | +--------------------------------------+------------+-------------+-------------+--------------+--------------------+-------------------------------+ ``` # CAN I ## Syntax ```sql CAN I [ [ CREATE_DATABASE | CREATE_STORE | CREATE_SCHEMA_REGISTRY | CREATE_DESCRIPTOR_SOURCE | CREATE_FUNCTION_SOURCE | CREATE_FUNCTION | CREATE_CONNECTOR | CREATE_QUERY | MANAGE_MEMBERS | MANAGE_GRANTS ] | [ [ USE | CREATE IN | SELECT FROM | INSERT INTO | DROP ] [ DATABASE | SCHEMA | STORE | DESCRIPTOR_SOURCE | QUERY | RELATION | FUNCTION_SOURCE | FUNCTION | ROLE | REGION ] target_name ] ]; ``` ## Description This command shows whether the current [Access Control](/overview/core-concepts/access-control#user) can execute specific commands in the current [Access Control](/overview/core-concepts/access-control#organiation). ### Arguments #### target\_name Name of the object to use for testing authorization for the privilege target. Use this only for testing authorization of a privilege type, — for example, `USE`. ## Examples #### Check whether the user can create a database ```bash /# CAN I CREATE_DATABASE; +------------+ | Can I | +============+ | true | +------------+ ``` #### Check whether the user can create a query ```bash /# CAN I CREATE_QUERY; +------------+ | Can I | +============+ | true | +------------+ ``` #### Check whether the user with the current role can create objects in another role ``` /# CAN I CREATE IN ROLE useradmin; +------------+ | Can I | +============+ | false | +------------+ ``` #### Check whether user has USAGE privilege on a database ```bash /# CAN I USE DATABASE db; +------------+ | Can I | +============+ | true | +------------+ ``` #### Check whether user can user another role ```bash /# CAN I USE ROLE useradmin; +------------+ | Can I | +============+ | false | +------------+ ``` #### Check whether user can drop a database ```bash main_db.public/pub_demo_msk# CAN I DROP DATABASE user_db; +------------+ | Can I | +============+ | false | +------------+ ``` #### Check whether user can select from an existing relation ```bash db.public/# CAN I SELECT FROM RELATION pageviews; +------------+ | Can I | +============+ | true | +------------+ ``` #### Check whether user can insert records into an existing relation ```bash db.public/# CAN I INSERT INTO RELATION pageviews; +------------+ | Can I | +============+ | true | +------------+ ``` #### Check whether user can manage grants ```bash db.public/# CAN I MANAGE_GRANTS; +------------+ | Can I | +============+ | false | +------------+ ``` # COPY DESCRIPTOR\_SOURCE ## Syntax ```sql COPY DESCRIPTOR_SOURCE descriptor_source_name TO 'file_path'; ``` ## Description This command downloads a [Descriptor Source](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) to your local machine. Descriptor sources are downloadable only if the current role has USAGE privileges. ### Arguments #### descriptor\_source\_name This is the name of the descriptor source to copy. If a name is case-sensitive names, you must wrap it in double quotes; otherwise, the lowercase name is used. #### file\_path This is the destination file path on the local machine to which the descriptor source file is copied. This is a string value and should be wrapped in single quotes. ## Examples #### Copy Descriptor Source ```sh demodb.public/demostore# COPY DESCRIPTOR_SOURCE pageviews TO '/tmp/pageviews'; Downloading descriptor source pageviews to /tmp/pageviews ``` # COPY FUNCTION\_SOURCE ## Syntax ```sql COPY FUNCTION_SOURCE function_source_name TO 'file_path'; ``` ## Description This command downloads a [Function Source](/overview/core-concepts/function) in the current organization to a specified file path on the user's local machine. You can download a function source only if the current role has USAGE privileges. ### Arguments #### function\_source\_name This is the name of the function source to copy. If the name is case-sensitive names, you must wrap it in double quotes; otherwise, the lowercase name is used. #### file\_path This is the destination file path on the local machine to which the function source file is copied. This is a string value and should be wrapped in single quotes. ### Return A message indicating the function source was successfully copied (or displays an error otherwise). ## Examples #### Copy Function Source ```sh demodb.public/demostore# COPY FUNCTION_SOURCE funcsrc TO '/tmp/funcsrc'; Downloading function source funcsrc to /tmp/funcsrc ``` # DESCRIBE ENTITY ## Syntax ```sql DESCRIBE Entity entity_name [ IN STORE store_name ]; ``` ## Description This command describes an entity in a [Data Store](/overview/core-concepts/store). Entities is visible only if the current [Access Control](/overview/core-concepts/access-control#role) has `USAGE` privileges on the store. ### Arguments #### entity\_name This is the name of the entity to describe. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the lowercase name is used. #### IN STORE store\_name Optionally, describe an entity in a specific store. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the lowercase name is used. ## Examples #### Describe an entity in the current Kafka store The following describes the `pageviews` entity in the current Kafka store `demostore`: ```sh demodb.public/demostore# DESCRIBE ENTITY pageviews; +-----------+--------------+-------------+-----------+-----------------+-------------------+-----------+ | Name | Entity Type | Partitions | Replicas | Key Descriptor | Value Descriptor | Configs | +===========+==============+=============+===========+=================+===================+===========+ | pageviews | Topic | 1 | 1 | | | {} | +-----------+--------------+-------------+-----------+-----------------+-------------------+-----------+ ``` The entity provides the Kafka topic-specific information and the Key/Value [Descriptor](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) defining the serialization format of both the key and value portions of a Kafka record. #### Describe an entity in a specific Kinesis store The following describes the `InterestCount` entity in the Amazon Kinesis store `kinesis_main`: ```sh demodb.public/demostore# DESCRIBE ENTITY "InterestCount" IN STORE kinesis_store; +---------------+--------------+---------+-------------+ | Name | Entity Type | Shards | Descriptor | +===============+==============+=========+=============+ | InterestCount | Data Stream | 6 | | +---------------+--------------+---------+-------------+ ``` The entity provides the Kinesis stream specific information and a [Descriptor](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) defining the serialization format of the Kinesis records. #### Describe an entity with descriptors The following describes the `pv_pb` entity in the Kafka Store called `KafkaStore2`. This entity's native format is ProtoBuf and the Key and Value descriptors assigned to it allow serialization to/from this format: ````sh demodb.analytics/demostore# DESCRIBE ENTITY pv_pb IN STORE "KafkaStore2"; ```csharp +-------+--------------+-------------+-----------+---------------------+--------------------+-----------+ | Name | Entity Type | Partitions | Replicas | Key Descriptor | Value Descriptor | Configs | +=======+==============+=============+===========+=====================+====================+===========+ | pv_pb | Topic | 1 | 2 | pb_key.PageviewsKey | pb_value.Pageviews | {} | +-------+--------------+-------------+-----------+---------------------+--------------------+-----------+ ```` #### Describe a Snowflake database ```sh demodb.public/sfstore# DESCRIBE ENTITY "FLINK_STREAMING"; +-----------------+--------------+------------+-------------------------------+-----------------+--------------+-----------+ | Name | Entity Type | Type | Created At | Retention Time | Owner | Comment | +=================+==============+============+===============================+=================+--------------+===========+ | FLINK_STREAMING | Database | STANDARD | 2024-07-17 00:19:33 +0000 UTC | 1 | ACCOUNTADMIN | | +-----------------+--------------+------------+-------------------------------+-----------------+--------------+-----------+ ``` #### Describe a Snowflake table ```sh demodb.public/sfstore# DESCRIBE ENTITY "FLINK_STREAMING"."PUBLIC"."SF_PAGEVIEWS"; +-------------------------------------+--------------+------------+-------------------------------+--------------+-------------------------------+-----------------+-------------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | Name | Entity Type | Type | Created At | Owner | Last DDLAt | Retention Time | Updated At | Comment | Columns | +=====================================+==============+============+===============================+==============+===============================+=================+===============================+==========+=========================================================================================================================================================+ | FLINK_STREAMING.PUBLIC.SF_PAGEVIEWS | Table | BASE TABLE | 2024-07-03 14:55:20 +0000 UTC | ACCOUNTADMIN | 2024-07-03 14:55:20 +0000 UTC | 1 | 2024-07-03 15:02:46 +0000 UTC | | [{"name":"TIMESTAMP","type":"NUMBER(38,0)","nullable":true,"default":null,"primaryKey":false,"uniqueKey":false,"comment":null,"policyName":null},{...}] | +-------------------------------------+--------------+------------+-------------------------------+--------------+-------------------------------+-----------------+-------------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` #### Describe a Databricks catalog ```sh demodb.public/databricks_store# DESCRIBE ENTITY catalog1; +------------------+--------------+-------------------------------+-----------------------------------------+-------------------------------+-----------------------------------------+----------+ | Name | Entity Type | Created At | Created By | Updated At | Updated By | Comment | +==================+==============+===============================+=========================================+===============================+=========================================+==========+ | catalog1 | Catalog | 2024-02-09 22:07:14 +0000 UTC | subscriptions+databricks@deltastream.io | 2024-02-09 22:07:14 +0000 UTC | subscriptions+databricks@deltastream.io | | +------------------+--------------+-------------------------------+-----------------------------------------+-------------------------------+-----------------------------------------+----------+ ``` #### Describe a Databricks schema ```sh demodb.public/databricks_store# DESCRIBE ENTITY catalog1.schema1; +------------------+--------------+-------------------------------+-------------+-------------------------------+-------------+----------+ | Name | Entity Type | Created At | Created By | Updated At | Updated By | Comment | +==================+==============+===============================+=============+===============================+=============+==========+ | catalog1.schema1 | Schema | 2024-02-13 23:43:59 +0000 UTC | sch_owner | 2024-02-13 23:43:59 +0000 UTC | sch_owner | | +------------------+--------------+-------------------------------+-------------+-------------------------------+-------------+----------+ ``` #### Describe a Databricks table ```sh demodb.public/dbx_store# DESCRIBE ENTITY catalog1.schema1.pageviews; +----------------------------+--------------+-------------------------------+-------------+-------------------------------+-------------+----------------------+------------------------------------+---------------------------------------------------------------------------+ | Name | Entity Type | Created At | Created By | Updated At | Updated By | Storage Location | Comment | Columns | +============================+==============+===============================+=============+===============================+=============+======================+====================================+===========================================================================+ | catalog1.schema1.pageviews | Table | 2024-04-10 15:24:07 +0000 UTC | tbl_owner | 2024-04-10 15:24:07 +0000 UTC | tbl_owner | s3://somebucket/test | {"deltastream.relation.fqn":"..."} | [{"name":"messageId","type":"string","nullable":true,"position":0},{...}] | +----------------------------+--------------+-------------------------------+-------------+-------------------------------+-------------+----------------------+----------------------------------+-----------------------------------------------------------------------------+ ``` #### Describe a Postgresql schema ```sh demodb.public/databricks_store# DESCRIBE ENTITY db.schema1; +-------------+--------------+-----------+ | Name | Entity Type | Owner | +=============+==============+===========+ | db.schema1 | Schema | sch_owner | +-------------+--------------+-----------+ ``` #### Describe a Postgresql table ```sh demodb.public/pgx_store# DESCRIBE ENTITY public.pageviews; +------------------+--------------+-----------+---------------------------------------------------------------------------+ | Name | Entity Type | Owner | Columns | +==================+==============+===========+===========================================================================+ | public.pageviews | Table | tbl_owner | [{"name":"viewtime","type":"bigint","nullable":true,"position":0},{...}] | +------------------+--------------+-----------+---------------------------------------------------------------------------+ ``` # DESCRIBE QUERY ## Syntax ```sql DESCRIBE QUERY [query_id | query_name[:query_version]]; ``` ## Description This command provides additional information about a [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query) created in the current [Organization](/overview/core-concepts/access-control#organiation). See [Query](/overview/core-concepts/queries) on how streaming queries operate in DeltaStream. The query is visible only if the current role has `USAGE` privileges. See also [DESCRIBE QUERY METRICS](/reference/sql-syntax/command/describe-query-metrics), [DESCRIBE QUERY EVENTS](/reference/sql-syntax/command/describe-query-events), [DESCRIBE QUERY STATE](/reference/sql-syntax/command/describe-query-state). ### Arguments #### query\_id This is the unique identifier of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which you have access. The first column (`ID`) in the response corresponds with the `query_id`. #### query\_name This is the name of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which you have access. The column (`Name`) in the response corresponds with the `query_name`. The `query_version` property disambiguates between multiple queries with the same name. #### query\_version This is the version of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which you have access. The column (`Version`) in the response corresponds with the `query_version`. ## Examples #### Describe a simple query The following describes a simple query with a [Changelog](/overview/core-concepts/databases#_changelog) source relation, `users_log`, filtering on a `userid` column and writing the results into a Changelog sink relation, `users2468_log`: ```sh demodb.analytics/demostore# DESCRIBE QUERY b2041101-c37c-4807-93d4-2de876b7bd39; +-----------+--------------------------------+ | Property | Value | +===========+================================+ | state | running | +-----------+--------------------------------+ | source | changelog users_log | +-----------+--------------------------------+ | sink | changelog users2468_log | +-----------+--------------------------------+ demodb.analytics/demostore# DESCRIBE QUERY demoquery:1; +-----------+--------------------------------+ | Property | Value | +===========+================================+ | state | running | +-----------+--------------------------------+ | source | changelog users_log | +-----------+--------------------------------+ | sink | changelog users2468_log | +-----------+--------------------------------+ ``` #### Describe a multi-source query The following describes a `JOIN` query that joins a [Stream](/overview/core-concepts/databases#_stream), `pageviews`, and a [Changelog](/overview/core-concepts/databases#_changelog) relation, `users_log`, on a `userid` column and writes the resulting records into a `pv_user2468_interest` changelog relation: ```sh demodb.analytics/demostore# DESCRIBE QUERY c81fc632-a043-472d-a1fe-aa81bc9078df; +-----------+--------------------------------+ | Property | Value | +===========+================================+ | state | running | +-----------+--------------------------------+ | source | changelog users_log | +-----------+--------------------------------+ | source | stream pageviews | +-----------+--------------------------------+ | sink | stream pv_user2468_interest | +-----------+--------------------------------+ demodb.analytics/demostore# DESCRIBE QUERY demoquery:1; +-----------+--------------------------------+ | Property | Value | +===========+================================+ | state | running | +-----------+--------------------------------+ | source | changelog users_log | +-----------+--------------------------------+ | source | stream pageviews | +-----------+--------------------------------+ | sink | stream pv_user2468_interest | +-----------+--------------------------------+ ``` #### Describe an ERRORED query The following describes a query stopped due to an unrecoverable failure, leaving it in the `ERRORED` state: ```sh demodb.analytics/demostore# DESCRIBE QUERY c81fc632-a043-472d-a1fe-aa81bc9078df; +-----------+-----------------------+ | Property | Value | +===========+=======================+ | state | errored | +-----------+-----------------------+ | source | stream pageviews | +-----------+-----------------------+ | sink | stream users | +-----------+-----------------------+ | error | | +-----------+-----------------------+ demodb.analytics/demostore# DESCRIBE QUERY demoquery:1; +-----------+-----------------------+ | Property | Value | +===========+=======================+ | state | errored | +-----------+-----------------------+ | source | stream pageviews | +-----------+-----------------------+ | sink | stream users | +-----------+-----------------------+ | error | | +-----------+-----------------------+ ``` # DESCRIBE QUERY METRICS ## Syntax ```sql DESCRIBE QUERY METRICS [query_id | query_name[:query_version]] [WITH (query_metrics_parameter = value [, ...])]; ``` ## Description This command provides a snapshot of metrics for a [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query).\ \ The 4 metrics shown are: 1. `NumRecordsInPerSecond` 2. `NumRecordsOutPerSecond` 3. `NumBytesInPerSecond` 4. `NumBytesOutPerSecond`. Returned metrics represent a snapshot of the value of each metric at the time this command was run. {% hint style="info" %} **Note** Metrics may not always be available for queries. In this case, stats are replaced with `n/a` or with a `02000` SQL error code. The example below displays a query in a terminal state. {% endhint %} See [Query](/overview/core-concepts/queries) on how streaming queries operate in DeltaStream. The query is visible only if the current role has `USAGE` privileges. ### Arguments #### query\_id This is the unique identifier of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The first column (`ID`) in the response corresponds with the `query_id`. #### query\_name This is the name of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Name`) in the response corresponds with the `query_name`. The `query_version` property disambiguates between multiple queries with the same name. #### query\_version This is the version of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Version`) in the response corresponds with the `query_version`. If `query_name` is provided without a `query_version`, the latest `query_version` is used. #### WITH (query\_metrics\_parameter = value \[, ...]) Optionally, this clause specifies [#query-metrics-parameters](#query-metrics-parameters "mention"). ### Query Metrics Parameters | Parameter Name | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FOR METRICS(name[,...])` |

Specifies the list of query metrics names to describe.


Required: Yes

Type: METRICS\_NAME

Valid values: numRecordsInPerSecond, numRecordsOutPerSecond, numBytesInPerSecond, numBytesOutPerSecond, consumerRecordsLagMax

| | `from` |

How far in the past from which to return metrics data points.

Required: Yes

Type: METRICS\_RANGE

Valid values: LAST 15 MINUTES, LAST 30 MINUTES, LAST HOUR, LAST 3 HOURS, LAST 6 HOURS, LAST 12 HOURS, LAST DAY

| ## Examples #### Describe a simple query The following describes a simple query with a [Changelog](/overview/core-concepts/databases#_changelog) source relation, `users_log`, filtering on a `userid` column and writing the results into a Changelog sink relation, `users2468_log`: ```sh demodb.analytics/demostore# DESCRIBE QUERY METRICS b2041101-c37c-4807-93d4-2de876b7bd39; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | 6 | +------------------------+--------+ | NumRecordsOutPerSecond | 5 | +------------------------+--------+ | NumBytesInPerSecond | 6870 | +------------------------+--------+ | NumBytesOutPerSecond | 4690 | +------------------------+--------+ demodb.analytics/demostore# DESCRIBE QUERY METRICS demoquery:1; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | 6 | +------------------------+--------+ | NumRecordsOutPerSecond | 5 | +------------------------+--------+ | NumBytesInPerSecond | 6870 | +------------------------+--------+ | NumBytesOutPerSecond | 4690 | +------------------------+--------+ ``` #### Describe a multi-source query The following describes a `JOIN` query that joins a [Stream](/overview/core-concepts/databases#_stream), `pageviews`, and a [Changelog](/overview/core-concepts/databases#_changelog) relation, `users_log`, on a `userid` column and writes the resulting records into a `pv_user2468_interest` Changelog relation: ```sh demodb.analytics/demostore# DESCRIBE QUERY METRICS c81fc632-a043-472d-a1fe-aa81bc9078df; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | 20 | +------------------------+--------+ | NumRecordsOutPerSecond | 25 | +------------------------+--------+ | NumBytesInPerSecond | 16870 | +------------------------+--------+ | NumBytesOutPerSecond | 10690 | +------------------------+--------+ demodb.analytics/demostore# DESCRIBE QUERY METRICS demoquery:1; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | 20 | +------------------------+--------+ | NumRecordsOutPerSecond | 25 | +------------------------+--------+ | NumBytesInPerSecond | 16870 | +------------------------+--------+ | NumBytesOutPerSecond | 10690 | +------------------------+--------+ ``` #### Describe an ERRORED query The following describes a query stopped due to an unrecoverable failure, leaving it in the `ERRORED` state: ```sh demodb.analytics/demostore# DESCRIBE QUERY METRICS c81fc632-a043-472d-a1fe-aa81bc9078df; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | n/a | +------------------------+--------+ | NumRecordsOutPerSecond | n/a | +------------------------+--------+ | NumBytesInPerSecond | n/a | +------------------------+--------+ | NumBytesOutPerSecond | n/a | +------------------------+--------+ demodb.analytics/demostore# DESCRIBE QUERY METRICS demoquery:1; +------------------------+--------+ | Key | Value | +========================+========+ | NumRecordsInPerSecond | n/a | +------------------------+--------+ | NumRecordsOutPerSecond | n/a | +------------------------+--------+ | NumBytesInPerSecond | n/a | +------------------------+--------+ | NumBytesOutPerSecond | n/a | +------------------------+--------+ ``` #### Describe a query's metrics from the last 15 minutes The following example shows metrics data points available from the last 15 minutes of a given query for the metrics name `numRecordsInPerSecond`, `numRecordsOutPerSecond`, `numBytesInPerSecond`, `numBytesOutPerSecond`, and `consumerRecordsLagMax`: ```bash demodb.public/demostore# DESCRIBE QUERY METRICS 9c36c3c1-c45e-42ff-b2e4-2e069b313ac3 WITH ('from' = LAST 15 MINUTES , FOR METRICS (numRecordsInPerSecond, numRecordsOutPerSecond, numBytesInPerSecond, numBytesOutPerSecond, consumerRecordsLagMax)); +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | Name | Reference Name | Values | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numRecordsInPerSecond | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"6"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numRecordsInPerSecond | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"1"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"1"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numRecordsOutPerSecond | KafkaSink:mr_metrics_082124:_Writer | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"6"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numBytesInPerSecond | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"570 B"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"532 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numBytesInPerSecond | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"177 B"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"169 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | numBytesOutPerSecond | KafkaSink:mr_metrics_082124:_Writer | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"931 B"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"942 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | consumerRecordsLagMax | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"0"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ | consumerRecordsLagMax | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T16:12:34.692-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:27:34.692-07:00","Value":"0"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ ``` #### Describe a query's metrics from the last 3 hours The following example shows metrics data points available from the last 15 minutes of a given query for the metrics name `numRecordsInPerSecond`, `numRecordsOutPerSecond`, `numBytesInPerSecond`, `numBytesOutPerSecond`, and `consumerRecordsLagMax`: ```bash demodb.public/demostore# DESCRIBE QUERY METRICS 9c36c3c1-c45e-42ff-b2e4-2e069b313ac3 WITH ('from' = LAST 3 HOURS, FOR METRICS (numRecordsInPerSecond, numRecordsOutPerSecond, numBytesInPerSecond, numBytesOutPerSecond, consumerRecordsLagMax)); +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Name | Reference Name | Values | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numRecordsInPerSecond | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"2"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"4"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numRecordsInPerSecond | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"1"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"1"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numRecordsOutPerSecond | KafkaSink:mr_metrics_082124:_Writer | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"2"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"6"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"5"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"4"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"6"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"5"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numBytesInPerSecond | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"426 B"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"169 B"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"492 B"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"447 B"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"418 B"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"495 B"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"495 B"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"402 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numBytesInPerSecond | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"138 B"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"69 B"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"165 B"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"142 B"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"138 B"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"153 B"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"157 B"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"119 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | numBytesOutPerSecond | KafkaSink:mr_metrics_082124:_Writer | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"787 B"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"243 B"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"904 B"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"729 B"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"578 B"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"950 B"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"872 B"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"714 B"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | consumerRecordsLagMax | Source:_KafkaSource:pv | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"0"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | consumerRecordsLagMax | Source:_KafkaSource:users_log | [{"Timestamp":"2024-08-21T14:43:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T14:58:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:13:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:28:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:43:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T15:58:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:13:22.883-07:00","Value":"0"},{"Timestamp":"2024-08-21T16:28:22.883-07:00","Value":"0"}] | +------------------------+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` #### Describe metrics for a query in the `STOPPED` state The following example show that when a query is in a terminated state — in this case, `STOPPED` — the system returns a SQL code of `02000` instead: ```bash demodb.public/demostore# DESCRIBE QUERY METRICS 4f5f6a0f-1f71-40ec-902a-0ec821bc36f3 WITH ('from' = LAST 3 HOURS, FOR METRICS (numRecordsInPerSecond)); sql error: no data found (SQLState: 02000) (request ID: ...) ``` # DESCRIBE QUERY EVENTS ## Syntax ```sql DESCRIBE QUERY EVENTS [query_id | query_name[:query_version]]; ``` ## Description This command provides a list of events for a [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query) created in the current [Organization](/overview/core-concepts/access-control#organiation). See [Query](/overview/core-concepts/queries) on how streaming queries operate in DeltaStream. The Query is visible only if the current role has `USAGE` privileges. ### Arguments #### query\_id This is the unique identifier of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The first column (`ID`) in the response corresponds with the `query_id`. #### query\_name This is the name of the Query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Name`) in the response corresponds with the `query_name`. The `query_version` property disambiguates between multiple queries with the same name. #### query\_version This is the version of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Version`) in the response corresponds with the `query_version`. If `query_name` is provided without a `query_version`, the system uses `query_version`. ## Examples #### Show query events when running and stopped ```bash demodb.public/demostore# DESCRIBE QUERY EVENTS 5b9ba120-bb59-4c95-82c5-6ee0f78dc75e; +-------------------------------+----------------------+---------+-----------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------+-----------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------+-----------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------+-----------+ demodb.public/demostore# DESCRIBE QUERY EVENTS demoquery:1; +-------------------------------+----------------------+---------+-----------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------+-----------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------+-----------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------+-----------+ demodb.public/demostore# DESCRIBE QUERY EVENTS 24f5a97b-97e9-40eb-8dbf-431b96474972; +-------------------------------+----------------------+---------------------+-----------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:57 +0000 UTC | system | terminated | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:40 +0000 UTC | delta@deltastream.io | terminate_requested | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------------------+-----------+ demodb.public/demostore# DESCRIBE QUERY EVENTS demoquery:1; +-------------------------------+----------------------+---------------------+-----------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:57 +0000 UTC | system | terminated | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:40 +0000 UTC | delta@deltastream.io | terminate_requested | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------------------+-----------+ ``` #### Show events for a query that has been rescheduled You can reschedule queries for any number of reasons. Typically you reschedule a query if a you must move it to another node. ```bash demodb.public/demostore# DESCRIBE QUERY EVENTS demoquery:1; +-------------------------------+----------------------+---------------------+-----------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:57 +0000 UTC | system | running | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:47:40 +0000 UTC | system | rescheduled | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------------------+-----------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------------------+-----------+ ``` The following example shows an errored query: ```bash testdb.public/internal_stage# DESCRIBE QUERY EVENTS 142ecf31-bc8d-496b-bd0b-28ebb4a8757e; +-------------------------------+----------------------+---------------------+------------------+ | Created At | Actor | Type | Messages | +-------------------------------+----------------------+---------------------+------------------+ | 2024-08-09 15:46:32 +0000 UTC | system | errored | | +-------------------------------+----------------------+---------------------+------------------+ | 2024-08-09 15:45:32 +0000 UTC | system | running | | +-------------------------------+----------------------+---------------------+------------------+ | 2024-08-09 15:45:01 +0000 UTC | delta@deltastream.io | created | | +-------------------------------+----------------------+---------------------+------------------+ ``` # DESCRIBE QUERY STATE ## Syntax ```sql DESCRIBE QUERY STATE [query_id | query_name[:query_version]]; ``` ## Description This command provides a snapshot of the source relations' position in a terminated [Streaming or Continuous Query](/overview/core-concepts/queries#_streaming_or_continuous_query) created in the current [Organization](/overview/core-concepts/access-control#organiation). See [Query](/overview/core-concepts/queries) on how streaming queries operate in DeltaStream. The query is visible only if the current role has `USAGE` privileges. ### Arguments #### query\_id This is the unique identifier of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The first column (`ID`) in the response corresponds with the `query_id`. #### query\_name This is the name of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Name`) in the response corresponds with the `query_name`. The `query_version` property disambiguates between multiple queries with the same name. #### query\_version This is the version of the query to describe. See [LIST QUERIES](/reference/sql-syntax/command/list-queries) to find out how to list all queries to which the user has access. The column (`Version`) in the response corresponds with the `query_version`. If `query_name` is provided without a `query_version`, the system uses the latest `query_version`. ## Examples #### State for an active query The following example shows that an active query does not report any state information: ```sh demodb.public/demostore# DESCRIBE QUERY STATE 24f5a97b-97e9-40eb-8dbf-431b96474972; +----------------+-------------+-------+ | Relation Name | Store Type | Info | +================+=============+=======+ +----------------+-------------+-------+ demodb.public/demostore# DESCRIBE QUERY STATE demoquery:1; +----------------+-------------+-------+ | Relation Name | Store Type | Info | +================+=============+=======+ +----------------+-------------+-------+ ``` #### State for a terminated query The following example shows that the described query was operating on a single source, `demodb.public.pagevviews`, in a store of type Kafka, and was at the offset `28899068` in partition `0` when the query was terminated: ```bash demodb.public/demostore# DESCRIBE QUERY STATE 24f5a97b-97e9-40eb-8dbf-431b96474972; +---------------------------+-------------+-----------------------------+ | Relation Name | Store Type | Info | +===========================+=============+=============================+ | demodb.public.pagevviews | KAFKA | partition:0,offset:28899068 | +---------------------------+-------------+-----------------------------+ demodb.public/demostore# DESCRIBE QUERY STATE demoquery:1; +---------------------------+-------------+-----------------------------+ | Relation Name | Store Type | Info | +===========================+=============+=============================+ | demodb.public.pagevviews | KAFKA | partition:0,offset:28899068 | +---------------------------+-------------+-----------------------------+ ``` # DESCRIBE RELATION ## Syntax ```sql DESCRIBE [ RELATION | STREAM | CHANGELOG | TABLE | MATERIALIZED VIEW | INDEX ] relation_name; ``` ## Description This command provides the runtime information of an existing [Relation](/overview/core-concepts/databases#relation) in a [Database's](/overview/core-concepts/databases) [schema](/overview/core-concepts/databases#schema). You can also use this command to inspect a specific relation type. See [LIST RELATIONS](/reference/sql-syntax/command/list-relations) for a list of available relations to describe. You can use this command to inspect an existing relation. The output of this command also provides any errors that may have prevented the relation and its underlying entity to be created in the namespace. Relations are visible only if the current role has `USAGE` privileges on the database and schema and `SELECT` or `INSERT` privilege on the relation. See also [DESCRIBE RELATION COLUMNS](/reference/sql-syntax/command/describe-relation-columns). ### Arguments #### relation\_name This is the name of the relation to describe. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the lowercase name is used. The relation name optionally can be fully qualified in the format `..` or `.`. If the database name or schema name are not provided, then the system uses the current session's database/schema (see [`USE`](/reference/sql-syntax/command/use)). ## Examples #### Describe a relation using a current database/schema The following describes the `pageviews` [Stream](/overview/core-concepts/databases#stream) relation in the current `demodb` database and `public` schema: ```sh demodb.public/demostore# DESCRIBE RELATION pageviews; +-----------+---------+ | Property | Value | +===========+=========+ | state | created | +-----------+---------+ ``` #### Describe a relation using a fully qualified name The following describes the `pageviews` [Stream](/overview/core-concepts/databases#stream) relation in the `demodb` database and `public` schema, using its fully-qualified relation name: ```sh demodb.public/demostore# DESCRIBE RELATION demodb.public.pageviews; +-----------+---------+ | Property | Value | +===========+=========+ | state | created | +-----------+---------+ ``` #### Describe a changelog The following describes the `users_log` [Changelog](/overview/core-concepts/databases#changelog) in the `demodb` database and `public` schema, using its fully-qualified relation name: ```sh demodb.public/demostore# DESCRIBE CHANGELOG demodb.public.users_log; +-----------+---------+ | Property | Value | +===========+=========+ | state | created | +-----------+---------+ ``` #### Describe a stream in a specific database/schema The following describes the `total_views` [Stream](/overview/core-concepts/databases#stream) in the `demodb` database and `analytics` schema, using its fully-qualified relation name: ```sh demodb.public/demostore# DESCRIBE RELATION demodb.analytics.total_views; +-----------+---------+ | Property | Value | +===========+=========+ | state | created | +-----------+---------+ ``` #### Describe a Table in errored state The following describes the `dbx_pv` [Table](/overview/core-concepts/databases#table) in the current database and schema that failed: ```sh demodb.analytics/demostore# DESCRIBE TABLE dbx_pv; +-----------+------------------------------------------+ | Property | Value | +===========+==========================================+ | state | errored | +-----------+------------------------------------------+ | error | schema not found: verify the spelling | | | and correctness of the Databricks | | | schema and catalog | +-----------+------------------------------------------+ ``` #### Describe a relation with a case-sensitive name The following describes the `Pageviews` [Stream](/overview/core-concepts/databases#stream) relation in the current `demodb` database and `public` schema: ```sh demodb.public/demostore# DESCRIBE RELATION "Pageviews"; +-----------+---------+ | Property | Value | +===========+=========+ | state | created | +-----------+---------+ ``` # DESCRIBE RELATION COLUMNS ## Syntax ```sql DESCRIBE [ RELATION | STREAM | CHANGELOG | MATERIALIZED VIEW ] COLUMNS fully_qualified_relation_name; ``` ## Description This provides the information about an existing [Database](/overview/core-concepts/databases#relation)'s columns in a [Database's](/overview/core-concepts/databases) [schema](/overview/core-concepts/databases#schema). This command can also be used to define a specific relation type's columns. See [LIST RELATIONS](/reference/sql-syntax/command/list-relations) for a list of available relations to describe. Relations are visible only if the current [Access Control](/overview/core-concepts/access-control#role) has `USAGE` privileges on the database and schema and `SELECT` or `INSERT` privilege on the relation. ### Arguments #### fully\_qualified\_relation\_name This is the name of the relation to describe its columns. If the name is case-sensitive you must wrap it in double quotes; otherwise, the system uses the lowercase name. Optionally, the relation name may be fully-qualified in the format of `..` or `.`. If the database or schema name is not provided, the system uses the current session's database or schema (see [USE](/reference/sql-syntax/command/use)). ## Examples #### Describe a Stream's columns ```sh demodb.public/demostore# DESCRIBE RELATION COLUMNS pageviews; +----------+---------+-----------+-------------+ | Name | Type | Nullable | Properties | +==========+=========+===========+=============+ | viewtime | BIGINT | true | {} | +----------+---------+-----------+-------------+ | userid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ | pageid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ ``` #### Describe a stream's columns in a specific database/schema The following describes the `total_views` [Database](/overview/core-concepts/databases#stream) relation columns in the `demodb` database and `analytics` schema, using its fully-qualified relation name: ```bash demodb.public/demostore# DESCRIBE RELATION COLUMNS pageviews; +----------+---------+-----------+-------------+ | Name | Type | Nullable | Properties | +==========+=========+===========+=============+ | viewtime | BIGINT | true | {} | +----------+---------+-----------+-------------+ | userid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ | pageid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ ``` #### Describe a materialized view's columns The following describes the columns for `pvcount` in the `demodb` database and `public` schema: ```bash demodb.public/demostore# DESCRIBE MATERIALIZED VIEW COLUMNS demodb."public".pvcount; +-------------+---------+-----------+-------------+ | Name | Type | Nullable | Properties | +=============+=========+===========+=============+ | page_name | VARCHAR | true | {} | +-------------+---------+-----------+-------------+ | pageid | VARCHAR | true | {} | +-------------+---------+-----------+-------------+ | cnt | BIGINT | false | {} | +-------------+---------+-----------+-------------+ ``` #### Describe a changelog The following describes the `users_log` [Database](/overview/core-concepts/databases#changelog) in the current database and schema: ```bash demodb.public/demostore# DESCRIBE CHANGELOG COLUMNS users_log; +--------------+-----------------------+-----------+-------------+ | Name | Type | Nullable | Properties | +==============+=======================+===========+=============+ | registertime | BIGINT | true | {} | +--------------+-----------------------+-----------+-------------+ | userid | VARCHAR | true | {} | +--------------+-----------------------+-----------+-------------+ | regionid | VARCHAR | true | {} | +--------------+-----------------------+-----------+-------------+ | gender | VARCHAR | true | {} | +--------------+-----------------------+-----------+-------------+ | interests | ARRAY | true | {} | +--------------+-----------------------+-----------+-------------+ | contactinfo | STRUCT | | | +--------------+-----------------------+-----------+-------------+ ``` #### Describe a Table's columns The following describes the columns for `pv_agg` [Database](/overview/core-concepts/databases#table) in the current database and schema: ```sh demodb.public/demostore# DESCRIBE RELATION COLUMNS pv_agg; +-----------+---------+-----------+-------------+ | Name | Type | Nullable | Properties | +===========+=========+===========+=============+ | userid | VARCHAR | true | {} | +-----------+---------+-----------+-------------+ | pagecount | BIGINT | true | {} | +-----------+---------+-----------+-------------+ ``` #### Describe a relation's columns with a case-sensitive name The following describes the columns for `Pageviews` [Database](/overview/core-concepts/databases#stream) relation in the current `demodb` database and `public` schema: ```sh demodb.public/demostore# DESCRIBE RELATION "Pageviews"; +----------+---------+-----------+-------------+ | Name | Type | Nullable | Properties | +==========+=========+===========+=============+ | viewtime | BIGINT | true | {} | +----------+---------+-----------+-------------+ | userid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ | pageid | VARCHAR | true | {} | +----------+---------+-----------+-------------+ ``` # DESCRIBE ROLE ## Syntax ```sql DESCRIBE ROLE role_name; ``` ## Description This command provides the description of an existing [Access Control](/overview/core-concepts/access-control#role) in the current [Access Control](/overview/core-concepts/access-control#organiation). Roles are visible only if the current role has `USAGE` privileges on the role. ### Arguments #### role\_name Name of the role to describe. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the system uses the lowercase name. ## Examples ```sh /# DESCRIBE ROLE useradmin; +--------------+------------+----------------+--------------------+-------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+============+================+====================+=============+ | role | public | usage | false | orgadmin | +--------------+------------+----------------+--------------------+-------------+ | organization | useradmin | manage_members | true | orgadmin | +--------------+------------+----------------+--------------------+-------------+ ``` # DESCRIBE SECURITY INTEGRATION ## Syntax ```sql DESCRIBE SECURITY INTEGRATION security_integration_name; ``` ## Description Describe configured security integrations. ### Arguments #### security\_integration\_name This is the name of a configured security integration. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the system uses the lowercase name. ## Examples #### Describe security integration ```shell /# DESCRIBE SECURITY INTEGRATION "okta"; +-------+---------+---------------+-------------------------------+-------------------------------+-------------------------------+ | Type | Client | Last Used At | Expires At | Created At | Updated At | +=======+=========+===============+===============================+===============================+===============================+ | scim | okta | | 2025-07-15 23:20:21 +0000 UTC | 2024-07-15 23:17:14 +0000 UTC | 2024-07-15 23:20:21 +0000 UTC | +-------+---------+---------------+-------------------------------+-------------------------------+-------------------------------+ ``` # DESCRIBE \ ## Syntax ```sql DESCRIBE statement; ``` ## Description This command provides details about the objects used by a `CREATE_RELATION`, `INSERT INTO`, or `CREATE AS SELECT` statement. The statement is described only if the current role has the `USAGE` privilege on the [Database](/overview/core-concepts/databases), [Schema](/overview/core-concepts/databases#schema), [Store](/overview/core-concepts/store), and [Relation](/overview/core-concepts/databases#relation) used in the statement. ### Arguments #### statement Any of the executable statements including [CREATE STREAM](/reference/sql-syntax/ddl/create-stream), [CREATE CHANGELOG](/reference/sql-syntax/ddl/create-changelog), [CREATE STREAM AS SELECT](/reference/sql-syntax/query/create-stream-as), [CREATE CHANGELOG AS SELECT](/reference/sql-syntax/query/create-changelog-as), [CREATE MATERIALIZED VIEW AS](/reference/sql-syntax/query/materialized-view/create-materialized-view-as), [CREATE TABLE AS SELECT](/reference/sql-syntax/query/create-table-as).
Response Columns * `type`: Provides the type of the statement being described. * `info`: A key-value list of objects used within the statement and specific information about them: * `ddl`/`sink`: A new relation the statement creates to write to — or, respectively, an existing relation to which the statement writes new records. * Provides `fqn`, `type`, `db_name`, `schema_name`, `name`, and `store_name`. A `fqn` is in the format of `org.db.schema.relation`. * `Sources`: List of relations the statement uses for reading records. * Provides `fqn`, `type`, `db_name`, `schema_name`, `name`, and `store_name`.
## Examples #### Describe a stream DDL statement In this example, a `CREATE STREAM` statement returns a `ddl` field for the `pageviews` stream that would be created if the statement was executed: ```sh demodb.public/demostore# DESCRIBE CREATE STREAM pageviews (viewtime BIGINT, userid VARCHAR, pageid VARCHAR) WITH ('value.format'='json', 'topic.partitions'='5', 'topic.replicas'='5'); +---------------+------------------------------------------+ | Type | Info | +===============+==========================================+ | CREATE_STREAM | {"ddl":{"fqn":"d39511ce-c918-4637-9524-1 | | | 63183452274.demodb.public.pageviews","ty | | | pe":"STREAM","db_name":"demodb","schema_ | | | name":"public","name":"pageviews","store | | | _name":"demostore"}} | +---------------+------------------------------------------+ ``` #### Describe a CCAS filter statement In this example, `pageviewscl` [Database](/overview/core-concepts/databases#changelog) is filtered and writes its result into `pageviewscl_6` changelog: ```sh demodb.public/demostore# DESCRIBE CREATE CHANGELOG PAGEVIEWSCL_6 AS SELECT userid, pageid FROM PAGEVIEWSCL WHERE userid = 'User_6'; +---------------------+------------------------------------------+ | Type | Info | +=====================+==========================================+ | CREATE_CHANGELOG_AS | {"ddl":{"fqn":"d39511ce-c918-4637-9524-1 | | | 63183452274.demodb.public.pageviewscl_6" | | | ,"type":"CHANGELOG","db_name":"demodb"," | | | schema_name":"public","name":"pageviewsc | | | l_6","store_name":"demostore"},"sources" | | | :[{"fqn":"d39511ce-c918-4637-9524-163183 | | | 452274.demodb.public.pageviewscl","type" | | | :"CHANGELOG","db_name":"demodb","schema_ | | | name":"public","name":"pageviewscl","sto | | | re_name":"demostore"}]} | +---------------------+------------------------------------------+ ``` #### Describe a CSAS JOIN statement In this example, the described statement joins the `pageviews` [Database](/overview/core-concepts/databases#stream) and `users1Changelog` changelog before filtering and writing its results into the `temporaljoin` stream: ```sh demodb.public/demostore# DESCRIBE CREATE STREAM temporaljoin AS SELECT p.userid AS pvuid, u.userid, u.gender, p.pageid, u.interests[1] AS top_interest FROM pageviews p JOIN "users1Changelog" u ON u.userid = p.userid WHERE p.userid != 'User_5'; +------------------+------------------------------------------+ | Type | Info | +==================+==========================================+ | CREATE_STREAM_AS | {"ddl":{"fqn":"d39511ce-c918-4637-9524-1 | | | 63183452274.demodb.public.temporaljoin", | | | "type":"STREAM","db_name":"demodb","sche | | | ma_name":"public","name":"temporaljoin", | | | "store_name":"demostore"},"sources":[{"f | | | qn":"d39511ce-c918-4637-9524-16318345227 | | | 4.demodb.public.pageviews","type":"STREA | | | M","db_name":"demodb","schema_name":"pub | | | lic","name":"pageviews","store_name":"de | | | mostore"},{"fqn":"d39511ce-c918-4637-952 | | | 4-163183452274.demodb.public.users1Chang | | | elog","type":"CHANGELOG","db_name":"demo | | | db","schema_name":"public","name":"users | | | 1Changelog","store_name":"demostore"}]} | +------------------+------------------------------------------+ ``` #### Describe an INSERT INTO statement The example below shows that the statement filters the `pageviews` stream and writes its results into the existing relation called `pageviews_6` of type `stream`: ```sh demodb.public/demostore# DESCRIBE INSERT INTO PAGEVIEWS_6 SELECT userid, pageid FROM PAGEVIEWS WHERE userid = 'User_6'; +-------------+------------------------------------------+ | Type | Info | +=============+==========================================+ | INSERT_INTO | {"sink":{"fqn":"d39511ce-c918-4637-9524- | | | 163183452274.demodb.public.pageviews_6", | | | "type":"STREAM","db_name":"demodb","sche | | | ma_name":"public","name":"pageviews_6"," | | | store_name":"demostore"},"sources":[{"fq | | | n":"d39511ce-c918-4637-9524-163183452274 | | | .demodb.public.pageviews","type":"STREAM | | | ","db_name":"demodb","schema_name":"publ | | | ic","name":"pageviews","store_name":"dem | | | ostore"}]} | +-------------+------------------------------------------+ ``` # DESCRIBE STORE ## Syntax ```sql DESCRIBE STORE store_name; ``` ## Description This provides the definition of an existing [Store](/overview/core-concepts/store). Stores are listed only if the current role has USAGE privileges. ### Arguments #### store\_name This is the name of the store to describe. If the name is case-sensitive, you must wrap it in double quotes; otherwise, the system uses the lowercase name. ## Examples #### Describe a store The following describes the store named `demostore`: ```sh demodb.public/demostore# DESCRIBE STORE demostore; +-------------+------------------------------------------+----------+--------------+------------------+------------------+ | Properties | Uri | Details | Tls Enabled | Verify Hostname | Schema Registry | +=============+==========================================+==========+==============+==================+==================+ | {} | b-1.kafka:9196,b-2.kafka:9196,b-3.kafka: | {} | true | true | | | | 9196 | | | | | +-------------+------------------------------------------+----------+--------------+------------------+------------------+ ``` #### Describe a case-sensitive store The following describes the store named `MyStore`: ```sh demodb.public/demostore# DESCRIBE STORE "MyStore"; +-------------+------------------------------------------+----------+--------------+------------------+------------------+ | Properties | Uri | Details | Tls Enabled | Verify Hostname | Schema Registry | +=============+==========================================+==========+==============+==================+==================+ | {} | b-1.kafka:9196,b-2.kafka:9196,b-3.kafka: | {} | true | false | | | | 9196 | | | | | +-------------+------------------------------------------+----------+--------------+------------------+------------------+ ``` # DESCRIBE USER ## Syntax ```sql DESCRIBE USER 'email_address'; ``` ## Description Describe a [Access Control](/overview/core-concepts/access-control#_user) and their information, including name, email, and granted [Access Control](/overview/core-concepts/access-control#_role)s. ### Arguments #### email\_address The email address associated with the user. ## Examples #### Describe user ```shell /# DESCRIBE USER 'someone@somewhere.com'; +------------+-------------+------------+-------------------------------+-------------------------------+ | Role Name | Granted By | Inherited | Updated At | Created At | +============+=============+============+===============================+===============================+ | orgadmin | orgadmin | false | 2023-09-19 16:17:03 +0000 UTC | 2023-09-19 16:17:03 +0000 UTC | +------------+-------------+------------+-------------------------------+-------------------------------+ | public | orgadmin | true | 2023-09-19 16:17:03 +0000 UTC | 2023-09-19 16:17:03 +0000 UTC | +------------+-------------+------------+-------------------------------+-------------------------------+ | securityad | orgadmin | true | 2023-09-19 16:17:03 +0000 UTC | 2023-09-19 16:17:03 +0000 UTC | | min | | | | | +------------+-------------+------------+-------------------------------+-------------------------------+ | sysadmin | orgadmin | true | 2023-09-19 16:17:03 +0000 UTC | 2023-09-19 16:17:03 +0000 UTC | +------------+-------------+------------+-------------------------------+-------------------------------+ | useradmin | orgadmin | true | 2023-09-19 16:17:03 +0000 UTC | 2023-09-19 16:17:03 +0000 UTC | +------------+-------------+------------+-------------------------------+-------------------------------+ ``` # GENERATE STREAM DDL ## Syntax {% code overflow="wrap" %} ```sql GENERATE STREAM DDL stream_name WITH (stream_parameter = value [, ...]); ``` {% endcode %} ## Description This command generates a CREATE DDL for STREAM from a given schema or data — for example, [Data Store](/overview/core-concepts/store#entity) — that has corresponding [Data Formats (Serialization)](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors). The command specially simplifies creating DDL for complex descriptors. You can generate DDL if the current role has [`USAGE`](/overview/core-concepts/access-control#privilege) privileges on the [Data Store](/overview/core-concepts/store). ### Stream Parameters | Parameter Name | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`stream_name`** | The name of the new stream. | | **`topic`** |

Name of the Kafka topic whose assigned data schema you're using (for Protobuf descriptor and Avro schema from SchemaRegistry).

Required? Yes, for AVRO, PROTOBUF

| | **`value. format`** |

AVRO, PROTOBUF, JSON

Required? Yes.

| | **`key.format`** |

AVRO, PROTOBUF, JSON

Required? No.

| | **`data.json.value.content`** |

A string containing one or more JSON objects separated by commas. It serves as a data sample you use to infer the schema of the value field in Kafka.

Required? Required only for JSON

| | **`data.json.key.content`** |

A string containing one or more JSON objects separated by commas. It serves as a data sample you used to infer the schema of the key in Kafka.

Required? Required only for JSON

| | **`postgresql.table.name`** |

Name of the source table in the POSTGRESQL store.

Required: only for POSTGRESQL store.

Type: String

| | **`postgresql.schema.name`** |

Name of the schema in the POSTGRESQL store containing the source table.

Required: only for POSTGRESQL store.

Type: String

| | **`postgresql.db.name`** |

Name of the database in the POSTGRESQL store containing the source table.

Required: only for POSTGRESQL store.

Type: String

| ## Generating "Create Stream" DDL examples **Generate a "create stream" ddl for an entity in the current store, which includes a protobuf descriptor.** {% code overflow="wrap" %} ```sql GENERATE STREAM DDL myStream with('topic'='pageviews_pb', 'value.format'='protobuf', 'key.format'='protobuf'); ``` {% endcode %} The output would be: {% code overflow="wrap" lineNumbers="true" fullWidth="false" %} ```sql CREATE STREAM myStream( "viewtime" BIGINT NOT NULL, "userid" VARCHAR NOT NULL, "pageid" VARCHAR NOT NULL ) WITH ('key.format'='protobuf', 'key.type'='STRUCT<"userid" VARCHAR>', 'topic'='pageviews_pb', 'value.format'='protobuf'); ``` {% endcode %} **Generate a "create stream" ddl for an entity in the current store, which includes AVRO in the Confluent Schema Registry.** {% code overflow="wrap" %} ```sql GENERATE STREAM DDL myStream with('topic'='pageviews_avro', 'value.format'='avro', 'key.format'='avro'); ``` {% endcode %} The output would be: {% code overflow="wrap" lineNumbers="true" fullWidth="false" %} ```sql CREATE STREAM myStream( "viewtime" BIGINT , "userid" VARCHAR , "pageid" VARCHAR ) WITH ('key.format'='avro', 'key.type'='STRUCT<"userid" VARCHAR>', 'topic'='pageviews_avro', 'value.format'='avro'); ``` {% endcode %} **Generate a "create stream" ddl for an entity whose content is JSON.** {% code overflow="wrap" %} ```sql GENERATE STREAM DDL myStream with ('value.format'='json', 'data.json.value.content'='{"viewtime":1629453600000,"userid":"user_123","pageid":"page_1"},{"viewtime":1629457200000,"userid":"user_456","pageid":"page_2"},{"viewtime":1629460800000,"userid":"user_789","pageid":"page_3"}' ); ``` {% endcode %} The output would be: {% code overflow="wrap" fullWidth="false" %} ```sql CREATE STREAM myStream( "viewtime" BIGINT, "userid" VARCHAR, "pageid" VARCHAR ) WITH ('value.format'='json'); ``` {% endcode %} **Generate a "create stream" ddl for a Postgres table in a CDC Pipeline** {% code overflow="wrap" %} ```sql GENERATE STREAM DDL myStream with('value.format'='json', 'store'='pgStore', 'postgresql.table.name'='pgTb', 'postgresql.schema.name'='pgSchema', 'postgresql.db.name'='pgDb'); ``` {% endcode %} {% code overflow="wrap" %} ```sql CREATE STREAM mystream( op VARCHAR, ts_ms BIGINT, "before" STRUCT<"id" BIGINT, "first_name" VARCHAR, "last_name" VARCHAR, "email" VARCHAR, "biography" VARCHAR>, "after" STRUCT<"id" BIGINT, "first_name" VARCHAR, "last_name" VARCHAR, "email" VARCHAR, "biography" VARCHAR>, "source" STRUCT<"db" VARCHAR, "schema" VARCHAR, "table" VARCHAR, "lsn" BIGINT> ) WITH ('postgresql.db.name'='pgDb', 'postgresql.schema.name'='pgSchema', 'postgresql.table.name'='pgTb', 'store'='pgStore'); ``` {% endcode %} # GENERATE CHANGELOG DDL ## Syntax {% code overflow="wrap" %} ```sql GENERATE CHANGLEOG DDL changelog_name WITH(stream_parameter = value [, ...]); GENERATE CHANGLEOG DDL changelog_name PRIMARY KEY(pk_name) WITH(stream_parameter = value [, ...]); ``` {% endcode %} ## Description This command generates a CREATE DDL for CHANGELOG from a given schema or data — for example, [Data Store](/overview/core-concepts/store#entity) — that has corresponding [Data Formats (Serialization)](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors). The command specially simplifies creating DDL for complex descriptors. You can generate DDL if the current role has [`USAGE`](/overview/core-concepts/access-control#privilege) privileges on the [Data Store](/overview/core-concepts/store). ## Changelog Parameters | Parameter Name | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`changelog_name`** | The name of the new changelog. | | **`primary key`** |

If you provide the primary key names they are added to the DDL. Otherwise, this is empty.

Required? No.

| | **`topic`** |

Name of the Kafka topic whose assigned data schema you're using (for Protobuf descriptor and Avro schema from SchemaRegistry).

Required? Yes, for AVRO, PROTOBUF

| | **`value.format`** |

AVRO, PROTOBUF, JSON

Required? Yes.

| | **`key.format`** |

AVRO, PROTOBUF, JSON

Required? No.

| | **`data.json.value.content`** |

A string containing one or more JSON objects separated by commas. It serves as a data sample you use to infer the schema of the value field in Kafka.

Required? Required only for JSON

| | **`data.json.key.content`** |

A string containing one or more JSON objects separated by commas. It serves as a data sample you used to infer the schema of the key in Kafka.

Required? Only for JSON

| ## Generating "Create Changelog" DDL examples **Generate a "create changelog" ddl for an entity in the current store, which includes a protobuf descriptor.** {% code overflow="wrap" %} ```sql GENERATE CHANGELOG DDL myChangelog with('topic'='pageviews_pb', 'value.format'='protobuf', 'key.format'='protobuf'); ``` {% endcode %} The output would be: {% code overflow="wrap" %} ```sql CREATE CHANGELOG myChangelog( "viewtime" BIGINT NOT NULL, "userid" VARCHAR NOT NULL, "pageid" VARCHAR NOT NULL, PRIMARY KEY() ) WITH ('key.format'='protobuf', 'key.type'='STRUCT<"userid" VARCHAR>', 'topic'='pageviews_pb', 'value.format'='protobuf'); ``` {% endcode %} **Generate a "create changelog" ddl for an entity in the current store, which includes AVRO in the Confluent Schema Registry.** {% code overflow="wrap" %} ```sql GENERATE CHANGELOG DDL myChangelog with('topic'='pageviews_avro', 'value.format'='avro', 'key.format'='avro'); ``` {% endcode %} The output would be {% code overflow="wrap" %} ```sql CREATE CHANGELOG myChangelog( "viewtime" BIGINT , "userid" VARCHAR , "pageid" VARCHAR , PRIMARY KEY() ) WITH ('key.format'='avro', 'key.type'='STRUCT<"userid" VARCHAR>', 'topic'='pageviews_avro', 'value.format'='avro'); ``` {% endcode %} **Generate a "create changelog" ddl for an entity whose content is JSON.** {% code overflow="wrap" %} ```sql GENERATE CHANGELOG DDL myChangelog with ('value.format'='json', 'data.json.value.content'='{"viewtime":1629453600000,"userid":"user_123","pageid":"page_1"},{"viewtime":1629457200000,"userid":"user_456","pageid":"page_2"},{"viewtime":1629460800000,"userid":"user_789","pageid":"page_3"}' ); ``` {% endcode %} The output would be ```sql CREATE CHANGELOG myChangelog( "viewtime" BIGINT, "userid" VARCHAR, "pageid" VARCHAR, PRIMARY KEY() ) WITH ('value.format'='json'); ``` # GENERATE TEMPLATE ## Syntax ```sql GENERATE TEMPLATE WITH (template_parameter = value [, ...]); ``` ## Description This command generates an AWS CloudFormation template for use in an AWS environment. ### Arguments #### WITH (template\_parameter = value \[, …​ ]) This clause specifies [#changelog\_parameters](#changelog_parameters "mention") and any other template-specific parameters mentioned below. ### Template Parameters | Parameter Name | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` |

Type of the CloudFormation template to generate.

Required: Yes
Valid values: KAFKA\_IAM\_ROLE, KINESIS\_IAM\_ROLE.
Type: String

| ### Kafka IAM Template Parameters | Parameter Name | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `template.msk.allows.topic.prefixes.create` |

List of comma-separated topic prefixes to allow creating — for example, 'ds\_'

Required: No
Type: String

| | `template.msk.allows.topic.prefixes.delete` |

List of comma-separated topic prefixes to allow deleting — for example, 'ds\_'

Required: No
Type: String

| | `template.msk.allows.topic.prefixes.read` |

List of comma-separated topic prefixes to allow reading — for example, 'my\*'

Required: No
Type: String

| | `template.msk.allows.topic.prefixes.write` |

List of comma-separated topic prefixes to allow writing — for example, 'ds\_'

Required: No
Type: String

| | `template.msk.cluster.arn` |

The Amazon Resource Name (ARN) that uniquely identifies the MSK cluster.

Required: Yes
Type: String

| ### Kinesis IAM Template Parameters | Parameter Name | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `template.kinesis.allows.streams.read` |

List of comma-separated name of data streams to give read access. This accepts wildcards in the names as well — for example,deliveries,shipment\*

Required: No
Type: String

| | `template.kinesis.allows.streams.write` |

List of comma-separated name of data streams to give write access.

Required: No
Type: String

| | `template.kinesis.allows.streams.create` |

List of comma-separated name of data streams to allow creating.

Required: No
Type: String

| | `template.kinesis.allows.streams.delete` |

List of comma-separated name of data streams to allow deleting.

Required: No
Type: String

| | `template.kinesis.allows.streams.list` |

List of comma-separated name of data streams to allow deleting. Accepts wildcard as well — for example, my\*

Required: No
Type: String

| | `template.kinesis.aws_account` |

AWS account ID number for which the template is valid.

Required: No
Type: String

| | `template.kinesis.aws_region` |

AWS region name for which the template is valid — for example, us-east-2.

Required: No
Type: String

| ## Examples #### Create a Kafka IAM role template ```sh demodb.public/demostore# GENERATE TEMPLATE > WITH ( > 'type' = KAFKA_IAM_ROLE, > 'template.msk.allows.topic.prefixes.create' = 'ds_' , > 'template.msk.allows.topic.prefixes.delete' = 'ds_' , > 'template.msk.allows.topic.prefixes.read' = '*' , > 'template.msk.allows.topic.prefixes.write' = 'ds_' , > 'template.msk.cluster.arn' = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4' ); +------------------------------------------+ | Template | +==========================================+ | | +------------------------------------------+ ``` #### Create a Kinesis IAM role template ```sh demodb.public/demostore# GENERATE TEMPLATE > WITH ( > 'type' = KINESIS_IAM_ROLE, > 'template.kinesis.allows.streams.read' = 'deliveries,shipment*', > 'template.kinesis.allows.streams.write' = 'issueswrite', > 'template.kinesis.allows.streams.create' = 'issuescreate', > 'template.kinesis.allows.streams.delete' = 'issuesdelete', > 'template.kinesis.allows.streams.list' = 'my*', > 'template.kinesis.aws_account' = '854977439775', > 'template.kinesis.aws_region' = 'us-east-2' > ); +------------------------------------------+ | Template | +==========================================+ | | +------------------------------------------+ ``` # GRANT OWNERSHIP ## Syntax ```sql GRANT OWNERSHIP ON ( COMPUTE_POOL compute_pool_name | DATABASE database_name | SCHEMA [database_name.]schema_name | RELATION [[database_name.]schema_name.]relation_name | STORE store_name | SCHEMA_REGISTRY schema_registry_name | QUERY query_id | DESCRIPTOR_SOURCE descriptor_source_name | FUNCTION_SOURCE function_source_name | FUNCTION function_identifier | ROLE role_name ) TO roleName ``` ## Description Transfer ownership or an object to a new [Role](/overview/core-concepts/access-control#_role). You can transfer ownership only if the current role is the owner of the object and has been granted the destination role. ## Examples ```sh demodb.public/# LIST DATABASES; +---------+-------------+----------+-------------------------------+ | Name | Is Default | Owner | Created At | +=========+=============+==========+===============================+ | demodb | true | sysadmin | 2023-09-10 19:41:23 +0000 UTC | +---------+-------------+----------+-------------------------------+ demodb.public/# GRANT OWNERSHIP ON DATABASE demodb TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | ownership grant | ALTER | Ownership of "main_db" granted to role | | | | "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/# USE ROLE securityadmin; +-------+---------------+----------+----------------------------+ | Type | Name | Command | Summary | +=======+===============+==========+============================+ | role | securityadmin | USE | using role "securityadmin" | +-------+---------------+----------+----------------------------+ demodb.public/# LIST DATABASES; +---------+-------------+----------+-------------------------------+ | Name | Is Default | Owner | Created At | +=========+=============+==========+===============================+ | user_db | false | orgadmin | 2024-06-05 19:41:23 +0000 UTC | +---------+-------------+----------+-------------------------------+ | demodb | true | rol1 | 2023-09-10 19:41:23 +0000 UTC | +---------+-------------+----------+-------------------------------+ ``` # GRANT PRIVILEGES ## Organization Privileges ### Syntax ```sql GRANT [ CREATE_COMPUTE_POOL | CREATE_DATABASE | CREATE_STORE | CREATE_SCHEMA_REGISTRY | CREATE_DESCRIPTOR_SOURCE | CREATE_FUNCTION_SOURCE | CREATE_FUNCTION | CREATE_QUERY | MANAGE_MEMBERS | MANAGE_GRANTS | ALL PRIVILEGES , ... ] ON ORGANIZATION TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [Organization](/overview/core-concepts/access-control#_organiation) [privileges](/overview/core-concepts/access-control#_privilege) to one or more roles. The current role requires one of the following privileges: * Ownership of Organization * `MANAGE_GRANTS` privilege on Organization * Privilege granted to the current role `WITH GRANT OPTION`. ### Arguments #### CREATE\_COMPUTE\_POOL Allow role to create [compute\_pools](/overview/core-concepts/compute-pools) under the organization. #### CREATE\_DATABASE Allow role to create [databases](/overview/core-concepts/databases) under the organization. #### CREATE\_STORE Allow role to define [stores](/overview/core-concepts/store) under the organization. #### CREATE\_SCHEMA\_REGISTRY Allow role to define [schema registries](/reference/sql-syntax/data-format-serialization#_schema_registry) under the organization. #### CREATE\_DESCRIPTOR\_SOURCE Allow role to upload [descriptor sources](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) to the organization. #### CREATE\_FUNCTION\_SOURCE Allow role to upload [UDF and UDAF sources](/overview/core-concepts/function) to the organization. #### CREATE\_FUNCTION Allow role to define a new [UDF or UDAF](/overview/core-concepts/function) under the organization. The role will also require `USAGE` privileges to the function source. #### CREATE\_QUERY Allow role to launch a new [query](/overview/core-concepts/queries) under the organization. The role also has additional privileges on database, schema, relations, and stores to launch the query. #### MANAGE\_MEMBERS Allow role to manage [roles](/overview/core-concepts/access-control#_role), invitations, and users. #### MANAGE\_GRANTS Allow role to manage all privilege grants within the organization. #### ALL PRIVILEGES Grants all the privileges listed above to the role. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to grant the privileges to. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```bash /# GRANT CREATE_DATABASE, CREATE_STORE ON ORGANIZATION TO rol1, rol2; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "create_database, | | | | create_store" on "MR main" granted to | | | | "rol1, rol2" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +--------------+---------+-----------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+=========+=================+====================+===============+ | role | public | usage | false | orgadmin | +--------------+---------+-----------------+--------------------+---------------+ | organization | MR main | create_database | false | securityadmin | +--------------+---------+-----------------+--------------------+---------------+ | organization | MR main | create_store | false | securityadmin | +--------------+---------+-----------------+--------------------+---------------+ ``` ```bash /# GRANT CREATE_DATABASE, CREATE_STORE ON ORGANIZATION TO rol1, rol2 WITH GRANT OPTION; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "create_database, | | | | create_store" on "MR main" granted to | | | | "rol1, rol2" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +--------------+---------+-----------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+=========+=================+====================+===============+ | role | public | usage | false | orgadmin | +--------------+---------+-----------------+--------------------+---------------+ | organization | MR main | create_database | true | securityadmin | +--------------+---------+-----------------+--------------------+---------------+ | organization | MR main | create_store | true | securityadmin | +--------------+---------+-----------------+--------------------+---------------+ ``` ## Compute\_pool Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON COMPUTE_POOL compute_pool_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the compute\_pool. #### CREATE Allow role to create compute\_pools. #### ALL PRIVILEGES Grants all the privileges listed above to the role. #### compute\_pool\_name The name of the compute\_pool to grant privileges on. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ## Database Privileges ```sql GRANT [ USAGE | CREATE | ALL PRIVILEGES , ... ] ON DATABASE database_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [database](/overview/core-concepts/databases) privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the database. The role also requires additional privileges on schema as well as relations to use them. #### CREATE Allow role to create schemas under the database. #### ALL PRIVILEGES Grants all the privileges listed above to the role. #### database\_name The name of the database to granted privileges on. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to grant the privileges to. #### WITH GRANT OPTION Grants privileges that allow the role to grant the same privileges to other roles. ### Example ```sh /# GRANT USAGE ON DATABASE user_db TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "user_db" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +----------+---------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==========+=========+============+====================+===============+ | role | public | usage | false | orgadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | usage | false | securityadmin | +----------+---------+------------+--------------------+---------------+ ``` ```sh /# GRANT USAGE,CREATE ON DATABASE user_db TO rol1 WITH GRANT OPTION; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "user_db" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +----------+---------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==========+=========+============+====================+===============+ | role | public | usage | false | orgadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | usage | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | create | true | securityadmin | +----------+---------+------------+--------------------+---------------+ ``` ## Database Schema Privileges ```sql GRANT [ USAGE | CREATE | ALL PRIVILEGES ] ON SCHEMA schema_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants schema privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the schemas. The role also has additional privileges on relations to use them. #### CREATE Allow role to create relations under the schema. #### ALL PRIVILEGES Grants all the privileges listed above to the role. #### schema\_name The qualified name of the schema to grant privileges on. This name can include a specific database name to form a fully-qualified name in the format of `.;` otherwise, the system uses the current database name in the session. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh /# GRANT USAGE,CREATE ON SCHEMA accounting_db.public TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage, create" on | | | | "public" granted to "rol1" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +----------+---------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==========+=========+============+====================+===============+ | role | public | usage | false | orgadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | usage | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | schema | public | usage | false | securityadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | create | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | schema | public | create | false | securityadmin | +----------+---------+------------+--------------------+---------------+ ``` ```sh /# GRANT USAGE,CREATE ON SCHEMA accounting_db.public TO rol1 WITH GRANT OPTION; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage, create" on | | | | "public" granted to "rol1" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +----------+---------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==========+=========+============+====================+===============+ | role | public | usage | false | orgadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | usage | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | schema | public | usage | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | database | user_db | create | true | securityadmin | +----------+---------+------------+--------------------+---------------+ | schema | public | create | true | securityadmin | +----------+---------+------------+--------------------+---------------+ ``` ## Store Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON STORE store_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants store privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the store. #### store\_name The name of the store on which to grant privileges. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh /# GRANT USAGE ON STORE kafka_pub TO rol2; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "kafka_pub" | | | | granted to "rol2" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol2; +--------------+-----------+-----------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+===========+=================+====================+===============+ | role | public | usage | false | orgadmin | +--------------+-----------+-----------------+--------------------+---------------+ | store | kafka_pub | usage | false | securityadmin | +--------------+-----------+-----------------+--------------------+---------------+ ``` ```sh /# GRANT USAGE ON STORE kafka_pub TO rol2 WITH GRANT OPTION; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "kafka_pub" | | | | granted to "rol2" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol2; +--------------+-----------+-----------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+===========+=================+====================+===============+ | role | public | usage | false | orgadmin | +--------------+-----------+-----------------+--------------------+---------------+ | store | kafka_pub | usage | true | securityadmin | +--------------+-----------+-----------------+--------------------+---------------+ ``` ## Descriptor Source Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON DESCRIPTOR_SOURCE descriptor_source_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [descriptor source](/reference/sql-syntax/data-format-serialization#protocol-buffers-and-descriptors) privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the descriptor source. #### descriptor\_source\_name The name of the descriptor source on which to grant privileges. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh demodb.public/demostore# GRANT USAGE ON DESCRIPTOR_SOURCE demosource TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "demosource" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/demostore# DESCRIBE ROLE rol1; +-------------------+--------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+==============+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+-------------------------+------------+--------------------+---------------+ | descriptor_source | demosource | usage | false | securityadmin | +-------------------+--------------+------------+--------------------+---------------+ ``` ## Relation Privileges ```sql GRANT [ SELECT | INSERT | ALL PRIVILEGES ] ON RELATION relation_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [relation](/overview/core-concepts/databases#_relation) privileges to one or more roles. ### Arguments #### SELECT Allow role to create a [query](/overview/core-concepts/queries) and use the relation as a source. #### INSERT Allow role to create a [query](/overview/core-concepts/queries) and use the relation as a sink. #### relation\_name The name of the relation to grant privileges on. Optionally, provide [database](/overview/core-concepts/databases) and [schema](/overview/core-concepts/databases#_schema) name for a fully-qualified relation name in the format of `[..]` — for example, `db1.public.pageviews`. Otherwise, the system uses the current database and schema to identify the relation. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh demodb.public/demostore# GRANT SELECT ON RELATION demodb."public".pv TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "select" on "pv" granted | | | | to "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/demostore# DESCRIBE ROLE rol1; +-------------------+-------------------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+=========================+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+-------------------------+------------+--------------------+---------------+ | relation | pv | select | false | securityadmin | +-------------------+-------------------------+------------+--------------------+---------------+ ``` ```sh demodb.public/demostore# GRANT INSERT ON RELATION demodb."public".pageviews TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "insert" on "pv" granted | | | | to "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/demostore# DESCRIBE ROLE rol1; +-------------------+-------------------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+=========================+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+-------------------------+------------+--------------------+---------------+ | relation | pv | insert | false | securityadmin | +-------------------+-------------------------+------------+--------------------+---------------+ ``` ## Function Source Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON FUNCTION_SOURCE function_source_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [function source](/overview/core-concepts/function) privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the runction source. #### function\_source\_name The name of the function source on which to grant privileges. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh demodb.public/demostore# GRANT USAGE ON FUNCTION_SOURCE demofnsrc TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "demofnsrc" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/demostore# DESCRIBE ROLE rol1; +-------------------+-----------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+===========+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+-----------+------------+--------------------+---------------+ | function_source | demofnsrc | usage | false | sysadmin | +-------------------+-----------+------------+--------------------+---------------+ ``` ## Function Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON FUNCTION function_identifier TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [function](/overview/core-concepts/function) privileges to one or more roles. ### Arguments #### USAGE Allow role to list and use the function. #### function\_identifier The name of the function on which to grant privileges. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which togrant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh demodb.public/demostore# LIST FUNCTIONS; +----------------------------+-------+--------------+------------------+--------------------+----------+-------------+-------------------------------+-------------------------------+ | Signature | Type | Source Name | Class Name | Egress Allow URIs | Owner | Properties | Created At | Updated At | +============================+=======+==============+==================+====================+==========+=============+===============================+===============================+ | upper(a VARCHAR) VARCHAR | udf | my_src | demo.DSUpperCase | | sysadmin | {} | 2024-06-06 03:35:52 +0000 UTC | 2024-06-06 03:35:52 +0000 UTC | +----------------------------+-------+--------------+------------------+--------------------+----------+-------------+-------------------------------+-------------------------------+ demodb.public/demostore# GRANT USAGE ON FUNCTION upper(a varchar) varchar TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "upper" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ demodb.public/demostore# DESCRIBE ROLE rol1; +-------------------+------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+============+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+------------+------------+--------------------+---------------+ | function_source | demofnsrc | usage | false | sysadmin | +-------------------+------------+------------+--------------------+---------------+ | function | my_func | usage | false | sysadmin | +-------------------+------------+------------+--------------------+---------------+ ``` ## Region Privileges ```sql GRANT [ USAGE | ALL PRIVILEGES ] ON REGION region_name TO ROLE role_name [, ...] [WITH GRANT OPTION]; ``` ### Description Grants [region](/overview/core-concepts/region) usage privileges to one or more roles. By default, the `public` role is granted access to all regions. A role with the `MANAGE_GRANTS` privilege can grant the region `USAGE` privilege to other roles, or revoke it. ### Arguments #### USAGE Allow role to list and use the region to create [stores](/overview/core-concepts/store) and launch [queries](/overview/core-concepts/queries). #### region\_name The name of the region on which to grant privileges. #### role\_name \[, ...] One or more [roles](/overview/core-concepts/access-control#_role) to which to grant the privileges. #### WITH GRANT OPTION Grants privileges that allow the role to grant those same privileges to other roles. ### Example ```sh /# DESCRIBE ROLE "public"; +--------------+----------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+================+============+====================+===============+ | role | public | usage | false | orgadmin | +--------------+----------------+------------+--------------------+---------------+ | region | AWS us-east-1 | usage | false | securityadmin | +--------------+----------------+------------+--------------------+---------------+ ``` ```sh /# GRANT USAGE ON REGION "AWS us-east-1" TO rol1; +-----------------+----------+------------------------------------------+ | Type | Command | Summary | +=================+==========+==========================================+ | privilege grant | ALTER | Privilege(s) "usage" on "AWS us-east-1" | | | | granted to "rol1" | +-----------------+----------+------------------------------------------+ /# DESCRIBE ROLE rol1; +-------------------+----------------+------------+--------------------+---------------+ | Type | Name | Privilege | With Grant Option | Granted By | +===================+================+============+====================+===============+ | role | public | usage | false | orgadmin | +-------------------+----------------+------------+--------------------+---------------+ | region | AWS us-east-1 | usage | false | securityadmin | +-------------------+----------------+------------+--------------------+---------------+ ``` # GRANT ROLE ## Syntax ```sql GRANT ROLE role_name [, role_name...] { TO USER user_email | TO ROLE role_name [, role_name...] }; ``` ## Description Grants access to [role](/overview/core-concepts/access-control#_role)(s) to a [user](/overview/core-concepts/access-control#_user) or other role(s). This allows the user to [USE ROLE](/reference/sql-syntax/command/use) and inherit all the [privileges](/overview/core-concepts/access-control#_privilege) granted to the role. The current role requires one of the following privileges: * Ownership of organization * `MANAGE_MEMBERS` privilege on organization * `OWNER` privilege on both parent and child roles (when granting to roles) ### Arguments #### role\_name \[, role\_name...] One or more [roles](/overview/core-concepts/access-control#_role) to grant. #### user\_email Email of the user when granting roles to a user. #### role\_name \[, role\_name...] One or more [roles](/overview/core-concepts/access-control#_role) that are granted the roles. ## Examples #### Grant role to a user ```sh /# GRANT ROLE custom_role TO USER 'user@domain.com'; +------------+------------+------------------------------------------+ | Type | Command | Summary | +============+============+==========================================+ | role grant | ALTER | Role(s) "custom_role" granted to user | | | | user "user@domain.com" | +------------+------------+------------------------------------------+ /# LIST USER ROLES; +---------------+-------------+-------------+---------------+ | Name | Is Current | Is Default | Is Inherited | +===============+=============+=============+===============+ | custom_role | false | false | false | +---------------+-------------+-------------+---------------+ | orgadmin | true | false | false | +---------------+-------------+-------------+---------------+ | public | false | false | true | +---------------+-------------+-------------+---------------+ ``` #### Grant role to another role ```sh /# GRANT ROLE useradmin TO ROLE sysadmin; +------------+------------+------------------------------------------+ | Type | Command | Summary | +============+============+==========================================+ | role grant | ALTER | Role(s) "useradmin" were granted to | | | | "sysadmin" | +------------+------------+------------------------------------------+ /# DESCRIBE ROLE sysadmin; +--------------+------------+------------------+--------------------+-------------+ | Type | Name | Privilege | With Grant Option | Granted By | +==============+============+==================+====================+=============+ | role | public | usage | false | orgadmin | +--------------+------------+------------------+--------------------+-------------+ | role | useradmin | usage | false | orgadmin | +--------------+------------+------------------+--------------------+-------------+ ``` # INVITE USER ## Syntax ```sql INVITE USER email_address [, ...] [ WITH (invitation_parameter = value [, ...]) ]; ``` ## Description This enables an organization user manager to invite users into an organization. The primary role name assigned to the user is reserved and kept inactive until the user accepts the invitation. If the user rejects the invitation the role is dropped. When you create the invitation you can assign additional roles, as well as a default role, to the user’s primary role. Only a [role](/overview/core-concepts/access-control#_role) with [MANAGE\_MEMBERS](/overview/core-concepts/access-control#_privilege) privilege can invite users. ### Arguments #### email\_address The email address of the user(s) to invite into the current organization. #### WITH invitation\_parameter = \[, ...] Optionally, this clause specifies [#store\_parameters](#store_parameters "mention"). ### Invitation Parameters
Parameter NameDescription
roles

Specifies a list of additional roles to grant to the user role once the invitation is accepted.


Required: No

Default value: []

Type: List
Valid values: See /pages/VqwCIosvUQkC2nR6FFJD

default_roleSpecifies the role that should be the default login role for the user.

Required: Yes
Default value: None
Type: String
Valid values: See /pages/VqwCIosvUQkC2nR6FFJD
## Example #### Invite a new user to an organization with specific roles 
INVITE USER 'someone@somewhere.com' WITH (
    'roles' = (a_role, sysadmin),
    'default_role' = sysadmin
);
#### Invite multiple users to an organization with specific roles ```sh /# INVITE USER 'someone@somewhere.com', 'someone@somewhereelse.com' WITH ('default_role' = sysadmin , 'roles' = (a_role, sysadmin)); +------------+--------------------------------------+----------+------------------------------------------+ | Type | Name | Command | Summary | +============+======================================+==========+==========================================+ | invitation | b4f99db5-9d1e-4682-a2b4-940a181b27e1 | CREATE | invitation for "someone@somewhere.com" | | | | | was successfully created | +------------+--------------------------------------+----------+------------------------------------------+ | invitation | 89ba9a85-9eef-4795-a84e-70c8480a823f | CREATE | invitation for | | | | | "someone@somewhereelse.com" was | | | | | successfully created | +------------+--------------------------------------+----------+------------------------------------------+ ``` # LIST API\_TOKENS ## Syntax ```sql [ LIST | SHOW ] API_TOKENS; ``` ## Description This command provides a list of a user's `API_TOKEN`s within the current [Access Control](/overview/core-concepts/access-control#organiation). ## Examples ```sh /# LIST API_TOKENS; +-----------+-------------------------------+-------------------------------+-------------------------------+ | Name | Last Used | Created At | Expires At | +===========+===============================+===============================+===============================+ | my_token | | 2024-06-04 22:42:18 +0000 UTC | 2024-09-02 22:42:18 +0000 UTC | +-----------+-------------------------------+-------------------------------+-------------------------------+ | mr_local | 2024-06-04 22:39:12 +0000 UTC | 2024-06-04 22:36:28 +0000 UTC | 2024-09-02 22:36:28 +0000 UTC | +-----------+-------------------------------+-------------------------------+-------------------------------+ ``` # LIST COMPUTE\_POOLS ## Syntax ```sql [LIST | SHOW] COMPUTE_POOLS; ``` ## Description This provides a list of[ ](https://docs.deltastream.io/overview/core-concepts/store)compute\_pools to which you have access. {% hint style="warning" %} **Important** You're only able to list compute\_pools if your current role has USAGE privileges. {% endhint %} ## Example ```sh mydb.public/mystore# show COMPUTE_POOLS; +-------+-----------------+---------------+-----------------+----------------+-------+----------+-------------+------------+-----------------------------------+----------------------------------+ | Name | Intended State | Actual State | Error Messages | Access Region | Size | Timeout | Is Default | Owner | Created At | Updated At | +-------+-----------------+---------------+-----------------+----------------+-------+----------+-------------+------------+-----------------------------------+----------------------------------+ | "abc" | stopped | stopped | | K3D us-east-1 | small | 120 | true | "sysadmin" | 2025-02-25 15:13:03.935 +0000 UTC | 2025-03-04 03:04:04.34 +0000 UTC | +-------+-----------------+---------------+-----------------+----------------+-------+----------+-------------+------------+-----------------------------------+----------------------------------+ ``` # LIST DATABASES ## Syntax ```sql [ LIST | SHOW ] DATABASES; ``` ## Description This command provides a list of [databases](/overview/core-concepts/databases) within the [organization](/overview/core-concepts/access-control#_organiation). A database is listed only if the current role has USAGE privileges. ## Examples ```sh demodb.public/demostore# LIST DATABASES; +------------+-------------+------------+-------------------------------+ | Name | Is Default | Owner | Created At | +============+=============+============+===============================+ | demodb | true | sysadmin | 2024-07-02 16:23:26 +0000 UTC | +------------+-------------+------------+-------------------------------+ | otherdb | false | sysadmin | 2024-06-28 21:21:18 +0000 UTC | +------------+-------------+------------+-------------------------------+ ``` # LIST DESCRIPTORS ## Syntax ```sql [ LIST | SHOW ] DESCRIPTORS [ IN DESCRIPTOR_SOURCE descriptor_source_name ]; ``` ## Description [Descriptor](https://github.com/deltastreaminc/ds-docs-gitbook/blob/main-dcap/overview/core-concepts/data-format-serde.md)[s](https://github.com/deltastreaminc/ds-docs-gitbook/blob/main-dcap/overview/core-concepts/data-format-serde.md) are used for supporting [protocol buffers](https://protobuf.dev/). Creating a descriptor source enables users to associate [ProtoBuf descriptors](https://protobuf.dev/) with specific [entities](/overview/core-concepts/store#entity) for data serialization and deserialization. This command provides a list of descriptors that the user has access to in the current organization. Descriptors are created when descriptor sources are created with [CREATE DESCRIPTOR\_SOURCE](/reference/sql-syntax/ddl/create-descriptor_source). This command provides a list of [descriptor](/reference/sql-syntax/data-format-serialization)[s](/reference/sql-syntax/data-format-serialization), optionally filtered by a specific descriptor source. Descriptors are listed only if the current role has `USAGE` privileges on the descriptor source. ### Arguments #### descriptor\_source\_name Optionally, this lists the descriptors belonging to the specified descriptor source. ## Examples #### List all descriptors ```sh demodb.public/demostore# LIST DESCRIPTORS; +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Name | Source Name | Type | Owner | Created At | Updated At | +============+==============+============+============+===============================+===============================+ | Message2 | src1 | protobuf | sysadmin | 2024-06-03 16:59:52 +0000 UTC | 2024-06-03 16:59:52 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Message1 | src1 | protobuf | sysadmin | 2024-06-03 16:59:52 +0000 UTC | 2024-06-03 16:59:52 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | StaffKey | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Manager | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Employee | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Address | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ ``` #### List all descriptors from a descriptor source From a list of all available descriptors, you can choose one of the descriptor source names — for example, `employee` — and list only the descriptors imported using that source: ```sh demodb.public/demostore# LIST DESCRIPTORS IN DESCRIPTOR_SOURCE employee; +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Name | Source Name | Type | Owner | Created At | Updated At | +============+==============+============+============+===============================+===============================+ | StaffKey | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Manager | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Employee | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ | Address | employee | protobuf | sysadmin | 2024-06-04 19:03:51 +0000 UTC | 2024-06-04 19:03:51 +0000 UTC | +------------+--------------+------------+------------+-------------------------------+-------------------------------+ ``` # LIST DESCRIPTOR\_SOURCES ## Syntax ```sql [ LIST | SHOW ] DESCRIPTOR_SOURCES; ``` ## Description This command provides a list of [descriptor sources](/reference/sql-syntax/data-format-serialization) within the [organization](/overview/core-concepts/access-control#_organiation). A descriptor source is listed only if the current role has `USAGE` privileges. ## Examples ```sh demodb.public/demostore# SHOW DESCRIPTOR_SOURCES; +-----------+-------+----------+----------------+----------+-------------------------------+-------------------------------+ | Name | Tags | Type | Url | Owner | Created At | Updated At | +===========+=======+==========+================+==========+===============================+===============================+ | pageviews | [] | protobuf | | sysadmin | 2024-06-05 17:38:56 +0000 UTC | 2024-06-05 17:38:56 +0000 UTC | +-----------+-------+----------+----------------+----------+-------------------------------+-------------------------------+ ``` # LIST ENTITIES ## Syntax ```sql [ LIST | SHOW ] ENTITIES [IN fully_qualified_entity_name] [IN STORE store_name]; ``` ## Description This command creates a new entity supported by a [store](/overview/core-concepts/store). You can then use the entity to host [relations](/overview/core-concepts/databases#_relation) created through [DDL](/reference/sql-syntax/ddl) or [query](/reference/sql-syntax/query). This command lists entities that are present under the entity name specified by `fully_qualified_entity_name`. If `fully_qualified_entity_name` is not provided, the system returns the entities under the root namespace. Entities are listed only if the current role has USAGE privileges on the store. > Note: For stores with a large number of entities, this command limit the list to 1000 entities. ### Arguments #### IN fully\_qualified\_entity\_name Optionally, this lists entities under the provided entity name. Otherwise, the system returns the root entities. #### IN STORE store\_name Optionally, this lists the entities in the specified store. By default, the system uses the current session's store. ## Examples #### List Kafka topics in the current store The following lists the entities in the current store: ```sh demodb.public/kafka_store# LIST ENTITIES; +--------------+----------+ | Name | Is Leaf | +==============+==========+ | pageviews | true | +--------------+----------+ | pageviews_pb | true | +--------------+----------+ | users | true | +--------------+----------+ ``` #### List Snowflake databases ```sh demodb.public/sfstore# LIST ENTITIES; +-----------------+------------+ | Name | Is Leaf | +=================+============+ | MY_SNOWFLAKE_DB | false | +-----------------+------------+ | SNOWFLAKE | false | +-----------------+------------+ | SNOWFLAKE_SAMPL | false | | E_DATA | | +-----------------+------------+ ``` #### List Snowflake schemas in a database ```sh demodb.public/sfstore# LIST ENTITIES IN "DELTA_STREAMING"; +---------------------+------------+ | Name | Is Leaf | +=====================+============+ | MY_STREAMING_SCHEMA | false | +---------------------+------------+ | PUBLIC | false | +---------------------+------------+ ``` #### List Snowflake tables in a schema ```sh demodb.public/sfstore# LIST ENTITIES IN "DELTA_STREAMING"."PUBLIC"; +---------------------+------------+ | Name | Is Leaf | +=====================+============+ | STREAM_DATA_TBL | true | +---------------------+------------+ | sf_pv_table | true | +---------------------+------------+ ``` #### List Databricks catalogs ```sh demodb.public/databricks_store# LIST ENTITIES; +------------+------------+ | Name | Is Leaf | +============+============+ | catalog1 | false | +------------+------------+ | main | false | +------------+------------+ | system | false | +------------+------------+ ``` #### List Databricks schemas in a catalog ```sh demodb.public/databricks_store# LIST ENTITIES in catalog1; +--------------------+------------+ | Name | Is Leaf | +====================+============+ | default | false | +--------------------+------------+ | information_schema | false | +--------------------+------------+ ``` #### List Databricks tables in a schema ```sh demodb.public/databricks_store# LIST ENTITIES IN catalog1.schema1; +--------------------+------------+ | Name | Is Leaf | +====================+============+ | pageviews | true | | pv_agg | true | +--------------------+------------+ ``` #### List schemas in a Postgresql store ```sh demodb.public/databricks_store# LIST ENTITIES; +------------+------------+ | Name | Is Leaf | +============+============+ | public | false | +------------+------------+ ``` #### List Postgresql tables in a schema ```sh demodb.public/psqlstore# LIST ENTITIES in public; +--------------------+------------+ | Name | Is Leaf | +====================+============+ | pageviews | true | +--------------------+------------+ ``` #### List Snowflake databases in a specific store ```sh demodb.public/kafka_store# LIST ENTITIES IN STORE sfstore; +-----------------+------------+ | Name | Is Leaf | +=================+============+ | MY_SNOWFLAKE_DB | false | +-----------------+------------+ | SNOWFLAKE | false | +-----------------+------------+ | SNOWFLAKE_SAMPL | false | | E_DATA | | +-----------------+------------+ ``` # LIST FUNCTIONS ## Syntax ```sql [ LIST | SHOW ] FUNCTIONS; ``` ## Description This command provides a list of [Function](/overview/core-concepts/function)s defined under the [Access Control](/overview/core-concepts/access-control#organiation). Functions are visible only if the current [Access Control](/overview/core-concepts/access-control#role) has `USAGE` privileges on them. ## Examples ```sh demodb.public/demostore# LIST FUNCTIONS; +----------------------------+-------+--------------+------------------+--------------------+----------+-------------+-------------------------------+-------------------------------+ | Signature | Type | Source Name | Class Name | Egress Allow URIs | Owner | Properties | Created At | Updated At | +============================+=======+==============+==================+====================+==========+=============+===============================+===============================+ | my_func(a VARCHAR) VARCHAR | udf | my_src | demo.DSUpperCase | | sysadmin | {} | 2024-06-06 03:35:52 +0000 UTC | 2024-06-06 03:35:52 +0000 UTC | +----------------------------+-------+--------------+------------------+--------------------+----------+-------------+-------------------------------+-------------------------------+ ``` --- [Next Page](/llms-full.txt/1)