CREATE ENTITY

Syntax

CREATE ENTITY fully_qualified_entity_name
[IN STORE store_name]
[WITH (entity_parameter = value [, ...])];

Description

This command creates a new entity supported by a Data Store. Use these entities to host DeltaStream Objects created through DDL or Query.

To list the entities created by this command, use LIST ENTITIES.

Arguments

fully_qualified_entity_name

The full name of the entity to create.

IN STORE store_name

Optionally, this creates the entity in the specified store. If the name is case sensitive you must wrap it in double quotes; otherwise the system uses the lowercase name.

WITH (entity_parameter = value [, …​])

This clause specifies Entity Parameters.

Entity Parameters

Parameter Name	Description
key.descriptor	A qualified descriptor name used to decode a record's key, if applicable. Reset the descriptor by setting it to NULL. Required: No Default value: None Type: String Valid values: See LIST DESCRIPTORS.
value.descriptor	A qualified descriptor name used to decode a record's value. Reset the descriptor by setting it to NULL. Required: No Default value: None Type: String Valid values: See LIST DESCRIPTORS.

Kafka-Specific Entity Parameters

Parameters to be used if the associated Data Store is type KAFKA:

Parameter Name	Description
topic.partitions	The number of partitions to use when creating the entity. Required: No Default value: 1 Type: Integer Valid values: [1,…]
topic.replicas	The number of replicas to use when creating the entity. Required: No Default value: 1 Type: Integer Valid values: [1,…]
kafka.topic.*	A configuration specific for the topic being created — for example, Kafka Entity Configuration for Confluent Platform. Required: No Default value: None Type: String Valid values: Kafka topic configuration specific to the underlying Data Storetype.

Kinesis-Specific Entity Parameters

Parameters to be used if the associated Data Store is type KINESIS:

Parameter Name	Description
kinesis.shards	The number of shards to use when creating the entity. Required: No Default value: 1 Type: Integer Valid values: [1,…]

Examples

Create a new Kafka topic with defaults

The following creates an entity called pv using the default parameters in your default store:

demodb.public/demostore# CREATE ENTITY pv;
+--------+-------+----------+------------------------------------------+
|  Type  |  Name |  Command |  Summary                                 |
+========+=======+==========+==========================================+
| entity | pv    | CREATE   | entity "pv" was successfully created in  |
|        |       |          | store "demostore"                        |
+--------+-------+----------+------------------------------------------+
demodb.public/demostore# LIST ENTITIES;
+--------------+----------+
|  Name        |  Is Leaf |
+==============+==========+
| pv_pb        | true     |
+--------------+----------+
| pv           | true     |
+--------------+----------+

Create a new Kafka topic with passthrough configuration for retention

CREATE ENTITY customers WITH (
  'store' = 'kafka_store', 
  'topic.partitions' = 1, 
  'topic.replicas' = 2, 
  'kafka.topic.retention.ms' = '172800000');

Create a new Kafka topic with additional topic configuration

The following creates an entity called pv_compact. It also overrides the default partitions, replicas, and cleanup.policy configuration of the Kafka topic in your default store:

demodb.public/demostore# CREATE ENTITY pv_compact WITH ( 'kafka.partitions' = 2, 'kafka.replicas' = 1, 'kafka.topic.cleanup.policy' = 'compact');
+--------+------------+----------+------------------------------------------+
|  Type  |  Name      |  Command |  Summary                                 |
+========+============+==========+==========================================+
| entity | pv_compact | CREATE   | entity "pv_compact" was successfully     |
|        |            |          | created in store "demostore"             |
+--------+------------+----------+------------------------------------------+
demodb.public/demostore# DESCRIBE ENTITY pv_compact;
+------------+-------------+-----------+-----------------+-------------------+------------------------------------------+
|  Name      |  Partitions |  Replicas |  Key Descriptor |  Value Descriptor |  Configs                                 |
+============+=============+===========+=================+===================+==========================================+
| pv_compact | 2           | 1         | <null>          | <null>            | { "cleanup.policy": "compact" }          |
+------------+-------------+-----------+-----------------+-------------------+------------------------------------------+

Create a new Kafka topic with key and value ProtoBuf Descriptors

The following creates an entity called pageviews_pb in your default store. It also sets the key and value descriptors necessary for serializing its records:

demodb.public/demostore# LIST DESCRIPTORS;
+-----------+--------------+----------+----------+-------------------------------+-------------------------------+
|  Name     |  Source Name |  Type    |  Owner   |  Created At                   |  Updated At                   |
+===========+==============+==========+==========+===============================+===============================+
| Pageviews | pb_value     | protobuf | sysadmin | 2024-07-16 18:30:39 +0000 UTC | 2024-07-16 18:30:39 +0000 UTC |
+-----------+--------------+----------+----------+-------------------------------+-------------------------------+
| Pageviews | pb_key       | protobuf | sysadmin | 2024-07-16 18:30:31 +0000 UTC | 2024-07-16 18:30:31 +0000 UTC |
| Key       |              |          |          |                               |                               |
+-----------+--------------+----------+----------+-------------------------------+-------------------------------+
demodb.public/demostore# CREATE ENTITY pv_pb WITH ( 'key.descriptor' = pb_key."PageviewsKey", 'value.descriptor' = pb_value."Pageviews");
+--------+-------+----------+------------------------------------------+
|  Type  |  Name |  Command |  Summary                                 |
+========+=======+==========+==========================================+
| entity | pv_pb | CREATE   | entity "pv_pb" was successfully created  |
|        |       |          | in store "demostore"                     |
+--------+-------+----------+------------------------------------------+
demodb.public/demostore# DESCRIBE ENTITY pv_pb;
+-------+-------------+-----------+---------------------+--------------------+-----------------------------------+
|  Name |  Partitions |  Replicas |  Key Descriptor     |  Value Descriptor  |  Configs                          |
+=======+=============+===========+=====================+====================+===================================+
| pv_pb | 1           | 1         | pb_key.PageviewsKey | pb_value.Pageviews | { "segment.bytes": "1073741824" } |
+-------+-------------+-----------+---------------------+--------------------+-----------------------------------+

Create a new entity in Kinesis Store with Kinesis parameters

The following creates an entity called pv_kinesis in the store named kinesis_store with 3 shards:

demodb.public/demostore# CREATE ENTITY pv_kinesis IN STORE kinesis_store WITH ('kinesis.shards' = 3);
+--------+------------+----------+-----------------------------------------------+
|  Type  |  Name      |  Command |  Summary                                      |
+========+============+==========+===============================================+
| entity | pv_kinesis | CREATE   | entity "pv_kinesis" was successfully created  |
|        |            |          | in store "kinesis_store"                      |
+--------+------------+----------+-----------------------------------------------+

Create a Snowflake database

db.public/sfstore# CREATE ENTITY "DELTA_STREAMING";
+------------+-----------------+------------+------------------------------------------+
|  Type      |  Name           |  Command   |  Summary                                 |
+============+=================+============+==========================================+
| entity     | DELTA_STREAMING | CREATE     | entity DELTA_STREAMING was successfully  |
|            |                 |            | created in store sfstore                 |
+------------+-----------------+------------+------------------------------------------+
db.public/sfstore# LIST ENTITIES;
+-----------------+------------+
|  Name           |  Is Leaf   |
+=================+============+
| DELTA_STREAMING | false      |
+-----------------+------------+
| FLINK_STREAMING | false      |
+-----------------+------------+

Create a Snowflake schema in a database

In this example, you create a new schema within the existing DELTA_STREAMING Snowflake database:

db.public/sfstore# CREATE ENTITY "DELTA_STREAMING"."MY_STREAMING_SCHEMA";
+------------+-------------------------------------+------------+------------------------------------------+
|  Type      |  Name                               |  Command   |  Summary                                 |
+============+=====================================+============+==========================================+
| entity     | DELTA_STREAMING.MY_STREAMING_SCHEMA | CREATE     | entity                                   |
|            |                                     |            | DELTA_STREAMING.MY_STREAMING_SCHEMA was  |
|            |                                     |            | successfully created in store            |
|            |                                     |            | sfstore                                  |
+------------+-------------------------------------+------------+------------------------------------------+
db.public/sfstore# LIST ENTITIES IN "DELTA_STREAMING";
+---------------------+------------+
|  Name               |  Is Leaf   |
+=====================+============+
| MY_STREAMING_SCHEMA | false      |
+---------------------+------------+
| PUBLIC              | false      |
+---------------------+------------+              

Create a Databricks catalog

demodb.public/databricks_store# CREATE ENTITY cat1;
+------------+------------+------------+------------------------------------------+
|  Type      |  Name      |  Command   |  Summary                                 |
+============+============+============+==========================================+
| entity     | cat1       | CREATE     | entity cat1 was successfully created in  |
|            |            |            | store databricks_store                   |
+------------+------------+------------+------------------------------------------+
demodb.public/test_databricks_store# LIST ENTITIES;
+----------------+------------+
|  Name          |  Is Leaf   |
+================+============+
| cat1           | false      |
+----------------+------------+
| system         | false      |
+----------------+------------+

Create a Databricks schema in a catalog

In this example, you create a new schema within the existing DELTA_STREAMING Snowflake database:

demodb.public/databricks_store# CREATE ENTITY cat1.schema1;
+------------+--------------+------------+------------------------------------------+
|  Type      |  Name        |  Command   |  Summary                                 |
+============+==============+============+==========================================+
| entity     | cat1.schema1 | CREATE     | entity cat1.schema1 was successfully     |
|            |              |            | created in store databricks_store        |
+------------+--------------+------------+------------------------------------------+
demodb.public/databricks_store# LIST ENTITIES IN cat1;
+--------------------+------------+
|  Name              |  Is Leaf   |
+====================+============+
| default            | false      |
+--------------------+------------+
| information_schema | false      |
+--------------------+------------+
| schema1            | false      |
+--------------------+------------+