Iceberg AWS Glue Catalog

Iceberg AWS GLUE

Apache Iceberg is a high-performance table format that supports large analytic tables.

This document walks through setting up Iceberg in DeltaStream using the AWS Glue catalog.

Note Iceberg is unique in DeltaStream in that, if you plan on reading from or querying Iceberg data, it requires you also define an object called a compute pool. A compute pool is a set of dedicated resources for running batch queries.

You do not need a compute pool if you are only writing to Iceberg – if, for example, you’re streaming filtered Kafka data into Iceberg tables. More information on compute pools.

Before You Begin

Work with your internal engineering team to set up an AWS Glue account. You can start with the AWS Glue documentation.

For this setup guide you must also have created a stream defined in DeltaStream labeled pageviews, which is backed by a topic in an Apache Kafka data Store. More details on creating a stream in DeltaStream.

Adding an Iceberg AWS GLUE data store

To set up Iceberg AWS Glue

1. Log onto DeltaStream. In the lefthand navigation, click Resources ( ) and, when the list of data stores displays, click + Add Data Store.

1. When the Choose a Data Store window opens, click Iceberg AWS Glue.

2. Click Next. The Add Data Store window opens.

1. Enter the requested authentication and connection values.

With the data store set up and the Kafka stream created, you can perform a simple filter on the pageviews stream and then sink the results back into Iceberg.

Write a CTAS (CREATE TABLE AS SELECT) Query to Sink Data into Iceberg

Here we’re reading data from Kafka and writing to AWS Glue. This ensures we’re using the correct data store.

1. In the lefthand navigation, click Workspace ( ).
2. In the SQL pane of your workspace, write the CREATE TABLE AS SELECT (CTAS) query to ingest from pageviews and output to a new table titled pageviews_iceberg_rest.

CREATE TABLE pageviews_iceberg_glue WITH (
'store' = 'iceberg_glue_store',
'iceberg.aws.glue.db.name' = 'gradient',
'iceberg.aws.glue.table.name' = 'pageviews_iceberg'
AS SELECT * FROM pageviews;

Notes

• iceberg.aws.glue.db.name is required. It creates the sink table in your DB.

• iceberg.aws.glue.table.name is optional. If you do not specify a table name, DeltaStream uses the object name on the first line – in this case, pageviews_iceberg_glue.

1. Click Run.

View the existing queries, including the query from the step immediately prior. To do this, in the left-hand navigation click Queries ( ).

Note It may take a few moments for the query to transition into the Running state. Keep refreshing your screen until the query transitions.

To see more details about the status of the query, click the query row:

View the results

1. In the left-hand navigation, click Resources ( ). This displays a list of the existing data stores.
2. To view the new table created by the above CTAS, navigate to the pageviews_iceberg table.

To view a sample of the data in your Iceberg table, click Print.

Process Streaming Data From Your Iceberg Data Store

Now it’s time to query the data stored in Iceberg.

1. Define a compute_pool to be able to query the iceberg table from above.

CREATE COMPUTE_POOL mypool 
WITH ( 'compute_pool.size' = 'small', 'compute_pool.timeout_min' = 3600');

The above statement creates and starts the compute_pool. If this is the first compute_pool in the organization, DeltaStream sets it as your default pool.

1. Run a batch query.

SELECT * FROM pv_table limit 10;

Inspect the Iceberg Data Store

1. In the lefthand navigation, click Resources ( ). This displays a list of the existing data stores.

1. Click the Iceberg Glue data store. The store page opens, displaying a list of any existing databases in your account.

2. (Optional) Create a new database. To do this:

• Click + Add Database. When prompted, enter a name for the new database and click Add. The new database displays in the list.

1. To view the tables that exist under a particular database, click the database name.

Clean up resources

STOP COMPUTE_POOL mypool;
TERMINATE QUERY <QUERY-ID>;