Create a Compute Pool to Work with Iceberg
A compute pool is a set of dedicated resources to run batch queries. Compute pools are Apache Spark clusters you use to perform real-time analytics on data you read from Iceberg tables. They are similar to Databricks’ all-purpose compute.
However, you never have to manage or interact with Spark directly. DeltaStream auto-configures and instantiates the pool based on a pool size you select.
Creating a Compute Pool
You create a compute pool much like you’d create any new DeltaStream object. Define it at the organization level and follow the same access control rules. Specific to compute pools, however, you also must select a pool size – S, M, or L.
Working with Compute Pools via the DeltaStream UI
To create a compute pool
In the lefthand navigation, click Resources (
) to display the Resources page.Click to activate the Compute Pools tab. Then click + Add Compute Pool.
When the Add Compute Pool window displays, enter the required information.
To specify a pool size, click the Pool Size down arrow and select from Small, Medium, or Large.
Compute pools by default shut off automatically after 60 minutes of inactivity. To change this value, click to activate the Enable Auto-Stop slider, then use the arrows to specify a different auto-shutoff length.
Click Add. The Resources page redisplays, with your new pool added to the list of pools under the Compute Pools tab.
After you create a compute pool, you can perform multiple actions:
Edit its pool size and auto-shutoff interval
Stop it
Delete it
View a full history of its activity.
You can access these actions in either of 2 ways:
Right-click on the compute pool you want and select from the context menu.
On the right, under the Action column, click the desired icon.
We'll use the right-click context menu to navigate for the next few procedures, but you can instead click an icon, if you prefer.
To edit a compute pool size and auto-shutoff interval
Right-click the compute pool, and from the menu that displays click Edit
<computepool name>Make the changes you wish, and then click Update.
To stop or start a compute pool
Right-click the compute pool, and from the menu that displays click Stop.
The compute pool redisplays with a Status of Stopped. Repeat this step to re-start the compute pool.
To delete a compute pool
Right-click the compute pool, and from the menu that displays click Delete.
When the Delete Compute Pool box displays, follow the instructions and then click Delete.
To view compute pool activity
Right-click the compute pool, and from the menu that displays click View Details. The Details pane displays on the right, with the Overview tab open. This tab only displays basic information (owner, status, and created/modified dates).displays, which
To view the complete history of this compute pool (created time, started/stopped time, ownership, and so on), click the Details tab.
Important Creating a compute pool essentially is a prerequisite for reading from/querying Iceberg tables. Please see Iceberg REST Catalog or Iceberg AWS Glue Catalog for detailed instructions on setting up Iceberg tables to work with DeltaStream.
Working with Compute Pools via the DeltaStream CLI
Follow the links below for instructions on using the DeltaStream SQL DDL to create and manage compute pools:
Last updated