Creating a Private Link from DeltaStream to Amazon MSK

Below is the procedure for creating private links for the data you’re streaming via DeltaStream. It includes the configuration needed within AWS MSK.

Note Currently DeltaStream supports private links only in AWS.

Before You Begin

• Review Introducing Private Links.

• You must have signed up with DeltaStream and created at least one organization. Private links function within the context of an organization; from a logical perspective, you enable your private links within a specific DeltaStream organization.

• Download the DeltaStream CLI if you don’t already use it. Currently you cannot create private links via the DeltaStream UI.

• Request DeltaStream Ops to enable private link connectivity feature for your account.

• Optionally, work with DeltaStream Operations if you wish to run all your queries (that is, stream processing) within a dedicated AWS dataplane. By default all DeltaStream customer queries run in a shared multi-tenant dataplane; network policies isolate all traffic among multiple customers. This dedicated data plane completely isolates your workload from other DeltaStream customers by using fully-isolated compute and VPC networking resources.

Creating a Private Link

This procedure involves building a SQL statement. When you run the statement, DeltaStream processes the link request automatically. Note that the private link is not established until it is accepted or approved by administrators from your organization who are responsible for maintaining Kafka stores.

The entire statement resembles the below; we’ll go through it segment by segment, showing how and where to find the information you need to complete the statement.

CREATE AWS PRIVATE LINK msktest 
             WITH ( 'access_region' = "AWS us-east-1",
             'private_link.target_type' = MSK_PRIVATE_LINK,
             'private_link.msk_cluster_arn' = ‘arn:aws:kafka:us-east-1:033434007612:cluster/ds-cc118-stage-he17-0/349e3343-1547-458a-86ab-7d4b424288f9-8‘,
             'private_link.msk_auth_type' = MSK AUTH_SASL_IAM, 
             'private_link.hosts' (
              ‘b-1.iam.dscc118stagehe170.f4tzmf.c8.kafka.us-east-1.amazonaws.com:14001‘ USING PORT 14001 IN ‘use1-az6‘, ) );

The private link created using the above SQL command is an inbound private link connection to your dedicated Kafka cluster within your Confluent cloud account.

Note There are separate but related instructions for creating private links for Confluent Kafka, Postgres RDS, and Confluent Enterprise edition.

Section 1: Enable Multi-VPC Configuration (private link) for AWS MSK

Follow the below guide to enable multi-VPC for your AWS cluster.

More details from AWS on MSK private link setup.

When you have enabled the private link for the MSK cluster, follow the below procedure to capture the details you need to configure the private link to MSK in the DeltaStream CLI.

1. Go to your Amazon MSK console page and navigate to Clusters > $$yourclustername to open the cluster details page.

2. In the Cluster summary section, locate and copy the MSK Cluster ARN.

3. Save the Cluster ARN to a Notepad or other text-based application. You will need to refer back to it again.

4. Note down the cluster ARN as $$yourclusterarn

1. Indicate the MSK auth type you must use for the connection. This varies, depending on your AWS MSK cluster setup and configuration; it could be SASL/SCRAM authentication or IAM role-based authentication. For the below example we use IAM-based authentication.

2. Capture the broker host URIs. To do this, return to the Amazon MSK Cluster details page, and toward the top right of the page click View client information.

3. In the Private endpoint (multi-VPC) column, multiple broker IAM endpoints display. Capture the bootstrap URIs next to the private endpoint column.

   • You must copy each of these endpoints into the SQL command you’re building in the DeltaStream UI.

4. Capture the availability zone associated with each broker in the MSK cluster. To do this, return to your Cluster details page. Below the Cluster summary section click Properties and then scroll down to the Broker details section.

5. Although there is an Availability Zone tab in this section, that is the virtual zone name, and it is not what you need. Instead, you must locate and copy the physical zone ID. To get the physical zone ID, click the corresponding link in the Client subnets column. (Each broker endpoint has an associated subnet.)

6. When the Client subnet details page displays, locate and copy the Availability Zone ID value for each broker subnet.

Section 2: Writing the SQL Statement in DeltaStream to Create the Private Link

1. Launch the DeltaStream CLI. The prompt opens to your default organization.

   • If you’re uncertain which organization this is, type list ORGANIZATIONS, and in the list that displays, scan down the is Current column and find the organization with a value of true.

2. Return to the prompt and specify the organization: Use ORGANIZATION [$$yourorganization]

The private link connection you’re creating will be in this specific DeltaStream organization. No other DeltaStream organization has access to this private link.

1. Create the private link. To do this, update the SQL with the variables captured in Section 1 above.

CREATE AWS PRIVATE LINK msktest
             WITH ( 'access_region' = "AWS us-east-1",
                         'private_link.target_type' = MSK_PRIVATE_LINK,
                         'private_link.msk.cluster.arn' = ‘$$Yourclusterarn’,
                         'private_link.msk_auth-type' = MSK_AUTH_SASL_IAM,
                         'private_link.hosts' (
‘$$brokerhost1:$$brokerhost1PORT’ USING PORT $$brokerhost1port IN ‘brokerhost1az’,
‘$$brokerhost2:$$brokerhost2PORT’ USING PORT $$brokerhost2port IN ‘brokerhost2az’,
‘$$brokerhost3:$$brokerhost3PORT’ USING PORT $$brokerhost3port IN ‘brokerhost3az’ ) ) ;

Your SQL statement in the DeltaStream CLI is now ready. When you run the statement, DeltaStream displays a confirmation and begins creating the private link.

Section 3 – Establishing Private Link Connectivity Between the DeltaStream Platform and the MSK Cluster

To complete the private link connection you must edit your MSK cluster policy to add the Discovery IAM Role ARN that the DeltaStream platform uses. To do this:

1. In the DeltaStream CLI, list your available private links by typing LIST AWS PRIVATE LINKS;

2. Locate your newly-created msktest private link entry, and copy the Discovery IAM role ARN.

3. Navigate back to your MSK Cluster details page and under the Properties tab scroll down to the Security settings section and click Edit cluster policy. The Edit cluster policy page displays.

4. Replace the code following “AWS”: with the ARN you just copied.

5. Click Save changes.

1. Return to the DeltaStream CLI and again list the status of the private link by typing list AWS private links. In a few moments you should see the private link transition from in progress to ready.

The cluster is ready to use as a Kafka store from within the DeltaStream platform.

Verifying Private Link Connectivity

To verify connectivity you can use the DeltaStream console UI to add a Kafka topic.

1. Launch the DeltaStream UI and create a new data store. Follow the instructions, if you need. Within about a minute, topics begin to populate your data store. This indicates the private connection is successful and working.

To do this you must set the number of partitions to 1 and the number of replicas to 3.

From DeltaStream you should get an alert that the operation was successful. You should also be able to see the newly-created topic in DeltaStream.