Join Kafka consumer purposes securely to your Amazon MSK cluster from completely different VPCs and AWS accounts

Now you can use Amazon Managed Streaming for Apache Kafka (Amazon MSK) multi-VPC personal connectivity (powered by AWS PrivateLink) and cluster coverage help for MSK clusters to simplify connectivity of your Kafka purchasers to your brokers. Amazon MSK is a completely managed service that makes it simple so that you can construct and run purposes that use Kafka to course of streaming information. While you create an MSK cluster, the cluster sources can be found to purchasers inside the identical Amazon VPC. This lets you launch the cluster inside particular subnets of the VPC, affiliate it with safety teams, and connect IP addresses out of your VPC’s handle area by means of elastic community interfaces (ENIs). Community site visitors between purchasers and the cluster stays inside the AWS community, with web entry to the cluster not potential by default.

If in case you have workloads segmented throughout a number of VPCs and AWS accounts, there could also be situations through which it’s good to make your MSK brokers accessible to Kafka purchasers throughout VPCs. With the launch of Amazon MSK multi-VPC personal connectivity, now you can privately entry your MSK brokers out of your consumer purposes in one other VPC inside the identical AWS account or one other AWS account with out enabling public entry or creating and managing your individual networking infrastructure for personal connectivity. A cluster coverage is an AWS Id and Entry Administration (IAM) resource-based coverage, which is outlined in your MSK cluster to offer cross-account IAM principals permissions to arrange personal connectivity to the cluster.

This publish introduces Amazon MSK multi-VPC connectivity and how one can privately entry your MSK clusters out of your purchasers in different VPCs. It additionally reveals methods to outline a cluster coverage in your MSK clusters. These new two capabilities simplify configuring cross-VPC community entry and establishing permissions wanted for Kafka purchasers to privately hook up with MSK brokers in a special account.

Earlier than Amazon MSK multi-VPC connectivity

Earlier than Amazon MSK multi-VPC connectivity, the community admin wanted to decide on one of many following safe connectivity patterns. Admins needed to repeat sure steps for every dealer within the cluster.

• Amazon VPC peering is the only networking assemble that allows bidirectional connectivity between two VPCs. On this method, the community admin needed to replace every VPC with the IP addresses of every dealer within the routing tables of all subnets. You may’t use this connectivity sample when there are overlapping IPv4 or IPv6 CIDR blocks within the VPCs.
• AWS Transit Gateway gives a extremely accessible and scalable design for connecting VPCs. On this method, the community admin continuously needed to replace the routing tables connected to every transit gateway. Not like VPC peering that may go cross-Area, AWS Transit Gateway is a regional service, however you need to use inter-Area peering between transit gateways to route site visitors throughout areas. AWS Transit Gateway has the utmost bandwidth (burst) per Availability Zone per VPC connection (50 Gbps). This might grow to be a problem for some workloads.
• AWS PrivateLink is an AWS networking service that gives personal entry to a particular service as a substitute of all sources inside a VPC and with out traversing the general public web. It additionally eliminates the necessity to expose your complete VPC or subnet, and prevents points like having to take care of overlapping CIDR blocks between the VPC that hosts the MSK cluster ENIs and the Kafka consumer VPC. AWS PrivateLink can scale to an infinite variety of VPCs and in contrast to the opposite choices, site visitors right here is unidirectional. Due to these advantages, AWS PrivateLink is a well-liked option to handle personal connectivity. Nevertheless, this connectivity sample comes with extra complexity. It requires creating a number of Community Load Balancers (NLBs) per cluster and creating personal service endpoints per NLB within the service account. Moreover, admins needed to create personal endpoints per personal service endpoint, and an Amazon Route 53 alias report per personal endpoint in each consumer account.

The next diagram illustrates the structure of customer-managed VPC endpoints between completely different VPCs in several AWS accounts with IAM authentication.

After Amazon MSK multi-VPC connectivity and cluster coverage

Now you can allow multi-VPC and cross-account connectivity in your MSK clusters in a number of easy steps and pay for what you employ. This eliminates the overhead of making and managing AWS PrivateLink infrastructure. When new brokers are added to a cluster, personal connectivity is maintained with out the necessity to make configuration modifications, saving you from the overhead and complexity of managing the underlying community infrastructure.

The next diagram illustrates this up to date structure of utilizing Amazon MSK multi-VPC connectivity to attach a consumer from a special AWS account.

Answer overview

Establishing multi-VPC personal connectivity includes turning on this characteristic for the cluster and configuring the Kafka purchasers to attach privately to the cluster.

The next are the high-level steps to configure the cluster:

1. Allow the multi-VPC personal connectivity characteristic for a subset of authentication schemes which are enabled in your MSK cluster.
2. If a Kafka consumer is in an AWS account that’s completely different than the cluster, connect a resource-based coverage to the MSK cluster to authorize IAM principals for creating cross-account connectivity.
3. Share the cluster ARN with the IAM principal related to the Kafka consumer that should create the cross-account entry to MSK cluster.

The next are the high-level steps to configure the purchasers:

1. Create a managed VPC endpoint for the consumer VPC that should join privately to the MSK cluster.
2. Replace the VPC endpoint’s safety group settings to allow outbound connectivity to the MSK cluster.
3. Arrange the consumer to make use of the cluster’s connection string to attach privately to the cluster.

Cluster setup

On this publish, we solely present the steps for enabling Amazon MSK multi-VPC connectivity for a provisioned cluster.

1. To allow Amazon MSK multi-VPC connectivity in your present cluster, select Activate multi-VPC connectivity on the Amazon MSK console.
   
   Observe that multi-VPC connectivity can’t be turned on with a cluster that enables unauthenticated entry. That is to forestall unauthenticated entry from completely different VPCs.
2. Choose the authentication strategies that you just enable purchasers in different VPCs to make use of.
   The record of authentication strategies is populated based mostly in your cluster’s safety configuration.
3. Overview the settings and select Activate choice. After the multi-VPC connectivity is enabled in your cluster, Amazon MSK will create the NLB and VPC endpoint service infrastructure required for personal connectivity. Amazon MSK will vend a brand new set of bootstrap dealer strings that can be utilized for personal connectivity. These may be accessed utilizing the View consumer information choice on the Amazon MSK console. The following step is to offer the IAM principals related along with your purchasers the permissions to attach privately to your cluster. To do that, it’s good to connect a cluster coverage to the cluster.
4. Select Edit cluster coverage within the Safety part of the cluster particulars web page on the Amazon MSK console.
   The brand new cluster coverage permits for outlining a Fundamental or Superior cluster coverage. With the Fundamental choice, you possibly can merely enter AWS account IDs of your consumer’s VPCs. This coverage permits all allowed principals in these AWS accounts to carry out CreateVPCConnection, GetBootstrapBrokers, DescribeCluster, and DescribeClusterV2 actions which are required for creating the cross-VPC connectivity to your cluster. Nevertheless, in different circumstances, it’s possible you’ll want a extra advanced coverage that enables for extra actions, or principals apart from AWS accounts, reminiscent of IAM roles, function classes, IAM customers, and extra. You may writer a cluster coverage in line with IAM JSON coverage steering and supply that to the cluster in Superior mode.
5. Outline your cluster coverage and select Save modifications.

Shopper setup

On the consumer aspect, first it’s good to connect an id coverage to the IAM principal who desires to create a managed VPC connection. The id coverage should present permission for making a managed VPC connection. The mandatory permissions are a part of the AWS managed coverage AmazonMSKFullAccess.

1. Within the different AWS account with the IAM principal you configured, use the brand new Managed VPC connection web page on the Amazon MSK console to create Amazon MSK managed VPC connections.
   A managed VPC connection maps to an AWS PrivateLink endpoint underneath the hood, and Amazon MSK makes use of the managed VPC connection to orchestrate personal connectivity to the cluster. You merely have to create the managed VPC connection and pay customary AWS PrivateLink fees for the underlying endpoint.
2. Enter the AWS Useful resource Title (ARN) of the cluster that you just wish to hook up with.
3. Select Confirm to confirm the cluster info and its minimal necessities for cross-connectivity.
4. Choose an authentication technique from the supplied values.
5. Select the VPC ID the place your Kafka purchasers are situated, and select their subnet IDs. You may add extra subnets utilizing the Add subnet choice.
   The desired consumer subnet should have Availability Zone IDs that match the cluster’s Availability Zone IDs. This makes positive the purchasers are situated in a identical bodily Availability Zone because the cluster brokers. Amazon MSK makes use of the port vary 14001:14100 for all authentication strategies. It is advisable to choose a safety group that enables outbound site visitors to this port. The next screenshot reveals an instance.
6. Overview the settings and select Create connection.
   The method will take a couple of minutes.
7. When it’s full, you possibly can acquire the purchasers’ connection string from the small print web page of your connection.
8. The following step is to replace the outbound guidelines for the VPC endpoint safety group to permit communication to the port vary 14001:14100.

Use the Amazon MSK-managed VPC connection

After you create the managed VPC connection, connecting privately to the cluster is simple. Merely use the brand new connection string to hook up with the cluster. For instance, it’s possible you’ll join from an Amazon Elastic Compute Cloud (Amazon EC2) occasion in your consumer VPC. Then run the next command to confirm should you can join and carry out actions towards the subjects within the MSK cluster:

export MSK_VPC=<YOUR CLIENT CONNECTION STRING GOES HERE>
bin/kafka-topics.sh --bootstrap-server $MSK_VPC -command-config /dwelling/ec2-user/kafka/config/client-config.properties –record

IAM authentication

Earlier than the launch of Amazon MSK multi-VPC connectivity, Kafka purchasers in different AWS accounts who opted in IAM authentication, wanted to imagine one other IAM function within the cluster’s account. To facilitate this, admins needed to create a number of IAM roles and write a belief coverage that enables authenticated principals from the consumer’s accounts to imagine corresponding roles by means of the sts:AssumeRole API name. This method was difficult to scale when the variety of VPCs or AWS accounts grew. With the launch of this cluster coverage, cross-account entry management is now simplified as a result of you possibly can connect a cluster coverage to your clusters to specify which cross-account purchasers have what permissions on sources inside the cluster.

This functionality means that you can handle all entry to the cluster and subjects in a single place. For instance, you possibly can management which IAM principals have write entry to sure subjects, and which principals can solely learn from them. Customers who’re utilizing IAM consumer authentication can even add permissions for required kafka-cluster actions within the cluster useful resource coverage.

Availability and pricing

Now you can use Amazon MSK multi-VPC connectivity in all business Areas the place Amazon MSK is obtainable, together with China and GovCloud (US) Areas.

You pay $0.006 per GB information processed for personal connectivity and $0.0225 per personal connectivity hour per authentication scheme in US East (Ohio). Confer with our Pricing web page for extra particulars.

Conclusion

With Amazon MSK multi-VPC personal connectivity, now you can privately entry your MSK brokers out of your consumer purposes in one other VPC inside the identical AWS account or one other AWS account, with minimal configuration. You not should create, handle, and replace a number of networking sources in a number of VPCs, or make Amazon MSK configuration modifications to attach your Kafka purchasers throughout VPCs and accounts. Amazon MSK creates and manages the sources for you. With Cluster coverage help, you possibly can simply present your cross-account consumer principals permissions to attach privately to your MSK cluster. Additional, in case you are utilizing IAM consumer authentication, you may as well leverage the cluster coverage to centrally management purchasers’ permissions to carry out operations on the cluster. Use the Amazon MSK multi-VPC connectivity and the cluster coverage characteristic right now to simplify your safe connectivity infrastructure.

For additional studying on Amazon MSK, go to the official product web page and our AWS Documentation.

In regards to the authors

Ali Alemi is a Streaming Specialist Options Architect at AWS. Ali advises AWS prospects with architectural finest practices and helps them design real-time analytics information techniques which are dependable, safe, environment friendly, and cost-effective. He works backward from prospects’ use circumstances and designs information options to resolve their enterprise issues. Previous to becoming a member of AWS, Ali supported a number of public sector prospects and AWS consulting companions of their utility modernization journey and migration to the cloud.