Resolve non-public DNS hostnames for Amazon MSK Join

Amazon MSK Join is a function of Amazon Managed Streaming for Apache Kafka (Amazon MSK) that provides a completely managed Apache Kafka Join setting on AWS. With MSK Join, you’ll be able to deploy totally managed connectors constructed for Kafka Join that transfer information into or pull information from fashionable information shops like Amazon S3 and Amazon OpenSearch Service. With the introduction of the Personal DNS help into MSK Join, connectors are capable of resolve non-public buyer domains, utilizing their DNS servers configured within the buyer VPC DHCP Choices set. This publish demonstrates an answer for resolving non-public DNS hostnames outlined in a buyer VPC for MSK Join.

You could wish to use non-public DNS hostname help for MSK Join for a number of causes. Earlier than the non-public DNS decision functionality included with MSK Join, it used the service VPC DNS resolver for DNS decision. MSK Join didn’t use the non-public DNS servers outlined within the buyer VPC DHCP possibility units for DNS decision. The connectors have been solely capable of reference hostnames within the connector configuration or plugin which might be publicly resolvable and couldn’t resolve non-public hostnames outlined in both a personal hosted zone or use DNS servers in one other buyer community.

Many purchasers be sure that their inside DNS functions are usually not publicly resolvable. For instance, you may need a MySQL or PostgreSQL database and will not need the DNS identify on your database to be publicly resolvable or accessible. Amazon Relational Database Service (Amazon RDS) or Amazon Aurora servers have DNS names which might be publicly resolvable however not accessible. You possibly can have a number of inside functions resembling databases, information warehouses, or different programs the place DNS names are usually not publicly resolvable.

With the latest launch of MSK Join non-public DNS help, you’ll be able to configure connectors to reference public or non-public domains. Connectors use the DNS servers configured in your VPC’s DHCP possibility set to resolve domains. Now you can use MSK Connect with privately join with databases, information warehouses, and different sources in your VPC to conform along with your safety wants.

When you have a MySQL or PostgreSQL database with non-public DNS, you’ll be able to configure it on a customized DNS server and configure the VPC-specific DHCP possibility set to do the DNS decision utilizing the customized DNS server native to the VPC as a substitute of utilizing the service DNS decision.

Answer overview

A buyer can have completely different structure choices to arrange their MSK Join. For instance, they’ll have Amazon MSK and MSK Join are in the identical VPC or supply system in VPC1 and Amazon MSK and MSK Join are in VPC2 or supply system, Amazon MSK and MSK Join are all in several VPCs.

The next setup makes use of two completely different VPCs, the place the MySQL VPC hosts the MySQL database and the MSK VPC hosts Amazon MSK, MSK Join, the DNS server, and varied different parts. You possibly can prolong this structure to help different deployment topologies utilizing acceptable AWS Id and Entry Administration (IAM) permissions and connectivity choices.

This publish supplies step-by-step directions to arrange MSK Join the place it would obtain information from a supply MySQL database with non-public DNS hostname within the MySQL VPC and ship information to Amazon MSK utilizing MSK Join in one other VPC. The next diagram illustrates the high-level structure.

The setup directions embody the next key steps:

1. Arrange the VPCs, subnets, and different core infrastructure parts.
2. Set up and configure the DNS server.
3. Add the info to the MySQL database.
4. Deploy Amazon MSK and MSK Join and devour the change information seize (CDC) data.

Stipulations

To observe the tutorial on this publish, you want the next:

Create the required infrastructure utilizing AWS CloudFormation

Earlier than configuring the MSK Join, we have to arrange the VPCs, subnets, and different core infrastructure parts. To arrange sources in your AWS account, full the next steps:

1. Select Launch Stack to launch the stack in a Area that helps Amazon MSK and MSK Join.
   
2. Specify the non-public key that you simply use to connect with the EC2 cases.
3. Replace the SSH location along with your native IP handle and preserve the opposite values as default.
4. Select Subsequent.
5. Assessment the main points on the ultimate web page and choose I acknowledge that AWS CloudFormation may create IAM sources.
6. Select Create stack and anticipate the required sources to get created.

The CloudFormation template creates the next key sources in your account:

• VPCs:
• Subnets within the MSK VPC:
  • Three non-public subnets for Amazon MSK
  • Personal subnet for DNS server
  • Personal subnet for MSKClient
  • Public subnet for bastion host
• Subnets within the MySQL VPC:
  • Personal subnet for MySQL database
  • Public subnet for bastion host
• Web gateway hooked up to the MySQL VPC and MSK VPC
• NAT gateways hooked up to MySQL public subnet and MSK public subnet
• Route tables to help the site visitors stream between completely different subnets in a VPC and throughout VPCs
• Peering connection between the MySQL VPC and MSK VPC
• MySQL database and configurations
• DNS server
• MSK consumer with respective libraries

Please notice, when you’re utilizing VPC peering or AWS Transit Gateway with MSK Join, don’t configure your connector for reaching the peered VPC sources with IPs within the CIDR ranges. For extra info, consult with Connecting from connectors.

Configure the DNS server

Full the next steps to configure the DNS server:

1. Connect with the DNS server. There are three configuration recordsdata obtainable on the DNS server below the /house/ec2-user folder:
   • named.conf
   • mysql.inside.zone
   • kafka.us-east-1.amazonaws.com.zone
2. Run the next instructions to put in and configure your DNS server:

   sudo yum set up bind bind-utils –y
   cp /house/ec2-user/named.conf /and so on/named.conf
   chmod 644 /and so on/named.conf
   cp mysql.inside.zone /var/named/mysql.inside.zone
   cp kafka.area.amazonaws.com.zone /var/named/kafka.area.amazonaws.com.zone
   

3. Replace /and so on/named.conf.

For the allow-transfer attribute, replace the DNS server inside IP handle to allow-transfer

{ localhost; <DNS Server inside IP handle>; };.

You will discover the DNS server IP handle on the CloudFormation template Outputs tab.

Observe that the MSK cluster remains to be not arrange at this stage. We have to replace the Kafka dealer DNS names and their respective inside IP addresses within the /var/named/kafka.area.amazonaws.com configuration file after organising the MSK cluster later on this publish. For directions, consult with right here.

Additionally notice that these settings configure the DNS server for this publish. In your personal setting, you’ll be able to configure the DNS server as per your wants.

4. Restart the DNS service:

   sudo su
   service named restart
   

It is best to see the next message:

Add the info to the MySQL database

Sometimes, we are able to use an Amazon RDS for MySQL database, however for this publish, we use customized MySQL database servers. The Amazon RDS DNS is publicly accessible and MSK Join helps it, nevertheless it was not capable of help databases or functions with non-public DNS up to now. With the most recent non-public DNS hostnames function launch, it may possibly help functions’ non-public DNS as properly, so we use a MySQL database on the EC2 occasion.

This set up supplies details about organising the MySQL database on a single-node EC2 occasion. This shouldn’t be used on your manufacturing setup. It is best to observe acceptable steerage for organising and configuring MySQL in your account.

The MySQL database is already arrange utilizing the CloudFormation template and is able to use now. To add the info, full the followings steps:

1. SSH to the MySQL EC2 occasion. For directions, consult with Connect with your Linux occasion. The info file salesdb.sql is already downloaded and obtainable below the /house/ec2-user listing.
2. Log in to mysqldb with the person identify grasp.
3. To entry the password, navigate to AWS Programs Supervisor and Parameter Retailer tab. Choose /Database/Credentials/grasp and click on on View Particulars and duplicate the worth for the important thing.
4. Log in to MySQL utilizing the next command:

   mysql -umaster -p<MySQLMasterUserPassword>
   

5. Run the next instructions to create the salesdb database and cargo the info to the desk:

   use salesdb;
   supply /house/ec2-user/salesdb.sql;

This may insert the data in varied completely different tables within the salesdb database.

6. Run present tables to see the next tables within the salesdb:

   mysql> present tables;
   +-----------------------+
   | Tables_in_salesdb |
   +-----------------------+
   | CUSTOMER |
   | CUSTOMER_SITE |
   | PRODUCT |
   | PRODUCT_CATEGORY |
   | SALES_ORDER |
   | SALES_ORDER_ALL |
   | SALES_ORDER_DETAIL |
   | SALES_ORDER_DETAIL_DS |
   | SALES_ORDER_V |
   | SUPPLIER |
   +-----------------------+

Create a DHCP possibility set

DHCP possibility units offer you management over the next features of routing in your digital community:

• You possibly can management the DNS servers, domains, or Community Time Protocol (NTP) servers utilized by the units in your VPC.
• You possibly can disable DNS decision utterly in your VPC.

To help non-public DNS, you should use an Amazon Route 53 non-public zone or your personal customized DNS server. Should you use a Route 53 non-public zone, the setup will work mechanically and there’s no have to make any adjustments to the default DHCP possibility set for the MSK VPC. For a customized DNS server, full the next steps to arrange a customized DHCP configuration utilizing Amazon Digital Personal Cloud (Amazon VPC) and fasten it to the MSK VPC.

There can be a default DHCP possibility set in your VPC hooked up to the Amazon supplied DNS server. At this stage, the requests will go to Amazon’s supplied DNS server for decision. Nonetheless, we create a brand new DHCP possibility set as a result of we’re utilizing a customized DNS server.

1. On the Amazon VPC console, select DHCP possibility set within the navigation pane.
2. Select Create DHCP possibility set.
   
3. For DHCP possibility set identify, enter MSKConnect_Private_DHCP_OptionSet.
4. For Area identify, enter mysql.inside.
5. For Area identify server, enter the DNS server IP handle.
6. Select Create DHCP possibility set.
7. Navigate to the MSK VPC and on the Actions menu, select Edit VPC settings.
8. Choose the newly created DHCP possibility set and put it aside.
   The next screenshot reveals the instance configurations.
   
9. On the Amazon EC2 console, navigate to privateDNS_bastion_host.
10. Select Occasion state and Reboot occasion.
11. Wait a couple of minutes after which run nslookup from the bastion host; it ought to be capable to resolve it utilizing your native DNS server as a substitute of Route 53:

nslookup native.mysql.inside

Now our base infrastructure setup is able to transfer to the subsequent stage. As a part of our base infrastructure, we now have arrange the next key parts efficiently:

• MSK and MySQL VPCs
• Subnets
• EC2 cases
• VPC peering
• Route tables
• NAT gateways and web gateways
• DNS server and configuration
• Acceptable safety teams and NACLs
• MySQL database with the required information

At this stage, the MySQL DB DNS identify is resolvable utilizing a customized DNS server as a substitute of Route 53.

Arrange the MSK cluster and MSK Join

The subsequent step is to deploy the MSK cluster and MSK Join, which is able to fetch data from the salesdb and ship it to an Amazon Easy Storage Service (Amazon S3) bucket. On this part, we offer a walkthrough of replicating the MySQL database (salesdb) to Amazon MSK utilizing Debezium, an open-source connector. The connector will monitor for any adjustments to the database and seize any adjustments to the tables.

With MSK Join, you’ll be able to run totally managed Apache Kafka Join workloads on AWS. MSK Join provisions the required sources and units up the cluster. It constantly displays the well being and supply state of connectors, patches and manages the underlying {hardware}, and auto scales connectors to match adjustments in throughput. Because of this, you’ll be able to focus your sources on constructing functions reasonably than managing infrastructure.

MSK Join will make use of the customized DNS server within the VPC and it received’t be depending on Route 53.

Create an MSK cluster configuration

Full the next steps to create an MSK cluster:

1. On the Amazon MSK console, select Cluster configurations below MSK clusters within the navigation pane.
2. Select Create configuration.
3. Identify the configuration mskc-tutorial-cluster-configuration.
4. Beneath Configuration properties, take away every thing and add the road auto.create.matters.allow=true.
   
5. Select Create.

Create an MSK cluster and fasten the configuration

Within the subsequent step, we connect this configuration to a cluster. Full the next steps:

1. On the Amazon MSK console, select Clusters below MSK clusters within the navigation pane.
2. Select Create clusters and Customized create.
3. For the cluster identify, enter mkc-tutorial-cluster.
4. Beneath Common cluster properties, select Provisioned for the cluster kind and use the Apache Kafka default model 2.8.1.
5. Use all of the default choices for the Brokers and Storage sections.
6. Beneath Configurations, select Customized configuration.
7. Choose mskc-tutorial-cluster-configuration with the suitable revision and select Subsequent.
8. Beneath Networking, select the MSK VPC.
9. Choose the Availability Zones relying upon your Area, resembling us-east1a, us-east1b, and us-east1c, and the respective non-public subnets MSK-Personal-1, MSK-Personal-2, and MSK-Personal-3 if you’re within the us-east-1 Area. Public entry to those brokers ought to be off.
10. Copy the safety group ID from Chosen safety teams.
11. Select Subsequent.
    
12. Beneath Entry management strategies, choose IAM role-based authentication.
13. Within the Encryption part, below Between shoppers and brokers, TLS encryption can be chosen by default.
    
14. For Encrypt information at relaxation, choose Use AWS managed key.
15. Use the default choices for Monitoring and choose Fundamental monitoring.
16. Choose Ship to Amazon CloudWatch Logs.
    
17. Beneath Log group, select go to Amazon CloudWatch Logs console.
18. Select Create log group.
19. Enter a log group identify and select Create.
    
20. Return to the Monitoring and tags web page and below Log teams, select Select log group
21. Select Subsequent.
22. Assessment the configurations and select Create cluster. You’re redirected to the main points web page of the cluster.
23. Beneath Safety teams utilized, notice the safety group ID to make use of in a later step.

Cluster creation can sometimes take 25–half-hour. Its standing adjustments to Energetic when it’s created efficiently.

Replace the /var/named/kafka.area.amazonaws.com zone file

Earlier than you create the MSK connector, replace the DNS server configurations with the MSK cluster particulars.

1. To get the record of bootstrap server DNS and respective IP addresses, navigate to the cluster and select View consumer info.
2. Copy the bootstrap server info with IAM authentication kind.
3. You possibly can determine the dealer IP addresses utilizing nslookup out of your native machine and it’ll present you the dealer native IP handle. Presently, your VPC factors to the most recent DHCP possibility set and your DNS server will be unable to resolve these DNS names out of your VPC.

   nslookup <dealer 1 DNS identify>

Now you’ll be able to log in to the DNS server and replace the data for various brokers and respective IP addresses within the /var/named/kafka.area.amazonaws.com file.

5. Add the msk-access.pem file to BastionHostInstance out of your native machine:

   scp -i "< your pem file>" Your pem file ec2-user@<BastionHostInstance IP handle>:/house/ec2-user/

6. Log in to the DNS server and open the /var/named/kafka.area.amazonaws.com file and replace the next strains with the right MSK dealer DNS names and respective IP addresses:

   <b-1.<clustername>.******.c6> IN A <Inside IP Deal with - dealer 1>
   <b-2.<clustername>.******.c6> IN A <Inside IP Deal with - dealer 2>
   <b-3.<clustername>.******.c6> IN A <Inside IP Deal with - dealer 3>
   

Observe that you have to present the dealer DNS as talked about earlier. Take away .kafka.<area id>.amazonaws.com from the dealer DNS identify.

7. Restart the DNS service:

   sudo su
   service named restart

It is best to see the next message:

Redirecting to /bin/systemctl restart named.service

Your customized DNS server is up and operating now and it’s best to be capable to resolve utilizing dealer DNS names utilizing the inner DNS server.

Replace the safety group for connectivity between the MySQL database and MSK Join

It’s vital to have the suitable connectivity in place between MSK Join and the MySQL database. Full the next steps:

1. On the Amazon MSK console, navigate to the MSK cluster and below Community settings, copy the safety group.
2. On the Amazon EC2 console, select Safety teams within the navigation pane.
3. Edit the safety group MySQL_SG and select Add rule.
4. Add a rule with MySQL/Aurora as the sort and the MSK safety group because the inbound useful resource for its supply.
5. Select Save guidelines.

Create the MSK connector

To create your MSK connector, full the next steps:

1. On the Amazon MSK console, select Connectors below MSK Join within the navigation pane.
2. Select Create connector.
3. Choose Create customized plugin.
4. Obtain the MySQL connector plugin for the most recent secure launch from the Debezium web site or obtain Debezium.zip.
5. Add the MySQL connector zip file to the S3 bucket.
6. Copy the URL for the file, resembling s3://<bucket identify>/Debezium.zip.
7. Return to the Select customized plugin web page and enter the S3 file path for S3 URI.
8. For Customized plugin identify, enter mysql-plugin.
   
9. Select Subsequent.
10. For Identify, enter mysql-connector.
11. For Description, enter an outline of the connector.
12. For Cluster kind, select MSK Cluster.
13. Choose the prevailing cluster from the record (for this publish, mkc-tutorial-cluster).
14. Specify the authentication kind as IAM.
15. Use the next values for Connector configuration:

    connector.class=io.debezium.connector.mysql.MySqlConnector
    database.historical past.producer.sasl.mechanism=AWS_MSK_IAM
    database.historical past.producer.sasl.jaas.config=software program.amazon.msk.auth.iam.IAMLoginModule required;
    database.allowPublicKeyRetrieval=true
    database.person=grasp
    database.server.id=123456
    duties.max=1
    database.historical past.client.sasl.jaas.config=software program.amazon.msk.auth.iam.IAMLoginModule required;
    database.historical past.producer.safety.protocol=SASL_SSL
    database.historical past.kafka.matter=dbhistory.salesdb
    database.historical past.kafka.bootstrap.servers=b-3.xxxxx.yyyy.zz.kafka.us-east-2.amazonaws.com:9098,b-1.xxxxx.yyyy.zz.kafka.us-east-2.amazonaws.com:9098,b-2. xxxxx.yyyy.zz.kafka.us-east-2.amazonaws.com:9098
    database.server.identify=salesdb-server
    database.historical past.producer.sasl.consumer.callback.handler.class=software program.amazon.msk.auth.iam.IAMClientCallbackHandler
    database.historical past.client.sasl.consumer.callback.handler.class=software program.amazon.msk.auth.iam.IAMClientCallbackHandler
    database.historical past.client.safety.protocol=SASL_SSL
    database.port=3306
    embody.schema.adjustments=true
    database.hostname=native.mysql.inside
    database.password=xxxxxx
    desk.embody.record=salesdb.SALES_ORDER,salesdb.SALES_ORDER_ALL,salesdb.CUSTOMER
    database.historical past.client.sasl.mechanism=AWS_MSK_IAM
    database.embody.record=salesdb
    

16. Replace the next connector configuration:

    database.person=grasp
    database.hostname=native.mysql.inside
    database.password=<MySQLMasterUserPassword>
    

17. For Capability kind, select Provisioned.
18. For MCU rely per employee, enter 1.
19. For Variety of employees, enter 1.
20. Choose Use the MSK default configuration.
21. Within the Entry Permissions part, on the Select service function menu, select MSK-Join-PrivateDNS-MySQLConnector*, then select Subsequent.
22. Within the Safety part, preserve the default settings.
23. Within the Logs part, choose Ship to Amazon CloudWatch logs.
24. Select go to Amazon CloudWatch Logs console.
25. Beneath Logs within the navigation pane, select Log group.
26. Select Create log group.
    
27. Enter the log group identify, retention settings, and tags, then select Create.
28. Return to the connector creation web page and select Browse log group.
29. Select the AmazonMSKConnect log group, then select Subsequent.
30. Assessment the configurations and select Create connector.

Await the connector creation course of to finish (about 10–quarter-hour).

The MSK Join connector is now up and operating. You possibly can log in to the MySQL database utilizing your person ID and make a few document adjustments to the client desk document. MSK Join will be capable to obtain CDC data and updates to the database can be obtainable within the MSK <Buyer> matter.

Eat messages from the MSK matter

To devour messages from the MSK matter, run the Kafka client on the MSK_Client EC2 occasion obtainable within the MSK VPC.

1. SSH to the MSK_Client EC2 occasion. The MSK_Client occasion has the required Kafka consumer libraries, Amazon MSK IAM JAR file, consumer.properties file, and an occasion profile hooked up to it, together with the suitable IAM function utilizing the CloudFormation template.
2. Add the MSKClientSG safety group because the supply for the MSK safety group with the next properties:
   • For Kind, select All Site visitors.
   • For Supply, select Customized and MSK Safety Group.

   Now you’re able to devour information.

3. To record the matters, run the next command:

   ./kafka-topics.sh --bootstrap-server <BootstrapServerString>

4. To devour information from the salesdb-server.salesdb.CUSTOMER matter, use the next command:

   ./kafka-console-consumer.sh --bootstrap-server <BootstrapServerString> --consumer.config consumer.properties --topic salesdb-server.salesdb.CUSTOMER --from-beginning

Run the Kafka client in your EC2 machine and it is possible for you to to log messages much like the next:

Struct{after=Struct{CUST_ID=1998.0,NAME=Buyer Identify 1998,MKTSEGMENT=Market Section 3},supply=Struct{model=1.9.5.Closing,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=true,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{after=Struct{CUST_ID=1999.0,NAME=Buyer Identify 1999,MKTSEGMENT=Market Section 7},supply=Struct{model=1.9.5.Closing,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=true,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{after=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Section 9},supply=Struct{model=1.9.5.Closing,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=final,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{earlier than=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Section 9},after=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Segment10},supply=Struct{model=1.9.5.Closing,connector=mysql,identify=salesdb-server,ts_ms=1678100372000,db=salesdb,desk=CUSTOMER,server_id=1,file=binlog.000001,pos=43298616,row=0,thread=67},op=u,ts_ms=1678100372612}

Whereas testing the applying, data with CUST_ID 1998, 1999, and 2000 have been up to date, and these data can be found within the logs.

Clear up

It’s all the time a great apply to scrub up all of the sources created as a part of this publish to keep away from any extra price. To wash up your sources, delete the MSK Cluster, MSK Join connection, EC2 cases, DNS server, bastion host, S3 bucket, VPC, subnets and CloudWatch logs.

Moreover, clear up all different AWS sources that you simply created utilizing AWS CloudFormation. You possibly can delete these sources on the AWS CloudFormation console by deleting the stack.

Conclusion

On this publish, we mentioned the method of organising MSK Join utilizing a personal DNS. This function permits you to configure connectors to reference public or non-public domains.

We’re capable of obtain the preliminary load and CDC data from a MySQL database hosted in a separate VPC and its DNS isn’t accessible or resolvable externally. MSK Join was ready to connect with the MySQL database and devour the data utilizing the MSK Join non-public DNS function. The customized DHCP possibility set was hooked up to the VPC, which ensured DNS decision was carried out utilizing the native DNS server as a substitute of Route 53.

With the MSK Join non-public DNS help function, you can also make your databases, information warehouses, and programs like secret managers that work with your personal VPC inaccessible to the web and be capable to overcome this limitation and comply along with your company safety posture.

To be taught extra and get began, consult with non-public DNS for MSK join.

Concerning the creator

Amar is a Senior Options Architect at Amazon AWS within the UK. He works throughout energy, utilities, manufacturing and automotive prospects on strategic implementations, specializing in utilizing AWS Streaming and superior information analytics options, to drive optimum enterprise outcomes.