Kinesis Agent is a standalone Java software program software that gives an easy strategy to accumulate and ship knowledge to Amazon Kinesis Knowledge Streams and Amazon Kinesis Knowledge Firehose. The agent repeatedly screens a set of recordsdata and sends new knowledge to the specified vacation spot. The agent handles file rotation, checkpointing, and retry upon failures. It delivers all your knowledge in a dependable, well timed, and easy method. It additionally emits Amazon CloudWatch metrics that can assist you higher monitor and troubleshoot the streaming course of.
This publish describes the steps to ship knowledge from a containerized software to Kinesis Knowledge Firehose utilizing Kinesis Agent. Extra particularly, we present how you can run Kinesis Agent as a sidecar container for an software operating in Amazon Elastic Container Service (Amazon ECS). After the info is in Kinesis Knowledge Firehose, it may be despatched to any supported vacation spot, reminiscent of Amazon Easy Storage Service (Amazon S3).
With a view to current the important thing factors required for this setup, we assume that you’re acquainted with Amazon ECS and dealing with containers. We additionally keep away from the implementation particulars and packaging technique of our check knowledge technology software, known as the producer.
Answer overview
As depicted within the following determine, we configure a Kinesis Agent container as a sidecar that may learn recordsdata created by the producer container. On this occasion, the producer and Kinesis Agent containers share knowledge through a bind mount in Amazon ECS.
Stipulations
It’s best to fulfill the next conditions for the profitable completion of this activity:
With these conditions in place, you’ll be able to start subsequent step to package deal a Kinesis Agent and your required agent configuration as a container in your native growth machine.
Create a Kinesis Agent configuration file
We use the Kinesis Agent configuration file to configure the supply and vacation spot, amongst different knowledge switch settings. The next code makes use of the minimal configuration required to learn the contents of recordsdata matching /var/log/producer/*.log
and publish them to a Kinesis Knowledge Firehose supply stream referred to as kinesis-agent-demo
:
Create a container picture for Kinesis Agent
To deploy Kinesis Agent as a sidecar in Amazon ECS, you first need to package deal it as a container picture. The container will need to have Kinesis Agent, which
and discover
binaries, and the Kinesis Agent configuration file that you simply ready earlier. Its entry level should be configured utilizing the start-aws-kinesis-agent
script. This command is put in while you run the yum set up aws-kinesis-agent
step. The ensuing Dockerfile ought to look as follows:
Run the docker construct
command to construct this container:
After the picture is constructed, it must be pushed to a container registry like Amazon ECR so as to reference it within the subsequent part.
Create an ECS activity definition with Kinesis Agent and the applying container
Now that you’ve got Kinesis Agent packaged as a container picture, you should use it in your ECS activity definitions to run as sidecar. To try this, you create an ECS activity definition along with your software container (referred to as producer
) and Kinesis Agent container. All containers in a activity definition are scheduled on the identical container host and due to this fact can share sources reminiscent of bind mounts.
Within the following pattern container definition, we use a bind mount referred to as logs_dir
to share a listing between the producer
container and kinesis-agent
container.
You should utilize the next template as a place to begin, however you should definitely change taskRoleArn
and executionRoleArn
to legitimate IAM roles in your AWS account. On this occasion, the IAM function used for taskRoleArn
will need to have write permissions to Kinesis Knowledge Firehose that you simply specified earlier within the agent.json
file. Moreover, guarantee that the ECR picture paths and awslogs-region
are modified as per your AWS account.
Register the duty definition with the next command:
Run a brand new ECS activity
Lastly, you’ll be able to run a brand new ECS activity utilizing the duty definition you simply created utilizing the aws ecs run-task
command. When the duty is began, it’s best to have the ability to see two containers operating below that activity on the Amazon ECS console.
Conclusion
This publish confirmed how simple it’s to run Kinesis Agent in a containerized atmosphere. Though we used Amazon ECS as our container orchestration service on this publish, you should use a Kinesis Agent container in different environments reminiscent of Amazon Elastic Kubernetes Service (Amazon EKS).
To study extra about utilizing Kinesis Agent, seek advice from Writing to Amazon Kinesis Knowledge Streams Utilizing Kinesis Agent. For extra details about Amazon ECS, seek advice from the Amazon ECS Developer Information.
Concerning the Creator
Buddhike de Silva is a Senior Specialist Options Architect at Amazon Net Companies. Buddhike helps prospects run giant scale streaming analytics workloads on AWS and make the most effective out of their cloud journey.