Automate alerting and reporting for AWS Glue job useful resource utilization

Knowledge transformation performs a pivotal position in offering the required information insights for companies in any group, small and enormous. To realize these insights, clients usually carry out ETL (extract, remodel, and cargo) jobs from their supply methods and output an enriched dataset. Many organizations at present are utilizing AWS Glue to construct ETL pipelines that carry information from disparate sources and retailer the info in repositories like an information lake, database, or information warehouse for additional consumption. These organizations are on the lookout for methods they will cut back price throughout their IT environments and nonetheless be operationally performant and environment friendly.

Image a state of affairs the place you, the VP of Knowledge and Analytics, are in command of your information and analytics environments and workloads operating on AWS the place you handle a crew of information engineers and analysts. This crew is allowed to create AWS Glue for Spark jobs in improvement, check, and manufacturing environments. Throughout testing, one of many jobs wasn’t configured to robotically scale its compute sources, leading to jobs timing out, costing the group greater than anticipated. The subsequent steps often embrace finishing an evaluation of the roles, price stories to see which account generated the spike in utilization, going by logs to see when what occurred with the job, and so forth. After the ETL job has been corrected, you might wish to implement monitoring and set commonplace alert thresholds to your AWS Glue surroundings.

This submit will assist organizations proactively monitor and value optimize their AWS Glue environments by offering a better path for groups to measure effectivity of their ETL jobs and align configuration particulars in keeping with organizational necessities. Included is an answer it is possible for you to to deploy that may notify your crew by way of electronic mail about any Glue job that has been configured incorrectly. Moreover, a weekly report is generated and despatched by way of electronic mail that aggregates useful resource utilization and supplies price estimates per job.

AWS Glue price concerns

AWS Glue for Apache Spark jobs are provisioned with a lot of staff and a employee kind. These jobs might be both G.1X, G.2X, G.4X, G.8X or Z.2X (Ray) employee varieties that map to information processing models (DPUs). DPUs embrace a specific amount of CPU, reminiscence, and disk house. The next desk incorporates extra particulars.

For instance, if a job is provisioned with 10 staff as G.1X employee kind, the job could have entry to 40 vCPU and 160 GB of RAM to course of information and double utilizing G.2X. Over-provisioning staff can result in elevated price, resulting from not all staff being utilized effectively.

In April 2022, Auto Scaling for AWS Glue was launched for AWS Glue model 3.0 and later, which incorporates AWS Glue for Apache Spark and streaming jobs. Enabling auto scaling in your Glue for Apache Spark jobs will will let you solely allocate staff as wanted, as much as the employee most you specify. We advocate enabling auto scaling to your AWS Glue 3.0 & 4.0 jobs as a result of this function will assist cut back price and optimize your ETL jobs.

Amazon CloudWatch metrics are additionally an effective way to observe your AWS Glue surroundings by creating alarms for sure metrics like common CPU or reminiscence utilization. To study extra about easy methods to use CloudWatch metrics with AWS Glue, seek advice from Monitoring AWS Glue utilizing Amazon CloudWatch metrics.

The next answer supplies a easy solution to set AWS Glue employee and job length thresholds, configure monitoring, and obtain emails for notifications on how your AWS Glue surroundings is performing. If a Glue job finishes and detects employee or job length thresholds had been exceeded, it is going to notify you after the job run has accomplished, failed, or timed out.

Resolution overview

The next diagram illustrates the answer structure.

Once you deploy this software by way of AWS Serverless Utility Mannequin (AWS SAM), it is going to ask what AWS Glue employee and job length thresholds you want to set to observe the AWS Glue for Apache Spark and AWS Glue for Ray jobs operating in that account. The answer will use these values as the choice standards when invoked. The next is a breakdown of every step within the structure:

1. Any AWS Glue for Apache Spark job that succeeds, fails, stops, or occasions out is shipped to Amazon EventBridge.
2. EventBridge picks up the occasion from AWS Glue and triggers an AWS Lambda operate.
3. The Lambda operate processes the occasion and determines if the info and analytics crew ought to be notified concerning the specific job run. The operate performs the next duties:
   1. The operate sends an electronic mail utilizing Amazon Easy Notification Service (Amazon SNS) if wanted.
      • If the AWS Glue job succeeded or was stopped with out going over the employee or job length thresholds, or is tagged to not be monitored, no alerts or notifications are despatched.
      • If the job succeeded however ran with a employee or job length thresholds greater than allowed, or the job both failed or timed out, Amazon SNS sends a notification to the designated electronic mail with details about the AWS Glue job, run ID, and motive for alerting, together with a hyperlink to the particular run ID on the AWS Glue console.
   2. The operate logs the job run info to Amazon DynamoDB for a weekly aggregated report delivered to electronic mail. The Dynamo desk has Time to Stay enabled for 7 days, which retains the storage to minimal.
4. As soon as per week, the info inside DynamoDB is aggregated by a separate Lambda operate with significant info like longest-running jobs, variety of retries, failures, timeouts, price evaluation, and extra.
5. Amazon Easy Electronic mail Service (Amazon SES) is used to ship the report as a result of it may be higher formatted than utilizing Amazon SNS. The e-mail is formatted by way of HTML output that gives tables for the aggregated job run information.
6. The info and analytics crew is notified concerning the ongoing job runs by Amazon SNS, and so they obtain the weekly aggregation report by Amazon SES.

Word that AWS Glue Python shell and streaming ETL jobs aren’t supported as a result of they’re not in scope of this answer.

Stipulations

You should have the next conditions:

• An AWS account to deploy the answer to
• Correct AWS Identification and Entry Administration (IAM) privileges to create the sources
• The AWS SAM CLI to construct and deploy the answer button under, to run template in your AWS surroundings

Deploy the answer

This AWS SAM software provisions the next sources:

• Two EventBridge guidelines
• Two Lambda features
• An SNS subject and subscription
• A DynamoDB desk
• An SES subscription
• The required IAM roles and insurance policies

To deploy the AWS SAM software, full the next steps:

Clone the aws-samples GitHub repository:

git clone https://github.com/aws-samples/aws-glue-job-tracker.git

Deploy the AWS SAM software:

cd aws-glue-job-tracker
sam deploy --guided

Present the next parameters:

• GlueJobWorkerThreshold – Enter the utmost variety of staff you need an AWS Glue job to have the ability to run with earlier than sending threshold alert. The default is 10. An alert will likely be despatched if a Glue job runs with greater staff than specified.

• GlueJobDurationThreshold – Enter the utmost length in minutes you need an AWS Glue job to run earlier than sending threshold alert. The default is 480 minutes (8 hours). An alert will likely be despatched if a Glue job runs with greater job length than specified.

• GlueJobNotifications – Enter an electronic mail or distribution record of those that must be notified by Amazon SNS and Amazon SES. You’ll be able to go to the SNS subject after the deployment is full and add emails as wanted.

To obtain emails from Amazon SNS and Amazon SES, you need to verify your subscriptions. After the stack is deployed, test your electronic mail that was specified within the template and make sure by selecting the hyperlink in every message. When the applying is efficiently provisioned, it is going to start monitoring your AWS Glue for Apache Spark job surroundings. The subsequent time a job fails, occasions out, or exceeds a specified threshold, you’ll obtain an electronic mail by way of Amazon SNS. For instance, the next screenshot reveals an SNS message a couple of job that succeeded however had a job length threshold violation.

You may need jobs that must run at the next employee or job length threshold, and also you don’t need the answer to guage them. You’ll be able to merely tag that job with the important thing/worth of remediate and false. The step operate will nonetheless be invoked, however will use the PASS state when it acknowledges the tag. For extra info on job tagging, seek advice from AWS tags in AWS Glue.

Configure weekly reporting

As talked about beforehand, when an AWS Glue for Apache Spark job succeeds, fails, occasions out, or is stopped, EventBridge forwards this occasion to Lambda, the place it logs particular details about every job run. As soon as per week, a separate Lambda operate queries DynamoDB and aggregates your job runs to offer significant insights and suggestions about your AWS Glue for Apache Spark surroundings. This report is shipped by way of electronic mail with a tabular construction as proven within the following screenshot. It’s meant for top-level visibility so that you’re in a position to see your longest job runs over time, jobs which have had many retries, failures, and extra. It additionally supplies an general price calculation as an estimate of what every AWS Glue job will price for that week. It shouldn’t be used as a assured price. If you need to see actual price per job, the AWS Value and Utilization Report is one of the best useful resource to make use of. The next screenshot reveals one desk (of 5 complete) from the AWS Glue report operate.

Clear up

In case you don’t wish to run the answer anymore, delete the AWS SAM software for every account that it was provisioned in. To delete your AWS SAM stack, run the next command out of your mission listing:

sam delete

Conclusion

On this submit, we mentioned how one can monitor and cost-optimize your AWS Glue job configurations to adjust to organizational requirements and coverage. This methodology can present price controls over AWS Glue jobs throughout your group. Another methods to assist management the prices of your AWS Glue for Apache Spark jobs embrace the newly launched AWS Glue Flex jobs and Auto Scaling. We additionally offered an AWS SAM software as an answer to deploy into your accounts. We encourage you to evaluation the sources offered on this submit to proceed studying about AWS Glue. To study extra about monitoring and optimizing for price utilizing AWS Glue, please go to this current weblog. It goes in depth on all the price optimization choices and features a template that builds a CloudWatch dashboard for you with metrics about all your Glue job runs.

Concerning the authors

Michael Hamilton is a Sr Analytics Options Architect specializing in serving to enterprise clients within the south east modernize and simplify their analytics workloads on AWS. He enjoys mountain biking and spending time together with his spouse and three youngsters when not working.

Angus Ferguson is a Options Architect at AWS who’s captivated with assembly clients the world over, serving to them clear up their technical challenges. Angus focuses on Knowledge & Analytics with a give attention to clients within the monetary companies business.