Price monitoring for Amazon EMR on Amazon EKS

Amazon EMR is the industry-leading cloud large information answer, offering a set of open-source frameworks corresponding to Spark, Hive, Hudi, and Presto, absolutely managed and with per-second billing. Amazon EMR on Amazon EKS is a deployment possibility permitting you to deploy Amazon EMR on the identical Amazon Elastic Kubernetes Service (Amazon EKS) clusters that’s multi-tenant and utilized by different purposes, bettering useful resource utilization, lowering price, and simplifying infrastructure administration. EMR on EKS present you as much as 5.37 instances higher efficiency than OSS Spark v3.3.1 with 76.8% price financial savings. It additionally offers all kinds of job submission strategies, like an AWS API referred to as StartJobRun, or via a declarative approach with a Kubernetes controller via the AWS Controllers for Kubernetes for Amazon EMR on EKS.

This consolidation comes with a trade-off of elevated problem measuring fine-grained prices for showback or chargeback by workforce or utility. In response to a CNCF and FinOps Basis survey, 68% of Kubernetes customers both depend on month-to-month estimates or don’t monitor Kubernetes prices in any respect. And for respondents reporting energetic Kubernetes price monitoring, AWS Price Explorer and Kubecost had been ranked as the most well-liked instruments getting used.

At the moment, you’ll be able to distribute prices per tenant utilizing a tough multi-tenancy with separate EKS clusters in devoted AWS accounts or a tender multi-tenancy utilizing separate node teams in a shared EKS cluster. To scale back prices and enhance useful resource utilization, you need to use namespace-based segregation, the place nodes are shared throughout totally different namespaces. Nevertheless, calculating and attributing prices to groups by workload or namespaces whereas considering compute optimization (like Saving Plans or Spot Occasion price) and the price of AWS companies like EMR on EKS is a difficult and non-trivial process.

On this put up, we current a price chargeback answer for EMR on EKS that mixes the AWS-native capabilities of AWS Price and Utilization Stories (AWS CUR) alongside the in-depth Kubernetes price visibility and insights utilizing Kubecost on Amazon EKS.

Resolution overview

A job in EMR on EKS incur prices primarily on two dimensions: compute assets and a marginal uplift cost for EMR on EKS utilization. To trace the associated fee related to every of the size, we use information from three sources:

• AWS CUR – We use this to get the EMR on EKS price uplift per job and for Kubecost to reconcile the compute price with any saving plans or reserved occasion used. The supporting infrastructure for CUR is deployed as outlined in Organising Athena utilizing AWS CloudFormation templates.
• Kubecost – We use this to get the compute price incurred by the executor and driver pods.

The associated fee allocation course of consists of the next elements:

• The compute price is offered by Kubecost. Nevertheless, with a view to do an in-depth evaluation, we outline an hourly Kubernetes CronJob on it that begins a pod to retrieve information from Kubecost and shops it in Amazon Easy Storage Service (Amazon S3).
• CUR recordsdata are saved in an S3 bucket.
• We use Amazon Athena to create a view and supply a consolidated view of the overall price to run an EMR on EKS job.
• Lastly, you’ll be able to join your most popular enterprise intelligence instruments utilizing the JDBC or ODBC connections to Athena. On this put up, we use Amazon QuickSight native integration for visualization functions.

The next diagram reveals the general structure in addition to how the totally different elements work together with one another.

We offer a shell script to deploy our the monitoring answer. The shell script configures the infrastructure utilizing an AWS CloudFormation template, the AWS Command Line Interface (AWS CLI), and eksctl and kubectl instructions. This script runs the next actions:

1. Begin the CloudFormation deployment.
2. Create and configure an AWS Price and Utilization Report.
3. Configure and deploy Kubecost backed by Amazon Managed Service for Prometheus.
4. Deploy a Kubernetes CronJob.

Conditions

You want the next stipulations:

This put up assumes you have already got an EKS cluster and run EMR on EKS jobs. For those who don’t have an EKS cluster prepared to check the answer, we advise beginning with a regular EMR on EKS blueprint that configures a cluster to submit EMR on EKS jobs.

Arrange the answer

To run the shell script, full the next steps:

1. Clone the next GitHub repository.
2. Go to the folder cost-tracking with the next command:

cd cost-tracking

3. Run the script with following command :

sh deploy-emr-eks-cost-tracking.sh REGION KUBECOST-VERSION EKS-CLUSTER-NAME ACCOUNT-ID

After you run the script, you’re prepared to make use of Kubecost and the CUR information to know the associated fee related together with your EMR on EKS jobs.

Monitoring price

On this part, we present you the right way to analyze the compute price that’s retrieved from Kubecost, the right way to question EMR on EKS uplift information, and the right way to mix them to have a single consolidated view for the associated fee.

Compute price

Kubecost affords numerous methods to trace price per Kubernetes object. For instance, you’ll be able to observe price by pod, controller, job, label, or deployment. It additionally permits you to perceive the price of idle assets, like Amazon Elastic Compute Cloud (Amazon EC2) situations that aren’t absolutely utilized by pods. On this put up, we assume that no nodes are provisioned if no EMR on EKS job is operating, and we use the Karpenter Cluster Autoscaler to provision nodes when jobs are submitted. Karpenter additionally does bin packing, which optimizes the EC2 useful resource utilization and in flip reduces the price of idle assets.

To trace compute price related to EMR on EKS pods, we question the Kubecost allocation API by passing pod and labels within the combination parameter. We use the emr-containers.amazonaws.com/job.id and emr-containers.amazonaws.com/virtual-cluster-id labels which can be all the time current in executor and driver pods. The labels are used to filter Kubecost information to get solely the associated fee related to EMR on EKS pods. You may evaluate numerous ranges of granularity on the pod, job, and digital cluster stage to know the price of a driver vs. executor, or of utilizing Spot Situations in jobs. You may as well use the digital cluster price to know the general price of a EMR on EMR when it’s utilized in a namespace that’s utilized by purposes apart from EMR on EKS.

We additionally present the instance_id, occasion dimension, and capability kind (On-Demand or Spot) that was used to run the pod. That is retrieved via querying the Kubecost property API. This information will be helpful to know the way you run your jobs and which capability you utilize extra usually.

The information about the price of operating the pods in addition to the property is retrieved with a Kubernetes CronJob that submits the request to the Kubecost API, joins the 2 information sources (allocation and property information) on the instance_id, cleans the info, and shops it in Amazon S3 in CSV format.

The compute price information has a number of fields which can be of curiosity, together with cpucost, ramcost (price of reminiscence), pvcost (price of Amazon EBS storage), effectivity of use of CPU and RAM, in addition to whole price, which represents the mixture price of all of the assets used, both at pod, job, or digital cluster stage.

To view this information, full the next steps:

1. On the Athena console, navigate to the question editor.
2. Select athenacurcfn_c_u_r for the database and cost_data for the desk.
3. Run the next question:

SELECT job_id,
vc_id,
sum(totalcost) as price
FROM "athenacurcfn_c_u_r"."compute_cost"
GROUP BY job_id, vc_id

The next screenshot reveals the question outcomes.

To question the info about info on the pod stage, you’ll be able to run the next SQL assertion:

SELECT
split_part(identify, '/', 1) as pod_name,
job_id,
vc_id,
totalcost,
instance_id,
"properties.labels.node_kubernetes_io_instance_type",
capacity_type
FROM "athenacurcfn_c_u_r"."compute_cost";

EMR on EKS uplift

The associated fee related to EMR on EKS uplift is obtainable via AWS CUT and is saved in an S3 bucket. The script you ran within the setup step created an Athena desk related to the info within the S3 bucket. The next steps take you thru how one can question the info:

1. On the Athena console, navigate to the question editor.
2. Select athenacurcfn_c_u_r for the database and cur_data for the desk.
3. Run the next question:

SELECT
split_part(line_item_resource_id, '/', 5) as job_id,
split_part(line_item_resource_id, '/', 3) as vc_id,
sum(line_item_blended_cost) as price
FROM athenacurcfn_c_u_r.automated
WHERE product_product_family='EMR Containers'
GROUP BY line_item_resource_id

This question offers you with the associated fee per job. The next screenshot reveals the outcomes.

You’ll have to wait as much as 24 hours for the CUR information to be obtainable. As such, you must solely run the previous question after the CUR information is obtainable and you’ve got run the EMR on EKS jobs.

Total price

To view the general price and carry out evaluation on it, create a view in Athena as follows:

CREATE VIEW emr_eks_cost AS
SELECT
split_part(line_item_resource_id, '/', 5) as job_id,
split_part(line_item_resource_id, '/', 3) as vc_id,
sum(line_item_blended_cost) as price,
'emr-uplift' as class
FROM athenacurcfn_c_u_r.cur_data
WHERE product_product_family='EMR Containers'
GROUP BY line_item_resource_id
UNION
SELECT
job_id,
vc_id,
sum(totalCost) as price,
'compute' as class
FROM "athenacurcfn_c_u_r"."compute_cost"
group by job_id, vc_id

Now that the view is created, you’ll be able to question and analyze the price of operating your EMR on EKS jobs:

SELECT sum(price) as total_cost, job_id, vc_id
FROM "athenacurcfn_c_u_r"."emr_eks_cost"
GROUP BY job_id, vc_id;

The next screenshot reveals an instance output of the question on the created view.

Lastly, you need to use QuickSight for a graphical high-level view in your EMR on EKS spend. The next screenshot reveals an instance dashboard.

Now you can adapt this answer to your particular wants and construct your customized evaluation.

Clear up

All through this put up, you deployed and configured the required infrastructure elements to trace price to your EMR on EKS workloads. To keep away from incurring further expenses for this answer, delete all of the assets you created:

1. Empty the S3 buckets cost-data-REGION-ACCOUNT_ID and aws-athena-query-results-cur-REGION-ACCOUNT_ID.
2. Delete the Athena workgroup kubecost-cur-workgroup.
3. Empty and delete the ECR repository emreks-compute-cost-exporter.
4. Run the script destroy-emr-eks-cost-tracking.sh, which is able to delete the AWS CloudFormation deployment, uninstall Kubecost, delete the CronJob, and delete the Price and Utilization Stories.

Conclusion

On this put up, we confirmed how you need to use Kubecost capabilities alongside Price and Utilization Stories to carefully monitor the prices for Amazon EMR on EKS per digital cluster or per job. This answer permits you to obtain extra granular prices for chargebacks utilizing Athena, Amazon Managed Service for Prometheus, and QuickSight.

The answer introduced steps to arrange Price and Utilization Stories and Kubecost, and configure a CronJob on an hourly foundation to get the price of operating pods spun by EMR on EKS. You may modify the introduced answer to run at longer intervals or to gather information on totally different EKS clusters. You may as well modify the Python script run by the CronJob to additional clear information or cut back the quantity of information saved by eliminating fields you don’t want. You need to use the insights offered to drive price optimization efforts over time, detect any enhance of prices, and measure the impression of recent deployments or specific occasions on useful resource utilization and value efficiency. For extra details about integrating EMR on EKS in your current Amazon EKS deployment, check with Design concerns for Amazon EMR on EKS in a multi-tenant Amazon EKS atmosphere

Concerning the Authors

Lotfi Mouhib is a Senior Options Architect working for the Public Sector workforce with Amazon Internet Providers. He helps public sector clients throughout EMEA understand their concepts, construct new companies, and innovate for residents. In his spare time, Lotfi enjoys biking and operating.

Hamza Mimi Principal Options Architect within the French Public sector workforce at Amazon Internet Providers (AWS). With an extended expertise within the telecommunications {industry}. He’s at present working as a buyer advisor on subjects starting from digital transformation to architectural steering.