Construct environment friendly ETL pipelines with AWS Step Capabilities distributed map and redrive characteristic

AWS Step Capabilities is a totally managed visible workflow service that allows you to construct complicated information processing pipelines involving a various set of extract, remodel, and cargo (ETL) applied sciences equivalent to AWS Glue, Amazon EMR, and Amazon Redshift. You may visually construct the workflow by wiring particular person information pipeline duties and configuring payloads, retries, and error dealing with with minimal code.

Whereas Step Capabilities helps automated retries and error dealing with when information pipeline duties fail as a consequence of momentary or transient errors, there will be everlasting failures equivalent to incorrect permissions, invalid information, and enterprise logic failure in the course of the pipeline run. This requires you to establish the difficulty within the step, repair the difficulty and restart the workflow. Beforehand, to rerun the failed step, you wanted to restart your complete workflow from the very starting. This results in delays in finishing the workflow, particularly if it’s a fancy, long-running ETL pipeline. If the pipeline has many steps utilizing map and parallel states, this additionally results in elevated value as a consequence of will increase within the state transition for working the pipeline from the start.

Step Capabilities now helps the flexibility so that you can redrive your workflow from a failed, aborted, or timed-out state so you possibly can full workflows quicker and at a decrease value, and spend extra time delivering enterprise worth. Now you possibly can get better from unhandled failures quicker by redriving failed workflow runs, after downstream points are resolved, utilizing the identical enter offered to the failed state.

On this submit, we present you an ETL pipeline job that exports information from Amazon Relational Database Service (Amazon RDS) tables utilizing the Step Capabilities distributed map state. Then we simulate a failure and display the best way to use the brand new redrive characteristic to restart the failed job from the purpose of failure.

Resolution overview

One of many widespread functionalities concerned in information pipelines is extracting information from a number of information sources and exporting it to a knowledge lake or synchronizing the information to a different database. You should use the Step Capabilities distributed map state to run a whole bunch of such export or synchronization jobs in parallel. Distributed map can learn thousands and thousands of objects from Amazon Easy Storage Service (Amazon S3) or thousands and thousands of data from a single S3 object, and distribute the data to downstream steps. Step Capabilities runs the steps throughout the distributed map as youngster workflows at a most parallelism of 10,000. A concurrency of 10,000 is properly above the concurrency supported by many different AWS providers equivalent to AWS Glue, which has a tender restrict of 1,000 job runs per job.

The pattern information pipeline sources product catalog information from Amazon DynamoDB and buyer order information from Amazon RDS for PostgreSQL database. The information is then cleansed, remodeled, and uploaded to Amazon S3 for additional processing. The information pipeline begins with an AWS Glue crawler to create the Information Catalog for the RDS database. As a result of beginning an AWS Glue crawler is asynchronous, the pipeline has a wait loop to examine if the crawler is full. After the AWS Glue crawler is full, the pipeline extracts information from the DynamoDB desk and RDS tables. As a result of these two steps are impartial, they’re run as parallel steps: one utilizing an AWS Lambda operate to export, remodel, and cargo the information from DynamoDB to an S3 bucket, and the opposite utilizing a distributed map with AWS Glue job sync integration to do the identical from the RDS tables to an S3 bucket. Notice that AWS Identification and Entry Administration (IAM) permissions are required for invoking an AWS Glue job from Step Capabilities. For extra data, check with IAM Insurance policies for invoking AWS Glue job from Step Capabilities.

The next diagram illustrates the Step Capabilities workflow.

There are a number of tables associated to clients and order information within the RDS database. Amazon S3 hosts the metadata of all of the tables as a .csv file. The pipeline makes use of the Step Capabilities distributed map to learn the desk metadata from Amazon S3, iterate on each single merchandise, and name the downstream AWS Glue job in parallel to export the information. See the next code:

"States": {
            "Map": {
              "Kind": "Map",
              "ItemProcessor": {
                "ProcessorConfig": {
                  "Mode": "DISTRIBUTED",
                  "ExecutionType": "STANDARD"
                },
                "StartAt": "Export information for a desk",
                "States": {
                  "Export information for a desk": {
                    "Kind": "Activity",
                    "Useful resource": "arn:aws:states:::glue:startJobRun.sync",
                    "Parameters": {
                      "JobName": "ExportTableData",
                      "Arguments": {
                        "--dbtable.$": "$.tables"
                      }
                    },
                    "Finish": true
                  }
                }
              },
              "Label": "Map",
              "ItemReader": {
                "Useful resource": "arn:aws:states:::s3:getObject",
                "ReaderConfig": {
                  "InputType": "CSV",
                  "CSVHeaderLocation": "FIRST_ROW"
                },
                "Parameters": {
                  "Bucket": "123456789012-stepfunction-redrive",
                  "Key": "tables.csv"
                }
              },
              "ResultPath": null,
              "Finish": true
            }
          }

Conditions

To deploy the answer, you want the next conditions:

Launch the CloudFormation template

Full the next steps to deploy the answer assets utilizing AWS CloudFormation:

1. Select Launch Stack to launch the CloudFormation stack:
   
2. Enter a stack title.
3. Choose all of the examine packing containers below Capabilities and transforms.
4. Select Create stack.

The CloudFormation template creates many assets, together with the next:

• The information pipeline described earlier as a Step Capabilities workflow
• An S3 bucket to retailer the exported information and the metadata of the tables in Amazon RDS
• A product catalog desk in DynamoDB
• An RDS for PostgreSQL database occasion with pre-loaded tables
• An AWS Glue crawler that crawls the RDS desk and creates an AWS Glue Information Catalog
• A parameterized AWS Glue job to export information from the RDS desk to an S3 bucket
• A Lambda operate to export information from DynamoDB to an S3 bucket

Simulate the failure

Full the next steps to check the answer:

1. On the Step Capabilities console, select State machines within the navigation pane.
2. Select the workflow named ETL_Process.
   
3. Run the workflow with default enter.

Inside just a few seconds, the workflow fails on the distributed map state.

You may examine the map run errors by accessing the Step Capabilities workflow execution occasions for map runs and youngster workflows. On this instance, you possibly can id the exception is because of Glue.ConcurrentRunsExceededException from AWS Glue. The error signifies there are extra concurrent requests to run an AWS Glue job than are configured. Distributed map reads the desk metadata from Amazon S3 and invokes as many AWS Glue jobs because the variety of rows within the .csv file, however AWS Glue job is about with the concurrency of three when it’s created. This resulted within the youngster workflow failure, cascading the failure to the distributed map state after which the parallel state. The opposite step within the parallel state to fetch the DynamoDB desk ran efficiently. If any step within the parallel state fails, the entire state fails, as seen with the cascading failure.

Deal with failures with distributed map

By default, when a state experiences an error, Step Capabilities causes the workflow to fail. There are a number of methods you possibly can deal with this failure with distributed map state:

• Step Capabilities allows you to catch errors, retry errors, and fail again to a different state to deal with errors gracefully. See the next code:

  Retry": [
                        {
                          "ErrorEquals": [
                            "Glue.ConcurrentRunsExceededException "
                          ],
                          "BackoffRate": 20,
                          "IntervalSeconds": 10,
                          "MaxAttempts": 3,
                          "Remark": "Exception",
                          "JitterStrategy": "FULL"
                        }
                      ]
  

• Typically, companies can tolerate failures. That is very true if you find yourself processing thousands and thousands of things and also you anticipate information high quality points within the dataset. By default, when an iteration of map state fails, all different iterations are aborted. With distributed map, you possibly can specify the utmost variety of, or share of, failed gadgets as a failure threshold. If the failure is throughout the tolerable degree, the distributed map doesn’t fail.
• The distributed map state permits you to management the concurrency of the kid workflows. You may set the concurrency to map it to the AWS Glue job concurrency. Bear in mind, this concurrency is relevant solely on the workflow execution degree—not throughout workflow executions.
• You may redrive the failed state from the purpose of failure after fixing the foundation reason for the error.

Redrive the failed state

The basis reason for the difficulty within the pattern answer is the AWS Glue job concurrency. To deal with this by redriving the failed state, full the next steps:

1. On the AWS Glue console, navigate to the job named ExportsTableData.
2. On the Job particulars tab, below Superior properties, replace Most concurrency to five.
   

With the launch of redrive characteristic, You should use redrive to restart executions of commonplace workflows that didn’t full efficiently within the final 14 days. These embody failed, aborted, or timed-out runs. You may solely redrive a failed workflow from the step the place it failed utilizing the identical enter because the final non-successful state. You may’t redrive a failed workflow utilizing a state machine definition that’s completely different from the preliminary workflow execution. After the failed state is redriven efficiently, Step Capabilities runs all of the downstream duties robotically. To study extra about how distributed map redrive works, check with Redriving Map Runs.

As a result of the distributed map runs the steps contained in the map as youngster workflows, the workflow IAM execution position wants permission to redrive the map run to restart the distributed map state:

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Effect": "Allow",
      "Action": [
        "states:RedriveExecution"
      ],
      "Useful resource": "arn:aws:states:us-east-2:123456789012:execution:myStateMachine/myMapRunLabel:*"
    }
  ]
}

You may redrive a workflow from its failed step programmatically, by way of the AWS Command Line Interface (AWS CLI) or AWS SDK, or utilizing the Step Capabilities console, which offers a visible operator expertise.

1. On the Step Capabilities console, navigate to the failed workflow you need to redrive.
2. On the Particulars tab, select Redrive from failure.
   

The pipeline now runs efficiently as a result of there may be sufficient concurrency to run the AWS Glue jobs.

To redrive a workflow programmatically from its level of failure, name the new Redrive Execution API motion. The identical workflow begins from the final non-successful state and makes use of the identical enter because the final non-successful state from the preliminary failed workflow. The state to redrive from the workflow definition and the earlier enter are immutable.

Notice the next concerning various kinds of youngster workflows:

• Redrive for categorical youngster workflows – For failed youngster workflows which might be categorical workflows inside a distributed map, the redrive functionality ensures a seamless restart from the start of the kid workflow. This lets you resolve points which might be particular to particular person iterations with out restarting your complete map.
• Redrive for traditional youngster workflows – For failed youngster workflows inside a distributed map which might be commonplace workflows, the redrive characteristic features the identical manner as with standalone commonplace workflows. You may restart the failed state inside every map iteration from its level of failure, skipping pointless steps which have already efficiently run.

You should use Step Capabilities standing change notifications with Amazon EventBridge for failure notifications equivalent to sending an e-mail on failure.

Clear up

To wash up your assets, delete the CloudFormation stack by way of the AWS CloudFormation console.

Conclusion

On this submit, we confirmed you the best way to use the Step Capabilities redrive characteristic to redrive a failed step inside a distributed map by restarting the failed step from the purpose of failure. The distributed map state permits you to write workflows that coordinate large-scale parallel workloads inside your serverless purposes. Step Capabilities runs the steps throughout the distributed map as youngster workflows at a most parallelism of 10,000, which is properly above the concurrency supported by many AWS providers.

To study extra about distributed map, check with Step Capabilities – Distributed Map. To study extra about redriving workflows, check with Redriving executions.

Concerning the Authors

Sriharsh Adari is a Senior Options Architect at Amazon Internet Companies (AWS), the place he helps clients work backwards from enterprise outcomes to develop modern options on AWS. Over time, he has helped a number of clients on information platform transformations throughout trade verticals. His core space of experience embody Know-how Technique, Information Analytics, and Information Science. In his spare time, he enjoys enjoying Tennis.

Joe Morotti is a Senior Options Architect at Amazon Internet Companies (AWS), working with Enterprise clients throughout the Midwest US to develop modern options on AWS. He has held a variety of technical roles and enjoys displaying clients the artwork of the potential. He has attained seven AWS certification and has a ardour for AI/ML and the contact middle house. In his free time, he enjoys spending high quality time together with his household exploring new locations and overanalyzing his sports activities group’s efficiency.

Uma Ramadoss is a specialist Options Architect at Amazon Internet Companies, centered on the Serverless platform. She is chargeable for serving to clients design and function event-driven cloud-native purposes and fashionable enterprise workflows utilizing providers like Lambda, EventBridge, Step Capabilities, and Amazon MWAA.