New – Create application-consistent snapshots utilizing Amazon Knowledge Lifecycle Supervisor and customized scripts

November 9, 2023

1

Amazon Knowledge Lifecycle Supervisor now helps the usage of pre-snapshot and post-snapshot scripts embedded in AWS Programs Supervisor paperwork. You need to use these scripts to make sure that Amazon Elastic Block Retailer (Amazon EBS) snapshots created by Knowledge Lifecycle Supervisor are application-consistent. Scripts can pause and resume I/O operations, flush buffered information to EBS volumes, and so forth. As a part of this launch we’re additionally publishing a set of detailed weblog posts that present you easy methods to use this characteristic with self-managed relational databases and Home windows Quantity Shadow Copy Service (VSS).

Knowledge Lifecycle Supervisor (DLM) Recap
As a fast recap, Knowledge Lifecycle Supervisor lets you automate the creation, retention, and deletion of Amazon EBS quantity snapshots. Upon getting accomplished the prerequisite steps corresponding to onboarding your EC2 occasion to AWS Programs Supervisor, establishing an IAM position for DLM, and tagging your SSM paperwork, you merely create a lifecycle coverage and point out (by way of tags) the relevant Amazon Elastic Compute Cloud (Amazon EC2) situations, set a retention mannequin, and let DLM do the remaining. The insurance policies specify when they’re to be run, what’s to be backed up, and the way lengthy the snapshots should be retained. For a full walk-through of DLM, learn my 2018 weblog publish, New – Lifecycle Administration for Amazon EBS Snapshots.

Utility Constant Snapshots
EBS snapshots are crash-consistent, that means that they symbolize the state of the related EBS quantity on the time that the snapshot was created. That is adequate for a lot of kinds of functions, together with these that don’t use snapshots to seize the state of an lively relational database. To make a snapshot that’s application-consistent, it’s essential to take pending transactions under consideration (both ready for them to complete or inflicting them to fail), momentarily pause additional write operations, take the snapshot, after which resume regular operations.

And that’s the place right this moment’s launch is available in. DLM now has the power to inform the occasion to organize for an application-consistent backup. The pre-snapshot script can handle pending transactions, flush in-memory information to persistent storage, freeze the filesystem, and even deliver the appliance or database to a cease. Then the post-snapshot script can deliver the appliance or database again to life, reload in-memory caches from persistent storage, thaw the filesystem, and so forth.

Along with the base-level help for customized scripts, it’s also possible to use this characteristic to automate the creation of VSS Backup snapshots:

Pre and Publish Scripts
The brand new scripts apply to DLM insurance policies for situations. Let’s assume that I’ve created a coverage that references SSM paperwork with pre-snapshot and post-snapshot scripts, and that it applies to a single occasion. Right here’s what occurs when the coverage is run per its schedule:

The pre-snapshot script is began from the SSM doc.
Every command within the script is run and the script-level standing (success or failure) is captured. If enabled within the coverage, DLM will retry failed scripts.
Multi-volume EBS snapshots are initiated for EBS volumes connected to the occasion, with additional management by way of the coverage.
The post-snapshot script is began from the SSM doc,
Every command within the script is run and and the script-level standing (success or failure) is captured.

The coverage accommodates choices that provide you with management over the actions which might be taken (retry, proceed, or skip) when both of the scripts instances out or fails. The standing is logged, Amazon CloudWatch metrics are printed, Amazon EventBridge occasions are emitted, and the standing can be encoded in tags which might be mechanically assigned to every snapshot.

The pre-snapshot and post-snapshot scripts can carry out any of the actions which might be allowed in a command doc: operating shell scripts, operating PowerShell scripts, and so forth. The actions should full throughout the timeout specified within the coverage, with an allowable vary of 10 seconds to 120 seconds.

Getting Began
You have to to have an in depth understanding of your software or database in an effort to construct a sturdy pair of scripts. Along with dealing with the “completely happy path” when all goes properly, your scripts must plan for a number of failure situations. For instance, a pre-snapshot script ought to fork a background process that can function a failsafe in case the post-snapshot script doesn’t work as anticipated. Every script should return a shell-level standing code, as detailed right here.

As soon as I’ve written and examined my scripts and packaged them as SSM paperwork, I open the Knowledge Lifecycle Supervisor web page within the EC2 Console, choose EBS snapshot coverage, and click on Subsequent step:

I goal all of my situations which might be tagged with a Mode of Manufacturing, and use the default IAM position (for those who use a distinct position, it should allow entry to SSM), depart the remainder of the values as-is, and proceed:

On the subsequent web page I scroll all the way down to Pre and publish scripts and increase the part. I click on Allow pre and publish scripts, select Customized SSM doc, after which choose my SSM doc from the menu. I additionally set the timeout and retry choices, and select to default to a crash-consistent backup if considered one of my scripts fails. I click on Overview coverage, do one ultimate test, and click on Create coverage on the next web page:

My coverage is created, and can take impact immediately. After it has run a minimum of as soon as, I can examine the CloudWatch metrics to test for begins, completions, and failures: