|
As you progress your machine studying (ML) workloads into manufacturing, that you must constantly monitor your deployed fashions and iterate while you observe a deviation in your mannequin efficiency. Whenever you construct a brand new mannequin, you sometimes begin validating the mannequin offline utilizing historic inference request knowledge. However this knowledge generally fails to account for present, real-world circumstances. For instance, new merchandise may grow to be trending that your product suggestion mannequin hasn’t seen but. Or, you expertise a sudden spike within the quantity of inference requests in manufacturing that you simply by no means uncovered your mannequin to earlier than.
Right this moment, I’m excited to announce Amazon SageMaker assist for shadow testing!
Deploying a mannequin in shadow mode allows you to conduct a extra holistic take a look at by routing a replica of the reside inference requests for a manufacturing mannequin to the brand new (shadow) mannequin. But, solely the responses from the manufacturing mannequin are returned to the calling utility. Shadow testing helps you construct additional confidence in your mannequin and catch potential configuration errors and efficiency points earlier than they impression finish customers. When you full a shadow take a look at, you need to use the deployment guardrails for SageMaker inference endpoints to soundly replace your mannequin in manufacturing.
Get Began with Amazon SageMaker Shadow Testing
You’ll be able to create shadow exams utilizing the brand new SageMaker Inference Console and APIs. Shadow testing provides you a totally managed expertise for setup, monitoring, viewing, and performing on the outcomes of shadow exams. When you’ve got present workflows constructed round SageMaker endpoints, you may as well deploy a mannequin in shadow mode utilizing the present SageMaker Inference APIs.
On the SageMaker console, choose Inference and Shadow exams to create, monitor, and deploy shadow exams.
To create a shadow take a look at, choose an present (or create a brand new) SageMaker endpoint and manufacturing variant you need to take a look at in opposition to.
Subsequent, configure the proportion of visitors to ship to the shadow variant, the comparability metrics you need to consider, and the length of the take a look at. You may as well allow knowledge seize in your manufacturing and shadow variant.
That’s it. SageMaker now mechanically deploys the brand new variant in shadow mode and routes a replica of the inference requests to it in actual time, all throughout the similar endpoint. The next diagram illustrates this workflow.
Notice that solely the responses of the manufacturing variant are returned to the calling utility. You’ll be able to select to both discard or log the responses of the shadow variant for offline comparability.
You may as well use shadow testing to validate adjustments you made to any part in your manufacturing variant, together with the serving container or ML occasion. This may be helpful while you’re upgrading to a brand new framework model of your serving container, making use of patches, or if you wish to guarantee that there is no such thing as a impression to latency or error price attributable to this modification. Equally, in the event you contemplate shifting to a different ML occasion sort, for instance, Amazon EC2 C7g cases based mostly on AWS Graviton processors, or EC2 G5 cases powered by NVIDIA A10G Tensor Core GPUs, you need to use shadow testing to guage the efficiency on manufacturing visitors previous to rollout.
You’ll be able to monitor the progress of the shadow take a look at and efficiency metrics akin to latency and error price by way of a reside dashboard. On the SageMaker console, choose Inference and Shadow exams, then choose the shadow take a look at you need to monitor.
In case you resolve to advertise the shadow mannequin to manufacturing, choose Deploy shadow variant and outline the infrastructure configuration to deploy the shadow variant.
You may as well use the SageMaker deployment guardrails if you wish to add linear or canary visitors shifting modes and auto rollbacks to your replace.
Availability and Pricing
SageMaker assist for shadow testing is offered as we speak in all AWS Areas the place SageMaker internet hosting is offered apart from the AWS GovCloud (US) Areas and AWS China Areas.
There isn’t a further cost for SageMaker shadow testing aside from utilization prices for the ML cases and ML storage provisioned to host the shadow variant. The pricing for ML cases and ML storage dimensions is identical because the real-time inference choice. There isn’t a further cost for knowledge processed out and in of shadow deployments. The SageMaker pricing web page has all the small print.
To be taught extra, go to Amazon SageMaker shadow testing.
Begin validating your new ML fashions with SageMaker shadow exams as we speak!
— Antje
