|
I’m completely satisfied to share which you can now consider, evaluate, and choose the most effective basis fashions (FMs) in your use case in Amazon Bedrock. Mannequin Analysis on Amazon Bedrock is obtainable immediately in preview.
Amazon Bedrock affords a alternative of automated analysis and human analysis. You need to use automated analysis with predefined metrics similar to accuracy, robustness, and toxicity. For subjective or customized metrics, similar to friendliness, fashion, and alignment to model voice, you possibly can arrange human analysis workflows with only a few clicks.
Mannequin evaluations are important in any respect phases of improvement. As a developer, you now have analysis instruments obtainable for constructing generative synthetic intelligence (AI) functions. You can begin by experimenting with completely different fashions within the playground surroundings. To iterate sooner, add automated evaluations of the fashions. Then, if you put together for an preliminary launch or restricted launch, you possibly can incorporate human opinions to assist guarantee high quality.
Let me provide you with a fast tour of Mannequin Analysis on Amazon Bedrock.
Computerized mannequin analysis
With automated mannequin analysis, you possibly can carry your individual information or use built-in, curated datasets and pre-defined metrics for particular duties similar to content material summarization, query and answering, textual content classification, and textual content era. This takes away the heavy lifting of designing and working your individual mannequin analysis benchmarks.
To get began, navigate to the Amazon Bedrock console, then choose Mannequin analysis beneath Evaluation & deployment within the left menu. Create a brand new mannequin analysis and select Computerized.
Subsequent, observe the setup dialog to decide on the FM you wish to consider and the kind of job, for instance, textual content summarization. Choose the analysis metrics and specify a dataset—both built-in or your individual.
When you carry your individual dataset, be certain it’s in JSON Traces format, and every line accommodates all the key-value pairs that you simply wish to consider your mannequin with for the mannequin dimension that you simply wish to consider. For instance, if you wish to consider the mannequin on a question-answer job, you’ll format your information as follows (with class being optionally available):
{"referenceResponse":"Cantal","class":"Capitals","immediate":"Aurillac is the capital of"}
{"referenceResponse":"Bamiyan Province","class":"Capitals","immediate":"Bamiyan metropolis is the capital of"}
{"referenceResponse":"Abkhazia","class":"Capitals","immediate":"Sokhumi is the capital of"}
...Then, create and run the analysis job to grasp the mannequin’s task-specific efficiency. As soon as the analysis job is full, you possibly can assessment the leads to the mannequin analysis report.
Human mannequin analysis
For human analysis, you possibly can have Amazon Bedrock arrange human assessment workflows with a couple of clicks. You possibly can carry your individual datasets and outline customized analysis metrics, similar to relevance, fashion, or alignment to model voice. You even have the selection to both leverage your individual inside groups as reviewers or interact an AWS managed workforce. This takes away the tedious effort of constructing and working human analysis workflows.
To get began, create a brand new mannequin analysis and choose Human: Carry your individual workforce or Human: AWS managed workforce.
When you select an AWS managed workforce for human analysis, describe your mannequin analysis wants, together with job kind, experience of the work workforce, and the approximate variety of prompts, alongside along with your contact info. Within the subsequent step, an AWS skilled will attain out to debate your mannequin analysis undertaking necessities in additional element. Upon assessment, the workforce will share a customized quote and undertaking timeline.
When you select to carry your individual workforce, observe the setup dialog to decide on the FMs you wish to consider and the kind of job, for instance, textual content summarization. Then, choose the analysis metrics, add your take a look at dataset, and arrange the work workforce.
For human analysis, you’ll format the instance information proven earlier than once more in JSON Traces format like this (with class and referenceResponse being optionally available):
{"immediate":"Aurillac is the capital of","referenceResponse":"Cantal","class":"Capitals"}
{"immediate":"Bamiyan metropolis is the capital of","referenceResponse":"Bamiyan Province","class":"Capitals"}
{"immediate":"Senftenberg is the capital of","referenceResponse":"Oberspreewald-Lausitz","class":"Capitals"}As soon as the human analysis is accomplished, Amazon Bedrock generates an analysis report with the mannequin’s efficiency towards your chosen metrics.
Issues to know
Listed here are a few vital issues to know:
Mannequin help – Throughout preview, you possibly can consider and evaluate text-based giant language fashions (LLMs) obtainable on Amazon Bedrock. Throughout preview, you possibly can choose one mannequin for every automated analysis job and as much as two fashions for every human analysis job utilizing your individual workforce. For human analysis utilizing an AWS managed workforce, you possibly can specify customized undertaking necessities.
Pricing – Throughout preview, AWS solely fees for the mannequin inference wanted to carry out the analysis (processed enter and output tokens for on-demand pricing). There will likely be no separate fees for human analysis or automated analysis. Amazon Bedrock Pricing has all the small print.
Be a part of the preview
Computerized analysis and human analysis utilizing your individual work workforce can be found immediately in public preview in AWS Areas US East (N. Virginia) and US West (Oregon). Human analysis utilizing an AWS managed workforce is obtainable in public preview in AWS Area US East (N. Virginia). To study extra, go to the Amazon Bedrock Developer Expertise net web page and take a look at the Consumer Information.
Get began
Log in to the AWS Administration Console and begin exploring mannequin analysis in Amazon Bedrock immediately!
— Antje
