Introduction
AWS IoT SiteWise is a managed service that helps prospects gather, retailer, manage and monitor information from their industrial tools at scale. Clients typically must deliver their historic tools measurement information from current methods akin to information historians and time collection databases into AWS IoT SiteWise for making certain information continuity, coaching synthetic intelligence (AI) & machine studying (ML) fashions that may predict tools failures, and deriving actionable insights.
On this weblog put up, we’ll present how one can get began with the BulkImportJob API and import historic tools information into AWS IoT SiteWise utilizing a code pattern.
You need to use this imported information to achieve insights by way of AWS IoT SiteWise Monitor and Amazon Managed Grafana, practice ML fashions on Amazon Lookout for Gear and Amazon SageMaker, and energy analytical functions.
To start a bulk import, prospects must add a CSV file to Amazon Easy Storage Service (Amazon S3) containing their historic information in a predefined format. After importing the CSV file, prospects can provoke the asynchronous import to AWS IoT SiteWise utilizing the CreateBulkImportJob operation, and monitor the progress utilizing the DescribeBulkImportJob and ListBulkImportJob operations.
Conditions
To observe by way of this weblog put up, you’ll need an AWS account and an AWS IoT SiteWise supported area. In case you are already utilizing AWS IoT SiteWise, select a special area for an remoted atmosphere. You might be additionally anticipated to have some familiarity with Python.
Setup the atmosphere
- Create an AWS Cloud9 atmosphere utilizingÂ
Amazon Linux 2
 platform - Utilizing the terminal in your Cloud9 atmosphere, set up Git and clone the sitewise-bulk-import-example repository from Github
sudo yum set up git git clone https://github.com/aws-samples/aws-iot-sitewise-bulk-import-example.git cd aws-iot-sitewise-bulk-import-example pip3 set up -r necessities.txt
Walkthrough
For the demonstration on this put up, we’ll use an AWS Cloud9 occasion to symbolize an on-premises developer workstation and simulate two months of historic information for a couple of manufacturing strains in an car manufacturing facility.
We’ll then put together the information and import it into AWS IoT SiteWise at scale, leveraging a number of bulk import jobs. Lastly, we’ll confirm whether or not the information was imported efficiently.
A bulk import job can import information into the 2 storage tiers supplied by AWS IoT SiteWise, relying on how the storage is configured. Earlier than we proceed, allow us to first outline these two storage tiers.
Scorching tier: Shops often accessed information with decrease write-to-read latency. This makes the recent tier ultimate for operational dashboards, alarm administration methods, and another functions that require quick entry to the current measurement values from tools.
Chilly tier: Shops less-frequently accessed information with increased learn latency, making it ultimate for functions that require entry to historic information. As an illustration, it may be utilized in enterprise intelligence (BI) dashboards, synthetic intelligence (AI), and machine studying (ML) coaching. To retailer information within the chilly tier, AWS IoT SiteWise makes use of an S3 bucket within the buyer’s account.
Retention Interval: Determines how lengthy your information is saved within the scorching tier earlier than it’s deleted.
Now that we discovered in regards to the storage tiers, allow us to perceive how a bulk import job handles writes for various situations. Seek advice from the desk under:
Worth | Timestamp | Write Conduct |
New | New | A brand new information level is created |
New | Present | Present information level is up to date with the brand new worth for the offered timestamp |
Present | Present | The import job identifies duplicate information and discards it. No modifications are made to current information. |
Within the subsequent part, we’ll observe step-by-step directions to import historic tools information into AWS IoT SiteWise.
Steps to import historic information
Step 1: Create a pattern asset hierarchy
For the aim of this demonstration, we’ll create a pattern asset hierarchy for a fictitious car producer with operations throughout 4 totally different cities. In a real-world situation, chances are you’ll have already got an current asset hierarchy in AWS IoT SiteWise, wherein case this step is elective.
Step 1.1: Evaluate the configuration
- From terminal, navigate to the basis of the Git repo.
- Evaluate the configuration for asset fashions and belongings.
cat config/assets_models.yml
- Evaluate the schema for asset properties.
cat schema/sample_stamping_press_properties.json
Step 1.2: Create asset fashions and belongings
- Run
python3 src/create_asset_hierarchy.py
to robotically create asset fashions, hierarchy definitions, belongings, asset associations. - Within the AWS Console, navigate to AWS IoT SiteWise, and confirm the newly created Fashions and Belongings.
- Confirm that you just see the asset hierarchy just like the one under.
Step 2: Put together historic information
Step 2.1: Simulate historic information
On this step, for demonstration objective, we’ll simulate two months of historic information for 4 stamping presses throughout two manufacturing strains. In a real-world situation, this information would usually come from supply methods akin to information historians and time collection databases.
The CreateBulkImportJob API has the next key necessities:
- To determine an asset property, you’ll need to specify both an
ASSET_ID
+PROPERTY_ID
mixture or theALIAS.
On this weblog, we shall be utilizing the previous. - The info must be in CSV format.
Observe the steps under to generate information in keeping with these expectations. For extra particulars in regards to the schema, confer with Ingesting information utilizing the CreateBulkImportJob API.
- Evaluate the configuration for information simulation.
cat config/data_simulation.yml
- Run
python3 src/simulate_historical_data.py
to generate simulated historic information for the chosen properties and time interval. If the whole rows exceedrows_per_job
as configured inbulk_import.yml
, a number of information recordsdata shall be created to help parallel processing. On this pattern, about 700,000+ information factors are simulated for the 4 stamping presses (A-D) throughout two manufacturing strains (Sample_Line 1 and Sample_Line 2). Since we configuredrows_per_job
as 20,000, a complete of 36 information recordsdata shall be created. - Confirm the generated information recordsdata underneath
information
listing. - The info schema will observe the
column_names
configured inbulk_import.yml
config file.
Step 2.2: Add historic information to Amazon S3
As AWS IoT SiteWise requires the historic information to be out there in Amazon S3, we’ll add the simulated information to the chosen S3 bucket.
- Replace the information bucket underneath
bulk_import.yml
with any current non permanent S3 bucket that may be deleted later. - Run
python3 src/upload_to_s3.py
to add the simulated historic information to the configured S3 bucket. - Navigate to Amazon S3 and confirm the objects had been uploaded efficiently.
Step 3: Import historic information into AWS IoT SiteWise
Earlier than you possibly can import historic information, AWS IoT SiteWise requires that you just allow Chilly tier storage. For added particulars, confer with Configuring storage settings.
If in case you have already activated chilly tier storage, think about modifying the S3 bucket to a brief one which will be later deleted whereas cleansing up the pattern sources.
Be aware that by altering the S3 bucket, not one of the information from current chilly tier S3 bucket is copied to the brand new bucket. When modifying S3 bucket location, make sure the IAM position configured underneath S3 entry position has permissions to entry the brand new S3 bucket.
Step 3.1: Configure storage settings
- Navigate to AWS IoT SiteWise, choose Storage, then choose Activate chilly tier storage.
- Choose an S3 bucket location of your selection.
- Choose Create a job from an AWS managed template.
- Verify Activate retention interval, enter
30 days
, and save.
Step 3.2: Present permissions for AWS IoT SiteWise to learn information from Amazon S3
- Navigate to AWS IAM, choose Insurance policies underneath Entry administration, and Create coverage.
- Change to JSON tab and substitute the content material with the next. Replace <bucket-name> with the identify of information S3 bucket configured in
bulk_import.yml
.{ "Model": "2012-10-17", "Assertion": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Useful resource": ["arn:aws:s3:::<bucket-name>"] } ] }
- Save the coverage with Identify as
SiteWiseBulkImportPolicy
. - Choose Roles underneath Entry administration, and Create position.
- Choose Customized belief coverage and substitute the content material with the next.
{ "Model": "2012-10-17", "Assertion": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "iotsitewise.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
- Click on Subsequent and choose the
SiteWiseBulkImportPolicy
IAM coverage created within the earlier steps. - Click on Subsequent and create the position with Function identify as
SiteWiseBulkImportRole
. - Choose Roles underneath Entry administration, seek for the newly created IAM position
SiteWiseBulkImportRole
, and click on on its identify. - Copy the ARNÂ of the IAM position utilizing the copy icon.
Step 3.3: Create AWS IoT SiteWise bulk import jobs
- Substitute the
role_arn
discipline inconfig/bulk_import.yml
with the ARN ofSiteWiseBulkImportRole
IAM position copied in earlier steps. - Replace the
config/bulk_import.yml
file:- Substitute the
role_arn
with the ARN ofSiteWiseBulkImportRole
IAM position. - Substitute the
error_bucket
with any current non permanent S3 bucket that may be deleted later.
- Substitute the
- Run
python3 src/create_bulk_import_job.py
to import historic information from the S3 bucket into AWS IoT SiteWise: - The script will create a number of jobs to concurrently import all the information recordsdata created into AWS IoT SiteWise. In a real-world situation, a number of terabytes of information will be shortly imported into AWS IoT SiteWise utilizing concurrently operating jobs.
- Verify the standing of jobs from the output:
- When you see the standing of any job as
COMPLETED_WITH_FAILURES
orFAILED
, confer with Troubleshoot widespread points part.
Step 4: Confirm the imported information
As soon as the majority import jobs are accomplished, we have to confirm if the historic information is efficiently imported into AWS IoT SiteWise. You possibly can confirm the information both by straight wanting on the chilly tier storage or by visually inspecting the charts out there in AWS IoT SiteWise Monitor.
Step 4.1: Utilizing the chilly tier storage
On this step, we’ll test if new S3 objects have been created within the bucket that was configured for chilly tier.
- Navigate to Amazon S3 and find the S3 bucket configured underneath AWS IoT SiteWise → Storage → S3 bucket location (in Step 3) for chilly tier storage.
- Confirm the partitions and objects underneath the
uncooked/
prefix.
Step 4.2: Utilizing AWS IoT SiteWise Monitor
On this step, we’ll visually examine if the charts present information for the imported date vary.
- Navigate to AWS IoT SiteWise and find Monitor.
- Create a portal to entry information saved in AWS IoT SiteWise.
- Present
AnyCompany Motor
because the Portal identify. - Select
IAM
for Person authentication. - Present your e mail handle for Help contact e mail, and click on Subsequent.
- Depart the default configuration for Extra options, and click on Create.
- Beneath Invite directors, choose your IAM consumer or IAM Function, and click on Subsequent.
- Click on on Assign Customers.
- Present
- Navigate to Portals and open the newly created portal.
- Navigate to Belongings and choose an asset, for instance, AnyCompany_Motor → Sample_Arlington → Sample_Stamping → Sample_Line 1 → Sample_Stamping Press A.
- Use Customized vary to match the date vary for the information uploaded.
- Confirm the information rendered within the time collection line chart.
Troubleshoot widespread points
On this part, we’ll cowl the widespread points encountered whereas importing information utilizing bulk import jobs and spotlight some potential causes.
If a bulk import job will not be efficiently accomplished, it’s best observe to confer with logs within the error S3 bucket configured in bulk_import.yml
and perceive the basis trigger.
No information imported
- Incorrect schema:
dataType doesn't match dataType tied to the asset-property
The schema offered at Ingesting information utilizing the CreateBulkImportJob API needs to be adopted precisely. Utilizing the console, confirm the offered DATA_TYPE offered matches with the information kind within the corresponding asset mannequin property. - Incorrect ASSET_ID or PROPERTY_ID:
Entry will not be modeled
Utilizing the console, confirm the corresponding asset and property exists. - Duplicate information:
A worth for this timestamp already exists
AWS IoT SiteWise detects and robotically discards any duplicate. Utilizing console, confirm if the information already exists.
Lacking solely sure components of information
- Lacking current information: BulkImportJob API imports the current information (that falls inside the scorching tier retention interval) into AWS IoT SiteWise scorching tier and doesn’t switch it instantly to Amazon S3 (chilly tier). You could want to attend for the subsequent scorching to chilly tier switch cycle, which is at the moment set to six hours.
Clear Up
To keep away from any recurring prices, take away the sources created on this weblog. Observe the steps to delete these sources:
- Navigate to AWS Cloud9 and delete your atmosphere.
- Run
python3 src/clean_up_asset_hierarchy.py
 to delete the next sources, so as, from AWS IoT SiteWise:- Asset associations
- Belongings
- Hierarchy definitions from asset fashions
- Asset fashions
- From AWS IoT SiteWise console, navigate to Monitor → Portals, choose the beforehand created portal, and delete.
- Navigate to Amazon S3 and carry out the next:
- Delete the
S3 bucket location
configured underneath the Storage part of AWS IoT SiteWise - Delete the information and error buckets configured within the
/config/bulk_import.yml
of Git repo
- Delete the
Conclusion
On this put up, you may have discovered how you can use the AWS IoT SiteWise BulkImportJob API to import historic tools information into AWS IoT SiteWise utilizing AWS Python SDK (Boto3). You can even use the AWS CLI or SDKs for different programming languages to carry out the identical operation. To study extra about all supported ingestion mechanisms for AWS IoT SiteWise, go to the documentation.
Concerning the authors
Raju Gottumukkala is an IoT Specialist Options Architect at AWS, serving to industrial producers of their sensible manufacturing journey. Raju has helped main enterprises throughout the power, life sciences, and automotive industries enhance operational effectivity and income progress by unlocking true potential of IoT information. Previous to AWS, he labored for Siemens and co-founded dDriven, an Trade 4.0 Knowledge Platform firm. |
Avik Ghosh is a Senior Product Supervisor on the AWS Industrial IoT crew, specializing in the AWS IoT SiteWise service. With over 18 years of expertise in know-how innovation and product supply, he makes a speciality of Industrial IoT, MES, Historian, and large-scale Trade 4.0 options. Avik contributes to the conceptualization, analysis, definition, and validation of Amazon IoT service choices. |