By Brendan Siebecker, Director of Alliances – Atlan
By Gaurav Malhotra, Options Architect – AWS
Extra persons are utilizing knowledge at the moment than ever earlier than, however it’s getting tougher and tougher for everybody to collaborate on the identical knowledge. This consists of knowledge engineers and analysts, product managers, entrepreneurs, researchers, and extra.
Atlan is an AWS Accomplice with the Amazon Redshift Prepared designation. It has pioneered a collaborative workspace serving to trendy knowledge groups work collectively higher. Atlan is a collaboration and orchestration layer—the glue that brings collectively your staff, the instruments you like, and the info you want.
With deep integrations throughout the trendy knowledge stack, Atlan helps groups create a single supply of reality for all of their knowledge belongings. Atlan is extending its suite of integrations by way of intensive collaboration with Amazon Internet Providers (AWS), and now you can discover Atlan on AWS Market.
On this publish, we’ll share how firms use Atlan and AWS to democratize their knowledge, collaborate extra successfully, and unify all of their information and context in a single place. We’ll additionally present learn how to combine Atlan and Amazon Redshift with a step-by-step walkthrough.
How Firms Use Atlan and AWS
Clients within the AWS ecosystem can profit from Atlan’s seamless integration with a set of AWS companies—together with analytics tooling equivalent to Amazon Redshift, Amazon Athena, and AWS Glue—in addition to well-liked instruments within the trendy knowledge stack equivalent to Tableau, Apache Airflow, and dbt.
For instance, Postman (an API platform utilized by greater than 500,000 firms worldwide) makes use of AWS and Atlan to open up their knowledge, construct belief, and turn out to be extra data-driven. That is vital as a result of Postman’s leaders staunchly imagine that everybody within the firm ought to have the ability to entry knowledge and acquire insights from it. Nevertheless, earlier than Atlan, their knowledge was typically a thriller and context lived within the heads of early staff members.
Prudhvi Vasa, Analytics Chief at Postman, defined the worth of democratizing and documenting knowledge with AWS and Atlan: “We’ve been capable of catalog and doc all of our knowledge, which acts as a single supply of reality for our knowledge. The consequence? Everybody is ready to discover the correct knowledge for his or her use case, and the info is constant throughout the board for all accessing it,” says Vasa.
“Having a dependable knowledge basis, the place folks can discover and perceive all our knowledge opens the opportunity of having everybody take part in analyzing knowledge. This permits our total firm to turn out to be extra data-aware and data-driven, which is the purpose for any main firm at the moment,” Vasa provides.
Atlan, AWS, and the Fashionable Knowledge Stack
Atlan acts as a virtualized layer throughout a wide range of instruments within the trendy knowledge stack. Its push- and pull-based metadata crawlers deliver metadata from completely different instruments within the knowledge platform to construct a unified collaboration platform.
First, Atlan creates a robust search and discovery layer. It acts as a Google-like search engine for your entire knowledge, the place you’ll be able to shortly uncover and entry any knowledge asset together with all of its related context and documentation. This search helps clever key phrase recognition, highly effective search filters, sorting by relevance or reputation, and even a Cmd+Okay shortcut.
This search doesn’t simply floor knowledge tables—it surfaces the whole lot about a company’s knowledge. In at the moment’s day and age, knowledge belongings will not be simply tables. That’s why Atlan lets folks search throughout each kind of knowledge asset—enterprise intelligence (BI) dashboards, pipelines, code, fashions, queries, metrics, directed acrylic graphs (DAGs), and extra.
Second, Atlan unifies context from all of the completely different instruments in your knowledge stack in a single place. The place does this knowledge come from? Who makes use of it? Can I belief it? The “asset profile” in Atlan solutions questions like these with info like a knowledge asset’s description, certification (verified, WIP, or deprecated), column previews, pattern knowledge, and Readme. This makes it simpler to grasp every knowledge asset (like lineage, documentation, and possession) in a single view.
Embedded Collaboration Integrations
Atlan is constructed on the premise of embedded collaboration, borrowing rules from GitHub, Figma, Superhuman, and different trendy future-of-work instruments.
Embedded collaboration is about work taking place the place you’re, with the least quantity of friction. What for those who may request entry to a knowledge asset while you get a hyperlink, and the proprietor may get the request on Slack and approve or reject it proper there?
What if, while you’re inspecting a knowledge asset and must report a problem, you might instantly set off a help request that’s completely built-in along with your engineering staff’s Jira workflow?
Embedded collaboration unifies these micro-workflows that waste time, trigger frustration, and result in software fatigue, turning time-consuming duties throughout a number of instruments into a number of clicks in whichever software you’re already utilizing.
Getting Began with Atlan and AWS
Atlan is constructed on prime of AWS’s highly effective companies. By immediately integrating with AWS companies like Amazon Redshift, Atlan helps knowledge groups accomplish extra by making collaboration a seamless a part of their course of.
Within the following step-by-step information, we’ll present you learn how to shortly combine Atlan with Redshift to open up a brand new world of collaboration, readability, and belief for contemporary knowledge groups.
Comply with the steps beneath to determine a connection and combine Atlan with a Redshift database.
Step 1: Choose the Supply
- Log into your Atlan workspace.
- Click on on the Workflow button within the left sidebar.
- You’ll see the Market web page with the listing of sources accessible in your workspace. Click on on New Workflow on the prime proper.
- Choose Redshift from the listing of choices within the integrations tab, and click on Setup Workflow.
Step 2: Present Credentials
- To arrange a brand new connection, fill in your Redshift credentials on the Credential web page. Under is an instance:
- Hostname: examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com
- Port: 5439
- Username: myusername
- Password: xxxxxx
- Default Database: dev
- Choose the proper authentication technique (fundamental authentication, or an IAM person or IAM position).
- After you have crammed within the particulars, click on on Take a look at Authentication after which Subsequent.
Step 3: Set Up Your Configuration
- On the Connection web page, identify your connection and choose the customers or teams who ought to have the ability to entry it.
- On the Metadata web page, specify any metadata you need to embody or exclude from crawling.
- Click on Run to run the crawler as soon as, or click on on Schedule & Run to schedule it for a day by day, weekly, or month-to-month run.
- When you click on Run or set a schedule, the workflow will begin operating.
Atlan will crawl your Amazon Redshift occasion and ingest all the metadata into Atlan. This course of varies relying on the dimensions of the warehouse you’re trying to crawl, however it typically takes lower than half-hour.
Step 4: Uncover Your Belongings
Now that you simply’ve efficiently linked Atlan to Amazon Redshift and the Atlan crawler has ingested the metadata, you can begin discovering your belongings inside Atlan.
Within the instance beneath, an organization has 362 Redshift belongings inside Atlan. These are seen on the Discovery web page, filtered by the Amazon Redshift integration.
The info staff can click on by way of to any asset to see related context, such because the column names, glossary phrases, classifications, standing, associated queries, and Readme.
They will additionally see the info asset’s lineage, which is auto-generated on the column stage for each Redshift asset. This helps knowledge groups see the place every asset comes from and which dashboards use it.
Conclusion
Establishing Atlan is simple for analysts and enterprise customers alike. From connecting your tables and dashboards inside Amazon Redshift to enriching imported knowledge belongings, the method doesn’t require main knowledge engineering sources or time. Crawling occurs seamlessly and stays in sync along with your AWS situations.
To see it for your self, take a look at Atlan’s guided demo. Atlan will be downloaded from AWS Market, or you’ll be able to get began on the Atlan website.
For extra info, take a look at these hyperlinks:
This text was initially printed on the AWS Accomplice Community (APN) weblog.
