Nearly two years in the past, we laid out our concept of recent metadata for the fashionable information stack, the Knowledge Catalog 3.0.
Since then, it’s been a dream seeing our imaginative and prescient come to life. Energetic metadata has change into a class and central speaking level within the information world. Gartner, Forrester, and G2 alike made main strikes to scrap previous concepts of information cataloging and introduce new ones round energetic metadata and trendy cataloging.
This transformation was pushed by actual product developments, ones that turned this summary dream into one thing that information groups might really use.
In 2022, our engineering crew shipped over 200 new options, together with a model new model of our product. These enhancements proved the facility of energetic metadata and even obtained us ranked as a Chief within the Forrester Wave, Q2 2022!
Earlier than we tackle the subsequent yr of latest options, let’s take a couple of minutes to look again on an incredible yr. Listed here are 15 of our favourite options from 2022.
TL;DR: 15 updates in Atlan it is best to find out about
From siloed instruments to embedded context
Now we have all the time been pushed by one core concept — variety is the one actuality. Knowledge groups are one of the numerous groups immediately, from the instruments they use and the individuals they work with to their very own use circumstances and personas. This variety is each a power and a weak spot.
Knowledge groups work throughout a various set of instruments — BI instruments like Looker, collaboration instruments like GitHub, communication instruments like Slack, ticketing programs like Jira, and extra. Too typically, these flip into siloed information programs. If you happen to’re in a dashboard, you wish to know if you happen to can belief it then and there, relatively than switching and looking throughout different instruments.
In 2022, we constructed variety into the core of our product. By activating metadata, Atlan sends essential context again into the workflows and instruments the place the people of information work day-after-day.
Improve collaboration and seize information with our Slack integration
With our Slack integration, launched in January 2022, we set the stage for intensive collaboration on this planet of metadata. Atlan’s deep integration with Slack goes past simply sending messages — it’s about sharing and capturing necessary information throughout your group.
Atlan’s Slack integration contains:
- Wealthy preview hyperlinks: Share property together with your crew on Slack and see detailed previews.
- Workflow alerts: Get alerts in Slack each time an Atlan workflow fails or succeeds.
- Ask a query from Atlan: Ask a query by clicking on the Slack icon for any asset. That Slack dialog may also be documented mechanically in Atlan for future reference.
- Doc threads instantly from Slack to Atlan: Simply share Slack conversations to particular property in Atlan.
- Search phrases and queries instantly from Slack: Seek for phrases and queries in Atlan with out leaving Slack.
Keep in move with our Atlan Chrome Extension
Energetic metadata makes it potential for metadata to move effortlessly and rapidly throughout the whole information stack, embedding enriched context and knowledge in each instrument within the information stack.
Think about you’re a dashboard and wish to know who owns it and what a metric means. How would you go about it? Properly, you’ll most likely have to determine who the proprietor was. You then would most certainly take a display seize of the dashboard, e mail or Slack it to that proprietor, and ask your questions. Not anymore!
Now you’ll be able to skip all these steps by merely clicking on the Atlan icon in your browser. It should open the Atlan Chrome extension proper in your dashboard, providing you with all of the context you want proper while you want it. Atlan makes this potential within the instruments the place your information groups and information shoppers spend most of their time — Looker, Tableau On-line, Snowsight, Mode, Energy BI, BigQuery, and Salesforce, with extra instruments on the best way.
Clear up points quicker with our Jira integration
Monitoring points and fixing issues simply obtained so much simpler with our Jira integration. For a lot of information groups, attending to the “root trigger” of damaged dashboards or inconsistent numbers could be a lengthy, drawn-out course of. Not anymore!
With Atlan’s Jira integration, customers can now rapidly open a brand new Jira ticket or handle an current ticket instantly from any Atlan asset. Simply click on the Atlan sidebar.
Reduce danger throughout information creation with our GitHub integration
When an information engineer makes adjustments to a dbt mannequin, they may unknowingly be affecting tens or a whole bunch of individuals, tables, or dashboards. How can we decrease this danger whereas additionally offering transparency for the information engineer and information client?
With our integration between Atlan and GitHub, information engineers can now entry all of the context they should decrease the chance for information shoppers.
Right here’s an instance of what this appears to be like like: say that you just’re an information engineer, and also you’ve created a pull request. When the GitHub motion runs, it should mechanically create an inventory of all downstream property that might be affected by this request — earlier than you make the change. From there, you’ll be able to attain out to customers upfront, or analysis and take a look at the property to see how they may be affected.
From dispersed, handbook instruments to automated 360° context
Simply as information groups immediately use numerous instruments, their information ecosystem is made up of numerous information property. These are not simply tables made up of rows and columns. Knowledge property are exams, metrics, pipelines, code, queries, dashboards, and extra — all scattered throughout completely different information instruments and programs.
At present, while you onboard an engineer, you simply share a hyperlink to a GitHub repository. This has the code, documentation, historical past, collaborators, and every little thing else you should know concerning the code, multi functional place.
Nonetheless, it takes loads of time and handbook effort to arrange and convey documentation into conventional information catalogs immediately — and simply as quickly as you’re achieved, they’re outdated. At Atlan, we wished to make information property simply as straightforward to share and perceive as software program on GitHub. Together with activating metadata again into every day workflows, our workspace offers customers a 360-degree view of each asset by mechanically pulling in and stitching collectively metadata from each instrument in your information stack.
Atlan Playbooks: the Zapier for metadata
One of many widespread questions we get from information groups is, “How can we automate our metadata?” Whereas different groups like advertising or gross sales can simply do action-based automation at scale with instruments like Zapier or Salesforce, information groups typically don’t have the bandwidth to code customized automations for every numerous use case. Why can’t there be a Zapier for metadata?
That’s the reason we’ve got developed the primary low-code/no-code metadata automation for information groups. With Atlan’s Playbooks, customers can now create rule-based automations at scale. These can be utilized to drive countless use circumstances throughout your group. Listed here are just a few examples:
- Deprecate property: Mark any property that haven’t been queried within the final 30 days as deprecated.
- Add/change possession: For Salesforce property which are lacking an proprietor, add RevOps because the proprietor.
- Report failed property: Put up an inventory on Slack of each desk and dashboard with a “failed” Airflow standing.
- Defend delicate information: Connect GDPR customized metadata on all property tagged as “PII”.
- Flag upstream alerts: Notify downstream homeowners when an upstream asset is tagged with a warning or announcement.
Scale back handbook documentation with Trident strategies
Conventional metadata administration could be a tiresome, handbook course of that requires loads of human intervention from information leaders and stewards.
That’s why we launched Trident, which offers automated strategies for brand new descriptions, homeowners, phrases, and classifications. Trident makes metadata enrichment simpler in order that information groups can spend extra time on significant tasks and fewer time writing descriptions and classifying information. Since its launch, one-third of all description updates on Atlan have been made with Trident!
P.S. We lately gave a preview of the potential of Trident by combining it with GPT-3 to create Trident AI — more and more clever automated descriptions, enterprise phrases, READMEs, and extra.
Handle your information and lower prices with asset reputation and utilization metrics
One of many greatest challenges dealing with information groups immediately is managing their information and lowering information infrastructure prices. That’s why we launched utilization and recognition metrics.
Knowledge groups can now analyze their Snowflake information property with out leaving Atlan. With these new metrics, organizations can establish their most used property, discover individuals with essentially the most context on property, establish unused property, and extra. These metrics embody:
- Asset question logs: The variety of instances an information asset has been queried (i.e. asset reputation), when it was final up to date, and many others.
- Consumer question logs: An information asset’s prime customers and most up-to-date customers, who final queried it, and many others.
- Value and efficiency optimization logs: Which queries are slowest, costliest, and many others.
Get full visibility and column-level lineage with our Fivetran Metadata API integration
Atlan’s deep partnership and integration with Fivetran creates the primary end-to-end view of the fashionable information stack. By leveraging Fivetran’s new Metadata API, Atlan now builds column-level lineage out of your Fivetran sources all the best way to BI instruments like Looker and Tableau — providing you with full visibility throughout your trendy information stack with highly effective root trigger and affect evaluation.
We’re excited to accomplice with Atlan to convey higher metadata administration to our prospects. Knowledge governance is a major battle for information groups immediately.
Collectively, Fivetran and Atlan are fixing this downside by serving to information groups acquire much-needed visibility on the place information got here from, who accessed it, and what adjustments have occurred within the pipeline.
Fraser Harris, VP of Product at Fivetran
Make metrics and fashions first-class residents with our dbt Semantic Layer integration
One other huge partnership from this yr was Atlan + dbt. With our new Semantic Layer integration, dbt metrics and fashions at the moment are first-class residents in Atlan.
Atlan’s deep integration with the Semantic Layer brings dbt’s wealthy metrics and fashions into the remainder of the information stack, permitting information groups to create repeatable, metadata properties — like desk homeowners and verified tags — and construct documentation requirements.
dbt’s metrics are additionally now part of Atlan’s column-level lineage, spanning from supply programs and information storage to transformation and BI. With elevated context, visibility, and self-service, this integration helps numerous information groups work collectively higher.
Get a birds-eye view of your information property with our Reporting Heart
At Atlan, we’ve seen a sample of questions come up as our prospects collect extra metadata. Our customers wish to perceive how their information panorama and its metadata are evolving.
That’s why we launched the Reporting Heart, which offers a fowl’s-eye view of all of the metadata updates in Atlan by visualizations that will help you monitor adoption, curation, and metadata administration. Briefly, Reporting Heart offers you information in your metadata. It doesn’t get extra meta than this.
With the Reporting Heart customers can now:
- Monitor their property
- Monitor glossary phrases and classes
- Implement governance by monitoring personas, functions, classifications and requests
- Perceive queries and question runs from insights in Atlan.
From generic instruments to personalised experiences
One other problem with conventional information catalogs and metadata options is that they deal with each person the identical, delivering a generic expertise throughout the group.
The issue with that method is, context means various things to completely different individuals. For instance, an information engineer appears to be like at a desk and so they’re taken with transformations, information high quality, and pipeline stability. In the meantime, an analyst could care about column names and descriptions, frequency distributions, and widespread queries ran in opposition to that desk. Lastly, a enterprise person would possibly simply see a metric and surprise “What does this metric imply?” and “Can I belief it?”
We wished to alter that narrative and construct a platform that delivers a customized expertise for every person throughout your group.
Create a Netflix-like information expertise with Personas and Functions
If Netflix can create a customized viewing expertise for his or her customers, we thought, why can’t Atlan do the identical factor for information groups? That’s precisely what we did with the introduction of personas and functions.
Personas and functions give admins the flexibility to regulate entry to property, insurance policies, and preferences tailor-made to the position of every particular person. Personas present asset-level controls for every crew, whereas functions group property collectively for cross-team tasks.
For instance, the advertising persona could give all advertising individuals entry to sure Snowflake tables and Tableau property. With a function, some members of the advertising crew may additionally get entry to different advertising marketing campaign information and dashboards which are related to campaigns they’re working this yr.
We’ve seen this having all types of fascinating implementations inside organizations. For example, many information groups utilizing Atlan have began desirous about implementing paradigms like the information mesh, which at its core has this idea of a site. Utilizing personas and functions, you’ll be able to create extraordinarily personalised experiences by area, which then permits your information mesh to start out being profitable.
Curate person experiences with customized metadata
Customized metadata is among the most useful options in Atlan. Together with creating their very own metadata (like an information freshness rating or information high quality metrics), admins can choose which customized metadata must be seen to every persona whereas limiting visibility to people who wouldn’t discover this data related.
For instance, it’s possible {that a} finance persona gained’t want entry to dbt transformations and Airflow DAGs, however they could have to dig into lineage to troubleshoot an odd Salesforce report or question a Snowflake desk to find out ARR.
With customized metadata, Atlan offers the flexibility to curate every particular person’s distinctive expertise, serving to them work extra effectively with the fitting data and guaranteeing they don’t get overwhelmed by the whole information property.
Energy personalised energetic metadata with an all-new Atlan person interface
At Atlan, we’ve all the time been dedicated to serving to information groups work collectively higher. We began out as an information crew ourselves and failed thrice in attempting to implement an information catalog. So we determined to construct Atlan to be the sort of platform and accomplice that we wished we had again then.
After over a yr of labor, we launched a model new model of Atlan final yr. From an entire redesign of the Atlan person interface to highly effective energetic metadata and personalization options, this replace made it simpler than ever for information groups to construct belief and democratize information with metadata.
From closed, siloed instruments to open programs
When you’ve gotten a various set of customers, utilizing a various set of instruments and property, all throughout industries, information stacks, and extra, there might be a variety in use circumstances. Each firm’s use case is completely different, and one dimension doesn’t match all. That’s why we care about constructing open and extensible platforms that enable information groups to resolve their very own issues in their very own methods.
Drive countless integrations and use circumstances with a absolutely open API structure
Offering a completely open API structure permits each motion to be programmatically pushed in Atlan. We now open all our metadata change occasions by AWS Eventbridge and a Kafka queue, Lamda webhooks, and extra.
On the core of the Atlan platform, every little thing is open by default — and this enables information groups throughout any group to construct for and remedy all kinds of use circumstances. Listed here are a pair that we’ve got seen organizations begin to implement.
Programmatic information archival and deletion
One in every of our prospects, a media firm, purchases loads of exterior information. The information has an expiration date, so that they arrange handbook processes to repeatedly handle and delete their expired information. This generally led to information not being deleted, leading to authorized and compliance dangers. This course of additionally pressured the information crew to repeatedly spend important time on handbook work, leading to misplaced productiveness and crew output.
Now, utilizing the Atlan API, a notification is triggered 30 days earlier than the contractual finish date for the information. This creates a Jira ticket, letting the suitable events know that the information can then be deleted or archived. By this new programmatic metadata workflow, the corporate was not solely in a position to eradicate authorized danger but additionally save important time and productiveness.
Energetic information belief administration
One other buyer has built-in Atlan with an information high quality instrument and energy energetic information belief administration. They use our open APIs to herald anomalies which are detected by the information high quality instrument, after which downstream BI property and probably impacted enterprise metrics are mechanically flagged. For instance, if a desk feeding a KPI has an information high quality situation, Atlan mechanically creates an Announcement for the KPI within the Atlan enterprise glossary.
Construct customized event-based automations with our AWS EventBridge integration
Many groups throughout organizations, from information observability to incident administration, are constructing automations that may set off actions primarily based on occasions. However there’s by no means been something accessible for metadata administration to create these kind of event-driven use circumstances for metadata… till now.
Knowledge groups can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an occasion and lets customers eat and construct use circumstances on it, to create Atlan metadata occasions into an EventBridge account.
Listed here are some examples of how this would possibly work in Atlan:
- Possession alerts: Get notifications in Slack when there’s a change within the possession of an asset.
- Propagation and classification: If somebody marks a discipline as PII in an upstream information supply, mechanically create a masking coverage to alter all associated fields downstream.
- And numerous extra use circumstances, similar to notifications round schema adjustments on property in Atlan for information engineering, login/logout occasions for safety groups, or triggering a Fivetran enrichment occasion to kick off an Atlan workflow.
—
💡 To be taught extra about the entire releases for 2022, take a look at shipped.atlan.com. You can even subscribe to our Product Updates publication for all the newest information.
💡 To take a look at Atlan for your self, go to our product tour.
💡 Prepared to start out utilizing these new options? Attain out to our Gross sales crew or your Buyer Success Supervisor to learn how.
