Should you’ve been on the web not too long ago, you in all probability noticed how OpenAI’s ChatGPT is taking the world by storm. From writing books and designing rooms to debugging code and explaining information ideas, it appears that evidently automation and AI can now reply any query you ask.
Compared, metadata can really feel like residing within the Stone Age. Historically, it’s a tiresome, guide course of that wants numerous human intervention from information leaders, stewards, engineers, and analysts alike. In our newest webinar, a ballot confirmed that 60% of attendees spend over 3 hours per week on guide information duties.
At Atlan, one in every of our objectives is to automate wherever attainable. How can we decrease the mundane and maximize the affect information groups can have on the enterprise?
In final week’s Atlan Activate, our quarterly product webinar, we launched new automation options to superpower your information. Listed below are the highest 5 options that may aid you map context throughout your information property and restrict the repetitive guide work that slows information groups down.
TL;DR: 5 new options in Atlan you must find out about
- Metadata Playbooks for rule-based actions: Like Zapier for information, that is the primary low-code/no-code metadata automation for information groups.
- Atlan + AWS EventBridge event-based actions: Create production-grade, event-driven automations for the world of metadata, reminiscent of alerts when possession adjustments or auto-tagged classifications.
- Profiling and Reputation Insights: Use new column-level profiling, reputation, and utilization metrics to evaluate information’s high quality, discover essentially the most broadly used queries, determine prime customers, and extra.
- Atlan to GitHub integration: Carry metadata proper to GitHub to attenuate danger and improve transparency earlier than any adjustments are made to your information.
- Trident AI powered by GPT-3: Say goodbye to guide documentation with more and more clever automated descriptions, enterprise phrases, READMEs, and extra.
Metadata Playbooks: Introducing Zapier to your information property
Create rule-based bulk automations at scale to routinely deprecate unused information belongings, assign possession, report failures or bulletins, and extra.
One of many widespread questions we get from information groups is, “How can we automate our metadata?” Whereas different groups like advertising or gross sales can do action-based automation at scale with instruments like Zapier or Salesforce, information groups don’t have the bandwidth to code customized automations for every various use case. Why can’t there be a Zapier for metadata?
That’s the reason we have now developed the primary low-code/no-code metadata automation for information groups. With Atlan’s Playbooks, customers can now create rule-based automations at scale. These can drive limitless use instances throughout your group. Listed below are a couple of examples:
- Deprecate belongings: Mark any belongings that haven’t been queried within the final 30 days as deprecated.
- Add/change possession: For Salesforce belongings which can be lacking an proprietor, add RevOps because the proprietor.
- Report failed belongings: Publish an inventory on Slack of each desk and dashboard with a “failed” Airflow standing.
- Defend delicate information: Connect GDPR customized metadata on all belongings tagged as “PII”.
- Flag upstream alerts: Notify downstream homeowners when an upstream asset is tagged with a warning or announcement.
Atlan + AWS EventBridge: Construct customized, event-based metadata automations
Create production-grade, event-driven automations for the world of metadata, reminiscent of alerts when possession adjustments or auto-tagged classifications.
Low-code/no-code, rule-based automations like Atlan playbooks are nice, however some information groups don’t wish to be restricted by guidelines. They wish to construct their very own automations for repetitive actions.
Different groups, from information observability to incident administration, are constructing automations that may set off actions primarily based on occasions. For instance, many entry administration groups use Okta to watch system log occasions for suspicious exercise and automate actions to mitigate dangers. Buyer engagement groups use Salesforce to create occasions that enrich assist instances with order information from clients. However what about metadata administration? There’s nothing obtainable to create these kind of event-driven use instances for metadata… till now.
We’re excited to announce that information groups can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an occasion and lets customers eat and construct use instances on it, to create Atlan metadata occasions into an EventBridge account.
Listed below are some examples of how this may work in Atlan:
- Possession alerts: Get notifications in Slack when there’s a change within the possession of an asset.
- Propagation and classification: If somebody marks a subject as PII in an upstream information supply, routinely create a masking coverage to alter all associated fields downstream.
- And numerous extra use instances, reminiscent of notifications round schema adjustments on belongings in Atlan for information engineering, login/logout occasions for safety groups, or triggering a Fivetran enrichment occasion to kick off an Atlan workflow.
Leverage Profiling and Reputation Insights to construct context and belief
Use new column-level profiling, reputation, and utilization metrics to evaluate information’s high quality, discover essentially the most broadly used queries, determine prime customers, and extra.
Having the ability to profile your information is highly effective. It builds provides the context and insights that folks want to grasp, use, and belief in information, regardless of whether or not they’re an information analyst or enterprise person.
That’s why a brand new Profiling characteristic is now obtainable inside Atlan. With a variety of recent metrics, customers in every single place can construct belief within the information they’re consuming.
- Enterprise metrics: Common, most, minimal, imply, row rely, sum, and extra.
- Superior metrics: Duplicates, frequencies, lacking information, uniqueness, customary deviation, and extra.
We’ve additionally added new Utilization and Reputation options to assist information groups handle their belongings. Firms can use these new metrics to avoid wasting prices on cloud information warehouse spending, determine essentially the most used belongings, discover folks with essentially the most context on an information asset, and extra.
- Asset question logs: The variety of occasions an asset has been queried (asset reputation), when it was final up to date, and so on.
- Person question logs: Prime customers, most up-to-date customers, who final queried an asset, and so on.
- Price and efficiency optimization logs: Which queries are slowest, which queries are most costly, and so on.
Atlan + GitHub: Allow information contracts by bringing metadata into your information creation course of
Carry energetic metadata proper to GitHub to attenuate danger and improve transparency earlier than any adjustments are made to your information.
An enormous dialogue these days is shifting information “to the left” — that’s, shifting vital information processes and checks nearer to when belongings are created, quite than when they’re distributed. This concept is a part of the information contracts debate, which has highlighted how vital it’s to enhance reliability and value between the info producers and customers.
Information contracts can play a important function in information pipeline execution, validation of information varieties, versioning, and extra. As an example, if an information engineer makes adjustments to a dbt mannequin, they could unknowingly be affecting tens or tons of of individuals, tables, or dashboards. How can we decrease this danger whereas additionally offering transparency for the info engineer and information client?
By bringing the ability of energetic metadata from Atlan to GitHub, information engineers can now entry all of the context they should decrease the chance for information customers.
Right here’s an instance of what this appears like: say that you just’re an information engineer, and also you’ve created a pull request. When the GitHub motion runs, it should routinely create an inventory of all downstream belongings that can be affected by this request — earlier than you make the change. From there, you’ll be able to attain out to customers prematurely, or analysis and take a look at the belongings to see how they could be affected. Because the saying goes, “prevention is healthier than remedy”.
Automate metadata with Trident AI powered by GPT-3
Say goodbye to guide documentation with more and more clever automated descriptions, enterprise phrases, READMEs, and extra.
An current characteristic in Atlan, Trident makes metadata enrichment enjoyable and straightforward by offering solutions for brand spanking new descriptions, homeowners, phrases, and classifications. Whereas this has been extraordinarily profitable — in truth, one-third of all description updates on Atlan up to now had been made utilizing Trident — our clients have been asking for extra. Extra AI and ML, that’s.
Introducing Trident AI, powered by GPT-3 — the powers of Trident mixed with the intelligence of GPT-3. Presently underneath improvement, this characteristic will assist use instances like:
- Creating descriptions for column names
- Creating descriptions for widespread enterprise phrases
- Writing a README for a enterprise time period
Earlier than, Trident would offer a suggestion for this stuff, which you could possibly then apply if you happen to preferred it. Now, with Trident AI, if you happen to don’t just like the suggestion, merely ask Trident AI for a change. Just like ChatGPT, Trident AI will present a extra sturdy suggestion primarily based on the ability of AI and GPT-3.
💡 If you wish to be taught extra, try the total recording of Atlan Activate: Supercharged Automations to Map Your Complete Information Property. It’s also possible to subscribe to our Product Updates publication for all the newest information.
💡 Prepared to start out utilizing these new options? Attain out to our Gross sales crew or your Buyer Success Supervisor to learn the way.
