Bettering Developer Productiveness and Supply Velocity with Atlan
The Energetic Metadata Pioneers collection options Atlan clients who’ve lately accomplished a radical analysis of the Energetic Metadata Administration market. Paying ahead what you’ve discovered to the following knowledge chief is the true spirit of the Atlan neighborhood! So that they’re right here to share their hard-earned perspective on an evolving market, what makes up their trendy knowledge stack, modern use instances for metadata, and extra.
On this installment of the collection, we meet Abraham Tom, Director of Knowledge at Generate Capital, who shares his 20-plus-year historical past in Knowledge & Analytics, why Knowledge Governance issues a lot to his group, and the way he and his staff use Atlan to enhance developer productiveness.
This interview has been edited for brevity and readability.
Would you thoughts introducing your self, and telling us the way you got here to work in Knowledge & Analytics?
I’ve been a knowledge skilled for over 20 years, constructing out Netezza, Oracle and SQL server warehouses, shifting on to Hadoop, and now Cloud techniques. I’ve labored on all points of information, starting from motion, governance, engineering, science (earlier than it was attractive), BI analytics, graphical illustration, and specializing in ground-up implementations of latest knowledge warehouse initiatives. My motivation is exhibiting to executives why data-driven choice making is effective, then constructing a roadmap to realize that state.
May you describe Generate Capital, and any vital initiatives you and your staff are engaged on?
Generate Capital PBC is within the enterprise of re-building the world to avoid wasting humanity. We construct, personal, function, and finance inexpensive and dependable infrastructure options for clear vitality, water, waste, transportation, and sensible cities’ infrastructure applied sciences. Knowledge Governance must be the cornerstone of our knowledge warehouse program so that everybody that must be knowledgeable or that should decide has the precise info to take action.
What does your stack appear to be, and the way has that developed over time?
I began because the second tech individual within the group. Generate Capital has the distinctive problem the place we get knowledge from totally different strains of enterprise and coalescing all that knowledge into an combination view. This required shortly devising an answer that may scale, and implementing a set of instruments that allowed for that. Our main repository was Snowflake as a cheap scalable platform, we use Streamsets as our main knowledge ingestion, dbt as our transformation engine, and Tableau as our main visualization and supply know-how. We additionally deal with a variety of the information integration between our ERP and CRM techniques, in addition to being the hub for knowledge transport to different techniques.
How did you come to search for an Energetic Metadata Administration resolution? Why did Atlan stand out?
Initially, I needed to discover a easy knowledge dictionary device. Whereas dbt has lineage capabilities and a few documentation constructs, it was restricted to what dbt is conscious of, and never all of the techniques we use. Atlan is a good resolution to not solely to accommodate our dictionary, however the lineage and glossary features have elevated our staff’s data in order that they’ll see the affect of their new growth.
What have you ever constructed with Atlan, to this point? What worth have you ever been in a position to drive?
Proper now, we’re utilizing Atlan as extra of a tech useful resource than a enterprise useful resource, and have discovered good worth.
Our product staff has develop into our first non-developer customers of the Atlan platform which has improved total ticket and specification writing. Tickets, function requests, and onboarding have all vastly improved. Our Product Managers can do analysis on how the information flows and may write very particular requests with higher acceptance standards, which has resulted in improved developer effectivity, much less back-and-forth with questions, and improved total supply, all of which elevated our velocity.
Now we have additionally built-in Atlan’s affect evaluation with our dbt pull requests as a GitHub merge dependency verify. This provides our builders consciousness of the downstream impacts of the adjustments they plan to merge earlier than the ultimate commit occurs. This has helped us guarantee any new function requests don’t adversely have an effect on current manufacturing gadgets. Normally, this requires senior sources to assessment code, and whereas we nonetheless require peer code assessment, that assessment is lots sooner, and everyone seems to be knowledgeable of the potential upstream change and affect.
As a part of our Governance initiative, we’re additionally beginning to leverage Atlan’s glossary operate as a seize mechanism for the enterprise definitions. Atlan assists us in tying that enterprise definition to the system dictionary.
Any last ideas?
Atlan is a good device for builders to see knowledge lineage and to assist them work out what must be finished and the way their work impacts downstream techniques. All of my builders repeatedly use this to know what they’re constructing. It additionally supplies different growth groups transparency of the work we do and the place their work finally ends up.
Photograph by Bastian Pudill on Unsplash
