Driving Self-service, Safe, Reliable Knowledge with Atlan
- Swapfiets, the world’s first bicycle subscription service, sought to launch a governance program to satisfy rising inside demand for information
- Selecting Atlan, Swapfiets launched their governance program with a self-service fashionable information catalog, providing metrics and supply definitions throughout six enterprise domains
- Having efficiently launched the catalog, Swapfiets’ information staff now routinely runs governance committees to strengthen cross-functional collaboration, and is driving a tradition of information savviness by means of coaching and enablement
Based within the Netherlands in 2014, Swapfiets is on a mission to make cities extra livable, introducing the world to the idea of “Bicycle as a Service” with a month-to-month subscription to a bicycle with restore companies included. Now with over 270,000 subscribers throughout eight international locations, Swapfiets’ achievement of sustainability targets, and its continued development, depends upon dependable, simply accessible information.
“I’m actually obsessed with information. Knowledge is so numerous, and it has not solely a technical half, but additionally a governance, or extra summary half, as properly,” shared Lisa Smits, Supervisor of Swapfiets’ Knowledge & Analytics staff. “That’s how I got here to Swapfiets, and that’s how I got here to guide this staff.”
Drawing upon a various skilled background spanning jobs from provide chain administration to pricing evaluation, and together with the assistance of her staff and colleagues, Lisa has helped shepherd in a brand new period of information savviness and collaboration throughout Swapfiets, constructing and rising a Knowledge Governance follow, and making use of best-of-breed expertise like Snowflake and Atlan.
Knowledge is a crucial a part of measuring the place we are actually, and what we have to do to develop additional. We now have initiatives like optimizing the elements we use and being extra sustainable, and that’s the place my staff, immediately and not directly, contributes to the entire firm.”
Lisa Smits, Knowledge Analytics Supervisor
As soon as structured as one giant staff, Swapfiets has since damaged up its information perform into three specialised elements. First is Knowledge Engineering, accountable for creating and sustaining Swapfiets’ information platform and driving information high quality and availability. Subsequent, Knowledge Science creates complicated information merchandise by means of superior modeling, equivalent to demand forecasting and buyer lifetime worth. Lastly, the analytics staff led by Lisa is accountable for Swapfiets’ BI environments and activating information merchandise. Powering this information staff evolution is a shortly modernizing information stack, atop Snowflake as their information warehouse.
“We migrated to Snowflake. It’s extra versatile to scale up and down, it takes so much much less time for our engineers to arrange, and permissions administration was an enormous factor that was simpler to do. This transitioned us into being a extra mature information staff,” Lisa shared.
Swapfiets additionally adopted Fivetran, automating the pipelines as soon as run on guide scripts, and dbt, using their semantic layer as a decentralized metrics retailer. Rounding out this modernization was the adoption of Airflow and Tableau.
“We matured in a single large step and actually specialised,” Lisa defined. “We targeted on having instruments to purchase, versus construct, making a aware determination on paying a bit extra, however saving a lot time in the long run, leading to decrease prices.”
Whereas Swapfiets was maturing considerably throughout technical and organizational domains, tribal information about their information property endured. Initially, the staff created a Google Sheet with an inventory of 25 metrics and definitions. However with out a easy interface to entry these metrics, or perceive the context round them, the quantity of questions posed to the Knowledge & Analytics staff elevated.
“We bought plenty of questions from individuals, like ‘How is that calculated?’ or ‘We see this quantity on this dashboard, however on this dashboard, it’s utterly totally different. How does that work?’,” Lisa defined. “There was no possession, no transparency, and no central place for us to write down down phrases, or clarify what metrics meant and the way they have been calculated. It took plenty of time to reply these ad-hoc issues.”
Normally, there was a wonderfully legitimate cause for variations between dashboards and property, however absent documentation, belief was eroding, main Lisa and her staff towards a brand new give attention to information governance.
To lose belief is straightforward, however to achieve that belief again is difficult. That’s once we began taking this critically. We needed to have a high-quality, reliable BI atmosphere, and that’s why we began trying into our choices for governance, and into Atlan.”
Lisa Smits, Knowledge Analytics Supervisor
Lisa and her staff started by trying to find glossary options to be able to enhance the transparency of their information property and cut back the quantity of questions from Swapfiets’ information customers. However as she and her staff discovered extra in regards to the Energetic Metadata Administration market, their expectations for the worth that such a platform might drive grew.
With a versatile person interface, information customers might uncover, perceive, and apply information in a self-service method, and with automated lineage, Swapfiets’ information practitioners might higher perceive how information flowed by means of their modernizing stack and the way it was being utilized. And with Atlan Insights, a metadata-based question builder, much more of Lisa’s colleagues can be empowered to make the most of information.
Slightly than only a glossary of phrases, Lisa and her staff envisioned a one-stop store for his or her colleagues who used information every single day, and for many who ought to, however hadn’t but.
You’ve gotten one place the place you may retailer the phrases, the place you may take a look at your whole information property, the place you’ve the descriptions, and the place you may see the homeowners. You probably have a query, you may instantly ship it by way of Slack. It’s all linked in a single place, and I feel having that full image, not solely governance but additionally self-service, was what actually stood out to us about Atlan.”
Lisa Smits, Knowledge Analytics Supervisor
After selecting Atlan, Lisa and her staff started by integrating Redshift (since migrating to Snowflake), dbt, and Tableau, shortly enabling lineage throughout their most crucial applied sciences. With their information property successfully crawled, and information property seen, they started the method of assigning homeowners and making a enterprise glossary.
Utilizing the definitions they’d already agreed upon with their enterprise colleagues and had saved in Google Sheets, Lisa’s staff enriched a baseline of a brand new enterprise glossary, then labored with their colleagues in Knowledge Engineering to doc the sources of every asset. Whereas utilizing pre-existing definitions saved useful time, the method of enriching information property and documenting their supply was nonetheless a cumbersome one. In Atlan Playbooks, rule-based bulk automations for duties like documentation, Swapfiets discovered a solution to considerably speed up enrichment.
Using Playbooks, Swapfiets constructed an automation guaranteeing that every time a brand new asset is launched into their metrics retailer, its pre-existing descriptions from upstream programs are mechanically handed by means of to Atlan. These descriptions are then verified, fairly than written from scratch, considerably accelerating their documentation rollout.
“Content material creation was plenty of work, however whenever you launched Playbooks, it was wonderful for us,” Lisa shared. “We now have 125 metrics, and it improved our time spent on documentation.”
Furthering their use of Playbooks, Lisa’s staff shortly moved to safe these newly accessible information property, mechanically tagging delicate or personally identifiable data, and utilizing personas inside Atlan to make sure delicate information was accessible solely to these approved to view it, or contribute to documenting it.
Searching for to deepen collaboration with material specialists, and to make information governance a routine a part of the best way Swapfiets operates, Lisa attended an Atlan Masterclass with WeWork to learn the way greatest to construct and keep a enterprise glossary.
Lisa’s staff maintains a single enterprise glossary, divided into enterprise domains that align material specialists with companions on the information staff. Six such enterprise domains exist inside the glossary, throughout groups like Operations, Human Assets, Product, and Client Direct. And for every of those six enterprise domains, Lisa’s staff organized cross-functional governance committees composed of information analysts, information homeowners, and information stewards that meet on a month-to-month foundation to debate their progress.
“In that month-to-month assembly, we give attention to three governance subjects, that are possession, documentation, and high quality. For documentation and possession, Atlan is basically useful. It’s additionally useful for high quality to seek out out the place within the information lifecycle high quality points come up,” Lisa shared. “In the course of the assembly, we write out motion factors and decide to duties like verifying extra phrases.”
To trace progress towards these commitments, Lisa’s staff makes use of Atlan’s reporting performance as a heads-up show of what number of phrases and information sources have been verified or enriched, and at what tempo. Additional simplifying the method, material specialists use filters in Atlan to show the phrases they’re accountable for, giving them an inventory of related phrases and information sources they’re tasked with enriching, or to mark unverified however necessary phrases and information sources as draft for future enrichment.
Whereas Swapfiets continues to seek out new methods to use Atlan and develop their governance follow, the worth yielded from self-service and their metrics catalog, alone, has been important.
“Colleagues continuously ask ‘What does this imply?’ or ‘How do you outline this metric?’ With an built-in workflow in Slack, we ship them to the Atlan glossary to self-serve their query. It’s been a good time saving for us,” Lisa defined.
For questions that may’t be answered by self-service, the Knowledge & Analytics staff advantages from clear, assigned possession for every information asset, making it easy to direct inquiries to material specialists, or to tag them immediately right into a Slack thread.
By standardizing the method for outlining metrics, and automating their preliminary enrichment utilizing Atlan Playbooks, Lisa’s staff has saved many hours of effort, up to now. However extra noteworthy to their staff is the truth that they’re now assured their course of will likely be adopted every time a brand new metric is launched.
The entire standardization of all of the metrics saves between 10 and half-hour for each metric. So in the event you multiply that by 125, it’s a pleasant quantity, nevertheless it’s not like, ‘Oh, we saved 5 weeks this yr.’ It’s simply good that we don’t have to consider this anymore. It’s not in our course of anymore whereas we create a metric. It’s already taken care of.”
Lisa Smits, Knowledge Analytics Supervisor
With a catalog of metrics in place, and a sustainable course of to additional enrich it, Lisa and her staff started a sequence of packages designed to extend information savviness throughout the Swapfiets group and additional cut back the burden of service on their group.
Knowledge Literacy Program
Foundational to how Swapfiets allows self-service is their Knowledge Literacy Program. A part of this program is a compulsory coaching for Atlan & Tableau. This teaches new Swapfiets workers about Atlan and its potentialities, and learn how to make the most of Swapfiets’ current Tableau dashboards. When full, customers are then given entry to Atlan and Tableau. On this program, the Analytics staff additionally gives a SQL-Atlan coaching for his or her Knowledge Champions.
For self-service questions, we like educating individuals the fundamentals of querying. And with Atlan, you’ve the visible question builder. Even when persons are not tremendous information literate, they will mess around with the question builder and it’s a lot simpler to grasp than one thing like ‘SELECT * FROM’. If they should filter, it’s truly referred to as filter as an alternative of the technical ‘WHERE’ assertion. It simply speaks to them much more.”
Lisa Smits, Knowledge Analytics Supervisor
With the mix of the Knowledge Literacy Program and Atlan Insights, the usage of Tableau is now reserved for long-term information merchandise and visualization, with the majority of customized querying now achieved utilizing Atlan, doubtlessly decreasing licensing prices over Tableau and decreasing the technical limitations to information evaluation with a visible question instrument.
Knowledge Champions
For Swapfiets workers with a better technical aptitude or ambition, or for information stewards accountable for updating their glossary, a extra superior stage of coaching dives deeper into visualization in Tableau, and learn how to discover property and translate them into queries in Atlan.
Utilizing fine-grained entry controls in Atlan, Lisa’s staff ensures these Knowledge Champions are capable of conduct self-service evaluation at a deeper stage, with out the chance of accessing and making use of improper or restricted information.
“We don’t have any PII in Tableau anymore, as a result of entry administration is less complicated in Atlan. Now, it’s clear what number of property have PII information, who can entry these, when they’re used, and who queries which tables,” Lisa shared.
Extract Champions
The most recent addition to Swapfiets’ democratization efforts are Extract Champions, data-savvy colleagues from every enterprise area that are actually empowered to assist Lisa’s staff when extract requests for information happen.
In Swapfiets’ information help channel on Slack, customers provoke a workflow to request information that’s unavailable in Tableau, or too superior to self-service. With one click on of an emoticon, Extract Champions from the related information area are tagged, settle for the duty, and start querying the information utilizing Atlan Insights for supply to the requester.
“Once they’re achieved, they simply click on the ‘End’ button on the ticket, and we don’t even have to take a look at the ticket. So it’s a win-win. I’m so passionate about this, it’s been nice,” Lisa shared.
Furthering the worth Extract Champions yield from Atlan, Lisa and her staff realized the repetitive nature of some requests, and labored to automate them utilizing Atlan Scheduled Queries.
“Atlan helped us schedule extracts to individuals outdoors of Atlan, which helped an Extract Champion who’s in control of communication with B2B clients,” Lisa defined. “And now he’s scheduled his guide work, and Atlan automated what he would normally share by way of e mail. He doesn’t have to achieve out to us anymore.”
Sustaining the unimaginable momentum of Swapfiets’ Knowledge Governance Program is a give attention to change administration, fixed communication, and driving cultural change.
Governance Committees proceed to satisfy on a month-to-month foundation, guaranteeing every enterprise area retains their enrichment and possession tasks top-of-mind. Updates and newsletters are routinely despatched to stakeholders, educating them on subjects like high quality, possession, safety, or accessibility, and updating them on essential subjects like how documentation is progressing towards their targets.
However essentially the most noteworthy change, past how shortly Swapfiets’ expertise has modernized and the way properly their Knowledge Governance Program has been stewarded, is a cultural shift. The place Lisa’s staff was as soon as accountable for answering questions on information and connecting the dots for information customers, that possession is shifting. Stewards and Knowledge House owners from throughout their group now really feel empowered to area questions on their information, and accountable for the standard and availability of those property.
We now have the information help channel, and we might reply most questions, however we gained’t anymore. We tag the information proprietor or steward and they’re going to look into it. This leads to extra motivation to search for sustainable options and take into consideration Knowledge Governance after they’re taking part in their half contributing to Swapfiets’ mission. I feel it’s created a tradition the place information is one thing that impacts all of us, and never simply the information staff. That’s been the most important change we’ve achieved, up to now.”
Lisa Smits, Knowledge Analytics Supervisor
Picture by Ernest Ojeh on Unsplash
