How a fast-growing fintech improved GDPR compliance with Atlan in hours, not months
At a Look
- Tide, a UK-based digital financial institution with almost 500,000 small enterprise prospects, sought to enhance their compliance with GDPR’s Proper to Erasure, generally often known as the “Proper to be forgotten”.
- After adopting Atlan as their metadata platform, Tide’s information and authorized groups collaborated to outline personally identifiable info as a way to propagate these definitions and tags throughout their information property.
- Tide used Atlan Playbooks (rule-based bulk automations) to routinely establish, tag, and safe private information, turning a 50-day guide course of into mere hours of labor.
Tide, a mobile-first monetary platform based mostly within the UK, provides quick, intuitive service to small enterprise prospects. Information is essential to Tide, having supported its unbelievable development to now almost 500,000 prospects in simply eight years. However in monetary companies, information acutely presents danger and calls for cautious and fastidious safety of delicate monetary info. These dangers solely enhance as enforcement of GDPR will increase, with nine-figure fines levied in opposition to offending corporations in simply the previous couple of years.
Recognizing the immense alternatives introduced by information, Tide’s CEO, Oliver Prill, recruited Hendrik Brackmann to construct a knowledge science group. “The ambition at that time wasn’t a lot to construct a knowledge group. It was about the place we may use machine studying at Tide”, Hendrik shared, “nevertheless it rapidly turned clear which you can’t understand that in case you don’t have a knowledge platform.”
The journey towards information maturity was a frightening one. Initially reporting into the Finance group at Tide, the info platform group consisted of simply two workers. It turned Hendrik’s duty to develop not simply a sophisticated information science group, however to decide on the proper information platform know-how, and to suggest, construct, and scale information and reporting groups.
“We seemed very deeply into how our group ought to look,” mentioned Hendrik. “We made a variety of adjustments, from splitting roles between analytics engineers and analysts, to beginning a knowledge governance group.” And together with personnel development and a extra mature help mannequin to help Tide’s development, Hendrik ensured that his group was aligned to enterprise wants, delivering transformational options like a transaction monitoring system, help for income identification, and machine studying–powered danger scoring.
In simply 4 years, Hendrik grew the perform to a group of 67 throughout information engineering, analytics, information science, and governance. It was throughout this time of utmost development that Hendrik acknowledged room for enchancment: “We grew in a short time, and we noticed we weren’t as environment friendly as we thought.”
Whereas Tide’s information group had matured by leaps and bounds, as a regulated entity, compliance was a excessive precedence that demanded large effort and a focus. “The authorized group hardly ever spoke with the engineering capabilities. It was a bit remoted,” Hendrik mentioned.
Early Days of Information Governance
Recognizing that collaboration between authorized and technical groups had to enhance, Hendrik started looking for a knowledge governance professional. He met Michal Szymanski, who would grow to be Tide’s Information Governance Supervisor. “The preliminary thought was to rent Michal as a bridge to the privateness perform,” Hendrik remarked.
Michal joined Tide as a one-man group. “My scope of duties elevated quite a bit,” mentioned Michal. “I needed to cope with an unlimited array of challenges, ranging from understanding the place information governance may assist in such a corporation.” He started by trying to grasp his stakeholders’ wants. “I needed to begin by interviewing many individuals throughout totally different enterprise areas to grasp what they wanted.”
Based in 2016, Tide had little of the technical debt or legacy know-how that sometimes burdens conventional monetary companies organizations. Their information stack consisted of dbt, Airflow, and Snowflake, with Looker downstream as their Enterprise Intelligence (BI) layer. Whereas Tide had invested in the proper know-how, Michal realized that his colleagues discovered it obscure how information traveled throughout their stack.
Hendrik noticed this problem as a chance for development.
We needed to embed information safety and privateness into our operating processes, moderately than discussing it on the finish of initiatives.
Hendrik Brackmann
By combining Michal’s new governance perform, an understanding of information lineage, and customary definitions of information, they may obtain the collaboration they’d been lacking.
Hendrik and Michal started looking for an answer. Summarizing the trail ahead, Michal defined, “We wanted to have a platform the place we may put all such attention-grabbing info to assist customers navigate the info that now we have. So my first job was to establish a knowledge catalog.”
Including a Context Layer
After an intensive analysis of the market, Hendrik and Michal selected Atlan as their information catalog.
[Atlan] built-in seamlessly with all of our instruments, and we felt it was very straightforward to make use of.
Hendrik Brackmann
Beginning with just a few key downside statements, Tide applied Atlan to enhance information discovery, visibility, and governance within the brief time period, and democratize information entry and understanding in the long term. To start out, Hendrik ensured that Atlan was correctly built-in with their information stack, and was capturing all related metadata.
With Atlan, technical and non-technical customers may discover the proper information asset for his or her wants, rapidly and intuitively, decreasing the time it as soon as took to search out, discover, and use information throughout instruments like Snowflake, Looker, and dbt. Utilizing Atlan’s information glossary and metrics, Tide started to get pleasure from higher context surrounding their information domains, which set the stage for standardizing classifications of delicate information like personally identifiable info. And lastly, Atlan’s automated lineage added transparency so Hendrik’s group may perceive the place information got here from, the way it reworked all through the info pipeline, and the place it was in the end consumed — one thing they couldn’t do earlier than.
Tide grew to make use of Atlan to help a big selection of customers and enterprise models, from Authorized and Privateness, to Information Science, Engineering, Governance, and BI colleagues. With improved context, greater belief in information, and democratized entry to Tide’s information, Hendrik started to contemplate new use instances: “We have been trying to establish how we may drive course of efficiencies in our analytics and engineering groups.”
With a 360-degree view of their information property, the stage was set for Hendrik’s group to construct broader, extra mission-critical options.
The GDPR Problem
After utilizing Atlan to raised perceive their information property, Hendrik’s group was able to help an important use case.
“Like each firm, we have to be compliant with GDPR,” mentioned Michal. And a key element of GDPR compliance is the proper to erasure, extra generally often known as the “Proper to be forgotten”, which provides Tide’s prospects throughout the European Union and the UK the proper to ask for his or her private information to be deleted.
Tide’s information group understood these obligations nicely, however the technique of compliance was troublesome.
Our manufacturing help group had a script, and each time somebody needed to delete information, they might undergo our back-end databases and delete private information fields.
Hendrik Brackmann
And whereas the help group’s script managed a major quantity of information deletion, guide effort was wanted to search out and delete information that persevered elsewhere in secondary methods that had native projections of the non-public information fields. Michal defined, “The method was not capturing information from all the brand new sources that stored showing within the group, simply the important thing information supply.”
Complicating this problem was a scarcity of shared definitions of non-public information, with differing opinions on what constituted personally identifiable info throughout organizations from Authorized to IT. This meant that finishing the “Proper to be forgotten” course of concerned regularly re-litigating definitions.
Whereas Tide was doing its greatest to adjust to GDPR, as its know-how stack and structure grew extra sophisticated, new services and products have been launched, and prospects elevated over time, the compliance course of took solely extra effort and time.
Automating this course of turned a precedence. In a super world, when a buyer exercised their proper to be forgotten, a single click on of a button would routinely establish and delete or archive all information concerning the buyer in accordance with GDPR. Immense guide effort, and the chance of delays or human error, could be eradicated.
That’s precisely what Hendrik set his group to do.
Driving Frequent Understanding
Earlier than pouring assets into fixing the issue, Hendrik and Michal wanted to justify the hassle to their colleagues. “It required element to be introduced to senior leaders as a way to determine that we might make investments money and time in fixing such an issue,” mentioned Michal. “That was essential, as a result of nobody actually desires to speculate until it means some enhance of income or value financial savings. We mentioned we will keep away from fines and we will be certain the corporate is dealing with private information at a excessive stage.”
The case was so robust that fixing the issue turned a group OKR. With their objective in hand, Hendrik requested his group to grasp the issue in higher element: “The very first step was to determine the place we had this type of information, then figuring out possession.”
In his position as a bridge between the info group and its enterprise counterparts, Michal labored with the Authorized group to ascertain what did or didn’t represent private information. And to make sure the groups have been collaborating easily, Hendrik established a cross-functional working group. “It’s simply getting the proper folks in a room after which getting them to speak,” mentioned Hendrik. “Our greatest contribution was bringing folks collectively and preserving them targeted.”
By bringing technical groups and area specialists collectively, Hendrik ensured each voice was heard and that his group remained targeted on collaboratively delivering worth, moderately than arcane technical ideas. Recalling an instance of how strongly the group collaborated, Hendrik shared, “We had our privateness lawyer on the decision after we mentioned structure. He may reply any questions that may come up immediately.”
With these definitions in hand, Hendrik and Michal started evaluating them in opposition to current documentation and processes. “There have been a few locations the place totally different folks have been making an attempt to listing private information. So the entrance finish group did this, and the again finish group did that. Some product managers did the identical, they usually weren’t constant,” Michal defined.
Additional, whereas his colleagues had a superb command of their information, they usually had hassle speaking the info’s definitions — a key a part of good information governance. Oftentimes, column names would function definitions. “In lots of instances, it was not exact sufficient,” mentioned Michal.
With clear misalignment, Tide wanted extra exact documentation and course of. Atlan introduced a simple option to remedy this problem. Hendrik’s group would take what they realized from their analysis (together with new definitions of non-public information, alternatives for enchancment, and homeowners of information) and doc it as soon as and for all of their catalog.
We mentioned: Okay, our supply of reality for private information is Atlan. We have been blessed by Authorized. Everybody, any more, may begin to perceive private information.
Michal Szymanski
From 50 Days to five Hours
With their information property built-in with and made navigable by Atlan, Tide used automated lineage to rapidly and simply decide the place personally identifiable information lived, and the way it moved via their structure. Beginning by figuring out the columns and tables the place private information persevered, the group then used Atlan to trace it downstream.
Michal defined simply how worthwhile lineage was to the group: “This was very helpful. It confirmed us how a lot information now we have in our information warehouse, after which we may additionally extrapolate this to the upstream sources of Snowflake. We knew we had it in Snowflake as a result of it’s coming from this and this database. So we knowledgeable the groups that they’d a number of private information and we would have liked to provide you with a design.”
Subsequent, Hendrik’s group determined to correctly tag personally identifiable information, and add their newly decided definitions. Belongings saved in Snowflake, like account numbers, electronic mail, cellphone numbers, and extra, could be searchable, however correctly secured and masked within the Atlan UI.
Whereas worthwhile, the guide effort concerned was daunting. Michal defined, “Individuals must go into the databases and attempt to translate my listing of non-public information components. There have been 31 components to search out in our databases, and now we have greater than 100 schemas, every with between 10 to twenty tables. So it could be a number of work to establish it.”
Making assumptions about which schemas would possibly comprise personally identifiable info may save time, however this wasn’t an possibility. The chance concerned meant Michal and his group needed to be exact, looking out and tagging location-by-location, or it could show expensive.
If we have been very diligent and did it for each schema, then it could most likely be half a day for every schema. So half a day, 100 occasions.
Michal Szymanski
After discussing this scope with the Atlan skilled companies group, Michal realized about Playbooks, a characteristic distinctive to Atlan. As a substitute of spending 50 days manually figuring out after which tagging personally identifiable info, Tide may use Playbooks to establish, tag, after which classify the info in a single, automated workflow.
Hendrik’s group was able to spend 50 days of effort on a job that might clarify enhancements to Tide’s danger profile. However after integrating their information property with Atlan and driving consensus on definitions, they used Playbooks’ automation to perform their objective in mere hours. Michal defined, “It was mainly just a few hours to debate what we would have liked.”
What’s Subsequent?
After saving almost 50 days of labor, Tide can now make additional enhancements to their course of, far earlier than anticipated.
Within the months to return, the group is constructing a microservices-based orchestrator to deal with requests from prospects about their private information. It would then be enhanced to anonymize information in accordance with GDPR requirements for de-identification and Tide’s information retention obligations as a regulated enterprise. Right here, too, Atlan has helped. Tide’s engineers can construct these options extra rapidly by referencing the data and lineage made doable by Hendrik’s group and Atlan.
I’d say I received nice help from the Atlan group, who have been with me on the entire journey. I’d have by no means thought of Playbooks. It was prompt in the proper method for the proper use case.
Michal Szymanski
As for Hendrik, his group’s accomplishments imply the belief of his imaginative and prescient from the very starting of his time at Tide. “Over the past 12 months, we’ve managed to maneuver nearer to the enterprise. Having the ability to create this type of organizational change is one thing that I really feel very happy with.”
With a major win for his group in hand, enabled by the proper know-how and guided by the proper technique, Hendrik shared his recommendation for fellow information leaders. “Give attention to enterprise worth, and the precise worth you’re producing in your group moderately than discovering a course of everybody within the business follows and adopting the identical factor. Don’t attempt to do governance all over the place. Determine what information units are related to you, and concentrate on these ends.”
Be taught extra about Atlan’s Playbooks and different supercharged automation options from 2022.
Header photograph: Dan Nelson on Unsplash
