Visitor weblog by Katarzyna Bodzioch-Marczewska, Options Architect at Brainly
Knowledge governance is a important facet of any group, and it turns into much more vital in a distributed mannequin (examine The Brainly Mannequin right here), the place groups are unbiased and have their very own knowledge. In such a state of affairs, the discoverability of information turns into a big problem, as groups want a method to share details about the info with different groups. With our groups rising quickly and in a distant setting, when every workforce has and owns their knowledge silos, it may be troublesome for different groups to seek out and entry the info they want.
To handle this problem, Brainly determined to implement a knowledge catalog.
The necessities we gathered included the next:
- Metadata of all of our knowledge belongings in a single place (S3, Tableau, Redshift, Snowflake, BigQuery)
- Making our knowledge belongings discoverable (easy and broad search capabilities — to have the ability to discover related knowledge shortly throughout all of our belongings, together with their context)
- Allow collaboration and belief (collect tribal data of varied groups in a single place)
- Scale back dependencies between enterprise, analysts, and engineers (giving everybody easy accessibility to documentation and the flexibility to seek out data-related solutions on their very own)
- Potential to indicate the place the info comes from (visible lineage of dependencies between the info objects and the way the info flows all through the group)
After evaluating numerous distributors and going by means of a number of Proofs of Idea, we selected Atlan as our knowledge catalog. The principle causes behind that selection embody:
- Desired functionalities had been working as we anticipated
- The instrument was very intuitive and easy to make use of
- All of our knowledge tech might be built-in
- Excellent assist from the seller
- Affordable value
However as we all know, instruments themself will not be fixing any issues… We built-in all of our belongings into Atlan… And that was the place the attention-grabbing half started…
As soon as we had the technical metadata in, we would have liked to deal with the context. And to seek out and acquire it, we would have liked (and nonetheless want) a change within the firm tradition among the many Knowledge Folks — to understand the worth of the info asset’s documentation as a part of the info product itself.
As a way to make that shift, we carried out a gamification plan to interact groups and create higher consciousness of the significance of documenting knowledge belongings. By way of this initiative, we had been in a position to recover from 200 tables documented and shared throughout groups. The gamification plan concerned establishing a leaderboard, the place groups may earn factors for documenting their knowledge belongings and sharing data in regards to the knowledge. This created a pleasant competitors and helped to lift consciousness in regards to the significance of information governance. We bought good prizes for the winners of the competitions, together with t-shirts that, by the way in which, turned legendary after a number of months.
However that was not sufficient. We discovered that the important thing to profitable knowledge governance is evident possession. Wherever the possession of information was clear, groups had been extra engaged and keen to doc and share their knowledge belongings. Nonetheless, in areas the place possession was unclear or blurry, the documentation remained poor. This highlights the significance of building clear roles and duties for knowledge possession and entry inside a company.
As we’re on our method to undertake Knowledge Mesh (examine our journey right here), we plan to handle these points throughout our migration to Snowflake. Knowledge Mesh is a cultural and technical idea that goals to decentralize knowledge administration and allow groups to personal and function their very own knowledge providers. By adopting a extra distributed strategy to knowledge possession and entry, we hope to enhance knowledge discoverability and governance throughout Brainly.
In conclusion, implementing a knowledge catalog and a gamification plan helped our firm enhance knowledge discoverability and governance. Clear possession and clear roles and duties for knowledge administration are essential. As we’re migrating to Snowflake, we’ll proceed to enhance our knowledge governance and make it possible for groups can simply entry and share knowledge throughout the group.
Keep tuned for updates on our progress.
Due to Brainly for scripting this wonderful article! 💙
This text was initially printed by Katarzyna Bodzioch-Marczewska on the Brainly Know-how Weblog.
