Activating and Governing a Rising Knowledge Platform with Atlan
The Energetic Metadata Pioneers collection options Atlan clients who’ve not too long ago accomplished an intensive analysis of the Energetic Metadata Administration market. Paying ahead what you’ve realized to the subsequent knowledge chief is the true spirit of the Atlan neighborhood! So that they’re right here to share their hard-earned perspective on an evolving market, what makes up their trendy knowledge stack, modern use circumstances for metadata, and extra.
On this installment of the collection, we meet Surj Rangi, Enterprise Cloud Knowledge Architect, Piyush Dhir, Senior Technical Lead, and Danni Garcia, Product Supervisor, at REA Group, the operator of main residential and business property web sites, mortgage brokering companies, and extra. Surj, Piyush, and Danni share REA’s evolving knowledge stack, their data-driven ambitions, and the factors and course of behind their selection of Atlan.
This interview has been edited for brevity and readability.
Might you inform us a bit about yourselves, your backgrounds, and what drew you to Knowledge & Analytics?
Surj Rangi:
I’m Surj Rangi, Architect in Knowledge Providers, and I’ve been at REA for 2 years now. I graduated in IT from the UK, then labored in quite a few consultancy corporations in Knowledge and Analytics and developed a powerful background in cloud platforms and knowledge structure. I migrated to Australia about seven years in the past, with 20 years of expertise in knowledge throughout varied industries together with Media, Telecommunications, Finance, E-commerce and Banking.
I joined REA and was very eager on the function that I used to be supplied and the staff I used to be coming into. What actually enticed me was working with an organization that had a startup mentality, and have been excited to push and ship outcomes. Beforehand, I’ve labored with massive banks the place there’s loads of paperwork and issues take time, and I used to be excited to see how issues work at a spot like REA.
Piyush Dhir:
I’m a Senior Technical Lead at REA. My journey goes again to school once I was ending my Bachelors in Software program Engineering and wanted to decide about what I wished to do subsequent.
I began as an Android developer again when it appeared like everyone’s subsequent factor was “What will be my subsequent Android challenge?” After I was doing that, I got here throughout SQL Server, studying how you must do operational modeling whenever you’re creating one thing like a front-end utility. That’s how I made my first step into knowledge. Since then, I’ve been working throughout quite a few completely different sorts of knowledge groups.
My first knowledge staff was a Knowledge Administration staff for a public firm in Australia. They have been ranging from zero, constructing an entire greenfield ecosystem for his or her knowledge utilizing the SAP merchandise. I spent about 5 years in that world, then moved into loads of small firms and large firms. I did a little bit of consulting, I labored for a financial institution within the center, after which lastly ended up at REA.
After I first joined a knowledge staff again in 2012, what actually stood out to me on the time was that knowledge was mentioned to be “the brand new oil”, and that Knowledge & Analytics have been going to be the subsequent large factor. Again then, some individuals began doing Machine Studying and enjoying round with R Studio, but it surely was by no means the “bread and butter” of any firm, simply a kind of “north star” sort of initiatives.
Instantly, now 10 years down the road, it’s turn out to be not solely the “bread and butter” of the corporate, but it surely’s a chance for monetization for lots of them, too. It’s good to see that transition taking place, and it’s been fascinating to observe.
Danni Garcia:
I’m a Product Supervisor in Knowledge Providers with a selected background in Knowledge research. I haven’t all the time been in Product. I’ve labored within the expertise business for nearly a decade now throughout many alternative areas and roles in each massive and small organizations, however I began out as a Knowledge Analyst.
Would you thoughts describing REA, and the way your knowledge staff helps the group?
Surj:
I feel it’s good to know that REA began in a storage in Australia within the early-to-mid ’90s, and since then the corporate has grown and scaled enormously throughout the globe. REA has a presence not solely in Australia, however Asia too and has robust ties with NewsCorp. We began by itemizing residential properties, and it’s grown from there to business properties and land, as effectively. We’ve additionally executed loads of mergers and acquisitions. For instance in Australia, we’ve purchased a agency referred to as Mortgage Selection that permits REA to be positioned not solely to promote listings, publications, and supply insights into property into the business in Australia, but in addition present mortgage dealer companies.
So if you wish to promote your property, REA offers the entire bundle. You’ll be able to promote your property, and should you want financing, we may also help you financial your subsequent funding.
We’ve gone by means of an extended journey, and have had a Knowledge Providers staff for an extended time period. The whole lot was decentralized, then it was centralized. Now it’s a little bit of a hybrid, the place we’ve got a centralized knowledge staff constructing out the centralized knowledge platform with key capabilities for use throughout the group, with decentralized knowledge possession. We are attempting to align with a Knowledge Mesh method when it comes to how we construct out our platform capabilities and adoption of “knowledge as a product” throughout the group.
We’re multi-cloud, each AWS and GCP, which brings its personal challenges, and we do all the pieces from ingestion of knowledge, event-driven structure to machine studying. We’re constructing knowledge property to share with exterior firms within the type of a knowledge market.
Danni:
Knowledge Providers exists to help all the inside strains of companies throughout our group. We’re not an operational staff, however a foundational one, that builds knowledge merchandise and capabilities to assist help groups to allow them to efficiently leverage knowledge for his or her merchandise. Our mission is to make it straightforward to grasp, shield and leverage REA knowledge.
Piyush:
I’ll add that over the past couple of years, REA has predominantly seen themselves as a listings enterprise. It’s nonetheless a listings enterprise, offering the very best listings info attainable out to clients and customers. However what’s occurred is that this wealthy knowledge evolution helps our enterprise turn out to be data-driven. Among the knowledge metrics you see on the REA web site and cellular utility are largely derived from the work that the group has put in to develop our Knowledge & Analytics and ML apply to drive higher choice making.
We have now loads of useful knowledge. There are loads of initiatives occurring now to develop the utilization of knowledge, and over the subsequent two years, we’ll develop our panorama and derive even higher outcomes for our clients and customers. to grasp, leverage, then showcase knowledge to our clients and their clients.
What does your knowledge stack appear to be?
Danni:
We have now a real-time ingestion platform referred to as Hydro utilizing MSK, which is a custom-built streaming platform. Then we’ve got our batch platform, which ingests batch knowledge utilizing Breeze, constructed on Airflow. Our knowledge lake resolution is BigQuery.
Piyush:
We have a look at ourselves as a poly-cloud firm, utilizing each AWS and Google Cloud Platform, in the intervening time.
From an AWS perspective, we’ve got most of our infrastructure workloads operating there. We have now EC2 cases and RDS operating there. We have now our personal VPC. We have now a number of load balancers.
From a Knowledge and Analytics perspective, nearly all of our workloads are in GCP. We’re presently utilizing BigQuery as a knowledge lake idea, and that’s the place most of our workloads run. We use SageMaker for ML, and there’s some groups which might be experimenting with BigQuery ML on the GCP aspect, as effectively. We even have a self-managed Airflow occasion, in order that’s our knowledge platform.
We’re presently within the technique of organising our personal event-driven structure framework utilizing Kafka, which is on AWS MSK.
Other than that, our Tableau entrance finish is used for reporting, so we’ve got each the Tableau desktop and the server model, in the intervening time.
Why seek for an Energetic Metadata Administration resolution? What was lacking?
Surj:
We have now an current open-source knowledge catalog that we’ve got been utilizing for just a few years now. Adoption has not been nice. As we’ve scaled and grown, we realized that we would have liked one thing that’s extra related for the trendy knowledge stack, which is the course that we’re going in the direction of.
There’s additionally a stronger push in our business towards higher safety of knowledge. We retailer loads of personally identifiable knowledge throughout the enterprise, and a few of our key methods we’ve got in Knowledge Providers are that we need to first perceive the information, shield it, then leverage it. We wish to have the ability to catalog our knowledge, and perceive how dispersed it’s throughout our warehouses, varied platforms, in batches, and streams.
We have now loads of knowledge, e.g. we’ve received over two petabytes of knowledge in GCP BigQuery alone. We wish to have the ability to perceive what knowledge is, the place it’s put collectively, and apply extra rigor to it. We have now good frameworks internally when it comes to governance, processes, and insurance policies, however we need to have the best tech stack to assist us use this knowledge.
Danni:
There have been some technical limitations, as our earlier knowledge catalog may solely help BigQuery, however we actually wished to help the course of the enterprise when it comes to scale and the way it could align extra broadly with our Knowledge Imaginative and prescient and Technique.
Our technique desires to implement Knowledge Mesh and ‘Knowledge as a Product’ mindset throughout the group. Each staff owns knowledge, they leverage it they usually have a accountability to handle it with governance frameworks.
So, with a view to embed Knowledge Governance practices and this cultural shift, we would have liked a device to help the frameworks, metadata technique, and tagging technique. We additionally wanted an answer to centralize all our Knowledge Belongings so we may have visibility of the place knowledge is and the way it’s being labeled which helps our Privateness initiatives.
We’re nonetheless on a metamorphosis journey at REA, which could be very thrilling. A brand new knowledge catalog was an actual alternative to push ourselves additional into that transformation with a brand new Knowledge Governance framework.
How did your analysis course of work? Did something stand out?
Surj:
We did some market analysis, chatting with Gartner and reviewing out there tooling throughout the business. We may have clearly stored utilizing our present Knowledge Catalog, however we wished to guage a large spectrum of instruments together with Atlan, Alation, and Open Metadata, to cowl Open Supply vs. Vendor managed.
We felt Atlan match the factors of a contemporary knowledge stack, offering us the capabilities we’d like, reminiscent of self-service tooling, an open API, and integrations to a wide range of expertise stacks which have been all essential to us.
We had an overwhelmingly good expertise partaking with Atlan, particularly with the Skilled Providers staff. The boldness that they gave us within the tooling after we went by means of our use circumstances drove a sense of robust alignment between REA and Atlan.
Piyush:
We did a three-phase analysis course of. Initially we went out to the market, did a few of our personal analysis, attempting to grasp which firms may match our use circumstances.
As soon as we did that, we went again and checked out completely different points reminiscent of pricing and used that as a filtering mechanism. We additionally appeared on the future roadmap of these firms to determine the place every firm is likely to be going, which was our second filtering course of. Once we have been executed selecting our choices, we had to determine which one would swimsuit us finest.
That’s after we did a lightweight proof of worth the place we created high-level analysis standards the place everyone concerned may rating completely different capabilities from 1-10. The staff included a supply supervisor, a product supervisor, an architect, and builders, simply to get a holistic view of the expertise everyone can be getting out of the device. After that scoring, we made a light-weight advice and introduced it to our executives.
A few of what we have been within the analysis standards have been issues like understanding what knowledge sources we may combine to, what safety appeared like, and ideas like extensibility so we could possibly be versatile sufficient to increase the catalog programmatically or by way of API. As a result of we’ve got our knowledge platform operating on Airflow, we additionally wished to grasp how effectively every choice labored with that.
Then we additionally checked out roadmaps and requested ourselves what may occur sooner or later, and if one thing like Atlan’s funding in AI is one thing we must be trying into, and different future enhancements Atlan or different distributors may present. We have been attempting to get an understanding of the subsequent two or three years, as a result of if we’re investing, we’re investing with a long-term perspective.
Surj:
In the event you have a look at the time period “Knowledge Catalog”, it’s been round for a really very long time. I’ve been working over 20 years, and I’ve used knowledge catalogs for a very long time, however the evolution has been vital.
When Piyush, Danni and I have been distributors, that’s one thing we have been excited about. Would you like a standard knowledge catalog, which we’ve in all probability seen in banks which have a powerful, ruled, centralized physique, or would you like one thing that’s evolving with the occasions, and evolving the place the business is heading?
I feel that’s why it was good to listen to from Atlan, and we favored the place they have been positioned in that evolution. We like that Atlan integrates with quite a few tech stacks. For instance, we use Nice Expectations for knowledge high quality in the intervening time, however we’re contemplating Soda or Monte Carlo, and we realized Atlan already has an integration with Soda and Monte Carlo. We’re discovering extra examples of that, the place Atlan is changing into extra related.
Conversely, after we have been addressing personally identifiable info, we wished to have the ability to scan our knowledge units. Atlan was fairly clear, saying “We’re not a scanning device, that’s not us.” It was good to have that differentiation. Once we checked out Open Metadata, they mentioned that they had scanning functionality, but it surely wasn’t as complete as we have been anticipating, and we all know now that this use case is in a unique realm.
It’s good to have that readability, and know which course Atlan goes to go.
How do you propose on rolling Atlan out to your customers?
Danni:
So typically in platforming and tooling, we’re very caught up specializing in the expertise and never specializing in the person expertise. That’s the place Atlan can actually assist.
We need to create one thing that’s tangible, and that individuals need to use, so we are able to drive mass adoption of the platform. With our earlier catalog, we didn’t have a lot adoption, so we’re making {that a} success metric, and one of many nice options in Atlan is that we are able to customise it to fulfill the wants of differing personas. An idea that hasn’t been historically pushed within the Knowledge Governance house!
We went out to the enterprise and undertook an enormous train, interviewing our stakeholders and potential customers. Now, we actually perceive the use circumstances, scale and what our customers need from the Knowledge Catalog. Our personas – analysts, producers, homeowners and customers will all be supported within the roll out of Atlan, ensuring that their expertise is personalized inside the device they usually can all perceive and use knowledge successfully for his or her roles.
