Main International Communications Company Improves Discoverability, Documentation, and Root Trigger and Impression Evaluation
Introduced collectively by multinational know-how firm Konica Minolta, Indicia Worldwide is the one international company that mixes inventive, information and know-how expertise with manufacturing and procurement experience to comprehend advertising and marketing ROI for a few of the world’s largest manufacturers.
Information permits Indicia Worldwide’s shoppers to speak with potential and present prospects, and ensures the best messages attain them on the excellent second. Supporting that data-driven mission is Graham Lannigan, Head of Information Platform at Indicia Worldwide.
“My function is taking care of the information platform that we use for our personal groups, and for the work we do with shoppers,” Graham shared. “Our information maturity has actually grown over time, and that led to the concept that we must be our personal information platform and capabilities, and utilizing that as a launchpad for backing up the providers we offer to shoppers, and enabling us to construct and energy a load of merchandise we are able to provide to shoppers and run ourselves.”
Indicia Worldwide is the product of a number of organizations acquired by Konica Minolta, together with Charterhouse, a inventive manufacturing company, Ergo, a print administration and advertising and marketing automation company, and the advertising and marketing company Indicia. Constant, previous and current, in every group is a service mannequin that stresses particular person consideration and care to every shopper. However regardless of this bespoke mannequin, Graham and his staff seen patterns rising in how every shopper was serviced.
“After we joined up the standard company mannequin with the worldwide print manufacturing experience, we realized that whereas we work throughout 40 totally different markets, we had quite a lot of frequent duties and customary issues,” Graham defined. “There have been quite a lot of occasions we have been doing the identical factor again and again, and generally utilizing totally different know-how.”
Having recognized these patterns of repetitive work, Graham and his staff have been constructing a set of information capabilities and merchandise that Indicia Worldwide can both provide to shoppers, or use internally to help the {custom} providers they supply.
The final couple years in my function as Head of Information Platform has been about figuring out the proper know-how companions, the proper issues we’re attempting to resolve with our information platform, and becoming these two collectively. It’s discovering the proper mix to be sure that now we have the capabilities we want, our information science staff has the instruments to get their job carried out, our information engineering staff has the instruments they want, and that every one of this comes collectively to ship the proper information to the proper individuals.”
Graham Lannigan, Head of Information Platform
Supporting this transformation begins with a various and specialised information staff, starting with Information Science and Analytics, supporting inside dashboarding and modeling wants, and making certain shoppers can carefully measure the worth derived from their advertising and marketing spend. Engineering groups embody Product Engineering, managing user-facing merchandise and internet purposes, and Information Engineering, accountable for Indicia’s Information Platform and making certain downstream groups like Information Science have accessible, high-quality information.
And powering this more and more specialised staff is a fast-modernizing information stack, underpinned by AWS as their major cloud platform, Matillion for ETL, Amazon Sagemaker for Information Science, and Energy BI for enterprise intelligence. However central to their success lately has been a dedication to Snowflake, starting in 2020.
“The core of our information platform is Snowflake. We began with some shopper workloads and we’ve slowly re-platformed our information capabilities onto Snowflake. Our staff has a extremely sturdy SQL background, so Snowflake was a very nice match,” Graham shared. Plus, with our international mannequin and a few of the issues we do with analytics like supporting wealthy shopper information, Snowflake has quite a lot of very nice capabilities round information sharing and information monetization that basically match with our ethos as an organization.”
With a specialised staff, and a rising trendy information platform primarily based on Snowflake at its heart, Graham’s staff started the seek for a know-how that might make these capabilities even simpler to make the most of.
“A part of my focus now could be engaged on how we shut the hole between the information in Snowflake and what the staff are doing in Sagemaker and our personal product providing, lowering information motion, and ensuring the obstacles to them doing cool stuff are eliminated as a lot as doable. We acknowledged the necessity for understanding what our information is, the place we’re holding it, and sharing a load of information, and that’s the place Atlan got here in,” Graham defined.
Traditionally, Graham’s staff would use its wiki to retailer documentation for every of their tasks, together with particulars round information buildings and fashions. And whereas this documentation was useful as every undertaking started, its accuracy would wane as change requests have been made and know-how advanced, and as extra documentation was added, it turned increasingly more tough to navigate.
“You’ll spend quite a lot of time monitoring down the individuals who had carried out some modifications and attempting to ask them if the documentation was nonetheless consultant of actuality or not,” Graham shared.
And whereas Indicia Worldwide’s engineering groups have been dedicated to creating its documentation work nicely, together with a custom-built device to crawl databases and synchronize data again to the wiki to replace documentation, the dimensions and complexity of buyer information made the device unworkable.
“Now we have some shoppers who’ve 40 million prospects and construct tens upon tens of billions of rows of information. Our Atlan occasion proper now has one thing like 500,000 belongings in it,” Graham defined. “There’s loads to control, so once we tried to hyperlink this information up with our personal instruments, we discovered that we simply weren’t maintaining.”
Starting their seek for an answer that might enhance information discovery and documentation, Graham and his staff thought-about choices from cloud suppliers and open-source options, however discovered they lacked a consumer interface that made their information belongings neatly accessible. In addition they thought-about legacy information catalog options, however have been discouraged by excessive price of possession, and located they didn’t align with Indicia’s burgeoning cloud-first mentality.
Atlan’s complete philosophy simply match rather well with us as a company, the place we have been, and what we have been seeking to do. We introduced Atlan in, began taking part in round with it, crawling a few of our belongings, and utilizing a few of the capabilities. And I believe in a short time the platform made a case for itself as a result of everybody I confirmed it to was like ‘Wow. That is sensible. This is able to be actually useful.’”
Graham Lannigan, Head of Information Platform
Starting by integrating Atlan with Snowflake, Graham’s aim was to dramatically enhance visibility into their information property. With this newfound visibility, he and his staff first seen a chance to optimize their storage prices, utilizing Atlan’s utilization and recognition metrics to determine tables that had not been accessed over a set time frame. Utilizing {custom} properties, Graham and his staff tag these belongings with a life stage, evaluate them with the groups that created them to find out their necessity, and resolve whether or not to retain or deprecate them.
“The reporting in Atlan has been actually useful for this,” Graham shared. “We’re beginning to get a greater deal with on (questions like) ‘Why preserve information if we don’t have to?’ and ‘Why have belongings if we’re not going to make use of them?’”
Transferring to complement these belongings, Graham has additionally used this newfound visibility to proactively ask subject material specialists so as to add documentation as their belongings change.
It simply offers this view of what’s occurring in our platform on a day-to-day foundation that might have been actually arduous to place collectively in any other case. And the truth that we are able to then tag somebody on Groups or share the hyperlink to the asset in Atlan and ask ‘What’s this factor?’ or ask ‘Please add some documentation to this.’ It’s amazingly useful.”
Graham Lannigan, Head of Information Platform
And whereas answering a query from his colleagues as soon as meant shifting system by system, trying up outdated documentation, or asking professional after professional for the knowledge he wanted, Graham can now use Atlan to rapidly discover the knowledge he wants, so he can concentrate on constructing and maturing their information platform.
“Personally, it’s been completely huge when it comes to my having the ability to reply questions or having the ability to perceive what’s occurring,” Graham defined. “When now we have a query about some information or the way it’s getting used, now my first cease is Atlan. I received’t log into Snowflake to take a look for an object. I’ll seek for it within the asset view, and if I have to, I’ll go into the lineage tab and I’ll discover how the information received there.”
Extending this worth past saving time in his personal day, Graham is starting to encourage Indicia Worldwide’s information shoppers to self-serve for solutions on information. At present, when groups are constructing a brand new information product and have problem discovering the proper information, Graham providers these questions personally, sending a hyperlink to the proper belongings in Atlan. However tomorrow, he sees dozens of information shoppers empowered to seek out this data, or enrich it, by themselves.
“It’s an enormous time-saver not only for me, however the remainder of the staff. The subsequent factor I’m attempting to instill is that relatively than pinging me a message and asking for assist, simply go to Atlan. Attempt to seek for what you assume it may be referred to as, and should you discover it and it’s not documented, add some documentation to tie it up so the subsequent particular person can discover it extra simply,” Graham defined.
Extending previous the worth of higher servicing inside shoppers, Graham’s staff is discovering new methods to enhance the practices of their engineering groups.
Starting with the frequent downside of dashboards and information pipelines breaking when new performance is deployed or when change requests are accomplished, Indicia Worldwide’s engineers can now use Atlan’s automated lineage for root trigger evaluation, considerably accelerating their time to decision.
One of many nice issues is that relatively than everybody attempting to scramble round and keep in mind what they modified and the place, we are able to actually simply pop into the lineage view in Atlan and hint that again to say ‘Right here’s the dashboard, there’s the dataset it’s getting the information from, that’s the desk, and that’s the place it’s ingested.’ We are able to monitor that every one the way in which again and discover the place the change was made.”
Graham Lannigan, Head of Information Platform
Going ahead, Graham and his staff intend to make use of Atlan’s automated lineage for proactive influence evaluation, figuring out the downstream belongings probably impacted by modifications earlier than these modifications are even made.
“The nicer approach is doing all of this upfront. Quite than ready till you could have an issue, preempt it by doing the influence evaluation upfront. That’s the bit the place I see huge worth for our groups,” Graham shared.
Lastly, Graham and his staff are working to combine Atlan into their CI/CD course of. Jenkins, Indicia Worldwide’s CI/CD orchestration device, would ship a notification to Atlan utilizing its open API when belongings are created, which might then ship a reminder to the asset proprietor to finish documentation.
“The place we’ve received individuals taking a function department and checking that into Dev and perhaps doing a pull request with an approval step, that now goes by means of our environments and will get deployed. However what if we may put an approval step in there that claims ‘Have you ever documented this in Atlan?’,” Graham defined.
Whereas early of their journey, Graham and his staff see a shiny future for Atlan, past the worth they’ve already yielded with higher visibility into their information property, quicker solutions to information shoppers, and more practical influence and root trigger evaluation.
As a advertising and marketing company to the world’s finest client manufacturers, Indicia Worldwide generally finds itself the steward of personally identifiable data, and intends to make use of Atlan to grasp what PII belongings they maintain, how previous they’re, how often used they’re, and to make sure that solely these licensed to entry them, can.
And through the use of Atlan’s native integration with Snowflake, and benefitting from the partnership between the 2 organizations, Indicia Worldwide is optimistic about higher activating Snowflake’s native tagging and masking capabilities with Atlan’s consumer interface, lineage, and automation capabilities.
To do any of this, we want that pane of glass to have the ability to take a look at the information and perceive what’s occurring inside our information property, what information we maintain, what its traits are, and the way it’s getting used. Atlan offers us that in a single device, which is unbelievable. So the extra Atlan is ready to combine natively with issues like Snowflake’s object tagging capabilities, the extra highly effective Atlan will get, and it turns into increasingly more vital in our stack.”
Graham Lannigan, Head of Information Platform
Picture by Financial institution Phrom on Unsplash
