Monetary and Expertise Heavyweight Lowering Time Spent on Information Discovery by One-third with Atlan
When one thinks of Nasdaq, their eponymous trade in New York Metropolis possible involves thoughts. It’s the world’s second-largest, with $18 trillion in market capitalization; house the world’s most prestigious know-how corporations.
Michael Weiss, Product Supervisor at Nasdaq, joined Atlan on the 2023 Gartner Information & Analytics Summit to share how Nasdaq is a lot greater than its flagship trade, and the way an organization that’s run on knowledge for over 5 many years is turning into a pacesetter in knowledge know-how, as effectively.
“It is likely to be a shock simply how massive Nasdaq is,” Michael defined. “Nasdaq operates over 30 exchanges throughout quite a lot of asset courses throughout the globe. We’re additionally house to over 4,100 publicly listed corporations globally. Within the U.S., we’re house to 73% of all publicly listed tech corporations, and the 4 largest by market cap. Nasdaq is really the house of know-how.”
A Decade of Cloud Expertise
Contemplating their place within the cautious and sometimes slow-moving monetary sector, it’d come as a shock to many who Nasdaq started their cloud journey with AWS way back to 2012. Initially, they constructed inner merchandise specializing in regulatory reporting, then income administration, supported by the cloud. “From there, we went to knowledge warehousing and analytics. Ever since then, we’ve been systematically shifting towards the cloud,” Michael shared. With greater than a decade of expertise working in AWS, Nasdaq is starting to maneuver the majority of its important workloads to the cloud, and is defining how the following era of capital markets will function.
Regardless of this deep expertise, Nasdaq sees ample alternative to enhance. First, buying and selling system knowledge is difficult in measurement, and in complexity, with as many as 140 billion occasions per day processed within the U.S., alone. “If you take a look at buying and selling system knowledge, it’s essential to realize it’s optimized for [operational] efficiency, not for analytics,” Michael defined.
Nasdaq’s course of for getting ready and presenting knowledge was one other focus, with their legacy ETL instruments unable to maintain up when each the kinds of knowledge, and the demand for knowledge, scaled considerably. “They have been a bit inflexible,” Michael shared. “They didn’t actually regulate to the place we have been attempting to get to on the time.”
Between the complexity of their knowledge, legacy instruments that have been sluggish to develop to Nasdaq’s ambitions, and a staff that had its fingers full sustaining their technical panorama, Nasdaq’s knowledge staff realized a possibility to raised assist their enterprise companions. “For those who have been a product proprietor or enterprise consumer who needed to get a brand new set of insights into your knowledge, it might take fairly a while to have somebody construct it out on the prevailing platform,” Michael shared. “We weren’t actually enabling our enterprise customers to get what they wanted.”
Because of this, parallel groups started to emerge, every with a singular method to creating knowledge options. “For those who have been a enterprise unit, you might go to one in all 4 groups to try to get a solution. As a matter of truth, they’d go to all 4 groups to see who would win first,” Michael defined.
It might take a compelling imaginative and prescient and technique, backed by a extra fashionable knowledge stack and staff construction, to raised assist Nasdaq’s rising inner demand for knowledge.
A Information Chief’s Information Stack
“It was solely about 18 months in the past that we determined to take a step again and re-evaluate our tech stack,” Michael defined. And in that quick time, Nasdaq has made a number of key investments.
First was an implementation of dbt, accelerating their staff’s means to construct knowledge fashions. “We actually needed a device that was versatile, and will actually allow individuals who had SQL expertise already to make the most of that know-how,” Michael shared. Nasdaq continued to function on AWS, embracing Redshift as their main knowledge retailer, alongside quite a lot of AWS companies akin to S3 as their knowledge lake, Glue for knowledge integration, and QuickSight for enterprise intelligence.
This new stack additionally included Monte Carlo, adopted for knowledge observability. “Loads of our knowledge monitoring on the time was very reactive. The pipeline may succeed however we wouldn’t know if the info was proper, mistaken, or detached,” Michael shared. “With a device like Monte Carlo, we’re in a position to catch a few of that earlier on.”
However the remaining piece of the puzzle was Atlan. Atlan was adopted as Nasdaq’s window to their modernizing knowledge stack, and as a vessel for a knowledge governance follow that was shortly maturing. “That is how we’re actually going to take ourselves to the following stage,” Michael remarked on Atlan’s adoption.
Evolving Groups and Processes
Past taking a leap ahead in technical maturity, Nasdaq’s knowledge staff understood that their staff construction and working mannequin would want to make an identical leap in the event that they have been to fulfill, then exceed enterprise wants. “You may herald new instruments and to maintain the identical methodology and the identical cultural approaches you don’t actually clear up the core drawback,” Michael shared. “How do you carry a data-first tradition to the complete group?”
Nasdaq lengthy had a centralized construction for reporting; applicable for his or her legacy know-how and duties, however a mannequin that struggled to allow their enterprise counterparts to make use of new instruments and applied sciences.
“We seemed on the talent units inside totally different groups to strive to determine one of the simplest ways to carry forth change at a extra basic organizational stage,” Michael defined.
This method led to rigorously focusing Nasdaq’s knowledge group into extra particular roles. First is the Platform Workforce, answerable for sustaining core know-how and decreasing cognitive load on analysts and customers. Subsequent is the Financial Analysis staff, centered on knowledge science and serving as a middle of experience for enterprise models conducting deeper evaluation.
Whereas a less-centralized mannequin was a greater match for the rising variety of knowledge customers, it required a extra mature engagement and onboarding mannequin as customers started to take advantage of new capabilities and knowledge.
“We began taking a look at how we essentially embed knowledge into every one in all our core companies to allow them to self-serve,” Michael shared. “So we got here up with an engagement mannequin the place we’d go to one in all our organizations. We do a survey to start out. We have been attempting to get an evaluation of the place they have been with their expertise after which from there we might really embed with them a set of information initiatives they needed to pursue and get them up and working.”
This new course of, contemplating the distinctive wants of Nasdaq’s enterprise models, and guiding them by way of necessities, joint growth of enterprise fashions, the creation of a metrics layer, and enablement to transition duty, meant that relatively than performing as a service heart, Nasdaq’s central knowledge staff might give attention to bigger initiatives, teaching and consulting when essential.
“We didn’t wish to simply say, ‘Hey, right here’s the platform. Have enjoyable, good luck. Come again inside six months and tell us what you consider it.’,” Michael defined. “We discovered that this enabling actually has helped drive success getting groups on board.”
Selecting Atlan for Energetic Metadata Administration
With core knowledge know-how in place, a staff construction that matched Nasdaq’s knowledge service technique, and an onboarding and engagement mannequin to make sure enterprise companions have been well-equipped to make use of new capabilities, Michael’s staff turned to Energetic Metadata Administration as a path for schooling, collaboration, and discovery.
“This isn’t the primary time we’ve launched new know-how at Nasdaq for knowledge. I’ve been at Nasdaq for 10 years. That is the third or fourth time we did it. Every time you be taught slightly one thing totally different,” Michael shared. “Once we launched this one of many key issues that our govt staff introduced up was ‘What’s going to be totally different this time?’ ‘What different issues are you going to herald the combo to cut back friction between knowledge producers and customers?’”
Essential to Nasdaq’s govt staff was the flexibility for customers to know what knowledge and instruments they’ve at their disposal, and Energetic Metadata Administration turned a vital know-how for answering this query. With no platform like Atlan, knowledge and capabilities would stay undiscoverable, decreasing the potential return on funding Michael had hoped for at first of Nasdaq’s journey.
“To get key stakeholders to purchase into all the pieces, it was important we had a plan in place. We as soon as discovered ourselves centered on BI and ETL however forgot about find out how to cut back friction between producers and customers of information. It was front-of-mind for all stakeholders that we deliberate to resolve this in any new knowledge method. With out this the remainder of our mannequin can be DOA,” Michael defined.
Amongst Nasdaq’s stringent necessities for Energetic Metadata Administration was a SaaS mannequin – avoiding the necessity to handle their very own infrastructure – the flexibility to rigorously mannequin entry factors throughout customers and roles, and a confirmed document of safety aligning with Nasdaq’s place as a extremely regulated enterprise entity.
Maybe most essential was cultural alignment. Explaining why Atlan proved to be the very best match for Nasdaq, Michael shared, “It’s not simply the know-how, however the group behind it. We actually needed a companion who would are available in and assist us actually perceive the facility and advantage of this device. We discovered that.”
Scoping the Information Service Alternative
An essential a part of Nasdaq’s path to implementing Atlan was surveying knowledge customers to raised perceive how they might enhance the invention and utility of information.
Almost 100 finish customers responded to the survey, and what Michael’s staff discovered was intriguing. 75% of respondents report spending time attempting to know the context round knowledge. And energy customers, who spend six or extra hours per week on knowledge, spend two of these hours attempting to know the context round what they have already got entry to.
Take into consideration that! A 3rd of their time each week is spent simply attempting to know what’s there. Think about if we might carry a product in that helps cut back that effort and actually permits them to get proper to the center of the issue. To drive knowledge merchandise from insights into the enterprise. And that’s what we’re attempting to get to.”
Michael Weiss, Product Supervisor
These findings have been constant throughout Nasdaq’s numerous knowledge and capabilities. Whether or not uncooked knowledge from their matching engine, or ready metrics and visuals, vital time was spent on amassing context, slowing supply on key insights and knowledge merchandise.
Searching for a beyond-the-numbers understanding of stakeholder sentiment, the survey requested for qualitative suggestions, resulting in the conclusion that siloed and misplaced data have been key issues to handle. With out sturdy possession of information property, or constant documentation, Michael’s stakeholders discovered it tough to choose up the place one other member of Nasdaq’s group left off.
“It was not unusual that you’d have somebody working in a single a part of the group, figuring out one a part of one thing. Then a knowledge product proprietor would are available in, and doesn’t perceive why selections have been made, and even what was accessible, in order that they’re again to sq. one,” Michael defined.
Driving Widespread Understanding of Nasdaq’s Information
“A few of the early outcomes have been actually astonishing,” Michael shared. Regardless of being early of their Atlan adoption journey, a follow-up survey exhibits indicators that Nasdaq’s alternative of Atlan for Energetic Metadata Administration is yielding constructive outcomes.
“The one remark that retains developing is ‘That is like having Google for our knowledge.’,” Michael defined. “Letting them begin with column names and baseline descriptions has enabled our customers to essentially perceive what’s in our lakes and warehouses already, after which begin asking the appropriate questions.”
Having made the appropriate know-how selections, and with a staff construction that helps their enterprise models distinctive knowledge wants, the introduction of Atlan into the Nasdaq ecosystem has catalyzed a standard understanding of their knowledge and the instruments at their disposal. And with a rising array of information, insights, and capabilities at customers’ fingertips in Atlan, confidence in Michael and his staff’s knowledge technique is bettering.
“Regardless that we’re early into this journey there’s already a little bit of a shift within the sentiment of our staff. Whereas we nonetheless have loads to do, this journey is already proving worthwhile and has been important for the chief staff having confidence we will proceed to ship on our promise.”
