Why semantics matter within the fashionable knowledge stack

Most organizations at the moment are nicely into re-platforming their enterprise knowledge stacks to cloud-first architectures. The shift in knowledge gravity to centralized cloud knowledge platforms brings monumental potential. Nevertheless, many organizations are nonetheless struggling to ship worth and reveal true enterprise outcomes from their knowledge and analytics investments.

The time period “fashionable knowledge stack” is often used to outline the ecosystem of applied sciences surrounding cloud knowledge platforms. To this point, the idea of a semantic layer hasn’t been formalized inside this stack.

When utilized accurately, a semantic layer types a brand new middle of information gravity that maintains the enterprise context and semantic that means crucial for customers to create worth from enterprise knowledge belongings. Additional, it turns into a hub for leveraging energetic and passive metadata to optimize the analytics expertise, enhance productiveness and handle cloud prices.

What’s the semantic layer?

Wikipedia describes the semantic layer as “a enterprise illustration of knowledge that lets customers work together with knowledge belongings utilizing enterprise phrases reminiscent of product, buyer or income to supply a unified, consolidated view of knowledge throughout the group.”

The time period was coined in an age of on-premise knowledge shops — a time when enterprise analytics infrastructure was expensive and extremely restricted in performance in comparison with at the moment’s choices. Whereas the semantic layer’s origins lie within the days of OLAP, the idea is much more related at the moment. 

What’s the fashionable knowledge stack?

Whereas the time period “fashionable knowledge stack” is ceaselessly used, there are numerous representations of what it means. In my view, Matt Bornstein, Jennifer Li and Martin Casado from Andreessen Horowitz (A16Z) supply the cleanest view in Rising Architectures for Fashionable Knowledge Infrastructure.

I’ll confer with this simplified diagram based mostly on their work beneath:

This illustration tracks the movement of knowledge from left to proper. Uncooked knowledge from varied sources transfer by means of ingestion and transport providers into core knowledge platforms that handle storage, question and processing and transformation previous to being consumed by customers in quite a lot of evaluation and output modalities. Along with storage, knowledge platforms supply SQL question engines and entry to Synthetic Intelligence (AI) and machine studying (ML) utilities. A set of shared providers cuts throughout the whole knowledge processing movement on the backside of the diagram. 

The place is the semantic layer?

A semantic layer is implicit any time people work together with knowledge: It arises organically until there’s an intentional technique carried out by knowledge groups. Traditionally, semantic layers had been carried out inside evaluation instruments (BI platforms) or inside a knowledge warehouse. Each approaches have limitations.

BI-tool semantic layers are use case particular; a number of semantic layers are inclined to come up throughout totally different use circumstances resulting in inconsistency and semantic confusion. Knowledge warehouse-based approaches are typically overly inflexible and too advanced for enterprise customers to work with straight; work teams will find yourself extracting knowledge to native analytics environments — once more resulting in a number of disconnected semantic layers. 

I exploit the time period “common semantic layer” to explain a skinny, logical layer sitting between the information platform and evaluation and output providers that summary the complexity of uncooked knowledge belongings in order that customers can work with business-oriented metrics and evaluation frameworks inside their most popular analytics instruments.

The problem is the way to assemble the minimal viable set of capabilities that offers knowledge groups enough management and governance whereas delivering end-users extra advantages than they might get by extracting knowledge into localized instruments.

Implementing the semantic layer utilizing transformation providers

The set of transformation providers within the A16Z knowledge stack contains metrics layer, knowledge modeling, workflow administration and entitlements and safety providers. When carried out, coordinated and orchestrated correctly, these providers kind a common semantic layer that delivers necessary capabilities, together with:

• Making a single supply of reality for enterprise metrics and hierarchical dimensions, accessible from any analytics software.
• Offering the agility to simply replace or outline new metrics, design domain-specific views of knowledge and incorporate new uncooked knowledge belongings.
• Optimize analytics efficiency whereas monitoring and optimizing cloud useful resource consumption.
• Implement governance insurance policies round entry management, definitions, efficiency and useful resource consumption.

Let’s step by means of every transformation service with an eye fixed towards how they have to work together to function an efficient semantic layer.

Knowledge modeling

Knowledge modeling is the creation of business-oriented, logical knowledge fashions which can be straight mapped to the bodily knowledge buildings within the warehouse or lakehouse. Knowledge modelers or analytics engineers give attention to three necessary modeling actions:

Making knowledge analytics-ready: Simplifying uncooked, normalized knowledge into clear, principally de-normalized knowledge that’s simpler to work with.

Definition of study dimensions: Implementing standardized definitions of hierarchical dimensions which can be utilized in enterprise evaluation — that’s, how a corporation maps months to fiscal quarters to fiscal years. 

Metrics design: Logical definition of key enterprise metrics utilized in analytics merchandise. Metrics will be easy definitions (how the enterprise defines income or ship amount). They are often calculations, like gross margin ([revenue-cost]/income). Or they are often time-relative (quarter-on-quarter change).

I wish to confer with the output of semantic layer-related knowledge modeling as a semantic mannequin. 

The metrics layer 

The metrics layer is the one supply of metrics reality for all analytics use circumstances. Its major operate is sustaining a metrics retailer that may be accessed from the complete vary of analytics shoppers and analytics instruments (BI platforms, purposes, reverse ETL, and knowledge science instruments).  

The time period “headless BI” describes a metrics layer service that helps person queries from quite a lot of BI instruments. That is the basic functionality for semantic layer success — if customers are unable to work together with a semantic layer straight utilizing their most popular analytics instruments, they’ll find yourself extracting knowledge into their software utilizing SQL and recreating a localized semantic layer.

Moreover, metrics layers have to assist 4 necessary providers:

Metrics curation: Metrics stewards will transfer between knowledge modeling and the metrics layer to curate the set of metrics supplied for various analytics use circumstances.

Metrics change administration: The metrics layer serves as an abstraction layer that shields the complexity of uncooked knowledge from knowledge shoppers. As a metrics definition adjustments, current reviews or dashboards are preserved. 

Metrics discoverability: Knowledge product creators want to simply discover and implement the right metrics for his or her objective. This turns into extra necessary because the record of curated metrics grows to incorporate a broader set of calculated or time-relative metrics.  

Metrics serving: Metrics layers are queried straight from analytics and output instruments. As finish customers request metrics from a dashboard, the metrics layer must serve the request quick sufficient to supply a constructive analytics person expertise.

Workflow administration

Transformation of uncooked knowledge into an analytics-ready state will be based mostly on bodily materialized transforms, digital views based mostly on SQL or some mixture of these. Workflow administration is the orchestration and automation of bodily and logical transforms that assist the semantic layer operate and straight influence the price and efficiency of analytics. 

Efficiency:  Analytics shoppers have a really low tolerance for question latency. A semantic layer can’t introduce a question efficiency penalty; in any other case, intelligent finish customers will once more go down the information extract route and create various semantic layers. Efficient efficiency administration workflows automate and orchestrate bodily materializations (creation of mixture tables) in addition to resolve what and when to materialize. This performance must be dynamic and adaptive based mostly on person question conduct, question runtimes and different energetic metadata. 

Price: The first price tradeoff for efficiency is expounded to cloud useful resource consumption. Bodily transformations executed within the knowledge platform (ELT transforms) eat compute cycles and value cash. Finish person queries do the identical. The selections made on what to materialize and what to virtualize straight influence cloud prices for analytics applications. 

Analytics performance-cost tradeoff turns into an attention-grabbing optimization downside that must be managed for every knowledge product and use case. That is the job of workflow administration providers.

Entitlements and safety

Transformation-related entitlements and safety providers relate to the energetic software of knowledge governance insurance policies to analytics. Past cataloging knowledge governance insurance policies, the fashionable knowledge stack should implement insurance policies at question time, as metrics are accessed by totally different customers. Many several types of entitlements could also be managed and enforced alongside (or embedded in) a semantic layer.

Entry management: Correct entry management providers guarantee all customers can get entry to all the knowledge they’re entitled to see.  

Mannequin and metrics consistency:  Sustaining semantic layer integrity requires some degree of centralized governance of how metrics are outlined, shared and used. 

Efficiency and useful resource consumption: As mentioned above, there are fixed tradeoffs being made on efficiency and useful resource consumption. Consumer entitlements and use case precedence can also issue into the optimization.

Actual time enforcement of governance insurance policies is crucial for sustaining semantic layer integrity.

Integrating the semantic layer inside the fashionable knowledge stack

Layers within the fashionable knowledge stack should seamlessly combine with different surrounding layers. The semantic layer requires deep integration with its knowledge cloth neighbors — most significantly, the question and processing providers within the knowledge platform and evaluation and output instruments. 

Knowledge platform integration

A common semantic layer shouldn’t persist knowledge outdoors of the information platform. A coordinated set of semantic layer providers must combine with the information platform in a couple of necessary methods:

Question engine orchestration: The semantic layer dynamically interprets incoming queries from shoppers (utilizing the metrics layer logical constructs) to platform-specific SQL (rewritten to mirror the logical to bodily mapping outlined within the semantic mannequin). 

Rework orchestration: Managing efficiency and value requires the aptitude to materialize sure views into bodily tables. This implies the semantic layer should be capable to orchestrate transformations within the knowledge platform. 

AI/ML integration: Whereas many knowledge science actions leverage specialised instruments and providers accessing uncooked knowledge belongings straight, a formalized semantic layer creates the chance to supply enterprise vetted options from the metrics layer to knowledge scientists and AI/ML pipelines. 

Tight knowledge platform integration ensures that the semantic layer stays skinny and may function with out persisting knowledge regionally or in a separate cluster.

Evaluation and output

A profitable semantic layer, together with a headless BI strategy to implementing the metrics layer, should be capable to assist quite a lot of inbound question protocols — together with SQL (Tableau), MDX (Microsoft Excel), DAX (Microsoft Energy BI), Python (knowledge science instruments), and RESTful interfaces (for software builders) — utilizing commonplace protocols reminiscent of ODBC, JDBC, HTTP(s) and XMLA.

Augmented analytics

Main organizations incorporate knowledge science and enterprise AI into on a regular basis decision-making within the type of augmented analytics. A semantic layer will be useful in efficiently implementing augmented analytics. For instance:

• Semantic layers can assist pure language question initiatives. “Alexa, what was our gross sales income final quarter?” will solely return the fitting outcomes if Alexa has a transparent understanding of what income and time imply. 
• Semantic layers can be utilized to publish AI/ML-generated insights (predictions and forecasts) to enterprise customers utilizing the identical analytics instruments they use to research historic knowledge. 
• Past simply prediction values, semantic layers could make broader inference knowledge obtainable to enterprise customers in a method that may improve explainability and belief in enterprise AI.

The middle of mass for data gravity within the fashionable knowledge stack

The A16Z mannequin implies that organizations might assemble a cloth of home-grown or single-purpose vendor choices to construct a semantic layer. Whereas definitely attainable, success shall be decided by how well-integrated particular person providers are. As famous, even when a single service or integration fails to ship on person wants, localized semantic layers are inevitable.

Moreover, you will need to think about how important enterprise data will get sprinkled throughout knowledge materials within the type of metadata. The semantic layer has the benefit of seeing a big portion of energetic and passive metadata created for analytics use circumstances. This creates a possibility for forward-thinking organizations to higher handle this data gravity and higher leverage metadata for bettering the analytics expertise and driving incremental enterprise worth.

Whereas the semantic layer remains to be rising as a know-how class, it can clearly play an necessary position within the evolution of the fashionable knowledge stack.

This text is a abstract of my present analysis round semantic layers inside the fashionable, cloud-first knowledge stack. I’ll be presenting my full findings on the upcoming digital Semantic Layer Summit on April 26, 2023. 

David P. Mariani is CTO and cofounder of AtScale, Inc.