How a semantic layer bridges BI and AI

This article was contributed by David Mariani, chief technology officer for AtScale.

An important step in the data maturity of an organization is moving beyond simple historical analysis to generating accurate predictions about the future. In the past, business analysts focused on historical analysis while data science teams attempted to surface interesting insights about the future. Today, with the advent of the semantic layer, these two siloed worlds are coming together. Enterprises that merge these two disciplines can deliver augmented analytics, helping everyone in the organization better understand the past and predict the future.

Types of analytics

Organizations leverage analytics to help them understand and improve their business operations and customer satisfaction. Before we go further, let’s define the four flavors of analysis we typically see in an organization, each with increasing levels of sophistication.

As illustrated by the table above, business users typically focus on historical analysis while data scientists are working to predict the future. It’s obvious that business users make better decisions if they can anticipate the future. It’s also obvious that data scientists build better models if they can compare their predictions to what actually happened. In other words, historical analysis and predictive analysis are relevant to both teams, but rarely do the two meet.

What is a semantic layer?

A semantic layer is a business representation of data that makes it easier for end users to access data using common, business-friendly terms. A semantic layer maps complex data relationships into easy-to-understand business terms to deliver a unified, consolidated view of data across the organization. A semantic layer provides the following benefits:

Usability

One of the biggest complaints from the business is that it takes way too long for IT to build or deliver reports for them. Users want to control their destiny, and subject-matter experts (not IT) are best suited to applying data to improve the business. A well-designed semantic layer hides the complexity of data’s physical form and location while translating data into understandable business constructs. A semantic layer frees business users and data scientists from the dependency on IT and data experts by making data easy to use.

Security and governance

Today, enterprises have strong and sometimes regulatory requirements to track “who” saw “which” data and “when.” A modern semantic layer allows users to appear as themselves to the underlying data platforms from any consumer tool. At the same time, a semantic layer ensures that data is consistent regardless of consumption style and makes sure everyone plays by the same (governance) rules. 

Agility

Analytics agility, also thought of as “time to insight,” is how long after data lands that it can be used to make decisions. BI tools that require data imports, extracts, or cube building take anywhere from minutes for small data to days/weeks for large data before data can be accessed. A modern semantic layer leverages data virtualization to enable new data landing in your data warehouse to be query-able by your BI tool immediately, regardless of size.

Performance and scale

Cubes and data extracts were introduced to overcome the performance issues of analytics and data platforms. This approach introduces data copies, adds complexity, destroys agility, and introduces latency. A modern semantic layer improves performance regardless of the underlying data model, whether it’s a snowflake, a star, or purely OLTP schema. By automatically creating and managing aggregates or materialized views inside the underlying data platform, a semantic layer learns from user query patterns and optimizes the data platform’s performance and cost without data movement.

The semantic layer: The unifying thread

With a semantic layer, you can bridge the gap between BI users and data science teams. This enables your teams to work transparently and cooperatively with the same information and with the same goals.

A semantic layer abstracts away the complexity of underlying raw data using a business model, allowing any data consumer to access quantitative metrics, attributes, features, predictions, business hierarchies, and complex calculations in an intuitive, easy-to-understand interface. A semantic layer solution presents this consumer-friendly interface in the “language” of their tooling (SQL, MDX, DAX, JDBC, ODBC, REST, or Python), translating queries into the dialect of the underlying cloud platform. With a common set of business terms, both teams can interact with the same data, with the same governance rules, with the same results, using the tooling of their choice. 

With both teams working on the same semantic layer solution, data scientists can share (or publish) their generated features and predictions with business analysts, while business analysts provide feedback to data science teams on the quality of their predictions and model drift. Once data becomes highly accessible, teams can collaborate not just within their four walls, but blend data from second and third-party data sources to unlock the power of data and analytics for everyone. Closing the gap between business intelligence and data science teams is the key to achieving a high level of data analytics maturity and applying all types of analytics at scale.

Augmented intelligence

When business and data science teams collaborate using a semantic layer, they enhance their historical data with predictive insights. Closing the gap between business intelligence and data science teams provides more visibility into the output of data science initiatives throughout the organization and enables organizations to leverage their data for predictive and prescriptive analytics. Augmented intelligence (also called augmented analytics or decision intelligence) brings AI-generated insights into traditional business intelligence workflows to improve data-driven decisions.

When most people think of augmented intelligence, they think about specific features that may appear in AI-enhanced business intelligence tools. For example, some BI tools add natural language query (NLQ) or outlier analysis to help their users ask better questions or find the needle in the haystack. These are valuable features, but they are confined to the particular tool and may work differently across different tools.

In contrast, augmenting data through the unification of BI and data science adds AI-enhanced data to the semantic layer, providing the same insights across the consumer spectrum, regardless of the tool used. Essentially, a semantic layer amplifies the effect of the data science team by sharing their work with a wider audience and providing that audience with the ability to deliver feedback on the quality and utility of their predictions — a win-win.

Unlocking the power of augmented intelligence

Augmented intelligence has the power to transform businesses into data-driven organizations. This starts with implementing the appropriate processes and tools to democratize data and empower individuals to utilize data through self-service analytics. 

Ultimately, every organization wants to empower every individual to make data-driven decisions. A semantic layer can become the vehicle for delivering augmented intelligence to a broader audience by publishing the results of data science programs through existing BI channels. By feeding data science model results back into the semantic layer, your organization can capture benefits beyond just historical analysis. Decision makers can consume predictive insights alongside historical data. They can also use the same governed data to reliably “drill down” into the details of a prediction. As a result, your organization can foster more self-service and greater data science literacy and generate a better return on data science investments.

David Mariani is the chief technology officer for AtScale.