DataOps: Still an unsolved challenge for many organizations

In our digital age, everyone has come to understand the importance of data — collecting information, sharing it across the organization and leveraging it to drive mission-critical decisions. Yet many organizations still don’t have a handle on how to do those things well enough and fast enough. The unfortunate reality is that some valuable data is never shared with the right people at the right time. Worse, decision-makers can’t always be sure the data they rely on is accurate and up-to-date. 

It is possible to build new technical processes and shape cultural expectations to manage data successfully. In fact, the software world has already achieved that goal with its development processes, and could be considered a model for dataops today. Data managers are just starting to raise their “product” to the same high level of organization, quality, and trust.

A uniquely valuable asset

Data is “a driver of growth and change” in the 21^st century, just as oil was 100 years ago, the Economist writes — and it’s just as valuable. “Flows of data have created new infrastructure, new businesses, new monopolies, new politics and — crucially — new economics.” 

In a data-driven world, everyone recognizes that data is no longer just a byproduct of key business processes, but a core asset with a unique value all its own. Every business, in every industry, can use its data to find new customers and retain customers, improve the brand experience for customers, study sales trends and refine marketing strategies.

But not every business is fully leveraging its data today. According to a Deloitte Analytics survey, data management is held back by three common challenges in particular:

• Low-quality data (just 34% of the survey participants considered that data “excellent” or “good,” defined as integrated, accurate and centralized)
• Lack of analytics technology (49% had only basic reporting tools and limited predictive analytics tools)
• Data ownership “power struggles,” attributed to inadequate executive leadership (38% reported localized analytics with limited sharing of tools, data and people; 20% reported “uncoordinated pockets” of analytical efforts)

Still, 49% of the participants agreed that analytics improved their decision-making capabilities. “Basically, analytics is about making good business decisions,” one analytics director told Deloitte. “Just giving reports with numbers doesn’t help. We must provide information in a way that best suits our decision-makers.”  

Finding balance in a deluge of data

Organizations are overwhelmed by the “deluge of data” flowing toward them every day. The data management burden most often falls on IT teams and specialized data teams, who are usually the only employees with the training to analyze data. Even when new specialists come on board to bolster a team, the learning curve is long and slow. 

It doesn’t help that teams aren’t unified, and neither is the data they consume. Information comes from many sources, which aren’t always easy to trace, and ends up in multiple silos. Diverse teams manage slices of data with no consistent processes in place, using different tools. The simple need for better tools all around further complicates the effort. At times it seems like data just disappears into an inexplicable black hole.

Truly insight-driven businesses — those that base decisions on data — remain in the minority today, according to the Deloitte survey, and “the most common culprit is culture.” The study concludes that “buying and using analytics tools is not hard — changing behaviors is.” The most fundamental change is “democratizing” data, which means training a broad range of employees in analytics, and arming them with tools that non-technical people can manage effectively.

For data managers, finding the right balance between changing technology and changing the culture is essential. It’s also hard to do. A Forrester Research study found that 88% are “neglecting either their technology and processes or culture and skills — or both.” A mere 12% say they’ve achieved a workable balance between culture and technology by learning to focus on both without forgetting either. Forrester dubs these rare organizations “data champions.”

What do best practices look like?

As defined in a Gartner glossary, dataops introduces collaborative data management across an organization in order to improve “communication, integration and data flows between data managers and data consumers.” dataops automates the design, deployment, and management of data delivery in a “dynamic environment,” using metadata to enhance the data’s value and usefulness. By ensuring “predictable delivery and change management of data, data models and related artifacts,” dataops almost inherently promises to usher in best practices.

Organizations can derive even more value from their data by adopting an advanced data lineage platform, one that can provide an automated, in-depth, multi-dimensional view of the data journey. This allows data managers — and other users, from executives to the front lines—high-level visibility into the data flow, with the ability to map where data is coming from and where it’s going, from original source all the way through to reporting and analytics. This type of lineage uses automated and augmented methods to build an extensive cross-system view of all an organization’s data, no matter which silo it lives in, including all data flows and dependencies. 

And, perhaps most significantly, the data lineage solution empowers everyone to become their own data expert. Anyone can view the entire data landscape on one screen, pulling data from any source onto one automated platform, and do it themselves, without assistance from IT or data specialists. This provides the answer to unifying many teams with new tools that work for everyone, and unifying data on a transparent, trusted platform. Data teams gain an added benefit as well — freedom from repetitive manual tasks.  

One source of truth

Enterprises may be overwhelmed with data today, but they want it to keep coming. A “thirst for data” keeps growing in data-based decision-makers, even as they struggle to absorb their data, Forrester reports. “Seventy percent of data decision-makers are gathering data faster than they can analyze and use it, yet 67% say they constantly need more data than their current capabilities provide.”

Add to that the familiar fact that “the cost of poor data” costs the world $3.1 trillion each year, and it’s not hard to imagine why companies are thinking about dataops, and fast-tracking the road to best practices. All of this will very much depend on whether they can establish one source of truth across the enterprise. That means the data any team wants is always there, always clear, and always trustworthy—with no more black holes.

And it’s good to remember that we all have the means to become data-based, insight-driven enterprises from top to bottom, across all teams and endeavors.

Yael Ben Arie is CEO of Octopai.