The complexity of the data estate is a big challenge for decision makers

A new report from Capital One and Forrester also reveals that respondents are struggling to manage data at scale overall.

Image: Adobe Stock

At a time when organizations are increasingly adopting a cloud-first approach and global spending on big data services is expected to reach $273 billion in 2026, 82% of data management decision makers are deeply concerned about controlling and predicting rising data costs, a new study finds.

Of most concern is that one of the biggest data challenges facing decision makers today is fully understanding the complexity of their organizations’ data estates, according to the Capital One and Forrester report. And nearly three-quarters of data management decision makers have yet to manage most of their organizations’ data in the cloud, according to the study.

More than half of decision makers surveyed (56%) indicated that their organizations maintain a centralized data estate that aggregates most or all organizational data. However, a small percentage (15%) are federated, which means there is a central set of data policies and platforms, but lines of business manage data independently through self-service, according to the report.

“A federated model supports the inevitable expansion of a data estate as data volumes, sources, and tools multiply, eliminating potential bottlenecks in the enterprise. However, while federating and enabling self-service, the data estate must still remain well-managed,” the report states.

Among other discoveries:

  • Talent challenges are the biggest barriers to data management excellence. Decision makers don’t have enough people, internal expertise, or cross-team collaboration and tools to achieve their results.
  • The costs of using data are increasingly unpredictable, with 82% of data management decision makers citing the difficulty of predicting and controlling costs as one of the top challenges in the data ecosystem.
  • Nearly 80% of decision makers cited the lack of data cataloging as a major challenge. Without data cataloging, decision makers struggle to understand what data they have, how it is used, and who owns it.
  • Other top concerns include poor data quality (80%), difficulty understanding the data (76%), and lack of data observability (74%).

SEE: Recruitment Kit: Database Engineer (TechRepublic Premium)

Monitoring and enforcement of data governance policies is another concern

Data governance is critical to business success because it “makes data relevant and ready for analysis, understanding, and action,” the report notes. “If users can’t access the data, they can’t mix it; if the data is of poor quality and/or does not correspond to their needs or area of ​​responsibility, they will not use it.

Yet one of the biggest challenges for data management decision makers is following and enforcing data governance policies. Only 20% say they have no difficulty governing data at scale. Eighty-two percent noted confusing data governance policies and 80% noted a lack of rights and enforcing role-based access to specific data as top challenges.

Data management capabilities expected to drive spending

Policymakers surveyed expect investment in rights to nearly double, from 17% today to 31% in one to two years, according to the report.

Nearly half of respondents said cloud data investment priorities going forward will include protection and security (49%), tools that make it easier to find and use data (47%) , as well as tools that reduce and manage costs (39%).

Multiple providers will be used for different use cases

The data management decision makers surveyed understand that partners are essential in helping them manage their organization’s data. Eighty-seven percent of respondents use commercial tools, while just over a quarter (28%) use a combination of in-house or open-source tools.

To create a more diverse data estate, organizations need to get their data from a variety of sources. Over the next two years, the majority of decision makers (84%) plan to use multiple vendors or different vendors for different use cases.

“This stands in stark contrast to previous decades where a single vendor provided most of the functionality needed to manage data estates,” the report notes. “Today, only 10% of respondents use different vendors for different use cases. Over the next two years, multi-vendor usage will nearly quadruple to 38%. »

SEE: Microsoft Power Platform: What You Need to Know About It (Free PDF) (TechRepublic)

Consider tools that provide visibility into data costs

As organizations progress in their cloud journey, they must adjust how they balance the use of global and local data with a federated approach to ensure governance requirements are met and the access, quality and observability of data are optimal, suggests the study.

Organizations need to invest in tools that control and forecast data costs. The inherent exponential costs associated with operating in the cloud must be addressed early to ensure success, the report states.

Additionally, “as organizations progress in their cloud journey, they must adjust how they balance the use of global and local data with a federated approach to ensure governance requirements are met” to ensure the best access, quality and observability to data, the report says.

Capital One commissioned this report and Forrester Consulting supplemented this research with custom survey questions posed to 157 North American data management decision makers. The survey was conducted in March 2022.

Sharon D. Cole