Why Data and Knowledge Silos Prevent The Digital Transformation

Author: Timm Grosser

Silos make data-driven use cases more difficult. In the worst case, they prevent them altogether. There are now plenty of solutions - both organizational and technological. This article kicks off a three-part blog series on the tensions surrounding data management in companies. Timm Grosser, Senior Analyst Data & Analytics at the Business Application Research Center (BARC), provides insights into the efficient use of data in times of complex data landscapes and diverse, enterprise-wide use cases.

Is 'data-driven' action the new normality and does it thus define the vision of a successful company? Is it already given that knowledge can be used properly in companies for corporate management, for the optimization of processes or even for the development of new, innovative products? In my opinion, the majority of companies are still a few years away from this. At many of our customers, data landscapes are disparate and not integrated. They are highly fragmented on-premise or in the cloud.  Valuable knowledge, absolutely essential for automation and decision making, is 'hiding' in expert minds. In short, silos in their various forms will still prevail in companies in 2021.

"Fragmented data landscapes are the
biggest challenge on the way to
becoming a data-driven enterprise."

For a long time, the problem solving culture was characterized by approaches limited to specific use cases such as business intelligence (with data warehouses, self-service BI approaches), advanced analytics applications (with data lakes), or operational applications such as recommendation engines for e-commerce systems. As a result, the attempt to integrate data from systems created new silos. At first glance, these isolated solutions are effective for the specific use case, but they fall short in terms of cross-company data usage. Non-integrated stand-alone solutions are thus in direct conflict with the objective of cross-system, flexible and cost-effective data utilization.

Adressing The Challenges

From our BARC perspective, data silos are becoming increasingly risky. They prevent informed and efficient decision-making processes, which are particularly sought after at management level. Yet the use of data is not new. They permeate operational processes and are used expediently for business management from subject matter experts to department heads. However, working with data or data silos in general still seems to bring some challenges, as the following excerpt from a BARC survey from 2020 shows.

At management level, recognition of the potential in data seems to be growing only gradually. Here in particular, we still see a great need for awareness-raising in order to create a clear understanding of what it means to work with data. We often see lip service such as "Yes, we want to be data-driven!" but no concepts or concrete measures (e.g., creating responsibilities and resources) to achieve this.

Use Cases Have Been Around For Some Time

Overall, the desire for better use of data is being accelerated by truly data-driven companies (e.g., in FinTech) that are slowly outpacing established companies. Those data-driven companies are expanding their market position with new innovative business models using more holistic use cases. By this we mean use cases that build on knowledge and data from across the enterprise, rather than just using data from a specific data warehouse or CRM application. There are already countless examples of this, for example:

• Demand-oriented control of production lines, in which the demand for products is calculated from current and historical purchasing behavior, external factors (e.g. weather, local events). Based on this, an optimized, partly 'automated' procurement of materials and control of production lines across companies follows. Here, many data points in the own company and from suppliers have to be linked with each other, partly in real time.
• Customer 360, in which customers are optimally provided with corresponding services along their entire life cycles in the company. From the initial contact, to the new customer, to the regular customer, to upselling, different areas (sales, marketing, portfolio management, customer care) in the company become involved. The aim is to increase customer loyalty by addressing the customer in a uniform and at the same time personal way via the best possible channels with individually tailored offers and services.

Data Itself Is Not The New Oil

The implementation of such use cases requires a fundamental rethinking of the way data is used. A mindset is needed that enables or promotes the unrestricted use of data in the company. However, the current initial situation is problematic, both technically and professionally:

• Data is fragmented. Identifying and accessing data becomes a challenge when it comes to integrating enterprise data and breaking down data silos. This is not necessarily about the physical integration of data. Ideally, there should be a simple, consistent data access interface that supports different types of applications. Though, this requires access to consolidated data. This is the only way to ensure consistent use of data, the same understanding of data, as well as data transparency.
• Data in itself does not deliver added value. It needs the right domain-oriented data for the respective use case. Data also needs context, and it must be interpretable. Documented knowledge about data helps to better understand data with its relationships and dependencies and to be able to apply or interpret it correctly. This knowledge is usually described in the form of metadata.

The extract from the BARC study 'Leverage your Data' from 2020 shows how essential knowledge about data is. 93 % of the respondents confirm the added value of documented expert knowledge. 87 % understand that they could achieve significantly more benefits for the company through better data management.

So, in order to survive in the increasingly volatile and dynamic reality, there is an urgent need for action to make data useful. It is time to put data silos to rest. Because from an analyst's point of view, use cases that require the holistic use of data will definitely increase.

Managing Data & Knowledge in Context

But what do we need to put data silos to rest? There are two main discussion points for this, besides a strategy:

• expand the organization and create data competencies, responsibilities, processes and thus develop a real data culture or
• the targeted use of technologies and architectural concepts, e.g. Data Fabric Data Catalogs and Knowledge Graphs.

Practice shows that both topics are closely connected and cannot be considered separately. For example, the implementation of technology or architecture cannot succeed without the corresponding organization. The structures and responsibilities for data are particularly relevant here. This is necessary to understand data in context. On the other hand, we often see that there is a lack of adequate architecture concepts and technologies to be able to deal with the new tasks and the complexity in data.

Our quintessence from the consulting projects is that without an actively involved organization, the introduction of a new software will not yield the best possible benefit or will even fail.

Data Is Nothing Without Meaning

Both aspects are essential to create a structure that makes finding and using data more efficient. For this to happen, knowledge about data, as well as systems, and organizational structures must be linked to the actual data in an understandable and usable form. The following excerpt from the same BARC study, in which we asked about current measures, shows that more than one in two companies is already actually tackling this. Here, most companies state that they want to create more information about their data.

Technologies such as data catalogs, knowledge graphs and data governance solutions are being fiercely debated as a technological response. At the lowest common denominator, all of them address one thing: being able to find, understand and use data. They help eliminate data silos and get a holistic picture of data, for everyone. They are meant to be easy to use and to encapsulate complexity. This is an attempt to solve primarily the challenges outlined in Figure 1.

Knowledge Graphs Build The Foundation

In consulting, we are seeing a large increase in demand for such technologies. Particularly growing interest is generated by Knowledge Graphs. The interest can be confirmed by the aforementioned BARC Survey. Already 22 % of the 354 respondents say they already use a Knowledge Graph and another 30 % are planning to. Knowledge graphs offer the potential to describe context and relationships between data objects directly on the data. By linking context to real data, knowledge graphs make data readable and interpretable by humans and machines alike. This avoids inconsistencies in the meaning of data and spares the analyst from unnecessary data preparation work or unnecessary communication loops.

The 'secret sauce' of knowledge graphs is
documenting business knowledge.

Navigation in the company data is more intuitive and thus also enables the more systematic use of automation and artificial intelligence. The latter can use the context information to develop better automatisms and suggestions to support the user in context management or also in data consumption. This is certainly one of the reasons why data catalog providers are also occasionally using knowledge graph technologies for their business glossaries. And that knowledge graph specialists are increasingly being used for overlapping use cases such as cross-enterprise production chains and data transparency.

The 'secret sauce' of knowledge graphs is documenting business knowledge – the knowledge of users – and linking it directly to the data. This means that head and data silos can be broken down in equal measure, or at least made transparent. The effort is worthwhile in most cases and is, in my opinion, inevitable to be able to act data-driven.

In the next part of the guest article series, Timm Grosser from BARC explains the functional requirements of a tool for breaking down data silos and elaborates on the argument why data is only useful with context.

——————————————————

Timm Grosser is Senior Analyst Data & Analytics at the Business Application Research Center (BARC). His expertise includes strategy definition, concept development and software selection for Data & Analytics with a focus on Data Management & Data Governance. He is a respected speaker and author and has been consulting companies of all sizes and industries since 2007.