What is Data Mesh?

Data mesh is a distributed data management model that helps solve advanced data sharing challenges through domain ownership, data productization, self-service, and federated governance. Data mesh moves away from centralized data concepts within the data warehouse and data lake architectures. These architectures can lead to siloed teams: overwhelmed data engineers, uninformed business teams, and disappointed data consumers. Organizations have multiple data sources from different lines of business that must be integrated for analytics. However, instead of uniting the disparate data sources and linking them together through a centrally managed data system, each domain in the data mesh serves its own data to other functions. Domains can be aligned to the business or to a consumer use case. Data instead conforms to organization-wide standards. A data mesh adds complexities to data architectures but also brings efficiency by improving data access, scalability, and accuracy.

Although organizations have access to ever-increasing data volumes, they have to sort, filter, process, and analyze the data to derive practical benefits. Organizations often use a centralized data team of engineers and scientists for managing data. The team uses a centralized data solution for the following purposes:

• Ingest the data from all the different business units (or business domains).
• Transform the data into a consistent, trustworthy, and useful format. For example, the team could make sure all dates in the system are in a common format or summarize daily reports.
• Prepare the data for data consumers, for example, generating reports for humans or preparing XML files for applications.

When the data volume increases, organizations face increasing costs to maintain the same agility as before. The monolithic system is difficult to scale for the following reasons.

Siloed data team

The central data team has specialist data scientists and engineers with limited business, consumer, and domain knowledge. However, they still have to provide data for a diverse set of operational and analytical needs without a clear understanding of motivation and intent.

Slow responsiveness to change

Data engineers typically implement data pipelines that ingest information and transform it over several steps before storing it in a central data lake. Any requested changes require modifications to the entire pipeline or the creation of a new pipeline. The central data team has to make these changes while managing conflicting priorities and with limited domain knowledge.

Reduced accuracy

Business units are disconnected from the data consumers and the central data team. As a result, they lack the incentive to provide meaningful, correct, and useful data.

Your organization must implement the following four principles to adopt the data mesh paradigm. These are the core principles that allow a mesh data infrastructure to work at scale.

Domain-driven and decentralized data architecture

The data mesh approach proposes that data management responsibility is organized around functions or domains. Domain teams are responsible for collecting, transforming, and providing data related to, or created by, their individual business functions or consumer use cases. Instead of domain data flowing from data sources into a central data solution, a specific team hosts and serves its datasets in an easily consumable format. For example, a retailer could have a clothing domain with data about their clothing products and a website behavior domain that contains site visitor behavior analytics.

Data products

For data mesh implementation to be successful, every domain team needs to apply product thinking to the datasets they provide. They must consider their data assets as their products and the rest of the organization's business and data teams as their customers.

For the best user experience, the domain data products should have the following basic qualities.

Discoverable

Each data product registers itself with a centralized data catalog for easy discoverability.

Addressable

Every data product should have a unique address that helps data consumers access it following standardized access patterns. The address typically follows centrally decided naming standards within the organization.

Trustworthy

Data products define acceptable service-level objectives around how closely the data reflects reality. For example, the website domain must first clean any bot traffic from its user analytics data product.

Self-describing

All data products have well-described syntax and semantics that follow standard naming conventions determined by the organization. Data must be understandable at a glance.

Interoperable

Data products must adhere to a common set of governance standards so that disparate teams immediately know how to access, manipulate, combine, and transform data.

Secure

Data products must use identity-based controls, including role-based access control (RBAC), to help secure the data.

Self-serve data infrastructure

A distributed data infrastructure requires every domain to set up its own data pipeline to clean, filter, and load its own data products. A data mesh introduces the concept of a self-serve data infrastructure solution to avoid duplication of this pipeline each time. Data engineers set up technologies so that all business units can ingest, transform, process, and host their data products. Self-serve infrastructure thus allows a division of responsibility. Data engineering teams manage the technology and infrastructure, while business teams manage the data.

Federated data governance tools

Data mesh architectures implement governance as a shared responsibility within the organization. Leadership determines global standards and policies that you can apply across domains. At the same time, the decentralized data architecture allows a large degree of autonomy within each domain, so long as data products have a standardized external interface. Data engineers from different domain teams can implement data governance tools to best meet the organization’s needs.

Over time, a centralized data architecture can result in disconnected data producers, an overloaded data management team, and frustrated data consumers. Data mesh architecture attempts to solve these challenges by delegating high autonomy and ownership within data domain teams, and allowing data management teams to focus on reusable, domain-agnostic infrastructure.

Here are the benefits of data mesh architecture.

Improved data accessibility

A data mesh transfers data control to domain experts who create meaningful data products within a decentralized governance framework. Data consumers request access to the data products and seek approvals or changes directly from data owners. As a result, everyone gets faster access to relevant data, without necessarily involving the data management team.

Increased flexibility

Centralized data infrastructure is more complex and requires collaboration to maintain and modify. Instead, the data mesh distributed the technical implementation of the central system across the business domains. This decentralized approach removes central data pipelines, reducing operational bottlenecks and technical strains on the system. The data management team create reusable pipelines for all domain teams, who can implement the solution with limited technical expertise, by self-service interfaces.

Improved domain data discovery and sharability

A data mesh model prevents bottlenecks from forming around a central data team. It also reduces the risk of data assets getting siloed within different business domain systems, due to its inherent sharability. Instead, the central data management framework governs and records the data available in the organization. For example, domain teams automatically register their data in a central registry.

Improved cross-domain governance

Data mesh architectures enforce data governance and security policies both within and between domains. They can assist in centralized monitoring and auditing of the data sharing process. For example, you can enforce log and trace data requirements on all domains. Your auditors can observe the usage and frequency of data access.

Improved interoperability with federated governance

A data mesh architecture improves interoperability by standardizing how domain data is processed, accessed, and consumed across a business. Every domain publishes data products using the same shared schemas and metadata conventions, according to a federated governance model.

Because data products are self-describing and registered centrally, business users can easily discover and integrate data products. This significantly reduces the backlog of single-solution work required of data engineers.

Data mesh supports all types of big data use cases. Here are some example use cases of the data mesh.

Data analytics

Multiple business functions host trusted, high-quality data for data analytics workloads. Your teams can use the data to create customized business intelligence solutions and dashboards showcasing project performance, marketing results, and operational data.

Customer care

Data mesh promotes the idea of customer domain use cases, which empowers support and marketing teams. For example, support teams can pull relevant data and reduce average handle time, and marketing teams can ensure they target the right customer demographics in their campaigns.

Chatbots and virtual assistants

A data mesh model gives virtual assistants and chatbots direct access to high-quality domain data across multiple business domains. When domains produce their own products, chatbots can access high-quality data from specific areas, such as support history or customer profiles. Because data ownership is defined within the data mesh system, there is a high degree of accountability that keeps data quality as high as possible.

You can implement data access controls to make sure that these automated chatbots only have permission to read and use certain datasets. This approach helps to uphold security and compliance requirements.

Regulatory reporting

The need for volume, timeliness, and accuracy in data that meets regulatory objectives places challenges on both regulators and regulated organizations. All parties can benefit from the application of data mesh technologies.

Third-party data

You can apply data mesh technology for use cases that need third-party and public datasets. You can treat external data as a separate domain and implement it in the mesh to maintain consistency with internal datasets.

Machine learning projects

Data scientists can accelerate machine learning and AI projects to derive the full benefits of business data. The design of the data mesh means that data scientists can discover, access, and aggregate data from different domains without having to duplicate pipelines. By checking for data quality and versioning across these integration pipelines, teams can make sure that the information arriving to machine learning systems is as accurate and useful as possible.

In a data mesh, there is the concept of data as a product, which means datasets are built and used similarly to customer-facing products. Each data product is a structured, reusable component that both serves a specific business purpose and can also be independently used by other teams.

Although raw data is normally closely connected to its source system, data products are structured, analytical system-ready data that teams can put to use immediately. Although a person might not have interacted with that data before, they can trust that the domain teams that worked on it adhered to all data governance obligations to make it trustworthy and ready for analytics.

Organizations implement shared data models and standards, which allow domains to publish their own interoperable data products. When it comes time to use these products, other domains can combine products without having to build custom data integration workflows. For example, you can join a data product from sales with one from marketing, as their structures and interfaces are centrally defined.

Operating a data mesh requires a structured approach to how different data teams structure information and how data ownership is established across your business. Unlike being able to rely on a single centralized data team, individual domain teams take responsibility under the data mesh architecture.

Every domain is responsible for its own data, from source to publishing a data product. From operational data to curated, ready-to-use analytical data, products become data services that teams can access at any time. Clearly defining who takes ownership of each dataset helps prevent quality issues from arising.

There is still a central data team in the data mesh structure. The difference is that the data team focuses on building out shared capabilities for data infrastructure, such as organizational standards and tooling that others can use. They handle less productization, but mandate data governance structures that influence every single domain.

The decentralized ownership uses these standards to ensure interoperability and help avoid data silos. Business users can access data directly from each domain that produces it, leveraging the common data model and structure to ensure it integrates easily.

Organizations can experiment with different technologies to build a data mesh architecture. There is no fixed path to data mesh implementation; however, here are some suggestions to help guide your structure.

Analyze your existing data and data owners and create a roadmap

Before building a data mesh, you must identify relevant business domains, consumer domains or use cases, catalog your existing data, and extract current data pipelines. Creating a data mesh architecture is an evolutionary process of your existing teams, practices, and technologies.

You create a roadmap for data mesh implementation, involving a phased approach, starting with a small subset of domains.

Support an organization-wide cultural shift

Today, we have the technology and tools required to easily build a data mesh with multiple data products. However, scaling your data mesh beyond small projects necessitates a paradigm shift away from centralized data architectures.

It requires a new, organization-wide approach that emphasizes the following:

• Data discovery and usage over extraction and loading
• Distributed data product ownership over a central data solution architecture
• Common data knowledge over data management team bottlenecks

Currently, data technology often drives architectural decisions. A data mesh reverses this flow, putting domain data products in the center so that they drive technology decisions.

Early on in your data mesh journey, you will need to begin educating and preparing your employees for this new way of approaching data. Because implementing a data mesh involves creating different teams, roles, and responsibilities for your existing employees, you will need to consider change management processes.

Implement global data governance policies

Federated data governance requires your central data team to identify global standards, reporting structures, security and authentication, and compliance standards for the data mesh. You can also define granular access controls that data product owners apply when hosting their datasets.

For example, you need to define global standards for field type formatting, use metadata management, and employ data product address conventions. Although a data mesh focuses on decentralized data ownership, the individual data products follow the same structure to help ensure data quality and interoperability.

Data producers define and measure data quality and central governance policies help guide their decisions.

Build your self-serve data solution

Your self-serve data solution should be generic, so that anyone can build new domain data products on it. It should also hide underlying technical complexity and provide infrastructure components in a self-serve manner.

Your data team builds generic, reusable data pipelines to ingest, process, and output data that are easy to follow for non-technical domain owners.

Here are some capabilities to include:

• Data encryption
• Data product schema
• Governance and access control
• Data product discovery, such as catalog registration or publishing
• Data product logging and monitoring
• Caching for improved performance

You can also build automation, such as configurations and scripts, to lower the lead time to create data products. This will help people within your business that aren’t data engineers use the system as a form of self-service data infrastructure.

Choose the right technologies to help implement data mesh

Your existing traditional storage systems, such as data warehouses and data lakes, can also power your data mesh. Here, you shift their use from monolithic systems to multiple decentralized data repositories. A data mesh enables the adoption of cloud platforms and cloud-centered technologies. Cloud infrastructure can reduce operational costs and the effort required to build a data mesh. Choose a cloud provider with rich data management services to support your data mesh architecture. You need to consider data integration requirements with legacy systems.

A data lake is a repository where you can store all your structured and unstructured data without any preprocessing and at any scale. In centralized data platforms, the data lake is the core technology for storing data from all possible sources.

A data mesh is a data management paradigm that uses data lakes differently. A data lake is no longer the centerpiece of the whole architecture. Instead, you can use multiple, single data lakes to implement data products as a part of the self-serve infrastructure.

A data fabric is another modern data architecture that seeks to help with a unified picture of data for all users. You can think of it as a technology layer over your underlying infrastructure that cohesively integrates and presents data to both technical and non-technical users. For example, decision-makers use the data fabric to view all their data in one place and make connections between disparate datasets. Existing data solutions and technologies integrate with the data fabric and share data and metadata.

Both data fabric and data mesh have similar goals for unified and effective data management. For instance, you have a central data lake and use AWS services for data ingestion. At the same time, you have legacy infrastructure for data transformations. Your data fabric integrates both systems and presents a unified view without changing the existing pipeline.

A data fabric architecture thus uses technology to work with your existing infrastructure. On the other hand, a data mesh implementation requires you to change the underlying infrastructure itself and your team roles. You have to change your data management's push-and-ingest model to a serve-and-pull model across each of your business domains and consumer use cases.

AWS offers a wide range of data services you can use to implement data mesh and other modern data architectures in your organization. You can rapidly build data products and data mesh infrastructure at a low cost without compromising performance.

Here are examples of AWS services you can use:

• Amazon Sagemaker for analytics and AI for your data products
• AWS Lake Formation to build the data mesh pattern at scale with tag-based access control
• AWS Data Exchange to integrate third-party data into your data mesh
• AWS Glue for sharing, hosting, and cataloging data products

Get started with your data mesh on AWS by creating a free account today.