Data Mesh: Delivering Data-Driven Value at Scale 1st Edition by Zhamak Dehghani, ISBN-13: 978-1492092391

Description

Data Mesh: Delivering Data-Driven Value at Scale 1st Edition by Zhamak Dehghani, ISBN-13: 978-1492092391

[PDF eBook eTextbook] – Available Instantly

• Publisher: ‎ O’Reilly Media; 1st edition (April 12, 2022)
• Language: ‎ English
• 384 pages (Size: 53 MB)

• ISBN-10: ‎ 1492092398
• ISBN-13: ‎ 978-1492092391

We’re at an inflection point in data, where our data management solutions no longer match the complexity of organizations, the proliferation of data sources, and the scope of our aspirations to get value from data with AI and analytics. In this practical book, author Zhamak Dehghani introduces data mesh, a decentralized sociotechnical paradigm drawn from modern distributed architecture that provides a new approach to sourcing, sharing, accessing, and managing analytical data at scale.

Dehghani guides practitioners, architects, technical leaders, and decision makers on their journey from traditional big data architecture to a distributed and multidimensional approach to analytical data management. Data mesh treats data as a product, considers domains as a primary concern, applies platform thinking to create self-serve data infrastructure, and introduces a federated computational model of data governance.

• Get a complete introduction to data mesh principles and its constituents
• Design a data mesh architecture
• Guide a data mesh strategy and execution
• Navigate organizational design to a decentralized data ownership model
• Move beyond traditional data warehouses and lakes to a distributed data mesh

Table of Contents:

Foreword

Preface

Why I Wrote This Book and Why Now

Who Should Read This Book

How to Read This Book

Conventions Used in This Book

O’Reilly Online Learning

How to Contact Us

Acknowledgments

Prologue: Imagine Data Mesh

Data Mesh in Action

A Culture of Data Curiosity and Experimentation

An Embedded Partnership with Data and ML

The Invisible Platform and Policies

Limitless Scale with Autonomous Data Products

The Positive Network Effect

Why Transform to Data Mesh?

The Way Forward

I. What Is Data Mesh?

1. Data Mesh in a Nutshell

The Outcomes

The Shifts

The Principles

Principle of Domain Ownership

Principle of Data as a Product

Principle of the Self-Serve Data Platform

Principle of Federated Computational Governance

Interplay of the Principles

Data Mesh Model at a Glance

The Data

Operational Data

Analytical Data

The Origin

2. Principle of Domain Ownership

A Brief Background on Domain-Driven Design

Applying DDD’s Strategic Design to Data

Domain Data Archetypes

Source-Aligned Domain Data

Aggregate Domain Data

Consumer-Aligned Domain Data

Transition to Domain Ownership

Push Data Ownership Upstream

Define Multiple Connected Models

Embrace the Most Relevant Domain Data: Don’t Expect a Single Source of Truth

Hide the Data Pipelines as Domains’ Internal Implementation

Recap

3. Principle of Data as a Product

Applying Product Thinking to Data

Baseline Usability Attributes of a Data Product

Transition to Data as a Product

Include Data Product Ownership in Domains

Reframe the Nomenclature to Create Change

Think of Data as a Product, Not a Mere Asset

Establish a Trust-But-Verify Data Culture

Join Data and Compute as One Logical Unit

Recap

4. Principle of the Self-Serve Data Platform

Data Mesh Platform: Compare and Contrast

Serving Autonomous Domain-Oriented Teams

Managing Autonomous and Interoperable Data Products

A Continuous Platform of Operational and Analytical Capabilities

Designed for a Generalist Majority

Favoring Decentralized Technologies

Domain Agnostic

Data Mesh Platform Thinking

Enable Autonomous Teams to Get Value from Data

Exchange Value with Autonomous and Interoperable Data Products

Accelerate Exchange of Value by Lowering the Cognitive Load

Scale Out Data Sharing

Support a Culture of Embedded Innovation

Transition to a Self-Serve Data Mesh Platform

Design the APIs and Protocols First

Prepare for Generalist Adoption

Do an Inventory and Simplify

Create Higher-Level APIs to Manage Data Products

Build Experiences, Not Mechanisms

Begin with the Simplest Foundation, Then Harvest to Evolve

Recap

5. Principle of Federated Computational Governance

Apply Systems Thinking to Data Mesh Governance

Maintain Dynamic Equilibrium Between Domain Autonomy and Global Interoperability

Embrace Dynamic Topology as a Default State

Utilize Automation and the Distributed Architecture

Apply Federation to the Governance Model

Federated Team

Guiding Values

Policies

Incentives

Apply Computation to the Governance Model

Standards as Code

Policies as Code

Automated Tests

Automated Monitoring

Transition to Federated Computational Governance

Delegate Accountability to Domains

Embed Policy Execution in Each Data Product

Automate Enablement and Monitoring over Interventions

Model the Gaps

Measure the Network Effect

Embrace Change over Constancy

Recap

II. Why Data Mesh?

6. The Inflection Point

Great Expectations of Data

The Great Divide of Data

Scale: Encounter of a New Kind

Beyond Order

Approaching the Plateau of Return

Recap

7. After the Inflection Point

Respond Gracefully to Change in a Complex Business

Align Business, Tech, and Now Analytical Data

Close the Gap Between Analytical and Operational Data

Localize Data Changes to Business Domains

Reduce Accidental Complexity of Pipelines and Copying Data

Sustain Agility in the Face of Growth

Remove Centralized and Monolithic Bottlenecks

Reduce Coordination of Data Pipelines

Reduce Coordination of Data Governance

Enable Autonomy

Increase the Ratio of Value from Data to Investment

Abstract Technical Complexity with a Data Platform

Embed Product Thinking Everywhere

Go Beyond the Boundaries

Recap

8. Before the Inflection Point

Evolution of Analytical Data Architectures

First Generation: Data Warehouse Architecture

Second Generation: Data Lake Architecture

Third Generation: Multimodal Cloud Architecture

Characteristics of Analytical Data Architecture

Monolithic

Centralized Data Ownership

Technology Oriented

Recap

III. How to Design the Data Mesh Architecture

9. The Logical Architecture

Domain-Oriented Analytical Data Sharing Interfaces

Operational Interface Design

Analytical Data Interface Design

Interdomain Analytical Data Dependencies

Data Product as an Architecture Quantum

A Data Product’s Structural Components

Data Product Data Sharing Interactions

Data Discovery and Observability APIs

The Multiplane Data Platform

A Platform Plane

Data Infrastructure (Utility) Plane

Data Product Experience Plane

Mesh Experience Plane

Example

Embedded Computational Policies

Data Product Sidecar

Data Product Computational Container

Control Port

Recap

10. The Multiplane Data Platform Architecture

Design a Platform Driven by User Journeys

Data Product Developer Journey

Incept, Explore, Bootstrap, and Source

Build, Test, Deploy, and Run

Maintain, Evolve, and Retire

Data Product Consumer Journey

Incept, Explore, Bootstrap, Source

Build, Test, Deploy, Run

Maintain, Evolve, and Retire

Recap

IV. How to Design the Data Product Architecture

11. Design a Data Product by Affordances

Data Product Affordances

Data Product Architecture Characteristics

Design Influenced by the Simplicity of Complex Adaptive Systems

Emergent Behavior from Simple Local Rules

No Central Orchestrator

Recap

12. Design Consuming, Transforming, and Serving Data

Serve Data

The Needs of Data Users

Serve Data Design Properties

Serve Data Design

Consume Data

Archetypes of Data Sources

Locality of Data Consumption

Data Consumption Design

Transform Data

Programmatic Versus Nonprogrammatic Transformation

Dataflow-Based Transformation

ML as Transformation

Time-Variant Transformation

Transformation Design

Recap

13. Design Discovering, Understanding, and Composing Data

Discover, Understand, Trust, and Explore

Begin Discovery with Self-Registration

Discover the Global URI

Understand Semantic and Syntax Models

Establish Trust with Data Guarantees

Explore the Shape of Data

Learn with Documentation

Discover, Explore, and Understand Design

Compose Data

Consume Data Design Properties

Traditional Approaches to Data Composability

Compose Data Design

Recap

14. Design Managing, Governing, and Observing Data

Manage the Life Cycle

Manage Life-Cycle Design

Data Product Manifest Components

Govern Data

Govern Data Design

Standardize Policies

Data and Policy Integration

Linking Policies

Observe, Debug, and Audit

Observability Design

Recap

V. How to Get Started

15. Strategy and Execution

Should You Adopt Data Mesh Today?

Data Mesh as an Element of Data Strategy

Data Mesh Execution Framework

Business-Driven Execution

End-to-End and Iterative Execution

Evolutionary Execution

Recap

16. Organization and Culture

Change

Culture

Values

Reward

Intrinsic Motivations

Extrinsic Motivations

Structure

Organization Structure Assumptions

Discover Data Product Boundaries

People

Roles

Skillset Development

Process

Key Process Changes

Recap

Index

About the Author

Zhamak Dehghani is a director of technology at ThoughtWorks, focusing on distributed systems architecture — big data and operational systems — in the enterprise. She’s a member of the company’s Technology Advisory Board and contributes to the creation of ThoughtWorks’s Technology Radar. She is an advocate for decentralization of all things – architecture, data and ultimately power. She is the founder of data mesh.

What makes us different?

• Instant Download

• Always Competitive Pricing

• 100% Privacy

• FREE Sample Available

• 24-7 LIVE Customer Support