Data Lattice

According to a recent survey by Broadridge in 2022, companies have reached the later stages of adoption of advanced data visualization and predictive analytics: 95% in Leader category, 74% in Advancer category, 44% in Implementer category, and only 11% in Beginner category. Despite the successful adoption, organizations have been facing a variety of obstacles and impediments. For example, the performance is slow when handling a large volume of data, and the system does not scale well. It is difficult to access and aggregate data without the assistance from data engineers. It is hard to establish a single source of truth because data disperses over many silos. The communications and collaborations between the stakeholders are typically disjointed and inefficient.

It is not only important but also necessary to have a holistic approach to address the above-mentioned barriers and challenges.

Data Lattice is a unified data architecture platform for next-generation data processing. It provides a machine-enabled integrated architectural framework that links, unifies, integrates, orchestrates, and governs disparate data sets, repos, and stores in a metadata-driven fashion. It improves synchronous and asynchronous accessibility, quality, reliability, scalability, and security with end-to-end integration of data, ML, and CICD pipelines on on-premises, cloud, multicloud, hybrid, edge, and field device environments. It facilitates data-centric applications and systems to process and manipulate data while working with diverse services. It offers an overarching discipline to standardize, synergize, and automate data management processes and best practices for data transactional processing, analysis, analytics, digital UX, metaverse, and ambient computing.

Data Lattice is composed of a pyramid and a set of pillars. The pyramid comprises 8 planes:

• Access
• The endpoints, connection methods, and protocols to feed, move, or retrieve data
• Orchestration
• Coordination, mediation, and choreography of processes and capabilities
• Composite
• Service mesh and aggregation of functions or data elements
• Analysis
• Business intelligence, data warehousing, visualization, and charting
• Analytics
• Machine learning, Artificial intelligence, statistical/quantitative analytics
• Persistence
• Persisted data in file/object stores, repos, databases, datastores and digital ledger in SQL and NoSQL types
• Environment
• The lower and production runtime settings and facilities with compute, memory, storage, and networking capacity
• Monitoring
• The current status, activities, and health of the systems to observe, track, analyze and manage services and applications

The 12 pillars in Data Lattice are: Metadata, Master & Reference Data, Quality, Document & Content, Digital, Design, Development, Engineering, XOps, Management, Security, and Governance.

Data Lattice provides business and technical values in a number of ways:

• Enhanced data orchestration
• Different techniques and enabling technologies for data routing, composition, transformation, masking, etc.
• Comprehensive data management
• Various tools to configure, monitoring and administrate data with UI console and dashboards
• More data types and methods
• Data in-motion and data at-rest from on-premises, cloud, edge, IoT devices, and third-party sources
• Automated pipeline management
• Data and AI pipelines with DataOps and MLOps practices
• Rapid integration and deployment
• CICD pipeline to facilitate GitOps from build to install
• Full compliance of data privacy
• Data access managed and audited, complying with regulations like GDPR
• Optimized total cost of ownership
• In-memory computing on commodity hardware, container hosting, or serverless with linear scalability and autoscaling
• Greater deployment flexibility
• Phase-based flexible deployment across distributed on-prem, hybrid, and multi-cloud environments

Details of Data Lattice are provided in a rich set of technical reports, practitioner's guides and workshops, which cover the "how" - applying Data Lattice in real-world projects and accelerating execution of data initiatives in enterprise via a community of practice.

For more information, please contact Tony Shan (blog@tonyshan.com) or leave your comments below.

  ©Tony Shan. All rights reserved. All standard disclaimers apply here.