METADATA-DRIVEN SCHEMA EVOLUTION With the objective of a company-wide data integration, the decision was made to create a common data model. This has the advantage of streamlining and standardizing down- stream processes. On the other hand, the common data model evolves over time, which leads to reengineering work. To ensure controlled and scalable schema evolution of the common data format, ded- icated Meta Marts were introduced. These marts contain structural metadata describ- ing every object, including column names, data types, descriptions, default values, and sample values. As the model specification evolves over time, the Meta Marts reflect these changes ac- cordingly. By centralizing structural definitions, manual adjustments across multiple models are avoided. Instead of adapting each downstream model individually, schema changes are managed once within the metadata layer and propagated automatically. COMPANY-WIDE DATA INTEGRATION The Siemens AG stood in front of a strategic decision and a pivotal shift in how data is unified across the organization, specifically when it comes to the integration of prod- ucts/assets from different business units. Supported by Scalefree’s strategic advisory, the initiative established a centralized data platform that serves as a single point of truth, moving away from decentralized, fragmented data silos. The unified data plat- form aims to provide a cross-functional, consistent access point for the data. The core of this project was architectural scalability. To execute this idea, Data Vault 2 provides the ideal framework by allowing the integration of data from different sourc- es. Furthermore, the possibility to scale in an efficient way as soon as new business units need to be integrated into the platform is supported by this modeling approach since the standardized framework allows scalability without reengineering overhead. The initiative directly addresses the complexity of the legacy systems by consolidating disparate systems into a unified, transparent data platform that eliminates redundant processes and drastically reduces maintenance overhead. Finally, this streamlined ap- proach replaces a patchwork of “black-box” solutions with a transparent, high-perfor- mance ecosystem.
The structural definitions are maintained in a governed docu- ment that serves as the single source of truth. Editing permis- sions are restricted to selected users to prevent inconsistencies and unintended changes that could compromise downstream data integrity. By querying the Meta Marts at runtime, models automatically incorporate structural changes as they occur, elim- inating the need for manual model adjustments. Models reference columns via their technical identifier rather than hard-coded names. This approach allows renaming, addi- tions, or data type adjustments within the format without requir- ing structural rewrites in downstream transformations.
The result is a metadata-driven framework that enables controlled schema evolution while minimizing operational overhead.
UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM
6/12
Powered by FlippingBook