This layer filters the good from the bad data and is crucial for ensuring high data qual- ity on the platform. This part was one of the most challenging, as the speed of the data transformations had to be very performant due to the requirements for near-real time. Solving this challenge requires a deep understanding of Snowflake’s query engine and of techniques on how to optimize its query execution. Snowflake’s underlying architec- ture helps in achieving the required performance, as the separation of storage and com- pute eliminates resource conflicts between layers, while an insert-only strategy and micro partitioning ensure fast, optimized data transformations. The Business Data Vault functions as a Golden Record Storage for the Siemens AG prod- ucts. Here, only those products are shown that successfully passed all quality checks from the Consistency Layer. The Information Delivery Layer is fully virtualized. Since most of the data transfor- mations were already handled in the underlying layers, this layer focuses on presenting data in a user-friendly format and enforcing role-based access control. As Data Vault is a very good approach to model data at scale, it is not the best modeling technique to represent the data to non-technical users. By using virtualized views instead of physical tables, the need for additional storage and data movement is eliminated entirely. The presented architecture was primarily implemented by Scalefree experts. Addition- ally, the open-source dbt package DataVault4dbt (created by Scalefree and available open source) was essential by providing standard loading patterns to generate the ma- jority of the Integration Layer code. For complex logic, developer-built templates (mac- ros) and generative AI-assisted development work. Developers work in fully isolated computing environments, ensuring that production is never impacted. Building on the foundation of the DataVault4dbt package, the de- velopment team worked on an additional automation layer that eliminates repetitive manual work and reduces error-prone- ness. Since factory source data follows a near-identical structure across integrations, a custom automation script was developed that automatically generates all DataVault4dbt configuration files required to build the Raw Data Vault entities. This shifts the team’s focus away from manually authoring and adjusting mod- els towards testing and validation, significantly accelerating the integration of new factory data sources.
UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM
5/12
Powered by FlippingBook