Siemens Success Story

BI,Sucess Story

SUCCESS STORY

UNIFYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

INTRODUCTION In modern manufacturing and enterprise environments, frag- mented data silos and the limitations of legacy on-premise infra- structure often create significant bottlenecks. Facing this exact challenge, Siemens AG - Digital Industries recognized the need for a unified, enterprise-wide solution. To simplify its IT landscape and unlock near-real-time insights, Siemens AG initiated a strategic migration in close partnership with Scalefree International GmbH. By leveraging state-of-the- art technologies and modern methodologies, the initiative aimed for a fundamental shift towards a transparent data ecosystem built for sustained scalability and high data quality. INITIAL SITUATION AND CHALLENGES The need for a new data platform originally arose in the Siemens AG business unit “Digital Industries”. The legacy on-premise solution was unable to store and process the vast amounts of data required. This was mainly due to the fact that the data processing was sequential, resulting in long loading cycles. Since many other Siemens business units had their own product management plat- forms, which led to scattered data silos, it was decided to integrate products from the whole Siemens company into one single platform. This approach was mainly driven by the goal to simplify the IT landscape and utilize the speed and flexibility of standard- ized processes. To overcome these challenges, a migration project was initiated. The selected target da- tabase is the Snowflake Cloud, with dbt Platform as a data transformation and orches- tration tool, as well as Data Vault 2 as a modeling approach and methodology.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

2/12

STRATEGIC OBJECTIVES AND KEY RESULTS To address the limitations of the legacy infrastructure and the fragmentation of data across business units, the joint Siemens AG and Scalefree team developed a structured migration strategy. This chapter outlines the primary technical and organizational ob- jectives defined to ensure the new platform meets the scalability and integration re- quirements.

— Evaluating & Validating the modern tech stack — Architecting for near-real-time delivery using Data Vault 2 — Company-wide data integration — Automate metadata-driven schema evolution — Data Quality Improvement — Ensuring platform transparency — Streamlining delivery

EVALUATING & VALIDATING THE MODERN TECH STACK

The entire project was launched with a workshop led by Scalefree that focused on iden- tifying the requirements for the entire data platform, particularly for the new architec- ture and the overarching goals. In the first phase of the project, a common understanding of the core business concepts and a common data model were created, which served as the basis for further imple- mentation and the derivation of business rules. Before the start of the first implemen- tation phase, important standards were defined, such as naming conventions, working methods, or project management. With this foundation, a proof of concept was conducted to elaborate on the capabilities of Snowflake and dbt Platform in combination with Data Vault 2. In this POC, the tech- nical setup of the Data Platform was initially created, as well as the first end-to-end data flows from the sources to the final information model. This phase demonstrated that the requirements of the new data platform could be met.

DATA STRATEGY WORKSHOP TAILORED EXPERIENCE FOR YOU AND YOUR TEAM Within two to three working days on-site,we focus on your currently urgent tasks and relevant topics to your project.

https://scalefr.ee/ds-workshop

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

3/12

ARCHITECTING FOR NEAR-REAL-TIME DELIVERY USING DATA VAULT 2 One big objective was the near-real-time capability of the whole architecture. Data ar- rives via Kafka streams in a semi-structured data format and should be visible in the information marts within several minutes. With Data Vault 2, teams can process differ- ent data formats (structured, semi-structured, or unstructured) at different velocities (real-time, batches) and integrate everything into a single platform. A major advantage is the independence of data sources, which allows data to be processed as soon as it arrives. If near-real-time is required, source systems can either send data on demand via an API or continuously via event streaming, depending on the use case, integration pattern, and the capabilities of the source system itself. Next to data streaming, the architecture supports the classic batch processing. In all cases, the data runs through all layers of the architecture, as shown in the figure.

The first layer is the Persistent Staging Area (PSA). Here, the incoming data arrives and is stored in its raw format without any modifications. This is particularly useful for the semi-structured data coming from the Kafka streams. As the structure can differ from file to file, the PSA is able to store all kinds of data, no matter their data structure. In the Integration Layer , the semi-structured data is broken down into individual log- ical parts and is partially converted into a relational structure. This is essential to ac- celerate the performance of downstream data transformations. At this place (Raw Data Vault), the semi-structured data and the relational data are integrated. Besides the Raw Data Vault, the Integration Layer features the consistency layer, where complex busi- ness logic is implemented.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

4/12

This layer filters the good from the bad data and is crucial for ensuring high data qual- ity on the platform. This part was one of the most challenging, as the speed of the data transformations had to be very performant due to the requirements for near-real time. Solving this challenge requires a deep understanding of Snowflake’s query engine and of techniques on how to optimize its query execution. Snowflake’s underlying architec- ture helps in achieving the required performance, as the separation of storage and com- pute eliminates resource conflicts between layers, while an insert-only strategy and micro partitioning ensure fast, optimized data transformations. The Business Data Vault functions as a Golden Record Storage for the Siemens AG prod- ucts. Here, only those products are shown that successfully passed all quality checks from the Consistency Layer. The Information Delivery Layer is fully virtualized. Since most of the data transfor- mations were already handled in the underlying layers, this layer focuses on presenting data in a user-friendly format and enforcing role-based access control. As Data Vault is a very good approach to model data at scale, it is not the best modeling technique to represent the data to non-technical users. By using virtualized views instead of physical tables, the need for additional storage and data movement is eliminated entirely. The presented architecture was primarily implemented by Scalefree experts. Addition- ally, the open-source dbt package DataVault4dbt (created by Scalefree and available open source) was essential by providing standard loading patterns to generate the ma- jority of the Integration Layer code. For complex logic, developer-built templates (mac- ros) and generative AI-assisted development work. Developers work in fully isolated computing environments, ensuring that production is never impacted. Building on the foundation of the DataVault4dbt package, the de- velopment team worked on an additional automation layer that eliminates repetitive manual work and reduces error-prone- ness. Since factory source data follows a near-identical structure across integrations, a custom automation script was developed that automatically generates all DataVault4dbt configuration files required to build the Raw Data Vault entities. This shifts the team’s focus away from manually authoring and adjusting mod- els towards testing and validation, significantly accelerating the integration of new factory data sources.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

5/12

METADATA-DRIVEN SCHEMA EVOLUTION With the objective of a company-wide data integration, the decision was made to create a common data model. This has the advantage of streamlining and standardizing down- stream processes. On the other hand, the common data model evolves over time, which leads to reengineering work. To ensure controlled and scalable schema evolution of the common data format, ded- icated Meta Marts were introduced. These marts contain structural metadata describ- ing every object, including column names, data types, descriptions, default values, and sample values. As the model specification evolves over time, the Meta Marts reflect these changes ac- cordingly. By centralizing structural definitions, manual adjustments across multiple models are avoided. Instead of adapting each downstream model individually, schema changes are managed once within the metadata layer and propagated automatically. COMPANY-WIDE DATA INTEGRATION The Siemens AG stood in front of a strategic decision and a pivotal shift in how data is unified across the organization, specifically when it comes to the integration of prod- ucts/assets from different business units. Supported by Scalefree’s strategic advisory, the initiative established a centralized data platform that serves as a single point of truth, moving away from decentralized, fragmented data silos. The unified data plat- form aims to provide a cross-functional, consistent access point for the data. The core of this project was architectural scalability. To execute this idea, Data Vault 2 provides the ideal framework by allowing the integration of data from different sourc- es. Furthermore, the possibility to scale in an efficient way as soon as new business units need to be integrated into the platform is supported by this modeling approach since the standardized framework allows scalability without reengineering overhead. The initiative directly addresses the complexity of the legacy systems by consolidating disparate systems into a unified, transparent data platform that eliminates redundant processes and drastically reduces maintenance overhead. Finally, this streamlined ap- proach replaces a patchwork of “black-box” solutions with a transparent, high-perfor- mance ecosystem.

The structural definitions are maintained in a governed docu- ment that serves as the single source of truth. Editing permis- sions are restricted to selected users to prevent inconsistencies and unintended changes that could compromise downstream data integrity. By querying the Meta Marts at runtime, models automatically incorporate structural changes as they occur, elim- inating the need for manual model adjustments. Models reference columns via their technical identifier rather than hard-coded names. This approach allows renaming, addi- tions, or data type adjustments within the format without requir- ing structural rewrites in downstream transformations.

The result is a metadata-driven framework that enables controlled schema evolution while minimizing operational overhead.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

6/12

DATA QUALITY IMPROVEMENT An important component of the project was improving data quality. This is necessary, for example, to identify duplicates or to enrich incomplete data records. However, not all data records can be covered by business logic. Therefore, a solution was needed that explicitly catches and processes those records that fail a business rule (bad data). With Scalefree’s guidance, the joint team developed dedicated quality marts to capture the failed records. These records were forwarded to a user interface to be displayed to qualified persons for analysis and correction. The enriched data is then sent back into the Data Vault via a Kafka Stream. In this way, the UI functions simultaneously as a data source and destination.

This architecture creates a data flow that gradually increases data quality and turns bad data into high-quality data.

ENSURING PLATFORM TRANSPARENCY Operational stability and fiscal responsibility are maintained through a comprehensive monitoring framework. Cost management is centralized within Grafana , providing re- al-time visibility into resource consumption across the Snowflake environment. To pre- vent budget overruns, automated email notifications are triggered when specific cost thresholds are met, ensuring that responsible stakeholders are alerted early enough to take proactive measures.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

7/12

In addition to financial oversight, the infrastructure includes dedicated performance monitoring to track the processing times of individual dbt models. This allows for the immediate identification of bottlenecks or inefficient logic within the data pipeline. Complementing these metrics is a specialized monitor that tracks the successful pro- cessing of data per business object . Based on individual logic implemented in the con- sistency layer (see chapter Architecture), this monitoring approach provides granular insights into how effectively the raw data is being transformed into business-ready in- formation. This ensures that any logic-driven processing gaps are identified immediate- ly, maintaining the reliability of the information delivery layer.

STREAMLINING DELIVERY ENVIRONMENTS AND BRANCHING STRATEGY

The development lifecycle is lean and robust, eliminating the costs of permanent re- dundant environments. With the benefits of Snowflake’s zero-copy cloning, isolated development environments are spun up on demand at no extra storage costs. The de- velopment strategy consists of three permanent environments, Development, Quality Assurance and Production, and follow the Git-flow approach.

Furthermore, automated CI pipelines validate every change before it can progress, en- suring only tested code is promoted. Completed increments are pushed to the main branch. This transition follows the same governance principles: merging is conditional upon successful CI validation in dedicated test schemas. Production releases follow a tag-driven approach that guarantees reproducibility while retaining manual sign-off for final deployment.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

8/12

IMPROVE CODE AND DATA QUALITY To enhance performance and maintainability as well as a high data quality standard, the Data Vault methodology, in combination with testing strategies, provides a high standard for the code and data quality. To ensure the architectural integrity of our Data Vault, automated checks for unique- ness, referential integrity, and null constraints were introduced . This approach provides a low-maintenance validation layer that secures maximal data quality. Unit tests are another strategy to ensure that the business logic behaves as wished. By simulating inputs from upstream models, unit tests are able to validate the logic before a single row of real data flows. This proactive validation test allows for a detailed anal- ysis of how exactly the processing logic works. Furthermore, potential errors can be identified in development rather than discovering them in production. The final step of validation is set in the information delivery layer. By introducing sin- gular tests , specific/individual SQL assertions that target the final outcome can identify any erroneous result sets that generic tests might miss. These tests are the final safety net that ensures that the end-user gets accurate results with the highest data quality possible. TEAM INTERNAL ORGANIZATION To further streamline the integration process, a rotating role has been established within the development team to oversee the merging of feature branches and to monitor general pipe- line health. By making this a rotating responsibility, ownership is always clearly defined, ensuring no time is wasted identify- ing who is responsible for specific tasks or troubleshooting.

FREE INITIAL CONSULTATION

Schedule a free consultation now and discover how our Project Consulting can empower you to solve your data challenges on your own. YOUR PROJECT SUCCESS IS OUR MISSION

https://scalefr.ee/project-consulting

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

9/12

SUMMARY OF BENEFITS AND RESULTS With the migration from a legacy, siloed infrastructure to a unified, cloud-native data platform, Siemens AG achieved significant technical and operational improvements: — Near-Real-Time Processing & Unification: Replaced slow, sequential batch processing with a high-performance architecture. Kafka Streams and Snowflake now seamlessly integrate diverse data types from decentralized business units into a single source of truth. — Automated, Metadata-Driven Scaling: The introduction of a metadata repository and dbt’s capabilities eliminated repet- itive manual coding. Schema changes are now managed centrally and propagated automatically, allowing the data model to evolve without heavy reengineering. — High Data Quality: Automated tests in all layers and the architectural capability of human interaction guarantee a reliable “Golden Record” of Siemens products. — Agile & Cost-Effective Delivery: Snowflake’s zero-copy cloning enables cost-free, isolated development and test environments. Combined with automated CI/CD pipelines and rigorous testing, the team drastically reduced deployment risks and accelerated time-to-value. — Platform Transparency: Centralized monitoring provides real-time visibility into resource consumption, au- tomated budget alerts, and model processing times to ensure operational stability.

“From the beginning, our collaboration with Scalefree followed a shared mindset of building something sustainable together. It was not just about delivery, but about co-creating a platform we can confidently grow and evolve, based on a deep understanding of Siemens’ challenges and a truly joint approach to finding the right solutions.”

Kathrin Postler Data Architect & Team Lead at Siemens AG

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

10/12

CONCLUSION The migration from a legacy on-premise solution to a modern, cloud-native data platform marks a significant milestone in Sie- mens AG’s data strategy, driven by a comprehensive partnership with Scalefree. By acting as both strategic consultants and the technical implementation partner, Scalefree helped translate Siemens’ internal vision into a highly functional reality. Together, they successfully consolidated fragmented, busi- ness-unit-specific platforms into a single, unified architecture, transforming how product data is managed, processed, and de- livered across the organization. This end-to-end collaboration proved that the combination of Snowflake, the dbt Platform, and Data Vault 2 provides a robust and highly scalable foundation. The newly engineered architec- ture successfully handles near-real-time requirements, supports both streaming and batch ingestion, and seamlessly integrates data from multiple business units. What was once a landscape of disconnected silos and black-box solutions has been entirely replaced by a transparent, governed, and high-performance ecosystem. Looking ahead, two key initiatives will further extend the plat- form’s capabilities. Currently delivering Type 1 Dimensions - reflecting only the most recent active state - the next step is to introduce point-in-time data access, enabling historical analysis and more advanced insights. In parallel, the platform is set to expand its integration scope to encompass data from more than a hundred factories, marking a significant step towards a truly enterprise-wide data foundation.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

11/12

Scalefree International GmbH Schützenallee 3 30519 Hannover Germany

P: +49 (511) 879 893 41 F: +49 (511) 879 893 49

contact@scalefree.com www.scalefree.com

CEOs: Michael Olschimke & Christof Wenzeritt

Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12

Powered by