Siemens Success Story

ARCHITECTING FOR NEAR-REAL-TIME DELIVERY USING DATA VAULT 2 One big objective was the near-real-time capability of the whole architecture. Data ar- rives via Kafka streams in a semi-structured data format and should be visible in the information marts within several minutes. With Data Vault 2, teams can process differ- ent data formats (structured, semi-structured, or unstructured) at different velocities (real-time, batches) and integrate everything into a single platform. A major advantage is the independence of data sources, which allows data to be processed as soon as it arrives. If near-real-time is required, source systems can either send data on demand via an API or continuously via event streaming, depending on the use case, integration pattern, and the capabilities of the source system itself. Next to data streaming, the architecture supports the classic batch processing. In all cases, the data runs through all layers of the architecture, as shown in the figure.

The first layer is the Persistent Staging Area (PSA). Here, the incoming data arrives and is stored in its raw format without any modifications. This is particularly useful for the semi-structured data coming from the Kafka streams. As the structure can differ from file to file, the PSA is able to store all kinds of data, no matter their data structure. In the Integration Layer , the semi-structured data is broken down into individual log- ical parts and is partially converted into a relational structure. This is essential to ac- celerate the performance of downstream data transformations. At this place (Raw Data Vault), the semi-structured data and the relational data are integrated. Besides the Raw Data Vault, the Integration Layer features the consistency layer, where complex busi- ness logic is implemented.

UNIFIYING ENTERPRISE DATA: SIEMENS’ JOURNEY TO A NEAR-REAL-TIME CLOUD PLATFORM

4/12

Powered by