The Data Vault Handbook - Concepts and Applications

Most companies tend to begin their staging process with a landing zone in the shape of a data lake—a centralized repository that enables the storage of large volumes of structured, semi-structured, and unstructured data at scale. This data lake could be persisted in order to host the history of the data sources. Following this, they design a relational Transient Staging Area that adapts to changes in data structures in the source and apply the necessary hard rules for further processing in the next layers. Finally, the choice between these designs is especially important in the Euro- pean Union, where laws like the General Data Protection Regulation (GDPR) require companies to address scenarios where users request the deletion of their personal data. A Transient Staging Area can simplify compliance by en- suring that data is only temporarily stored, reducing the complexity of data deletion requests. However, this approach also introduces the aforemen- tioned challenges related to reprocessing and data retrieval. On the other hand, companies opting for a Persistent Staging Area will need to implement additional measures, such as data encryption protocols.

2.1.2. LOAD DATE TIMESTAMP, RECORD SOURCE & HASH DIFF

We will provide a brief overview of the technical attributes included in the Staging Area and their significance for Data Vault. It’s important to note that Hash Keys are not covered here, as they were already discussed in Section 1.3.3.

RECORD SOURCE

The Record Source attribute is essential for tracking the origin of data, pro- viding a clear and detailed reference to the source system from which the data was extracted. Therefore, its goal is to maintain traceability and allow