Additionally, metadata tests were implemented as well, so that no standard tests were missing during the migration. Testing of the end result was the final step for an iteration. A copy of the Information Marts from the on-premise solution to the cloud solution, and a standardized comparison testing approach against the migrated Information Marts, helped a lot to ensure its quality and to measure the differences due to code improvements and adjustments during the migration process. EVOLVING THE DATA ACQUISITION LANDSCAPE At the beginning, only Snowflake for data storage, the software Prefect for orchestration, and a custom-developed component were available to copy data from the source to Snowflake. It worked reasonably well, but was cumbersome to maintain, wherefore the data platform team decided to test the Python library dltHub. dltHub supports a wide range of data sources and destinations and could be used for delivering data out of the data platform at a later point in time. Additionally, the team decided to build a framework with a GUI on top of dltHub, that could be used with different clients to set up new integrations without the need for deep dltHub or programming knowledge. Support has now been built for various source types, incremental loading, different transformations after data ingestion, triggering of Prefect flow runs, and launching dbt Core jobs. Once dltHub completes the data transfer from source to destination, an additional transformation layer takes over. This is not a replacement for dbt Core, but rather an addition that processes staging data before it reaches the final repositories. The layer consists of a Jinja- based template system that transforms the staged data from dltHub into structured sources that dbt Core can further process. The main templates convert the data into CDC (Change Data Capture) structures, which are stored in a Persistent Staging Area (PSA). In addition, some templates store data in snapshot format and other structures for different use cases.
SUCCESS STORY: MIGRATION TO A CLOUD-BASED DATA FABRIC - PEOPLE, TECHNOLOGY & METHODOLOGY
4/12
Powered by FlippingBook