Migration to a Cloud-Based Data Fabric - Sparebank 1

LOADING PROCEDURES FOR THE DATA LAKEHOUSE

BRONZE With dltHub, data is retrieved directly from the source systems at a frequency needed. The data gets written directly into Snowflake’s internal tables, the Bronze layer, without any modifications. The tool stack is Snowflake for data storage, Prefect for orchestration, and a custom framework built on dltHub for data ingestion. The framework uses a Django web application with a built-in admin module to manage the process via a graphical user interface (GUI) and a command-line interface (CLI). This integrated system allows for efficient data loading and transformation, providing a user-friendly way to manage the data pipeline. SILVER The dbt package DataVault4dbt (created by Scalefree and available open source) provided all Data Vault standard loading patterns to generate the code for almost the whole Silver layer. Complex business logics are written manually by developers, as this requires individual thinking. But also here, templates (in dbt they are called “macros”) based on best practices, as well as generative AI, helped to generate this code as much as possible. Additionally, database queries and micro partitioning got optimized in Snowflake for faster data transformations. Query optimizations needed to be utilized as the database technology is different in Snowflake compared to SQL Server. While SQL Server is a row-based database technology, designed for efficient writing and reading datasets, Snowflake comes with a different data storage method, designed for analytical purposes. An insert-only strategy helped to leverage the strengths of Snowflake infrastructure. Furthermore, the separation of storage and computing in Snowflake allows for increased loading and querying performance, as there are resource conflicts between the different layers (Bronze, Silver, and Gold) as well as developers will never slow down the production environment, as they are using completely isolated computing resources. GOLD The Gold layer got completely virtualized. All data transformations were already handled in the Silver layer, so that only a last step is missing: representing the results in a user-friendly way and applying role-based access control. As Data Vault is a very good approach to model data at scale, it is not the best modeling technique to represent the data to non-technical end users. As almost no computationally intensive transformations are necessary, only a structural change, the Gold layer uses virtualized entities to represent the data to end users. The benefits are to save storage as well as eliminate data movement processes in the Gold layer. An additional business value is achieved by adding data descriptions to the Gold layer by using the data catalog functionalities from dbt Cloud. Information like table descriptions, column descriptions, data owners, granularity of the data, data test definitions, and orchestration information are part of the data catalog, which is the menu for the end users when selecting their data.

SUCCESS STORY: MIGRATION TO A CLOUD-BASED DATA FABRIC - PEOPLE, TECHNOLOGY & METHODOLOGY

6/12

Powered by