Migration to a Cloud-Based Data Fabric - Sparebank 1

“ A Data Mesh approach was adopted to facilitate organizational scalability and remove bottlenecks.”

ADOPT A DATA MESH-LIKE APPROACH TO FACILITATE ORGANIZATIONAL SCALABILITY Decentralizing data ownership and management empowered individual business units to manage their own data products by collecting, storing, and managing their data. This made the business units more agile and responsive to customers. Implementing a data mesh architecture enabled seamless data sharing and integration across different business units

Additional benefits of decentralizing data ownership and management, implementing a data mesh architecture, and utilizing data virtualization techniques included improved data governance, reduced data silos, increased data agility, improved data security, and reduced costs. By implementing these strategies, organizations improved their data management practices and gained a competitive advantage. GOVERNED DATA SHARING Data sharing was enabled by implementing access control mechanisms, such as role-based access control and Row-Level-Security, to regulate data access based on user permissions and attributes. Guidelines for data anonymization were established to protect sensitive information. With dbt Cloud’s data lineage functionality, it is possible to provide lineage tracking visualizations to trace the origin and transformation of data throughout its lifecycle.

The data governance policies were regularly reviewed and updated to adapt to changing regulatory requirements and business needs.

COST MONITORING The more people get access to the Data Fabric and use the cloud resources, the more expensive it gets. It is crucial to keep an eye on the costs to figure out the high, unnecessary cost reasons quickly. Snowflake itself provides good cost monitoring on a high level, but a deeper analysis was implemented in addition to point to expensive single loading procedures, especially when the same code gets executed regularly, i.e., the daily production load, which consists of thousands of table loadings and tests.

The overall strategy is to load as much as possible with an incremental “insert-only” approach, instead of full-table loads. This allows stable reporting and very efficient loading procedures.

Additionally, virtualization is used as much as possible, where no complex calculations take place. This affects especially the Gold layer.

SUCCESS STORY: MIGRATION TO A CLOUD-BASED DATA FABRIC - PEOPLE, TECHNOLOGY & METHODOLOGY

9/12

Powered by