SETUP STANDARDS AT THE BEGINNING (Q2 2023) Naming conventions for several technical entities, like databases, schemas, tables/views, and columns, were defined. Additionally, business terms were standardized as much as possible at the beginning and mapped to a common understanding. Integrating data across different systems into an enterprise data platform by using business terminology was a fundamental base for scalability in the future. By working focused in an agile manner, the first Information Marts were organized and prioritized in a kanban board upfront by the product owner, and the way of (agile) working was discussed. The full Scrum ceremonies were not used, as not all fit into the team(s) way of working, but the agile mindset of short iterations for delivery was adapted as much as possible. Thus, a proper deployment strategy was in the plans as well from the very beginning. Additionally, how to deal with identified challenges from the previous proof of concept and the definition of done before a deployment was discussed. A kick-off meeting with the expectations from each team member was also very useful, to get to know each other and to increase the level of mutual respect. A lot of first technical steps, like setting up the data platform (Snowflake) or the dbt Core setup, were already done in the proof of concept, which continued to be used.
“ The Data Vault methodology helped a lot with its scalable modeling approach and standardized loading patterns.”
DEFINE A MIGRATION STRATEGY FOR INFORMATION MARTS FROM THE ON- PREMISE SOLUTION TO THE SNOWFLAKE CLOUD To ensure a smooth and agile data migration from SQL Server to Snowflake, a prioritization of the data products (Information Marts) was set by the Product Owner to migrate those step by step. The core model could be extended very easily in multiple iterations by just adding new entities instead of re-engineering them every time. The migration strategy followed a tracer bullet approach.
By identifying the target first (the information marts), and then going back through the lineage of data until the source, enabled an accelerated migration process was enabled by focusing on specific data sources for each iteration. Furthermore, a lot of code could be generated by using Scalefree’s dbt package DataVault4dbt. Data validation and quality assurance processes were implemented as well to guarantee the accuracy and integrity of the migrated data. Standard tests were defined to ensure this.
SUCCESS STORY: MIGRATION TO A CLOUD-BASED DATA FABRIC - PEOPLE, TECHNOLOGY & METHODOLOGY
3/12
Powered by FlippingBook