REFACTOR INEFFICIENCIES AND IMPROVE CODE QUALITY DURING THE MIGRATION To enhance performance and maintainability, the Data Vault methodology provides highly standardized methods from industrial best practices of building a Data Fabric. Especially the integration of the raw data by business keys from the very beginning was very useful for downstream processing, as the data got well “integrated” by business terminology. Furthermore, inefficiencies were identified and eliminated by reviewing the codebase for redundant code, removing existing technical debts, and automating as much as possible. Code refactoring techniques were implemented by breaking down large functions, simplifying expressions, implementing consistent naming conventions, and coding styles. Additionally, coding standards and best practices were established by code review processes and version control systems. By implementing these measures, the codebase became more efficient, better understandable, maintainable, and extensible, making it easier for developers to work with the code, identify and fix bugs, and add new features.
DEVELOPMENT, TESTING, AND DEPLOYMENT STRATEGY
Common development strategies consist of at least 3 permanent environments: Development, Testing (or Quality Assurance), and a Production environment. Those cause additional costs due to maintenance and redundant storage, especially in the cloud, where you pay per usage. The Data Fabric explained so far consists of Snowflake and dbt Core, which allows the usage of just one permanent environment, the Production environment. The Development as well as the Testing environment are dynamic environments, starting up when needed, shutting down when no longer needed. With Snowflake’s “zero-copy cloning” as well as dbt’s “defer” functionality, there is no need to have all 3 environments permanently online, but just the Production. A new feature or fix is developed on a freshly created database. By using the defer functionality from dbt Core, there is no need to load all predecessor tables. You can directly access the data from the Production environment and focus on the entities where the changes need to be applied. A zero-copy clone on the table itself can be helpful in case the table does not need to be fully reloaded, but continues incremental loading.
Remember
Due to the separation of storage and computing, this approach will not interrupt anything that runs on the Production environment.
SUCCESS STORY: MIGRATION TO A CLOUD-BASED DATA FABRIC - PEOPLE, TECHNOLOGY & METHODOLOGY
7/12
Powered by FlippingBook