INTRODUCTION A leading full flowsheet technology and service provider to the global mining and cement industries sought to enhance operational independence and, as a consequence, efficiency by separating data warehouses for distinct departments. This case study explores how Scalefree addressed the client’s challenges by separating data warehouses for distinct departments. CASE DESCRIPTION The client, a prominent player in the mining and cement industries, faced a challenge: their shared data warehouse setup was hindering operational autonomy and efficiency in light of evolving business requirements. Scalefree had previously developed a Data Vault 2.0 based data warehouse, leveraging Snowflake as the database platform and dbt for automation. After two years of successful utilization, changes in the client’s organization and evolving business needs necessitated a reevaluation of the data management approach. While the existing setup, initially facilitated data management and report sharing, changing dynamics within the industry had led to increased interdepartmental dependencies, hindering autonomy and potential divestment efforts. With the client’s trust in Scalefree’s expertise reaffirmed, the focus shifted towards finding a solution that would address the emerging challenges comprehensively. The primary challenge lay in untangling the shared data warehouse while minimizing disruptions. This involved addressing various risks associated with data integrity, ensuring a seamless transition without impacting operations, and customizing CI/CD pipelines to accommodate department-specific requirements. Ensuring departmental autonomy within the shared data warehouse environment was crucial, as was mitigating risks of data integrity and operational interruptions during separation. Following comprehensive consultation and analysis of the new requirements, Scalefree proposed a solution tailored to the client’s needs. This involved establishing separate namespaces within the Snowflake database to isolate departmental data, creating distinct dbt projects, ADF pipelines, and Docker images for each department, and splitting up automated CI/CD pipelines to ensure efficiency and consistency.
2/3 ENABLING DEPARTMENTAL INDEPENDENCE: DATA WAREHOUSE SEPARATION FOR GLOBAL MINING AND CEMENT SUPPLIER © SCALEFREE INTERNATIONAL GMBH 2024
Powered by FlippingBook