A Strategic Shift Towards Flexibility and Compliance

INTRODUCTION The client was a global healthcare provider with the aim to improve patient care and advance healthcare worldwide with their innovative products. CASE DESCRIPTION The client sought to enhance their data flow efficiency by implementing a flexible and scalable data warehouse solution based on the Data Vault 2.0 framework. Their primary goals were to improve platform performance, reduce costs, and ensure compliance with GDPR guidelines and data retention policies. They also aimed to adopt a hybrid operational model, integrating both centralized and decentralized development teams for optimal project execution. The challenges encompassed transitioning existing domain-specific solutions to the updated Data Vault 2.0 platform and tackling ETL hurdles without a Persistent Staging Area (PSA) or data lake. Additionally, we needed to pinpoint an automation tool capable of integrating diverse ad-hoc SQL and Python processes. Completing the platform involved developing a sophisticated access framework capable of automating grants and providing row-level access. To overcome these challenges, Scalefree assisted the client by providing knowledge transfer and additional manpower. They collaborated with the client to select a suitable automation tool, introduced dbt Cloud with Scalefree’s Data Vault 2.0 package (datavault4dbt) and TurboVault4dbt for standardized development processes. Additionally, they crafted dbt macros for interpreting an Access Control List (ACL) and devised a consolidation strategy for the ETL process to establish a Persistent Staging Area (PSA).

The implemented solution resulted in a standardized data flow, ensuring the timely provision of accurate data to designated recipients. This optimization of reporting systems enhanced data quality and freshness, ultimately facilitating better decision-making processes and increasing organizational efficiency. Furthermore, the hybrid operational model, combining centralized core teams with decentralized development teams, proved instrumental in achieving agility and efficiency in project execution. The collaboration fostered between the teams enhanced communication, streamlined workflows, and accelerated the implementation of solutions. Technologies involved in the solution included Snowflake, dbt Cloud with datavault4dbt, TurboVault4dbt, and Microsoft Azure DevOps. The adoption of these technologies not only addressed the immediate challenges but also positioned the client for future scalability and innovation in their data management processes.

2/3