Also, as a strategic decision, we did not plan to fully use the potential of 3D right from the start, i.e. some of the more advanced Data Vault structures beyond the standard hubs, link and satellites were only integrated at a later point into the model conversion rule set. This was done because the initial effort for implementation was estimated to be too high due to the inhomogeneous modeling of the source systems. But thanks to the automation capabilities of the tool, a later redesign and redeployment of these entities repaid the technical debts without any major issues. Another challenge was that the provided initial model conversion rules aimed for databases in 3NF did not match the design of the rolling mill database which was rather hierarchical. So, with their first sub-project, Dillinger had to dive deeper into the 3D model conversion rules to generate a simple Raw Vault. But it should be remembered that they were able to implement these refactorings in a relatively short time once the ruleset was up and running to deal with their individual scenario. To be further noted, changes could be implemented rapidly which shows the strengths of the Wherescape tool suite. The combination of the less manpower and limited tool experience at Dillinger, including an initial lack of Data Vault knowledge, unfortunately led to a steep learning curve. As a positive result, Dillinger acquired a lot of experience during the course of the project. The support from Wherescape was very helpful and swift, though the difficulty was mostly found in communicating to get on the same page. CHANGE DATA CAPTURE A major technical problem was the susceptibility to faults of the productive mill database. In their initial attempts to perform a full load, it quickly became clear that full loads were only feasible in seldom, exceptional cases. They cannot be used for regular loading of the warehouse and consequently, Dillinger had to find a solution to establish regular loading. One of their DBAs with prior data warehouse experience developed a custom trigger-based solution: Change data capture, or CDC for short. Every change on tables that have this trigger is written to a copy of the table that has additional information like the CSCN number, the commit timestamp and the type of operation. The CSCN number is a unique, database-given number ordering the sequence of the changes. This brings some advantages: ● No interference with the original tables ● Low, constant, additional load on the database, no peaks ● Complete historization ● Loading cycle for the warehouse is variable, with CDC giving a 48 hours buffer for data
On the other hand, it also has a few disadvantages:
● Very high volume of data on tables with a lot of traffic ● More historization than necessary in some cases
In RED, Dillinger added the CSCN number in milliseconds to the load date, thus giving each row a unique load date and preserving the sequence of the changes.
Made of Steel - Data Vault 2.0 at Dillinger | Scalefree International GmbH
8/13
Powered by FlippingBook