Datawarehouse testing using MiniDBs for IT
In Business Intelligence(BI) applications there are Datawarehouse(DW) which holds data in Tera Bytes and sometimes go up to as many as 50-100 data sources. Testing such applications requires huge test servers and longer data testing cycles. The risk of source systems being down or in restore mode when ETL was running is not in team’s control. To address all such issues and improve test effectiveness/productivity, Microsoft India BI Team came up with a organization wide concept of using MiniDB’s across all BI projects.
Above diagram shows a typical DW architecture and storage needs, ETL Run time, Internal Job run times. MiniDB is basically a smaller data set version with exactly same schema/objects of source systems, application databases and a working model but with different topology than the production environment. Using the SQL server trimming mechanism we trim the databases to fit into a single
box, and call it as One Box environment which is used by Dev/Test teams to do most of their functional, data, and ETL testing. The entire test strategy resulted in many advantages as reducing test cycles by 50%, more test coverage and shorter regression cycles.
This paper will talk about how to create MiniDB’s, replicate working model of a DW, test strategy to do BVT in less than 2 hours in a large data warehouse applications, and all practical challenges/benefits in the course of the implementing this technique in any data warehouse application.
Attached is the complete white paper for your continued reading , feedback and comments....
-Naren
Views or Opinions expressed in this blog represnt only author and not any company or organization or person.