DBTimes

DBTimes

Frameworks implement different phases of a Data Warehouse life cycle. They are used for a DW that follows Kimball Dimensional modeling approach.

SQL Server/.Net (C#) Framework Download code SQLServerDotNetFramework.zip
Components included	.Net Project - applications	· Data Source Loader – loads data into versioned staging tables in SQL Server. Load actions are described in .xml configuration file. The source can be .csv or delimited files, or SQL Server tables, views or stored procedures, or SQL code
		· ETL Runner – runs ETL to load dimensions, fact tables and SSAS cube. For each of three stages the load can be full or incremental. Has many other features
		· Data Comparer – used for reconciliation.
	SQL Server – Implementation of sample Data Warehouse – NFL Games. About 50% of the code is generic, the other 50% is used as a template to develop the ETL for specific dimensional model	· Sample stored procedures to implement dimensions load.
		· Sample stored procedures to implement facts load.
		· Generic stored procedures to implement SSAS load.

Spark/Scala Framework

The framework is implemented as a library using Configuration First approach – with the emphasis on configuration as customization method.

Download the source code and build the library as part of your application.

Download data for sample implementation FrameworkSampleSourceData-NFL.zip

Components included

Spark/Scala Project

Implementation of sample Data Warehouse – NFL Games.

DW Data Source Loader – loads data into staging parquet files.

· Load actions are described in .json configuration file.

· Features include data versioning, incremental load, the data source, and others.

· The source can be .csv or delimited files, or SQL Server tables or views.

· Can be easily extended to accept other sources

DW Data Source Comparer – a reconciliation tool - compares two data sources.

· Sources can be parquet or csv file or DBMS table

· Produces csv file with differences

DW ETL – runs ETL to load dimensions, fact tables into parquet files.

· Maintains surrogate keys, type 1 and type 2 changes (and by extension type 3 and type 4), full and incremental load, load re-runs

· Each of three stages of the load can be full or incremental.

· Features support full life cycle of the dimensional model