DBTimes

Home

Products

Frameworks

Contact us

 

Frameworks help to implement Data Warehouse that follows Kimball Dimensional modeling approach.

 

Both frameworks automate three phases of Data Warehouse life cycle: load into staging area, etl process, and reconciliation.

 

 

dwtools - JVM (Scala) / Apache Spark

 

dwtools libraries

 

·         Libraries use Configuration First approach – with the emphasis on configuration as customization method.

·         All three libraries are independent from usage standpoint, but packaged together in a single JAR – dwtools - for end-to-end Data Warehouse implementation. Each library has its own configuration schema, set of rules and APIs.

 

 

DW Data Source Loader – loads data into staging parquet files.

·         Load actions are described in .json configuration file.

·         Features include data versioning, incremental load.

·         Sources include csv or delimited files, or SQL Server, or Oracle, or Mongo DB databases, or custom source

·         Can be extended to accept other sources

 

Documentation for DW Data Source Loader

 

 

 

DW ETL – implements ETL to load dimensions and fact tables into parquet files.

·         Dimensional Model is defined in configuration file

·         Supports creation of surrogate keys, type 1, 2, 0 and 3 changes, full and incremental load, re-runs

·         Full or incremental load.

 

Documentation for DW ETL

 

 

 

DW Data Source Comparer – a reconciliation tool - compares two data sources.

·         Sources can be parquet or csv file or SQL Server or Oracle, or Mongo DB data

·         Produces csv file with differences

 

Documentation for DW Data Source Comparer

 

Sample Implementation

 

Sample implementation - NFL Data Warehouse - has separate Scala application for each library

 

 

 

 

https://github.com/vshulman111/dwspark/tree/main/nfl_sample_dw_for_dwtools_2.12-3.3 - implementation for dwtools_2.12-3.3

 

https://github.com/vshulman111/dwspark/tree/main/nfl_sample_dw_for_dwtools_2.13-4.0 - implementation for dwtools_2.13-4.0

 

dwtools versions

 

 

dwtools_2.12-3.3 – Spark 3.3.2, Scala 2.12, Java 11

·         version 1.0.0 – released 2026-08-19

·         Version 1.0.1 – released 2026-08-25

o    Bug fix –new data with different effective date was overriding previous effective dates for non-versioned destination

dwtools_2.13-4.0 –Spark 4.0.4, Scala 2.13, Java 17

·         Version 1.0.1 – released 2026-09-01

·         Version 1.0.2 – released 2026-09-13

o    Added schema evolution to DW Data Source Loader

 

 

 

 

 

.Net (C#) / SQL Server / SSAS

 

Download code SQLServerDotNetFramework.zip

Components included

.Net Project - applications

·         Data Source Loader – loads data into versioned staging tables in SQL Server. Load actions are described in .xml configuration file. The source can be .csv or delimited files, or SQL Server tables, views or stored procedures, or SQL code

·         ETL Runner – runs ETL to load dimensions, fact tables and SSAS cube. For each of three stages the load can be full or incremental. Has many other features

·         Data Comparer – used for reconciliation.

SQL Server –

Implementation of sample Data Warehouse – NFL Games. About 50% of the code is generic, the other 50% is used as a template to develop the ETL for specific dimensional model

·         Sample stored procedures to implement dimensions load.

·         Sample stored procedures to implement facts load.

·         Generic stored procedures to implement SSAS load.