DBTimes
Frameworks help to implement Data Warehouse that follows Kimball Dimensional modeling approach.
Both frameworks automate three phases of Data Warehouse life cycle: load into staging area, etl process, and reconciliation.
dwtools - JVM (Scala)
/ Apache Spark
|
Libraries ·
Libraries use Configuration First approach –
with the emphasis on configuration as customization method. ·
All three libraries are independent from usage
standpoint, but packaged together in a single JAR – dwtools - for end-to-end Data
Warehouse implementation. Each library has its own configuration schema, set
of rules and APIs. In the examples provided a separate application is created
for each library |
|
DW
Data Source Loader – loads data into
staging parquet files. ·
Load
actions are described in .json configuration file. ·
Features
include data versioning, incremental load. ·
Sources
include csv or delimited files, or SQL Server, or Oracle,
or Mongo DB databases, or custom source ·
Can
be extended to accept other sources Documentation for DW Data Source Loader |
|
|
DW
ETL – implements ETL
to load dimensions and fact tables into parquet files. ·
Dimensional
Model is defined in configuration file ·
Supports
creation of surrogate keys, type 1, 2, 0 and 3 changes, full and incremental
load, re-runs ·
Full
or incremental load. |
|
|
|
DW
Data Source Comparer – a
reconciliation tool - compares two data sources. ·
Sources
can be parquet or csv file or SQL Server or Oracle,
or Mongo DB data ·
Produces
csv file with differences Documentation for DW Data Source Comparer |
|
|
Sample Implementation ·
NFL Data Warehouse. |
|
https://github.com/vshulman111/dwspark/tree/main/nfl_sample_dw_for_dwtools_2.12-3.3 |
|
dwtools versions
|
|
dwtools_2.12-3.3 version 1.0.0 –
2026 August - for Spark 3.3.x, scala 2.12 |
.Net (C#) / SQL Server / SSAS
|
Download code SQLServerDotNetFramework.zip |
||
|
Components included |
.Net Project - applications |
·
Data Source Loader – loads data into
versioned staging tables in SQL Server. Load actions are described in .xml
configuration file. The source can be .csv or
delimited files, or SQL Server tables, views or stored procedures, or SQL
code |
|
·
ETL Runner – runs ETL to
load dimensions, fact tables and SSAS cube. For each of three stages the load
can be full or incremental. Has many other features |
||
|
·
Data Comparer – used for
reconciliation. |
||
|
SQL Server – Implementation
of sample Data Warehouse – NFL Games. About 50% of the code is generic, the
other 50% is used as a template to develop the ETL for specific dimensional
model |
·
Sample stored procedures to implement dimensions
load. |
|
|
·
Sample stored
procedures to implement facts load. |
||
|
·
Generic stored procedures to implement SSAS load. |
||