dwEtl objectRequired
Configuration for DW Etl process
JSON Pointer: /properties/dwEtl
Nested schema details (15)
dwEtlConfigVersion stringRequired
version of DW ETL configuration
JSON Pointer: /properties/dwEtl/properties/dwEtlConfigVersion
Additional schema keywords (1)
enum["1"]
isDebugDwLib booleanOptional
If the value is true, there will be some dataframe related output sent to the standard output from the library causing slowdown in job execution.
JSON Pointer: /properties/dwEtl/properties/isDebugDwLib
jobType stringRequired
The ETL process consists of two parts – ETL dimensions and ETL facts. jobType is used to run either both or one of these parts. The allowed values for this attribute are "LoadDimsFacts", or "LoadDims", or "LoadFacts". The main scenario for running ETL dimensions separate from ETL facts is loading dimensions in parallel by defining separate configurations for one or more dimensions (using isLoadFlag) and running ETL facts after - serially or in parallel.
JSON Pointer: /properties/dwEtl/properties/jobType
Additional schema keywords (1)
enum["LoadDimsFacts","LoadDims","LoadFacts"]
isInitialLoad booleanRequired
The flag indicates if the load is initial or subsequent. If the load is initial, then the existing files for dimensions and/or facts are deleted. Which ones are deleted depends on the jobType - for "LoadDimsFacts" both dimensions and fact files are deleted, for "LoadDims" - dimensions are deleted, and for "LoadFacts" - fact files are deleted.
There is one use case when this flag should be set to true for ongoing loads: when dimension or fact has to be recreated every time based on current sources, i.e., when rows for non-existent natural keys deleted and all attributes are Type 1.
JSON Pointer: /properties/dwEtl/properties/isInitialLoad
rerunEtlAfter stringOptional
The attribute is used to rerun subsequent load for effective dates processed earlier after the timestamp value specified based on the log.
JSON Pointer: /properties/dwEtl/properties/rerunEtlAfter
String constraints
format"date-time"
fileEtlLog objectOptional
When this attribute is defined, the loader will create a file with summary info for each load. The info includes record counts, description of source and destination, and the action timing.
JSON Pointer: /properties/dwEtl/properties/fileEtlLog
Nested schema details (4)
parquet objectRequired
File type is parquet for load control.
JSON Pointer: /properties/dwEtl/properties/fileEtlLog/properties/parquet
path stringRequired
A parquet file path.
JSON Pointer: /properties/dwEtl/properties/fileEtlLog/properties/path
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/fileEtlLog/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/fileEtlLog/unevaluatedProperties
stgSources arrayOptional
Staging sources. This are sources for loading dimensions and fact tables.
JSON Pointer: /properties/dwEtl/properties/stgSources
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/stgSources/items
Nested schema details (10)
moniker stringRequired
Staging source moniker is a unique identifier of a staging source. It is used as view name for respective source in Spark SQL.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/moniker
description stringOptional
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/description
applicationSpecific objectOptional
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/applicationSpecific
Nested schema details (1)
Unevaluated properties
Boolean schema: true - all instance values are accepted at this location.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/applicationSpecific/unevaluatedProperties
effectiveDateColumn stringOptional
A column in a staging source data that is effective date for the current staging source.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/effectiveDateColumn
isDefaultForEffectiveDate booleanOptional
There can be only one default effective date across all staging sources. When is set to true it designates effective date in current staging source as a default effective date.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/isDefaultForEffectiveDate
timestampColumn stringOptional
A column in a staging source data that is has load timestamp. Timestamp can be used for incremental processing of dimension in subsequent ETL runs.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/timestampColumn
effectiveDateRule stringOptional
An attribute to specify how to derive effective date from effectiveDateColumn. The values can be
"DISTINCT" - only the dates in the stg source are used in the etl,
"WEEKDAYS" - all days between min and max values excluding weekends, or
"ALL" - all days between min and max values
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/effectiveDateRule
Additional schema keywords (1)
enum["ALL","DISTINCT","WEEKDAYS"]
fileSource objectRequired
Defines file staging source.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/fileSource
Nested schema details (4)
parquet objectRequired
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/fileSource/properties/parquet
path stringRequired
A parquet file path.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/fileSource/properties/path
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/fileSource/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/properties/fileSource/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/stgSources/items/unevaluatedProperties
dimAuthority objectOptional
A section of specifications for attributes common to all dimensions.
JSON Pointer: /properties/dwEtl/properties/dimAuthority
Nested schema details (5)
packageName stringRequired
Package name is used to dynamically create an instance of a dimension class. All dimension classes described in a configuration must be defined in the same package.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/packageName
dimensionNamePattern stringRequired
A regular expression pattern for dimension name. Each dimension’s name must match this pattern.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/dimensionNamePattern
fileDimensionsDestination objectRequired
A section to describe a dimension destination file. The file for each dimension will be created and maintained by the ETL library.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination
Nested schema details (5)
parquet objectRequired
File type is parquet for dimension destination.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination/properties/parquet
path stringRequired
Defines destination file path for all dimensions. The file name in path must contain three underscores as a placeholder for the dimension name
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination/properties/path
String constraints
pattern"^.*___.*$"
Additional schema keywords (1)
customMessage{"pattern":"The file name in path for fileDimensionsDestination must contain three underscores as a placeholder for the dimension name"}
previousCopyPath stringOptional
If defined, DW ETL will preserve a previous version of the dimension and place it in this path. A file name must contain three underscores as a placeholder for the dimension name
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination/properties/previousCopyPath
String constraints
pattern"^.*___.*$"
Additional schema keywords (1)
customMessage{"pattern":"The file name in previousCopyPath must contain three underscores which will be replaced with the name of the dimension"}
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/properties/fileDimensionsDestination/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimAuthority/unevaluatedProperties
dimensions arrayOptional
Describes dimensions. Since dimensions can be loaded in parallel via independent ETL runs this array can have some dimensions absent or inactive (i.e., with 'isLoad' flag set to false)
JSON Pointer: /properties/dwEtl/properties/dimensions
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/dimensions/items
Nested schema details (8)
name stringRequired
Name of the dimension. Must match the pattern defined in dimensionNamePattern field.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/name
isLoad booleanRequired
If set to false the dimension will not be loaded. Can be used to create separate configurations to load specific dimensions or during development to skip the load of already tested dimensions.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/isLoad
stgSourcesMonikers arrayOptional
Monikers of staging sources used to load this dimension. DW ETL will call a loadDim override for each source. If there are no staging sources loadDimm will be called just once.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/stgSourcesMonikers
Nested schema details (1)
Array items string
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/stgSourcesMonikers/items
dimSources arrayOptional
Other dimensions that are sources for loading this dimension. A dimension that is a source must be loaded prior to loading this dimension.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items
Nested schema details (5)
name stringRequired
Name of the dimension.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/name
schema arrayOptional
Schema lists a subset of columns used to load current dimension (used for optimization to reduce number of columns). If schema is absent all columns can be referenced in loading current dimendion.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/schema
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/schema/items
Nested schema details (3)
colName stringRequired
Column name
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/schema/items/properties/colName
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/schema/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/schema/items/unevaluatedProperties
unevaluatedProperties Optional
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/properties/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/dimSources/items/unevaluatedProperties
schema arrayRequired
Specifications for dimension columns
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema
Array constraints
minItems1
Additional schema keywords (1)
customMessage{"minItems":"Dimension's schema does not have any columns defined. It must have at least one column"}
Nested schema details (3)
Array items object
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items
Nested schema details (14)
colName stringRequired
A name of the dimension column.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/colName
colType stringRequired
Data type of dimension column. colType must be Short, Integer or Long if the column is a dimension key (i.e., "isKey": true)
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/colType
Additional schema keywords (1)
enum["Short","Integer","Long","Date","Timestamp","String","Double"]
isKey booleanOptional
If true, this column is a dimension key.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/isKey
isSurrogateKey booleanOptional
isSurrogateKey can only be defined for dimension key column, i.e., when "isKey": true. If true, this column is a surrogate key and its value will be generated using increasing – but not necessarily sequential - distinct values. The data type for a surrogate key must be “Long”.
If isSurrogateKey is false or not defined, the value for the key column must be provided together with the rest of the dimension values.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/isSurrogateKey
unknownValue stringnullOptional
Unknown value must be the valid value for column data type. Use yyyy-mm-dd format for Date type. If unknown value is not specified it will be set to NULL to any column but the dimension key – the default unknown value for dimension key is 0 (zero).
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/unknownValue
isNaturalKey booleanOptional
A flag to indicate that a column is a natural key for the dimension. Multiple columns can be defined as natural key to create a compound natural key.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/isNaturalKey
scdType numberOptional
Slowly changing dimension column modeling technique: 0 - Retain Original Non-Null Value, 1 - Overwrite, 2 - Add New Row, 3 - Retain Latest Non-Null Value. Default - 1
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/properties/scdType
Additional schema keywords (2)
enum[0,1,2,3]customMessage{"enum":"The allowed values for "scdType" property are 0, 1(default), 2 or 3. 0 - Retain Original Non-Null Value, 1 - Overwrite, 2 - Add New Row, 3 - Retain Latest Non-Null Value"}
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/patternProperties/^__
allOf option 1 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/0
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/0/if
Nested schema details (1)
isKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/0/if/properties/isKey
Additional schema keywords (1)
consttrue
Then Any type
'scdType' is not applicable for a key column
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/0/then
Additional schema keywords (1)
customMessage{"not":""scdType" is not applicable when "isKey" is true "}
Nested schema details (1)
Not Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/0/then/not
allOf option 2 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/1
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/1/if
Nested schema details (1)
isNaturalKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/1/if/properties/isNaturalKey
Additional schema keywords (1)
consttrue
Then Any type
'scdType' is not applicable for a Natural Key column
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/1/then
Additional schema keywords (1)
customMessage{"not":""scdType" is not applicable when "isNaturalKey" is true "}
Nested schema details (1)
Not Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/1/then/not
allOf option 3 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/2
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/2/if
Nested schema details (1)
isKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/2/if/properties/isKey
Additional schema keywords (1)
consttrue
Then Any type
'isSurrogateKey' and 'unknownValue' are required for a key column
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/2/then
Additional schema keywords (1)
customMessage{"required":""isSurrogateKey" and "unknownValue" attributes are required when "isKey" is true "}
allOf option 4 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/3
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/3/if
Nested schema details (1)
isNaturalKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/3/if/properties/isNaturalKey
Additional schema keywords (1)
consttrue
Then Any type
'unknownValue' is required for a natural key column
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/3/then
Additional schema keywords (1)
customMessage{"required":""unknownValue" is required for a natural key column, e.g., ... "isNaturalKey" : true, "unknownValue" : "some-value" or null ... "}
allOf option 5 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/4
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/4/if
Nested schema details (1)
isSurrogateKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/4/if/properties/isSurrogateKey
Additional schema keywords (1)
consttrue
Then Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/4/then
Nested schema details (1)
colType Any typeRequired
'colType' value must be Long if the column is Surrogate key as surrogate key is derived from Spark monotonically_increasing_id
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/allOf/4/then/properties/colType
Additional schema keywords (2)
const"Long"customMessage{"const":""colType" must be "Long" when dimension key is a surrogate key, i.e., ... "colType": "Long", "isSurrogateKey" : true, ... "}
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/items/unevaluatedProperties
allOf option 1 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/0
Array constraints
minContains1
Additional schema keywords (1)
customMessage{"contains":"Dimension's schema does not have natural key defined. At least one column must have "isNaturalKey" flag set to true, i.e., ... "isNaturalKey": true, ... "}
Nested schema details (1)
Contains object
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/0/contains
Nested schema details (1)
isNaturalKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/0/contains/properties/isNaturalKey
Additional schema keywords (1)
consttrue
allOf option 2 Any type
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/1
Array constraints
maxContains1minContains1
Additional schema keywords (1)
customMessage{"contains":"Dimension's schema must have one dimension key defined. One and only one column must/can have "isKey" flag set to true"}
Nested schema details (1)
Contains object
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/1/contains
Nested schema details (1)
isKey Any typeRequired
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/schema/allOf/1/contains/properties/isKey
Additional schema keywords (1)
consttrue
unevaluatedProperties Optional
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/properties/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dimensions/items/unevaluatedProperties
dataMart objectOptional
A section of specifications for attributes common to all fact tables.
JSON Pointer: /properties/dwEtl/properties/dataMart
Nested schema details (5)
packageName stringRequired
All fact classes described in a configuration must be defined in the same package. Package name is used to dynamically create an instance of a fact class.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/packageName
factNamePattern stringRequired
A regular expression pattern for fact table name. Each fact table’s name must match this pattern.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/factNamePattern
fileFactsDestination objectRequired
A section to describe a fact table destination file. The file for each fact table will be created and maintained by the ETL library.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination
Nested schema details (5)
parquet objectRequired
File type is parquet for fact table destination.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination/properties/parquet
path stringRequired
Defines destination file path for all fact tables. For example, "path" : "/dw/data/dm/FactNamePlaceholder.parquet". Text FactNamePlaceholder in path must match the factNamePattern above. Text 'FactNamePlaceholder' will be replaced with actual fact table name which also has to match the pattern.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination/properties/path
String constraints
pattern"^.*___.*$"
Additional schema keywords (1)
customMessage{"pattern":"The file name in path for fileFactsDestination must contain three underscores which will be replaced with the name of the fact table"}
pathWithKeySetters stringOptional
If defined, DW ETL will create fact table with natural keys used to set fact table keys in addition to regulat fact table. This file is used mostly in development to insure the correct setting on dimension keys on a fact table and troubleshoot the unknown keys. The help in troubleshooting the unknown keys can make this file useful in production. Creating of this file has relatively low performance cost.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination/properties/pathWithKeySetters
String constraints
pattern"^.*___.*$"
Additional schema keywords (1)
customMessage{"pattern":"The file name in pathWithKeySetters for fileFactsDestination must contain three underscores for fact table placeholder which will be replaced with the name of the fact table"}
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dataMart/properties/fileFactsDestination/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/dataMart/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/dataMart/unevaluatedProperties
facts arrayOptional
Specifications for fact tables or facts (fact table and fact are used interchangeably). Since facts can be loaded in parallel via independent ETL runs this array can have some facta absent or inactive (i.e., with 'isLoad' flag set to false)
JSON Pointer: /properties/dwEtl/properties/facts
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/facts/items
Nested schema details (9)
name stringRequired
The name of the fact table. The name must match the pattern defined in factNamePattern attribute.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/name
isLoad booleanRequired
If set to false the dimension will not be loaded. Set it to false during development to skip loading of some fact table. Can also be used in production to create different configurations to load fact tables in parallel.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/isLoad
processingMode stringOptional
The following fact processing modes are supported on subsequent ETL runs:
•\tREPLACE – new fact table replaces the old one. Fact table can have partition columns (one or more), i.e., columns with isForPartition flag set to true. If partition columns are defined the fact table will be replaced and partitioned.
•\tREPLACE_PARTITION – this mode is similar to REPLACE, but requires to have partition columns. In this mode new partitions replace old partitions with the same name, but if new partition does not exist for some existing old partitions – the old partitions will remain.
•\tMERGE – old fact table rows are merged with new based on the merge key columns. All rows in old fact table with the same merge key are replaced with new ones. The old rows with merge keys that do not exist in the new fact table will remain.
•\tMERGE_PARTITION – this mode is similar to MERGE, only in addition to merge key takes partition column into consideration.
•\tADD – new rows are added to existing ones. Fact table can have one or more partition columns. If partition columns are defined the new rows are added to respective partitions.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/processingMode
Additional schema keywords (1)
enum["REPLACE","REPLACE_PARTITION","MERGE","MERGE_PARTITION","ADD"]
stgSourcesMonikers arrayOptional
Monikers of staging sources used to load this fact table. DW ETL will call a loadFact override for each source. If there are no staging sources loadFact will be called just once.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/stgSourcesMonikers
Nested schema details (1)
Array items string
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/stgSourcesMonikers/items
dimSources arrayOptional
Other dimensions that are used to load this fact table. This dimensions must be distingushed from dimentions used to set key. The dimension needs to be listed here is it used to load fact values as opposed to setting keys. Dimensions to set keys are defined in schema.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items
Nested schema details (5)
name stringRequired
Name of the dimension.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/name
schema arrayOptional
Specifications for dimension columns used in loading fact table
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/schema
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/schema/items
Nested schema details (3)
colName stringRequired
Column name
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/schema/items/properties/colName
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/schema/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/schema/items/unevaluatedProperties
unevaluatedProperties Optional
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/properties/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/dimSources/items/unevaluatedProperties
schema arrayRequired
Specifications for fact columns.
The fact table is constructed from schema using following rule:
•\tIf "isForPartition" column or columns are defined the fact table will be partioned on this columns using Spark partitioning feature.
•\tDistinct "foreignKey" values will be used to create fact table foreign keys. If multiple columns are needed to set a foreign key, the schema would have multiple elements with the same value of a foreign key – one for each column needed to asset a key. For example, if there are two columns from the underlying dimension are needed to set a key, the fact table schema will have two elements with the same value of "foreignKey".
•\tColumns with "isMeasure" attribute set will be added to a fact table as measures
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema
Nested schema details (1)
Array items object
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items
Nested schema details (14)
colName stringRequired
Name of the column for the result of select in loadFact method. This will be the name of the column for measures (i.e., when "isMeasure": true), but will not be a name of the foreign key column. See "foreignKey" attribute for names of the foreign keys
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/colName
colType stringRequired
Type of the column whose name is defined in "colName". A data type of a measure column must be Double, Short, Integer or Long. The type of foreign keys is determined by the type of the respective key on the underlying dimension
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/colType
Additional schema keywords (1)
enum["Short","Integer","Long","Date","Timestamp","String","Double"]
isForPartition booleanOptional
A flag to indicate that this column is used for partitioning fact data. For example, for column { "colName" : "SeasonYear", "colType" : "Integer", "isForPartition" : true } the fact directory structure would look like below
FactPlays.parquet
\tSeasonYear=2013
\tSeasonYear=2014
\tSeasonYear=2015
\tSeasonYear=2016
\tSeasonYear=2017
To query partitioned Fact tables in, for example Apache Drill, use syntax like this
select * from dfs.`fact-file-path/FactPlay.parquet` where dir0=’SeasonYear=2016’
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/isForPartition
isMergeKey booleanOptional
A flag to indicate that the column is used for merge of fact data during subsequent loads. All fact table records that have a merge key value that is present in a new fact table will be deleted and replaced with new values
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/isMergeKey
isEffDateForTypeTwo booleanOptional
This flag can be set on a column of Date type to designate a column as an effective date for a dimension with scd type 2 columns. The column will be used to set the key of dimension via equation: eff-date-for-type-two-column between dimension-row-start-date and dimension-row-end-date. Only one column in fact table schema can have this attribute set to true
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/isEffDateForTypeTwo
isForSettingForeignKey booleanOptional
When isForSettingForeignKey is set, this column is used to set a foreign key from a dimension on a fact row. The column is either a part of natural dimension key or some other column in a dimension that is used in a custom logic to set a foreign dimension key.
If this attribute of a column is set to true, there should be three more properties defined: foreignKey, underlyingDim, and underlyingDimCol. The underlyingDimCol is optional.
For example, if some dimension has three natural key columns that comprise a unique compound key, in the schema there would be three columns - one for each natural key – to set one foreign key on the resulting fact table.
If underlyingDimCol is not defined the setForeignKeyOnFactTable method must be overridden to implement custom logic for setting the foreign key on the fact table
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/isForSettingForeignKey
foreignKey stringOptional
The name of the foreign key column on the fact table. If the foreign key is set using multiple fields (compound key) from the underlying dimension, there will be multiple schema elements with the same "foreignKey" value. There will be only one column with that name in the fact table
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/foreignKey
underlyingDim stringOptional
The name of the dimension with the primary key for the the given "foreignKey"
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/underlyingDim
underlyingDimCol stringOptional
The name of the column in the underlying dimension that is used to set given "foreignKey". If all underlying dimension columns have isNaturalKey set to true, the default implementation of setForeignKeyOnFactTable will be used, otherwise this method will have to be overridden in the corresponding dimension class.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/underlyingDimCol
isMeasure booleanOptional
This flag designates a column as a fact table measure (or fact). The measure column must be one of the numeric types – Float, Short, Integer, or Long.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/properties/isMeasure
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/patternProperties/^__
allOf option 1 Any type
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/0
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/0/if
Then Any type
If this attribute of a column is set to true, there should be three more properties defined: foreignKey, underlyingDim, and underlyingDimCol. The underlyingDimCol is optional.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/0/then
Additional schema keywords (1)
customMessage{"required":""foreignKey", "underlyingDim", and optionally "underlyingDimCol" are required when "isForSettingForeignKey" is true. If "underlyingDimCol" is not defined then setForeignKeyOnFactTable method must be overridden in class a class of a dimension defined in "underlyingDim""}
allOf option 2 Any type
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/1
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/1/if
Nested schema details (1)
isMeasure Any typeRequired
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/1/if/properties/isMeasure
Additional schema keywords (1)
consttrue
Then Any type
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/1/then
Nested schema details (1)
colType Any typeRequired
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/allOf/1/then/properties/colType
Additional schema keywords (2)
enum["Double","Short","Integer","Long"]customMessage{"enum":"For columns that are measures ("isMeasure": true) the column type can only be one of the numeric types: "Double" , "Short", "Integer", "Long" "}
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/schema/items/unevaluatedProperties
unevaluatedProperties Optional
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/properties/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/properties/facts/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/properties/facts/items/unevaluatedProperties
allOf option 1 Any type
JSON Pointer: /properties/dwEtl/allOf/0
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/allOf/0/if
Nested schema details (1)
jobType Any typeRequired
JSON Pointer: /properties/dwEtl/allOf/0/if/properties/jobType
Additional schema keywords (1)
enum["LoadDimsFacts","LoadDims"]
Then Any type
JSON Pointer: /properties/dwEtl/allOf/0/then
allOf option 2 Any type
JSON Pointer: /properties/dwEtl/allOf/1
Nested schema details (2)
If Any type
JSON Pointer: /properties/dwEtl/allOf/1/if
Nested schema details (1)
jobType Any typeRequired
JSON Pointer: /properties/dwEtl/allOf/1/if/properties/jobType
Additional schema keywords (1)
enum["LoadDimsFacts","LoadFacts"]
Then Any type
JSON Pointer: /properties/dwEtl/allOf/1/then
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/dwEtl/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/dwEtl/unevaluatedProperties