DBTimes
Defines required fields for the application that uses DW Source Comparer library. The application configuration can have other root level fields.
Configuration for source compare
Version of a Source Comparer json configuration
Specific value:"1"
Specifications for comparing sources
No Additional ItemsA name of the compare scenario to differentiate multiple scenarios. Names must be unique.
If the value of isActive is false, the compare scenario will not be processed.
If the value is true, there will be some dataframe related output sent to the standard output from the library causing slowdown in job execution.
When comparing two floating point values, they will be considered equal if the absolute value of a difference is less than floatThreshold.
When comparing two timestamp fields they will be truncated using defined unit prior to being compared.
Must match regular expression:^(?i)(year|yyyy|yy|month|mon|mm|day|dd|microsecond|millisecond|second|minute|hour|week|quarter)$
When comparing two date fields they will be truncated using defined unit prior to being compared.
Must match regular expression:^(?i)(year|yyyy|yy|month|mon|mm|week|quarter)$
When defined, the number of reported differences for each column will not exceed the specified value. That will reduce the size of resulting differences file.
Specification for the result of the comparison
Properties for csv file. By default the saved file is multiline
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
Must contain a minimum of 1 items
Column name
At least one column must have this property set to true.
Multiple columns can be designated as unique key to create a compound unique key.
By default the unique key is included unless isInclude value is set to false
By default all columns are compared and included in comparison result, unless at least one column has explicit isInclude flag set (to true or false).
If that's the case only columns with isInclude flag set to true will be compared and included into comparison result.
If columns are listed that do not have isInclude flag, they will be ignored, unless they are keys.
For keys (isUniqueKey is set to true) the default for isInclude is true
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
true
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
First or left source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
Options for SQL Server jdbc connection.
"partitionColumn" is excluded from the list of options for jdbc source since this column has to be defined in "schema" with "isPartitionColumn" flag
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
Properties for mongoDb can be any of the options supported by MongoDb Spark connector in batch mode. Schema must be defined
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^partitioner\.options\.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
A meaningful name of the source to show in compare results. If none defined – the default is “left” or “right”
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
true
If "isPartitionColumn" attribute of a column is set to true, colType must be defined (needed to determine the type of upper and lower bounds.
Column nmae
Column type
For DBMS source one column can be defined as partition column to read data on multiple connections in parallel. isPartitionColumn must be defined if numPartitions is defined.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
Query to create a subset of columns and/or rows to compare. It can be any valid select statement using Spark SQL syntax to be executed on a data loaded from the Source. Source table must be desinated by three underscores, e.g., SELECT CAST(id AS INT) AS id FROM ___ .If not defined all fields and all rows from the source will be used
^.*___.*$
^.*___.*$
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
Second or right source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.
Same definition as leftSourceAll properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.
All properties whose name matches the following regular expression must respect the following conditions
Property name regular expression:^__
Comment field begins with two underscores. There can be any number of comment fields.