RequiredFieldsFromTable

RequiredFieldsFromTable

Retrieves a filtered list of field names from sourceTable, and verifies that those fields exist on targetable

Used to verify that data inputs are as expected.

sourceTable    The table that contains the searchField

searchField    The column on sourceTable that contains field names to search for

targetable    The table to search for required fields

dataset        The filter to apply to the discrete values returned from searchField

{
  "method": "RequiredFieldsFromTable",
  "description": "Required Fields",
  "sourceTable": "01_DATA_MODEL",
  "searchField": "Client Source Field",
  "targetTable": "02_DATA_ETL",
  "dataSet": {
    "logic": "or",
    "name": "DS_RequiredFields_1",
    "strict": true,
    "set": [
      {
        "logic": "and",
        "stype": "FIELD",
        "entity": {
          "type": "field",
          "name": "01_DATA_MODEL.DS_RequiredFields",
          "table": "01_DATA_MODEL",
          "valueFilter": []
        },
        "op": "=",
        "values": [
          "1"
        ]
      }
    ]
  },
  "project": "TestProject"
}

Get list of fieldnames from sourceTable.searchField ([01_DATA_MODEL.Client Source Field])

Apply filter to list of field names (in this case, 01_DATA_MODEL.DS_RequiredFields=1)

Confirm that all fieldnames in list of filtered names are present on targetTable (02_DATA_ETL)

If any fields are missing, raise an error, listing all missing fields.