PRE-Release Note: DataJet v 6.12.04.01

NOTE: These release notes are still in progress - the software is not yet released

Current test version 6.12.04.01

Key Features

The key features of this release are: 

• Phase 3 Injection & Package Management
  • Object architecture has been changed to better support compound (2nd+ tier) injections
  • Injection and file dependencies are now tracked in each target project
  • Users are notified if they open a project with out-of-date dependencies
• Phase 3 Eyeota
  • It is now possible to perform rapid base/target counts on segments (API: AnalyseSegments)
  • A single call can be used to create a pivot table (API: CreateAudiencesFromWorkbook)
  • Campaign data path info is included with a package
• Metadata and PII Improvements
  • Metadata is automatically included when using APIs CopyFromSourceTable, ExportIntoTable, CreatePackage, CopyUp and CopyDown
  • It is no longer necessary to call SetProjectProperties when injecting a package - metadata in the package will be automatically available.
  • A single excel workbook can now be applied as metadata across multiple tables that require the same metadata configuration
  • Metadata is consistently present in packages and projects that inject packages
  • PII settings are preserved as data is copied
• Core Module upgrades and Patches
• General bug fixes and improvements
• Tech ops usability issues

Functional Summary

Administration & Package Management

• AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
• Add Explorer and Engine version logging to Access Server
• Ability to delete or archive packages
• Add a "Packages" option to Admin menu that brings up the package wizard.
• *P5 AMP-140: Include campaign info in package management setup
• Provide an API to track injections, versions, packages etc
• Opening wizard to edit injected package will go to package (ctrl-w)
• ProcessTextCommand "sysfdump" and "sysxdump" commands added
• GetOrphs - "both", "reverse" and "mongoOnly" optional flags added

Engineering and Analytics

• Make Engineering | Find Function dialog put the focus into the search text box when it opens.
• Ability to copy correlation matrix to clipboard
• AMP-68: Build aggregate on an Integer produces a double when an integer is needed.
• *P6 AMP-179, 180 Export to file and export to table source Fields
• AMP-170 Add total segment count to Venn diagram legend labels
• Add support for compound joins to cross-realm injection of packages
• *P2 AMP-151 Unable to append larger data to a smaller table

Scripting

• AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"
• Add ignoreIfNotFound to DeleteTemplate
• Make date visible in Script Editor | Execution History tab
• Add a warning to explorer CLOSE button if a script is executing in script editor
• *P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed
• When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent
• Add support for Local Push/Pull to Supplementary Viewer in Script Editor
• AMP-137 SetObjectProperty hideintree: Hides all fields so that only the names of required fields need to be stored in script.
• AMP-201: Issues with STOPIF suddenly not working
• Script Editor:  targetDataSets keys support the dropping of Dataset collections

Database Build, Export  and Management

• Add flag to CreateTableFromFile to skip if directory is empty and not create a table
• *P4 AMP-157 Add to SetObjectProperty error message the name of missing object
• AMP-156: Make sure metadata from input packages is included in output package
• AMP-38: Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
• AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields
• includeProjectMetadata flag to automatically add project properties to target project
• Data Audit - Add Copy current view to clipboard button
• AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
• AMP-177 Metadata not working on injected tables
• AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown
• AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel
• Add to Data Audit | Tables tab the origin of a table, eg, Injected/Local and Source (Package or Project name)

General Desktop UI

• PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions
• FIELD TEMPLATE: If an invalid template is dropped onto a DataView, display error message
• AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields
• AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible
• Panel Form Table and field selectors are removing underscores from object names
• DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog
• AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree
• AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built
• Tree-view sort button is not sorting fields alphabetically
• AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables
• *P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear
• JSON in runner can be re-formatted
• Runner has support for Data Model and grid view

Audiences

• AMP-154: Make counts on a saved audience match live audience counts
• AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list
• Add process indicator to audience calculation
• Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime
• EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script
• AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile
• Can drop dataset on AnalyseSegments JSON to append targetSet to targetSet list or replace base dataset
• Drop dataset collection on campaign grid to build AnalyseSegments JSON call on clipboard
• Segment grid content can be copied to clipboard

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
• Export
• Introduction to Data Injection
• Categories and Metadata
• XProcess to python

Module Upgrades

• Essential C# and .NET 8 upgrade
• Upgrade Explorer to .NET 8
• Upgrade JSON parser
• Update to C++ core

Breaking Changes

Package Injection

AMP-139 Modify object ownership so that data is stored on local project rather than source project.

Required changes:

• Native projects (i.e., projects that do not contain injected data): No change required
• Source projects (i.e., projects that provide data for injection): 
  • All packages to be rebuilt.  
  • No need to reload data.  
  • Metadata needs to be re-applied (see below)
• Consumer projects (i.e., projects that have packages injected into them): 
  • Projects need to be rebuilt. 
  • No need to reload native data.
  • Metadata needs to be re-applied (see below)

Metadata

AMP 156, 177 - Metadata on packages and injected tables

Required changes:

• All Metadata from ALL PACKAGES and PROJECTS will be unavailable when opening a project in v6.12.04 or later (apart from the Categoriesproperty)
  • Note - this does not delete the metadata from the underlying package, but the metadata will not be visible on servers running v6.12.04 or later
• ALL PACKAGES and PROJECTS will need to have metadata re-applied using SetPropertiesFromWorkbook or SetPropertiesForTableFromWorkbook
• ALL PACKAGES will need to be rebuilt (after {SetProperties...} has been applied in the package's source project).

Bugs

AMP-164 Nationwide project losing injected joins after restart. They come back after re-running refresh script.

AMP-193 Large number of Orphan files generated on Metro

The prototype injection architecture (i.e., prior to version 6.11.11.01) implemented an object ownership architecture where data and system files for fields derived/created on an injected table (i.e, created in the consumer project) were stored with the source project.  This included joins between the consumer project and the injected data.

In practice this means that if a client consumer project created data on injected tables, "orphan files" would appear in the source project.   They were "orphan" because GetOrphs could not reconcile them and so reported them as Orphan files (i.e., files that do not belong to a current project).

This is resolved with the new object ownership architecture - system and data files for fields derived on injected tables are now stored in the consumer project.

AMP-196 Data Audit fails with message "Error getting project details. Cannot process GetDetails"

This issue was due to the large number of fields in the project.  The Data Audit report has been modified to support up to XXXX fields.  It is also now possible to audit a single table by selecting the table via Project Explorer | Table | Right-click | Audit.   

Export To Table from the Data Audit will now honour the current display - only what is displayed will be exported.

*P2 AMP-151 Unable to append larger data to a smaller table

This issue was due to a compiler issue, and only happened on Rocky Linux.  Appending a continuous string (width 65) of 152m rows to a table of 98m rows caused a failure in ExportIntoTable (Append). It is now resolved.

AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built

The Add Layer button is now disabled whilst a join layer is added from the Engineering | Joins dialog:

BUG: Panel Form Table and field selectors are removing underscores from object names

Previous releases would remove underscores from the Engineering | Panel Form | Stats Panel | Modify drop-downs.  This is now fixed.

API Methods

New

Changed

Deprecated

Feature Details

Administration & Package Management

AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.

Selected templates will now be imported as template objects:

Explorer and Engine version logging added to Access Server

Access Server now has an Engine Versions category:

The Events section now records the version of Desktop/Explorer that was used when logging in:

Add a "Packages" option to Admin menu that brings up the package wizard.

This has the ability to delete packages, and is only accessible to Admin users:

*P5 AMP-140: Include campaign info in package management setup

When creating a package using CreatePackage, use the key "includeCampaignConfiguration":true to include the path to campaign data within the package.

When the package is injected (via InjectPackage), use the same key "includeCampaignConfiguration":true and the consumer project will be campaign enabled.

This removes the need to manually set up campaign definition files for each project that is going to inject campaign (e.g., Eyeota) data.

If multiple packages with campaign data are injected, the most recent injection will determine the path.

{
  "method": "CreatePackage",
  "includeProjectMetadata": true,
  "createMetafile": true,
  "autoLock": false,
  "push": true,
  "name": "REGRESS_campaign_current",
  "targetProject": "campaign-audience",
  "includeCampaignConfiguration": true,
  "project": "Regression"
}

{
  "method": "InjectPackage",
  "homePath": "",
  "includeProjectMetaData": true,
  "errorOnMissingDataFiles": false,
  "name": "REGRESS_campaign_current",
  "pull": true,
  "tables": [
    "DailyLoad",
    "DATA_CAMP_FULL",
    "LoadHistory"
  ],
  "reports": [
    "Segment Type",
    "Segment Dashboard",
    "Historic Segment Type",
    "Segment Dashboard2"
  ],
  "includeCampaignConfiguration": true,
  "project": "Test Campaign Injection"
}

The campaign menu item will be enabled in the consumer project:

Provide an API to track injections, versions, packages etc

GetDependencies returns a report listing current dependencies for a project:

{
  "method": "GetDependencies",
  "project": "CONSUMER_Test1", 
  "changed": false, 
}

For more information on dependencies, see Introduction to Data Injection

Engineering and Analytics

Make Engineering | Find Function dialog put the focus into the search text box when it opens.

Ability to copy correlation matrix to clipboard

The correlation grids can now be copied to the clipboard

AMP-68: Build aggregate on an Integer produces a double when an integer is needed.

long/integer min/max will now produce a long field (not a double)

*P6 AMP-179, 180 Export to file and export to table source Fields

In order to export source fields, the project must first be configured with the source field suffix using SetProjectProperties (alternateFieldNameSuffix): 

If the project has not been configured to export source fields, selecting Export Source fields will have no effect:

{
  "method": "SetProjectProperties",
  "project": "REGRESS_ExportSource_FULL", 
  "alternateFieldNameSuffix": "_source"
}

To test whether or not a project has been configured to export source, fields, use the following in Admin | Runner:

{
  "method": "ReadMongo",
  "query":{"class":"BatProjectMarker"}  
}

The following is an example of the Export method:

{
  "method": "Export",
  "project": "REGRESS_AWI_Load_Local_FULL",
  "exportHeaders": true,
  "exportAlternateFields": true,
  "urns": false,
  "quotedStringHeaders": true,
  "overwrite": true,
  "eid": "672ba94a8557e1d3504a7354",
  "delimiter": "Tab",
  "createImportScript": false,
 
  "fields": [
    "DATA_AWI.address_region",
    "DATA_AWI.allant_currenthouseholdkey"
  ],
  "dataSet": {
    "logic": "or",
    "name": "SubSet",
    "strict": true,
    "set": [
      {
        "logic": "or",
        "stype": "TABLE",
        "entity": "DATA_AWI"
      },
      {
        "logic": "or",
        "stype": "INTRINSIC",
        "intrinsic": {
          "function": "top",
          "params": [
            "10"
          ]
        }
      }
    ]
  },
  "exportFilename": "%MYOUTPUT%REGESS_ExportSource.txt"
}

When exporting source fields, the header in the export file will now match the display field header.  i.e., exporting Customer_Name as source will have header Customer_Name, but export data from Customer_Name_Source

AMP-170 Total segment count added to Venn diagram legend labels and data grid is now visible

It is also now possible to copy and paste data from the grid:

Add support for compound joins to cross-realm injection of packages

{TODO}

Scripting

AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"

Selecting the run button in script editor for a script that has changes will give the following dialog:

Add ignoreIfNotFound to DeleteTemplate

When calling DeleteTemplate, if the flag ignoreIfNotFound is set to true, no error will be raised id the template is not found

{
  "method": "DeleteTemplate",
  "name": "BTD - Random",
  "ignoreIfNotFound": true,
  "project": "CONSUMER_Test1"
}

Make date visible in Script Editor | Execution History tab

The Script | Script Editor | Pull | Script | Script History tab now includes the date of when a particular version was added to the Script Hub:

Add a warning to explorer CLOSE button if a script is executing in script editor

If Explorer/Desktop is closed whilst a script is running, a warning dialog will be displayed:

*P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed

Script Editor now has a right-click option in the JSON window to "View Changes".  This shows the current display next to the stored method details:

When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent

The description of Script Editor | Push | Description will now match the currently stored description for the script that is about to be updated:

Add support for Local Push/Pull to Supplementary Viewer in Script Editor

AMP-137 SetObjectProperty - new property: hideintree makes it simpler to control visibility of fields. 

Use SetObjectProperty hideintree property with value or true or false to determine which fields in a table to hide/show.   Quickly hide or show all fields using "table.*" for names[]

Hide all fields in table:

{
  "method": "SetObjectProperty",
  "property": "hideintree",
  "value": true,
  "names": [
    "BikeTripData.*"
  ],
  "project": "CONSUMER_Test1"
}

Make 2 fields visible:

{
  "method": "SetObjectProperty",
  "property": "hideintree",
  "value": false,
  "names": [
    "BikeTripData.Bike Id",
    "BikeTripData.commonName_End"
  ],
  "project": "CONSUMER_Test1"
}

AMP-201: Issues with STOPIF suddenly not working

In some rare situations, the STOPIF parser would remove digits from a number and evaluate the resulting parameter as a date.   This would then provide an incorrect evaluation of a condition, causing the condition to return True, even though the condition returns False if the parameters are correctly evaluated as numbers.

Additional checking has been added to catch these occurrences, as well as the ability to explicitly prevent the parser from evaluating a number a date. 

An "Evaluate Condition" method is now available in the JSON window:

Database Build, Export  and Management

Changes to Metadata

Issues:

• AMP-156: Make sure metadata from input packages is included in output package
• Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
• includeProjectMetadata flag to automatically add project properties to target project
• AMP-177 Metadata not working on injected tables

Storage of metadata properties has been modified so that once metadata is attached to a field, it remains attached to all copies of that field (Note - this does include Copy Up and Copy Down, but does not include fields that are derived from the source field).

This means that output packages (created using CreatePackage) will automatically contain the metadata that is in the source project.   

When a package is injected - using InjectPackage - into a consumer project, the consumer project will automatically display the metadata for the injected package - there is no need to set up project properties first (e.g., using SetProjectProperties)

See the following for further details of packages, metadata and injection:

• Categories and Metadata
• Introduction to Data Injection
• InjectPackage
• CreatePackage

AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.

AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown

SetPII blanks out the contents of fields:

ExportIntoTable  will now preserve these settings:

As will CopySourceTable:

Add flag to CreateTableFromFile to skip if directory is empty and not create a table

The flag "ignoreEmptyFolder" has been added to CreateTableFromFile.  If this flag is true, and an empty bulk load directory has been specified, the method will skip without raising an error.

*P4 AMP-157 Add to SetObjectProperty error message the name of missing object

AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields

Data Audit - Add Copy current view to clipboard button

The Data Audit report now has the capability to copy the current view to the clipboard:

Data Audit has also been extended to support the following:

• Project Explorer | Table | Right-Click | Audit - displays "All" tab just for selected table
• Tables tab includes details of source projects and packages for a table.

AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel

Auto-guess has been modified so that {TODO}

Add to Data Audit | Tables tab the origin of a table, eg, Injected/Local and Source (Package or Project name)

Details of source package and source project are now included in the Data Audit | Tables tab.  See Data Audit and Introduction to Data Injection for further details.

General Desktop UI

PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions

All five aggregate functions - count, sum, min, max, avg - are now available from the drag and drop shortcut option for aggregates.

FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message

If a template from a table other than the DataView resolution table is dropped onto the dataview, the message "Template not from Dataset resolution table" will be displayed.

AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields

When data is decoded, the original field is renamed with a suffix (for example "_source") to be [FIELDNAME_]source.   These fields are usually hidden at the point of decode.  They can be made visible via Project | Show Source Fields. 

AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible

If the Project Explorer contains data that requires a horizontal scrollbar, this will now be visible all the time.

DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog

The Bottom By and Top By dialogs have been modified to give a clearer indication of what is required as data entry:

• Field - field to select from
• N - Number of records to select

AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree

Previously, a dataset that was created in UI (for example, in Project Explorer | Context Panel) that used the in clause could be viewed in Dataset Scratch but not edited.   The edit dialog has been modified to support editing of in clauses:

See DataSet HotBar - Functional Overview (operators) for more details on Dataset operations.

Tree-view sort button is not sorting fields alphabetically

The Project Explorer tree-view supports 2 display orders:

• Chronological - items are displayed in the order in which they were created
• Alphabetical - items are sorted alphabetically

AMP-156: Modify SetPropertiesFromWorkbook and Metadata so that a single hygiene file can be applied to multiple tables

A new API call has been added - SetPropertiesForTableFromWorkbook

This works similarly to SetPropertiesFromWorkbook but does not specify the owner table of each field to be modified.   The table is set using the "table" property. This allows the same field definition to be applied to multiple tables.

{
  "method": "SetPropertiesForTableFromWorkbook",
  "table": "Base1_AWIData_Foreign",
  "filename": "%DATAPATH%REGRESS/OneTouch/OneTouch_Definitions/DEF_OT_HYGIENE_Metadata_20241128.xlsx",
  "delimiter": ",",
  "properties": [
    "Displayname",
    "MetadataType",
    "Code",
    "Allant ID",
    "Categories",
    "Value Type",
    "DSARDelete",
    "DisplayText"
  ],
  "project": "REGRESS_Base1_Create_Sample"
}

*P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear

See provide an api to track injections, versions, packages etc

Audiences

AMP-154: Make counts on a saved audience match live audience counts

AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list

Method CreateAudiencesFromWorkbook reads a list of audience IDs and names from an excel workbook and creates a series of dataset audiences on the primary contact table. 

It includes the option to export a subset of data as a "pivot table" (see property exportAsTable)

Add process indicator to audience calculation

Audiences are generally calculated via the web portal (https://webhome.allantgroup.com/amp/).   The following message is displayed while the audience is calculating:

"Changes to rules blocked while calculating counts"

In the desktop application, a process indicator is shown in the Campaign | Audiences tab:

Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime

The ProcessSegments report now contains:

• start - timestamp of when processing began
• end - timestamp of when processing ended
• duration - duration of processing in seconds

EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script

Changes to the segment processor mean that the record count now matches the unique ID count in the segment.

Counts for all segments can be output to Datajet table or Excel spreadsheet using GetCampaignHistory:

AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile

A new method - AnalyseSegments - makes it possible to count intersections and overlaps between segments

Can drop dataset on AnalyseSegments JSON to append targetSet to targetSet list or replace base dataset

Datasets can be dropped into the JSON window of script editor to replace or append targetDataSets and base datasets

Dataset collections can be dropped into the JSON window of script editor to replace targetDataSets or append to existing Target DataSets  

Drop dataset collection on campaign grid to build AnalyseSegments JSON call on clipboard

Dropping a dataset collection onto the campaign grid will add an AnalyseSegments JSON which has the collection set as targetDataSets to the clipboard.

Use paste or CTRL V to copy the content from the clipboard into script editor or runner.

Segment grid content can be copied to clipboard

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
• Introduction to Data Injection
• Export

Module Upgrades

Essential C# and .NET 8 upgrade

Upgrade Explorer to .NET 8

Upgrade JSON parser

Update to C++ core

 - c++ lib updated

•     AWSSDK.S3 Version=3.7.308.8
•     BouncyCastle.Cryptography Version=2.4.0
•     ClosedXML Version=0.102.3
•     ExcelDataReader Version=3.6.0
•     ExcelDataReader.DataSet Version=3.6.0 
•     JWT Version=10.1.1 
•     MathNet.Numerics Version=5.0.0 
•     MongoDB.Bson Version=2.26.0 
•     MongoDB.Driver Version=2.26.0 
•     MongoDB.Driver.Core Version=2.26.0 
•     MySql.Data Version=8.4.0 
•     Newtonsoft.Json Version=13.0.3 
•     Npgsql Version=8.0.3 
•     SharpCompress Version=0.37.2 
•     SlackBotMessages Version=2.1.0 
•     SSH.NET Version=2024.0.0 
•     System.Data.Odbc Version=8.0.0 
•     System.Data.SqlClient Version=4.8.6 
•     System.Text.Encoding.CodePages Version=8.0.0 
•     xFunc.Maths Version=4.4.1