PRE-Release Note: DataJet v 6.11.11

NOTE: These release notes are still in progress - the software is not yet released

Current test version 6.11.07.01

Known Issues in current pre-release:

• Issue detaching joins when detaching tables - associated joins not reliably detached
• ExportIntoTable not including metadata
• Error processing SOURCE_FILENAME in CreateTableFromFile
• Hotbar disappearing when resizing desktop window
• Issues with Project Has Changed Instance

Key Features

The key features of this release are: 

• Phase 3 Injection & Package Management
  • Object architecture has been changed to better support compound (2nd+ tier) injections
  • Injection and file dependencies are now tracked in each target project
  • Users are notified if they open a project with out-of-date dependencies
• Phase 3 Eyeota
  • It is now possible to perform rapid base/target counts on segments (API: AnalyseSegments)
  • A single call can be used to create a pivot table (API: CreateAudiencesFromWorkbook)
  • Campaign data path info is included with a package
• Metadata and PII Improvements
  • Metadata is automatically included when using APIs CopyFromSourceTable, ExportIntoTable, CreatePackage, CopyUp and CopyDown
  • TODO: SetProjectProperties changes
  • A single excel workbook can now be applied as metadata across multiple tables that require the same metadata configuration
  • Metadata is consistently present in packages and projects that inject packages
  • PII settings are preserved as data is copied
• Core Module upgrades and Patches
• General bug fixes and improvements
• Tech ops usability issues

Functional Summary

Administration & Package Management

• AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
• Add Explorer and Engine version logging to Access Server
• Ability to delete or archive packages
• Add a "Packages" option to Admin menu that brings up the package wizard.
• *P5 AMP-140: Include campaign info in package management setup
• Provide an API to track injections, versions, packages etc

Engineering and Analytics

• Make Engineering | Find Function dialog put the focus into the search text box when it opens.
• Ability to copy correlation matrix to clipboard
• AMP-68: Build aggregate on an Integer produces a double when an integer is needed.
• *P6 AMP-179, 180 Export to file and export to table source Fields
• AMP-170 Add total segment count to Venn diagram legend labels
• Add support for compound joins to cross-realm injection of packages
• *P2 AMP-151 Unable to append larger data to a smaller table

Scripting

• AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"
• Add ignoreIfNotFound to DeleteTemplate
• Make date visible in Script Editor | Execution History tab
• Add a warning to explorer CLOSE button if a script is executing in script editor
• *P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed
• When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent
• Add support for Local Push/Pull to Supplementary Viewer in Script Editor
• AMP-137 SetObjectProperty hideintree: Hides all fields so that only the names of required fields need to be stored in script.

Database Build, Export  and Management

• Add flag to CreateTableFromFile to skip if directory is empty and not create a table
• *P4 AMP-157 Add to SetObjectProperty error message the name of missing object
• AMP-156: Make sure metadata from input packages is included in output package
• Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
• AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields
• includeProjectMetadata flag to automatically add project properties to target project
• Data Audit - Add Copy current view to clipboard button
• AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
• AMP-177 Metadata not working on injected tables
• AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown
• AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel

General Desktop UI

• PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions
• FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message
• AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields
• AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible
• Panel Form Table and field selectors are removing underscores from object names
• DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog
• AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree
• AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built
• Tree-view sort button is not sorting fields alphabetically
• AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables
• *P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear

Audiences

• AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results
• AMP-154: Make counts on a saved audience match live audience counts
• AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list
• Add process indicator to audience calculation
• Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime
• AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script
• AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
• Changes to object architecture
• Export
• Introduction to Data Injection
• XProcess to python

Module Upgrades

• Essential C# and .NET 8 upgrade
• Upgrade Explorer to .NET 8
• Upgrade JSON parser
• Update to C++ core

Breaking Changes

Package Injection

AMP-139 Modify object ownership so that data is stored on local project rather than source project.

Required changes:

• Native projects (i.e., projects that do not contain injected data): No change required
• Source projects (i.e., projects that provide data for injection): All packages to be rebuilt.  No need to reload data.
• Consumer projects (i.e., projects that have packages injected into them): Projects need to be rebuilt. No need to reload native data.

Bugs

AMP-164 Nationwide project losing injected joins after restart. They come back after re-running refresh script.

AMP-193 Large number of Orphan files generated on Metro

The prototype injection architecture (i.e., prior to version 6.11.11.01) implemented an object ownership architecture where data and system files for fields derived/created on an injected table (i.e, created in the consumer project) were stored with the source project.  This included joins between the consumer project and the injected data.

In practice this means that if a client consumer project created data on injected tables, "orphan files" would appear in the source project.   They were "orphan" because GetOrphs could not reconcile them and so reported them as Orphan files (i.e., files that do not belong to a current project).

This is resolved with the new object ownership architecture - system and data files for fields derived on injected tables are now stored in the consumer project.

AMP-196 Data Audit fails with message "Error getting project details. Cannot process GetDetails"

This issue was due to the large number of fields in the project.  The Data Audit report has been modified to support up to XXXX fields.  It is also now possible to audit a single table by selecting the table via Project Explorer | Table | Right-click | Audit.   

Export To Table from the Data Audit will now honour the current display - only what is displayed will be exported.

*P2 AMP-151 Unable to append larger data to a smaller table

This issue was due to a compiler issue, and only happened on Rocky Linux.  Appending a continuous string (width 65) of 152m rows to a table of 98m rows caused a failure in ExportIntoTable (Append). It is now resolved.

API Methods

New

Changed

Deprecated

Feature Details

Administration & Package Management

AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.

Selected templates will now be imported as template objects:

Explorer and Engine version logging added to Access Server

Access Server now has an Engine Versions category:

The Events section now records the version of Desktop/Explorer that was used when logging in:

Add a "Packages" option to Admin menu that brings up the package wizard.

This has the ability to delete packages, and is only accessible to Admin users:

*P5 AMP-140: Include campaign info in package management setup

When creating a package using CreatePackage, use the key "includeCampaignConfiguration":true to include the path to campaign data within the package.

When the package is injected (via InjectPackage), use the same key "includeCampaignConfiguration":true and the consumer project will be campaign enabled.

This removes the need to manually set up campaign definition files for each project that is going to inject campaign (e.g., Eyeota) data.

If multiple packages with campaign data are injected, the most recent injection will determine the path.

{
  "method": "CreatePackage",
  "includeProjectMetadata": true,
  "createMetafile": true,
  "autoLock": false,
  "push": true,
  "name": "REGRESS_campaign_current",
  "targetProject": "campaign-audience",
  "includeCampaignConfiguration": true,
  "project": "Regression"
}

{
  "method": "InjectPackage",
  "homePath": "",
  "includeProjectMetaData": true,
  "errorOnMissingDataFiles": false,
  "name": "REGRESS_campaign_current",
  "pull": true,
  "tables": [
    "DailyLoad",
    "DATA_CAMP_FULL",
    "LoadHistory"
  ],
  "reports": [
    "Segment Type",
    "Segment Dashboard",
    "Historic Segment Type",
    "Segment Dashboard2"
  ],
  "includeCampaignConfiguration": true,
  "project": "Test Campaign Injection"
}

The campaign menu item will be enabled in the consumer project:

Provide an API to track injections, versions, packages etc

GetDependencies returns a report listing current dependencies for a project:

{
  "method": "GetDependencies",
  "project": "CONSUMER_Test1", 
  "changed": false, 
}

For more information on dependencies, see Introduction to Data Injection

Engineering and Analytics

Make Engineering | Find Function dialog put the focus into the search text box when it opens.

Ability to copy correlation matrix to clipboard

The correlation grids can now be copied to the clipboard

AMP-68: Build aggregate on an Integer produces a double when an integer is needed.

long/integer min/max will now produce a long field (not a double)

*P6 AMP-179, 180 Export to file and export to table source Fields

In order to export source fields, the project must first be configured with the source field suffix using SetProjectProperties (alternateFieldNameSuffix): 

If the project has not been configured to export source fields, selecting Export Source fields will have no effect:

{
  "method": "SetProjectProperties",
  "project": "REGRESS_ExportSource_FULL", 
  "alternateFieldNameSuffix": "_source"
}

To test whether or not a project has been configured to export source, fields, use the following in Admin | Runner:

{
  "method": "ReadMongo",
  "query":{"class":"BatProjectMarker"}  
}

The following is an example of the Export method:

{
  "method": "Export",
  "project": "REGRESS_AWI_Load_Local_FULL",
  "exportHeaders": true,
  "exportAlternateFields": true,
  "urns": false,
  "quotedStringHeaders": true,
  "overwrite": true,
  "eid": "672ba94a8557e1d3504a7354",
  "delimiter": "Tab",
  "createImportScript": false,
 
  "fields": [
    "DATA_AWI.address_region",
    "DATA_AWI.allant_currenthouseholdkey"
  ],
  "dataSet": {
    "logic": "or",
    "name": "SubSet",
    "strict": true,
    "set": [
      {
        "logic": "or",
        "stype": "TABLE",
        "entity": "DATA_AWI"
      },
      {
        "logic": "or",
        "stype": "INTRINSIC",
        "intrinsic": {
          "function": "top",
          "params": [
            "10"
          ]
        }
      }
    ]
  },
  "exportFilename": "%MYOUTPUT%REGESS_ExportSource.txt"
}

When exporting source fields, the header in the export file will now match the display field header.  i.e., exporting Customer_Name as source will have header Customer_Name, but export data from Customer_Name_Source

AMP-170 Total segment count added to Venn diagram legend labels and data grid is now visible

It is also now possible to copy and paste data from the grid:

Add support for compound joins to cross-realm injection of packages

{TODO}

Scripting

AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"

Selecting the run button in script editor for a script that has changes will give the following dialog:

Add ignoreIfNotFound to DeleteTemplate

When calling DeleteTemplate, if the flag ignoreIfNotFound is set to true, no error will be raised id the template is not found

{
  "method": "DeleteTemplate",
  "name": "BTD - Random",
  "ignoreIfNotFound": true,
  "project": "CONSUMER_Test1"
}

Make date visible in Script Editor | Execution History tab

The Script | Script Editor | Pull | Script | Script History tab now includes the date of when a particular version was added to the Script Hub:

Add a warning to explorer CLOSE button if a script is executing in script editor

If Explorer/Desktop is closed whilst a script is running, a warning dialog will be displayed:

*P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed

Script Editor now has a right-click option in the JSON window to "View Changes".  This shows the current display next to the stored method details:

When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent

The description of Script Editor | Push | Description will now match the currently stored description for the script that is about to be updated:

Add support for Local Push/Pull to Supplementary Viewer in Script Editor

AMP-137 SetObjectProperty - new property: hideintree makes it simpler to control visibility of fields. 

Use SetObjectProperty hideintree property with value or true or false to determine which fields in a table to hide/show.   Quickly hide or show all fields using "table.*" for names[]

Hide all fields in table:

{
  "method": "SetObjectProperty",
  "property": "hideintree",
  "value": true,
  "names": [
    "BikeTripData.*"
  ],
  "project": "CONSUMER_Test1"
}

Make 2 fields visible:

{
  "method": "SetObjectProperty",
  "property": "hideintree",
  "value": false,
  "names": [
    "BikeTripData.Bike Id",
    "BikeTripData.commonName_End"
  ],
  "project": "CONSUMER_Test1"
}

Database Build, Export  and Management

Changes to Metadata and PII

Issues:

• AMP-156: Make sure metadata from input packages is included in output package
• Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
• includeProjectMetadata flag to automatically add project properties to target project
• AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
• AMP-177 Metadata not working on injected tables
• AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown

Add flag to CreateTableFromFile to skip if directory is empty and not create a table

*P4 AMP-157 Add to SetObjectProperty error message the name of missing object

AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields

Data Audit - Add Copy current view to clipboard button

AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel

General Desktop UI

PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions

FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message

AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields

AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible

Panel Form Table and field selectors are removing underscores from object names

DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog

AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree

AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built

Tree-view sort button is not sorting fields alphabetically

AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables

*P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear

Audiences

AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results

AMP-154: Make counts on a saved audience match live audience counts

AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list

Add process indicator to audience calculation

Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime

AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script

AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either

Module Upgrades

Essential C# and .net 8 upgrade

Upgrade Explorer to .NET 8

Upgrade JSON parser

Update to C++ core

 - c++ lib updated

•     AWSSDK.S3 Version=3.7.308.8
•     BouncyCastle.Cryptography Version=2.4.0
•     ClosedXML Version=0.102.3
•     ExcelDataReader Version=3.6.0
•     ExcelDataReader.DataSet Version=3.6.0 
•     JWT Version=10.1.1 
•     MathNet.Numerics Version=5.0.0 
•     MongoDB.Bson Version=2.26.0 
•     MongoDB.Driver Version=2.26.0 
•     MongoDB.Driver.Core Version=2.26.0 
•     MySql.Data Version=8.4.0 
•     Newtonsoft.Json Version=13.0.3 
•     Npgsql Version=8.0.3 
•     SharpCompress Version=0.37.2 
•     SlackBotMessages Version=2.1.0 
•     SSH.NET Version=2024.0.0 
•     System.Data.Odbc Version=8.0.0 
•     System.Data.SqlClient Version=4.8.6 
•     System.Text.Encoding.CodePages Version=8.0.0 
•     xFunc.Maths Version=4.4.1