PRE-Release Note: DataJet v 6.11.11

NOTE: These release notes are still in progress - the software is not yet released

Current test version 6.11.04.01

Key Features

The key features of this release are: 

• Phase 3 Injection & Package Management
  • Object architecture has been changed to better support compound (2nd+ tier) injections
  • Injection and file dependencies are now tracked in each target project
  • Users are notified if they open a project with out-of-date dependencies
• Phase 3 Eyeota
  • It is now possible to perform rapid base/target counts on segments (API: AnalyseSegments)
  • A single call can be used to create a pivot table (API: CreateAudiencesFromWorkbook)
  • Campaign data path info is included with a package
• Metadata and PII Improvements
  • Metadata is automatically included when using APIs CopyFromSourceTable, ExportIntoTable, CreatePackage
  • TODO: SetProjectProperties changes
  • A single excel workbook can now be applied as metadata across multiple tables that require the same metadata configuration
  • Metadata is consistently present in packages and projects that inject packages
  • PII settings are preserved as data is copied
• Core Module upgrades and Patches
• General bug fixes and improvements
• Tech ops usability issues

Functional Summary

Administration & Package Management

• AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
• Add Explorer and Engine version logging to Access Server
• Ability to delete or archive packages
• Add a "Packages" option to Admin menu that brings up the package wizard.
• *P5 AMP-140: Include campaign info in package management setup
• Provide an API to track injections, versions, packages etc

Engineering and Analytics

• Make Engineering | Find Function dialog put the focus into the search text box when it opens.
• AMP-XX Ability to export correlation matrix to excel
• AMP-68: Build aggregate on an Integer produces a double when an integer is needed.
• *P6 AMP-179, 180 Export to file and export to table source Fields
• AMP-170 Add total segment count to Venn diagram legend labels
• Add support for compound joins to cross-realm injection of packages
• *P2 AMP-151 Investigate Unable to append larger data to a smaller table

Scripting

• AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"
• Add ignoreIfNotFound to DeleteTemplate
• Make date visible in Script Editor | Execution History tab
• Add a warning to explorer CLOSE button if a script is executing in script editor
• *P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed
• When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent
• Add support for Local Push/Pull to Supplementary Viewer in Script Editor
• AMP-137 New APIs: HideAllFields and ShowFields: Hides all fields so that only the names of required fields need to be stored in script.

Database Build, Export  and Management

• Add flag to CreateTableFromFile to skip if directory is empty and not create a table
• *P4 AMP-157 Add to SetObjectProperty error message the name of missing object
• AMP-156: Make sure metadata from input packages is included in output package
• Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
• AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields
• includeProjectMetadata flag to automatically add project properties to target project
• Data Audit - Add Copy current view to clipboard button
• AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
• AMP-177 Metadata not working on injected tables
• AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown
• AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel

General Desktop UI

• PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions
• FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message
• AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields
• AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible
• Panel Form Table and field selectors are removing underscores from object names
• DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog
• AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree
• AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built
• Tree-view sort button is not sorting fields alphabetically
• AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables
• *P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear

Audiences

• AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results
• AMP-154: Make counts on a saved audience match live audience counts
• AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list
• Add process indicator to audience calculation
• Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime
• AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script
• AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
• Changes to object architecture
• Export
• Introduction to Data Injection
• XProcess to python

Module Upgrades

• Essential C# and .NET 8 upgrade
• Upgrade Explorer to .NET 8
• Upgrade JSON parser
• Update to C++ core

Breaking Changes

Package Injection

AMP-139 Modify object ownership so that data is stored on local project rather than source project.

Required changes:

• Native projects (i.e., projects that do not contain injected data): No change required
• Source projects (i.e., projects that provide data for injection): All packages to be rebuilt.  No need to reload data.
• Consumer projects (i.e., projects that have packages injected into them): Projects need to be rebuilt.  No need to reload native data.

Bugs

AMP-164 Nationwide project losing injected joins after restart. They come back after re-running refresh script.

AMP-193 Large number of Orphan files generated on Metro

The prototype injection architecture (i.e., prior to version 6.11.11.01) implemented an object ownership architecture where data and system files for fields derived/created on an injected table (i.e, created in the consumer project) were stored with the source project.  This included joins between the consumer project and the injected data.

In practice this means that if a client consumer project created data on injected tables, "orphan files" would appear in the source project.   They were "orphan" because GetOrphs could not reconcile them and so reported them as Orphan files (i.e., files that do not belong to a current project).

This is resolved with the new object ownership architecture - system and data files for fields derived on injected tables are now stored in the consumer project.

AMP-196 Data Audit fails with message "Error getting project details. Cannot process GetDetails"

This issue was due to the large number of fields in the project.  The Data Audit report has been modified to support up to XXXX fields.  It is also now possible to audit a single table by selecting the table via Project Explorer | Table | Right-click | Audit.   

Export To Table from the Data Audit will now honour the current display - only what is displayed will be exported.

*P2 AMP-151 Unable to append larger data to a smaller table

This issue was due to a compiler issue, and only happened on Rocky Linux.  Appending a continuous string (width 65) of 152m rows to a table of 98m rows caused a failure in ExportIntoTable (Append). It is now resolved.

API Methods

New

Changed

Deprecated

Feature Details

Administration & Package Management

AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.

Selected templates will now be imported as template objects:

Explorer and Engine version logging added to Access Server

Access Server now has an Engine Versions category:

The Events section now records the version of Desktop/Explorer that was used when logging in:

Add a "Packages" option to Admin menu that brings up the package wizard.

This has the ability to delete packages, and is only accessible to Admin users:

*P5 AMP-140: Include campaign info in package management setup

When creating a package using CreatePackage, use the key "includeCampaignConfiguration":true to include the path to campaign data within the package.

When the package is injected (via InjectPackage), use the same key "includeCampaignConfiguration":true and the consumer project will be campaign enabled.

This removes the need to manually set up campaign definition files for each project that is going to inject campaign (e.g., Eyeota) data.

If multiple packages with campaign data are injected, the most recent injection will determine the path.

{
  "method": "CreatePackage",
  "includeProjectMetadata": true,
  "createMetafile": true,
  "autoLock": false,
  "push": true,
  "name": "REGRESS_campaign_current",
  "targetProject": "campaign-audience",
  "includeCampaignConfiguration": true,
  "project": "Regression"
}

{
  "method": "InjectPackage",
  "homePath": "",
  "includeProjectMetaData": true,
  "errorOnMissingDataFiles": false,
  "name": "REGRESS_campaign_current",
  "pull": true,
  "tables": [
    "DailyLoad",
    "DATA_CAMP_FULL",
    "LoadHistory"
  ],
  "reports": [
    "Segment Type",
    "Segment Dashboard",
    "Historic Segment Type",
    "Segment Dashboard2"
  ],
  "includeCampaignConfiguration": true,
  "project": "Test Campaign Injection"
}

The campaign menu item will be enabled in the consumer project:

Provide an API to track injections, versions, packages etc

GetDependencies returns a report listing current dependencies for a project:

Engineering and Analytics

Make Engineering | Find Function dialog put the focus into the search text box when it opens.

AMP-XX Ability to export correlation matrix to excel

AMP-68: Build aggregate on an Integer produces a double when an integer is needed.

*P6 AMP-179, 180 Export to file and export to table source Fields

AMP-170 Add total segment count to Venn diagram legend labels and make grid visible

TODO: diagram showing zones

Add support for compound joins to cross-realm injection of packages

Scripting

AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"

Add ignoreIfNotFound to DeleteTemplate

TODO: DeleteTemplate

{
  "method": "Delete Template",
  "name": "BTD - Random",
  "ignoreIfNotFound": true,
  "project": "CONSUMER_Test1"
}

Make date visible in Script Editor | Execution History tab

Add a warning to explorer CLOSE button if a script is executing in script editor

*P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed

When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent

Add support for Local Push/Pull to Supplementary Viewer in Script Editor

AMP-137 New APIs: HideAllFields and ShowFields: Hides all fields so that only the names of required fields need to be stored in script.

Database Build, Export  and Management

Add flag to CreateTableFromFile to skip if directory is empty and not create a table

*P4 AMP-157 Add to SetObjectProperty error message the name of missing object

AMP-156: Make sure metadata from input packages is included in output package

Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first

AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields

includeProjectMetadata flag to automatically add project properties to target project

Data Audit - Add Copy current view to clipboard button

AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.

AMP-177 Metadata not working on injected tables

AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown

AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel

General Desktop UI

PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions

FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message

AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields

AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible

Panel Form Table and field selectors are removing underscores from object names

DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog

AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree

AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built

Tree-view sort button is not sorting fields alphabetically

AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables

*P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear

Audiences

AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results

AMP-154: Make counts on a saved audience match live audience counts

AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list

Add process indicator to audience calculation

Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime

AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script

AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile

Documentation

• Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either

Module Upgrades

Essential C# and .net 8 upgrade

Upgrade Explorer to .NET 8

Upgrade JSON parser

Update to C++ core

 - c++ lib updated

•     AWSSDK.S3 Version=3.7.308.8
•     BouncyCastle.Cryptography Version=2.4.0
•     ClosedXML Version=0.102.3
•     ExcelDataReader Version=3.6.0
•     ExcelDataReader.DataSet Version=3.6.0 
•     JWT Version=10.1.1 
•     MathNet.Numerics Version=5.0.0 
•     MongoDB.Bson Version=2.26.0 
•     MongoDB.Driver Version=2.26.0 
•     MongoDB.Driver.Core Version=2.26.0 
•     MySql.Data Version=8.4.0 
•     Newtonsoft.Json Version=13.0.3 
•     Npgsql Version=8.0.3 
•     SharpCompress Version=0.37.2 
•     SlackBotMessages Version=2.1.0 
•     SSH.NET Version=2024.0.0 
•     System.Data.Odbc Version=8.0.0 
•     System.Data.SqlClient Version=4.8.6 
•     System.Text.Encoding.CodePages Version=8.0.0 
•     xFunc.Maths Version=4.4.1