- 10 Minutes to read
- Print
- DarkLight
- PDF
PRE-Release Note: DataJet v 6.11.11
- 10 Minutes to read
- Print
- DarkLight
- PDF
NOTE: These release notes are still in progress - the software is not yet released
Current test version 6.11.04.01
Key Features
The key features of this release are:
- Phase 3 Injection & Package Management
- Object architecture has been changed to better support compound (2nd+ tier) injections
- Injection and file dependencies are now tracked in each target project
- Users are notified if they open a project with out-of-date dependencies
- Phase 3 Eyeota
- It is now possible to perform rapid base/target counts on segments (API: AnalyseSegments)
- A single call can be used to create a pivot table (API: CreateAudiencesFromWorkbook)
- Campaign data path info is included with a package
- Metadata and PII Improvements
- Metadata is automatically included when using APIs CopyFromSourceTable, ExportIntoTable, CreatePackage
- TODO: SetProjectProperties changes
- A single excel workbook can now be applied as metadata across multiple tables that require the same metadata configuration
- Metadata is consistently present in packages and projects that inject packages
- PII settings are preserved as data is copied
- Core Module upgrades and Patches
- General bug fixes and improvements
- Tech ops usability issues
Functional Summary
Administration & Package Management
- AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
- Add Explorer and Engine version logging to Access Server
- Ability to delete or archive packages
- Add a "Packages" option to Admin menu that brings up the package wizard.
- *P5 AMP-140: Include campaign info in package management setup
- Provide an API to track injections, versions, packages etc
Engineering and Analytics
- Make Engineering | Find Function dialog put the focus into the search text box when it opens.
- AMP-XX Ability to export correlation matrix to excel
- AMP-68: Build aggregate on an Integer produces a double when an integer is needed.
- *P6 AMP-179, 180 Export to file and export to table source Fields
- AMP-170 Add total segment count to Venn diagram legend labels
- Add support for compound joins to cross-realm injection of packages
- *P2 AMP-151 Investigate Unable to append larger data to a smaller table
Scripting
- AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"
- Add ignoreIfNotFound to DeleteTemplate
- Make date visible in Script Editor | Execution History tab
- Add a warning to explorer CLOSE button if a script is executing in script editor
- *P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed
- When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent
- Add support for Local Push/Pull to Supplementary Viewer in Script Editor
- AMP-137 New APIs: HideAllFields and ShowFields: Hides all fields so that only the names of required fields need to be stored in script.
Database Build, Export and Management
- Add flag to CreateTableFromFile to skip if directory is empty and not create a table
- *P4 AMP-157 Add to SetObjectProperty error message the name of missing object
- AMP-156: Make sure metadata from input packages is included in output package
- Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
- AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields
- includeProjectMetadata flag to automatically add project properties to target project
- Data Audit - Add Copy current view to clipboard button
- AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
- AMP-177 Metadata not working on injected tables
- AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown
- AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel
General Desktop UI
- PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions
- FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message
- AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields
- AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible
- Panel Form Table and field selectors are removing underscores from object names
- DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog
- AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree
- AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built
- Tree-view sort button is not sorting fields alphabetically
- AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables
- *P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear
Audiences
- AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results
- AMP-154: Make counts on a saved audience match live audience counts
- AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list
- Add process indicator to audience calculation
- Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime
- AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script
- AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile
Documentation
- Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
- Changes to object architecture
- Export
- Introduction to Data Injection
- XProcess to python
Module Upgrades
- Essential C# and .NET 8 upgrade
- Upgrade Explorer to .NET 8
- Upgrade JSON parser
- Update to C++ core
Breaking Changes
Package Injection
AMP-139 Modify object ownership so that data is stored on local project rather than source project.
Required changes:
- Native projects (i.e., projects that do not contain injected data): No change required
- Source projects (i.e., projects that provide data for injection): All packages to be rebuilt. No need to reload data.
- Consumer projects (i.e., projects that have packages injected into them): Projects need to be rebuilt. No need to reload native data.
Bugs
AMP-164 Nationwide project losing injected joins after restart. They come back after re-running refresh script.
AMP-193 Large number of Orphan files generated on Metro
The prototype injection architecture (i.e., prior to version 6.11.11.01) implemented an object ownership architecture where data and system files for fields derived/created on an injected table (i.e, created in the consumer project) were stored with the source project. This included joins between the consumer project and the injected data.
In practice this means that if a client consumer project created data on injected tables, "orphan files" would appear in the source project. They were "orphan" because GetOrphs could not reconcile them and so reported them as Orphan files (i.e., files that do not belong to a current project).
This is resolved with the new object ownership architecture - system and data files for fields derived on injected tables are now stored in the consumer project.
AMP-196 Data Audit fails with message "Error getting project details. Cannot process GetDetails"
This issue was due to the large number of fields in the project. The Data Audit report has been modified to support up to XXXX fields. It is also now possible to audit a single table by selecting the table via Project Explorer | Table | Right-click | Audit.
Export To Table from the Data Audit will now honour the current display - only what is displayed will be exported.
*P2 AMP-151 Unable to append larger data to a smaller table
This issue was due to a compiler issue, and only happened on Rocky Linux. Appending a continuous string (width 65) of 152m rows to a table of 98m rows caused a failure in ExportIntoTable (Append). It is now resolved.
API Methods
New
CreateAudiencesFromWorkbook | Reads list of audience IDs and names from excel workbook and creates a series of dataset audiences on primary contact table. Includes the option to export a subset of data as a pivot table. |
AnalyseSegments | Creates a spreadsheet of intersection counts for a base filter and set of target filters for segmentation data |
GetDependencies | Returns a list of project dependencies |
DeleteDependencies | Deletes dependencies from a project's dependency list |
DeletePackage | Deletes a named package from the storage hub |
DetachPackage | Detaches all items injected from the named package |
Changed
Method | Change |
---|---|
ProcessSegments | "maxLines" and "collated" deprecated - no longer required as the default is now to process the max possible, and all data is collated. |
CreatePackage | includes campaignRoot if |
InjectPackage | field templates can be imported as objects |
ExportIntoTable | |
BulkDecode | Adds workbook as dependency object |
BulkBanding | Adds workbook as dependency object |
CreateTableFromWorkbook | Adds workbook as dependency object |
CopySourceTable | |
SetPropertiesFromWorkbook | |
CreateTableFromFile | |
CopyUp | |
CopyDown | |
BuildAggregate | |
STOPIF | |
SetObjectProperty | new property: hideintree |
Export | |
DeleteTemplate | key "ignoreIfNotFound" added. |
Deprecated
InjectTable | |
InjectTables | |
ProcessSegments.maxLines | |
ProcessSegments.collated |
Feature Details
Administration & Package Management
AMP-78: Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
Selected templates will now be imported as template objects:
Explorer and Engine version logging added to Access Server
Access Server now has an Engine Versions category:
The Events section now records the version of Desktop/Explorer that was used when logging in:
Add a "Packages" option to Admin menu that brings up the package wizard.
This has the ability to delete packages, and is only accessible to Admin users:
*P5 AMP-140: Include campaign info in package management setup
When creating a package using CreatePackage, use the key "includeCampaignConfiguration":true to include the path to campaign data within the package.
When the package is injected (via InjectPackage), use the same key "includeCampaignConfiguration":true and the consumer project will be campaign enabled.
This removes the need to manually set up campaign definition files for each project that is going to inject campaign (e.g., Eyeota) data.
If multiple packages with campaign data are injected, the most recent injection will determine the path.
{
"method": "CreatePackage",
"includeProjectMetadata": true,
"createMetafile": true,
"autoLock": false,
"push": true,
"name": "REGRESS_campaign_current",
"targetProject": "campaign-audience",
"includeCampaignConfiguration": true,
"project": "Regression"
}
{
"method": "InjectPackage",
"homePath": "",
"includeProjectMetaData": true,
"errorOnMissingDataFiles": false,
"name": "REGRESS_campaign_current",
"pull": true,
"tables": [
"DailyLoad",
"DATA_CAMP_FULL",
"LoadHistory"
],
"reports": [
"Segment Type",
"Segment Dashboard",
"Historic Segment Type",
"Segment Dashboard2"
],
"includeCampaignConfiguration": true,
"project": "Test Campaign Injection"
}
The campaign menu item will be enabled in the consumer project:
Provide an API to track injections, versions, packages etc
GetDependencies returns a report listing current dependencies for a project:
Engineering and Analytics
Make Engineering | Find Function dialog put the focus into the search text box when it opens.
AMP-XX Ability to export correlation matrix to excel
AMP-68: Build aggregate on an Integer produces a double when an integer is needed.
*P6 AMP-179, 180 Export to file and export to table source Fields
AMP-170 Add total segment count to Venn diagram legend labels and make grid visible
TODO: diagram showing zones
Add support for compound joins to cross-realm injection of packages
Scripting
AMP-43: Change "Save/Run" and "Push/Run" option in Script Editor | Run to be "Cancel/Save/Run"
Add ignoreIfNotFound to DeleteTemplate
TODO: DeleteTemplate
{
"method": "Delete Template",
"name": "BTD - Random",
"ignoreIfNotFound": true,
"project": "CONSUMER_Test1"
}
Make date visible in Script Editor | Execution History tab
Add a warning to explorer CLOSE button if a script is executing in script editor
*P3 AMP-158 Add to Script Editor JSON Right-click the ability to see the currently saved method details which a method has been changed
When pushing a script, the current description of the script that is about to be updated is missing if the script was retrieved using Pull Recent
Add support for Local Push/Pull to Supplementary Viewer in Script Editor
AMP-137 New APIs: HideAllFields and ShowFields: Hides all fields so that only the names of required fields need to be stored in script.
Database Build, Export and Management
Add flag to CreateTableFromFile to skip if directory is empty and not create a table
*P4 AMP-157 Add to SetObjectProperty error message the name of missing object
AMP-156: Make sure metadata from input packages is included in output package
Automatically include metadata when using ExportIntoTable and CopySourceTable without needing SetProjectProperties to be run first
AMP-185 ExportIntoTable needs to maintain hidden/visible status of _source fields
includeProjectMetadata flag to automatically add project properties to target project
Data Audit - Add Copy current view to clipboard button
AMP-155: SetPropertiesFromWorkbook does not set metadata for the last line.
AMP-177 Metadata not working on injected tables
AMP-198 Preserve SetPII settings in ExportToTable, CopyUp and CopyDown
AMP-81 Auto-guess for CreateTableFromFile breaks if mismatched single quote - need to be able to cancel
General Desktop UI
PROJECT EXPLORER: Extend AGGREGATE SHORTCUT menu to include all functions
FIELD TEMPLATE: If an invalid template is dropped onto a dataview, display error message
AMP-137 Change Project | Show Hidden fields to say Project | Show Source Fields
AMP-159 Pin horizontal scroll bar to window in table tree view so it's always visible
Panel Form Table and field selectors are removing underscores from object names
DATASET: Intrinsic Function "Bottom By" - add a label for "Number" to the dialog
AMP-73: Add support to UI to allow NULL to be added to an existing "in" clause in Query Rule Tree
AMP-109 Add Join Layer button not being correctly disabled whilst layer is being built
Tree-view sort button is not sorting fields alphabetically
AMP-156: Modify SetPropertiesFromWorkbook and MetaData so that a single hygiene file can be applied to multiple tables
*P1 AMP-125 Notify user when a project that contains out of date or invalid package is opened - at the moment tables just disappear
Audiences
AMP-181 Explain difference in audience counts between collated and non-collated and verify collated results
AMP-154: Make counts on a saved audience match live audience counts
AMP-143 API: CreateAudiencesFromWorkbook - single call to generate pivot table from a segment list
Add process indicator to audience calculation
Add additional details to ProcessSegments report: Calculation Time, StartExecution DateTime
AMP-XX EyeotaSegmentProfiler - Summary counts for eyeota segments, run as part of load script
AMP-XX AnalyseSegments - report detailing overlap counts between eyeota segments and a base profile
Documentation
- Doc: InjectPackage "sourceName" property does not work. Doesn't work on CreatePackage either
Module Upgrades
Essential C# and .net 8 upgrade
Upgrade Explorer to .NET 8
Upgrade JSON parser
Update to C++ core
- c++ lib updated
- AWSSDK.S3 Version=3.7.308.8
- BouncyCastle.Cryptography Version=2.4.0
- ClosedXML Version=0.102.3
- ExcelDataReader Version=3.6.0
- ExcelDataReader.DataSet Version=3.6.0
- JWT Version=10.1.1
- MathNet.Numerics Version=5.0.0
- MongoDB.Bson Version=2.26.0
- MongoDB.Driver Version=2.26.0
- MongoDB.Driver.Core Version=2.26.0
- MySql.Data Version=8.4.0
- Newtonsoft.Json Version=13.0.3
- Npgsql Version=8.0.3
- SharpCompress Version=0.37.2
- SlackBotMessages Version=2.1.0
- SSH.NET Version=2024.0.0
- System.Data.Odbc Version=8.0.0
- System.Data.SqlClient Version=4.8.6
- System.Text.Encoding.CodePages Version=8.0.0
- xFunc.Maths Version=4.4.1