Release Note: DataJet v5.5.13.1 - 13/05/2023

7 Minutes to read

Print
Dark
Light
PDF

Release Note: DataJet v5.5.13.1 - 13/05/2023

7 Minutes to read

Print
Dark
Light
PDF

Article summary

Did you find this summary helpful?

Thank you for your feedback

Overview

The key features of this release are:

Single Sign On support for OKTA
Increased Debugging and Logging features
Upgraded plugin architecture and default plugins
Enhancement to Remote File management

Administration

Scripting

API

Engineering and Analytics

Modelling

DataBase Build, Export and Management

General Desktop UI

Breaking Changes

OKTA integration

From version 5.5.13.1 onwards, DataJet provides support for Single Sign On. DataJet Administrator should now be used to manage users, groups and realms. Users must be configured in the DataJet Administrator to be able to login to DataJet. For more information on DataJet Administrator, see "Using the DataJet Administrator".

Versions v5.5.13.1 or higher of DataJet's Desktop Client will be needed to login to DataJet. Using an earlier version will result in a failed login.

Backup method requires "targetProject"

The Backup method stores the Mongo data for the specified project in a zip file so that it can be restored or imported back into a realm. Previously the method ran against the active project. This has now been changed to require "targetProject to be specified." Any existing calls to Backup which do not have targetProject in them will fail as a result.

Old Method:

{
    "method": "Backup",
    "path": "%DATAPATH%mybackup.zip",
    "project": "Demonstration"
  }

New Method:

{
  "method": "Backup",
  "path": "%DATAPATH%Backups",
  "description": "Backup Existing Project",
  "targetProject": "MainProject1",
  "system": false,
  "project": "TestProject"
}

Mongo DB

DataJet v5.5.13.1 now requires MongoDB v5.0 or higher.

BUG Fixes

Fix to Admin | System Information : Additional exception handling to prevent freezing when Context menu is open.

Export formats persist for no-state users: Some export formats (e.g., delimiter) were not preserved between export instances.

Script Editor support for underscores in object names: Previously some underscores were stripped from lists of tables and fields.

Feature Details

Switchable Logging for Server API calls

Detailed server logging can be turned on or off using the Desktop | Admin | Command Interface window. See Command Line for more information.

"Enable Script" text command

Ability to force the re-enabling of Script Editor. See Command Line for more information.

Changes to Job Window Cancel

Models can now be cancelled from the job window. To cancel a job, right click the job and select the Cancel Model option.

See Jobs for more information.

Script name added to Script Execution History

History for Plugins, Stored Scripts and scripts executed using ExecuteScript is accessible from Desktop | Script | Execute. The display now includes the name of the executed script:

Script Editor: Preview

Script Editor now allows a preview of resolved variables to be viewed.

Use [CTRL ^ L ] to bring up the Script Variables window:

Select the preview tab to see the preview for the currently selected method. Alternatively, right-click on a method and choose Preview:

Note that %DATAPATH% and %DSTAMP% are not resolved in preview. This is by design.

Script Editor: view resolved script

It is now possible to view a fully resolved script in the Supplementary Script Viewer.

To view a resolved script, select Script Editor | File | View Resolved Script.

When viewing a resolved script in the Supplementary viewer, File | Save As will be enabled, allowing a copy of the resolved script to be made.

See System Variables and Defines for more information.

New system variables: %CURRENT_PROJECT% and %CALLING_PROJECT%

%CURRENT_PROJECT% and %CALLING_PROJECT% can be access from within script logging commands such as LogWrite, LogDataSetCount etc. This can be useful when nesting scripts and/or projects via Stored Scripts and ExecuteStoredScript to test that project variables are being correctly used in child scripts.

See System Variables for more information.

ExecuteStoredScript: optional debug flag

Putting "debug": true into ExecuteStoredScript will output details of variables that have been passed into the child script into the script log (Note - the script log is found in Admin | Remote Files | Team Export | djscriptlogs ) . This can be useful for debugging nested scripts, for example in a scenario where a Master script is calling multiple separate Load Scripts for different data sources.

In the example below, DaughterProject:ChildScript is called, and the variables "%LoadProjectName%" and "%CallingScript% are passed through to the script:

Within the script (DaughterProject:ChildScript ), 2 variables are declared in the DEFINES section of the script:

%LoadProjectName%
%ActiveScript%

In the log, details of what was passed and what was kept can be seen:

The example above shows that there are 3 classes of variable:

Declared Variable - Variables declared in the child script
Passed Overwrite Variable - Variables defined in the child script and in ExecuteStoredScript. The declared variable will be overwritten
Passed Additional Variable - Variable defined in ExecuteStoredScript, but not declared in child script.

Script Logs output to provided filename

Script Logs can be started via the StartErrorLog method. The output will be stored in the djscriptlogs sub-folder of the Team Export Folder

{
  "method": "StartErrorLog",
  "filename": "ScriptLog_%DSTAMP%",
  "mailTo": [
    "me@email.com"
  ],
  "project": "DaughterProject"
}

New API methods

The following new API methods have been added

Category	Method	Description
Project	OpenTargetProject	Opens a project using a name that is not modified by ScriptEditor
Project	CreateTargetProject	Creates a project using a name that is not modified by ScriptEditor
Project	DropTargetProject	Drops a project using a name that is not modified by ScriptEditor
Data Model	MoveFields	Moves fields from a source table into a new table.
Data Model	SetHiddenTables	Controls which tables are visible/hidden. See Set Table Visibility for more information.

Extended Totals in Multi-Function Profiles

Increased ability to calculate totals in Multi-Function Profiles. Use Options | Extended Totals to turn on totals. If a total is not valid for the column then it will be blank.

Quantiles: ability to reverse labels

The numbering of Quantile Bins can now be reversed.

MakeKeyTable filter support

It is now possible to apply a filter to a fields before using it to make a Key Table. The filter will be applied to the field, and then the FirstDiscrete will be run against the filtered field to obtain the list of unique values from which to form the Key Table:

Changes to Plugins

Plugins can auto open a report on completion

If a plugin or extension creates a report as part of its processing, adding the following onView segment at the end of the plugin script will cause the report to be automatically opened

 "onView": [
    {
      "action": "OpenReport",
      "report": "%MODEL% Non Linear Regression"
    }
  ]

Control of Plugin menu location

Plugins can be accessed from the following locations:

Analytics | Extensions
Engineering | Extensions
Modelling

Edit the "classification" key of the Plugin script to control plugin location:

{
  ...
  "classification": "ANALYTICS",
  ...
}

New Plugin: Scan rank

Model Setup List selections

It is now possible to add list selections to Plugin Dialog variable input. To do so, specify a type of "choice" and include the "options" key:

 {
      "variable": "%METHOD%",
      "name": "Method",
      "type": "choice",
      "options": [ "KMEANS", "SPECTRAL", "AGGLOMERATIVE" ],
      "description": "Clustering Method"
    }

Updates to 2d and 3d cluster modelling

"Help" section support for Plugins

Plugin dialogs now support an inline Help comment, as well as a link to an external help page. Use the "help" and "helpURL" keys to configure plugin help.

CopySourceTable accepts field list

CopySourceTable allows a table to be moved or copied between projects on the same realm. If "allFields" is false, then "fields" array can be used to specify the list of fields to move or copy.

{
  "method": "CopySourceTable",
  "sourceProject": "LoadProject1",
  "table": "DATA_CUST",
  "targetName": "DATA_CUST_ARCHIVE",
  "allFields": false,
  "fields": [
    "customer_key",
    "address_line1",
    "address_line2",    
    "dj_dt_timestamp",
    "email",
    "phone"
  ],
  "description": "Removed Fields",
  "moveFiles": true,
  "project": "ArchiveProject1"
}

Project menu "Set Table Visibility"

It is possible to hide tables within a project by using Project | Set Table Visibility. See SetHiddenTables for more information.

Metadata support for CopySourceTable

If Metadata has been applied to a table in its source location, the metadata will be brought with the table as part of CopySourceTable. See Getting Started - Categories and Metadata for more information on Metadata.