Release Note: DataJet v5.11.27 - 27/11/2023

6 Minutes to read

Print
Dark
Light
PDF

Release Note: DataJet v5.11.27 - 27/11/2023

6 Minutes to read

Print
Dark
Light
PDF

Article summary

Did you find this summary helpful?

Thank you for your feedback

Key Features

The key features of this release are:

Overview

Administration

API

Engineering and Analytics

Scripting

Reporting

Web reports are mobile / small screen compatible

Database Build, Export and Management

General Desktop UI

Breaking Changes

JoinLayers

JoinLayers built in earlier versions will need to be rebuilt.

The JoinLayer function "Score" has been deprecated and should no longer be used.

Statistics

Spreadsheets or Dashboard Excel Templates that have embedded statistics in them may need alteration due to the addition of "variance", "skew", "kurtosis", "mode" and "range" to core statistics.

BUG Fixes

FIX: Context panel needs to be refreshed when a new project is opened	Sometime the context panel does not respond to changes in the database explorer selection. Workaround was to select templates or reports tab and then go back to database explorer
FIX: SP Mass Profiler - allows a collection to be dropped on a crosstab, and menu comes up, but then doesn't do anything	Crosstabs can now be mass profiled.
FIX: Issues with unreliable tree update/display	The tree would sometimes fail to update after engineering. It should now update automatically
FIX: Scan Rank Module (REF: AMP+20)	Engineering \| Rank has been extended to implement "alwaysIncrement" - this should now be used in place of scanrank
FIX: Tables disappear from project after updating Hidden Tables list (REF: AMP+28)	Updated to account for scenario where tables are deleted but hidden tables are in effect

Changes to API Methods

Category	Method	Description
NEW	CopyDownFromTemplate	Copies down all fields in a field template from a PRIMARY to a FOREIGN table.
Modified	SortTable	Extended to allow sorting of tables that contain variable continuous string fields
NEW	ExportIntoTables	Allows a single table to be split into multiple tables, based on a dataset collection
Modified	ExportIntoTable	"optimization":true is now the default

Feature Details

Accelerator Files

Accelerator files which support the building of rapid joins and join layers can be added to suitable fields as part of the database build process.

Use ExtendIndexing with a type of "PrimaryKey" to add an accelerator file to a field:

{
  "method": "ExtendIndexing",
  "project": "Demo",
  "fields": [
    "DATA.dj_Key_1",
    "DATA.dj_Key_2"
  ],
  "type": "PrimaryKey",
  "description": "build accelerator files on join keys"
}

Build an extended index on all fields on the primary table that will be used as part of either a join or a joinlayer.

(NOTE: Extended Indexes may take a relatively long-time to build - they are generally only built on large tables (100million+ records) that will be linked to many times by many different data sources)

Extensions to Join Layers

It is now possible to build the following:

Main Join (e.g., AddJoin)
1-16 Join layers (e.g., AddJoinLayer)

(Previously a maximum of 15 JoinLayers could be added to a join)

Extended info in Admin | Status | Threads

Executing threads now displays the thread owner, as well as lifetime since the thread started.

Job window Execution Time

Execution Time (in seconds) is reported in the Jobs window for core functions.

Open project added to Manage Project Dialog

A project can be opened directly from the Project Management Console:

Configuration options for increasing available memory

"Requestable Memory Size" can be configured as part of the server configuration. This option is stored in the encrypted djserver.cnfg file. If more memory is required (see Command Line for options to test memory requirements) contact DataJet Support for a new version of this file.

Plugins added to Admin | Remote File

Plugin extensions can be configured via the Remote Files dialog.

See Introduction to Plugins and Extensions for further details.

Additional configuration options in Query Matrix

See Query Matrix for more information

Favor Integer - displays decimal data as integer if there is no loss of information
No Errors - continues executing the matrix if an error is encountered

Extended core and multi-field statistics

Variance is now part of core statistics:

Multi-Field Statistics has the option to include Extended Stats:

Multi-Field Statistics has "Table Export" option:

New statistical functions

Various statistical functions are available to work with Missing Data:

ZEROTONULL - converts fields with value 0 to null
NULLTOZERO - converts NULL records to 0
NULLTOAVG - converts NULL records to average for the field
NULLTOMED - converts NULL records to median for the field

Extension to Query Rules: Support added to FirstDiscrete for variable continuous strings

It is possible to apply an extended index to a continuous variable string so that it can be used in the FirstDiscrete intrinsic function:

{
    "method": "ExtendIndexing",
    "field": "DATA.StringField",
    "type": "PrimaryKey",
    "project": "VSJoin"
  }

NOTE: If an attempt is made to do FirstDiscrete on a continuous variable string that has not had its index extended, the following error will be generated:

"unhandled intrinsic function"

API: SortTable extended to allow sorting of tables that contain variable continuous string fields

It is possible to sort tables that contain variable length strings:

NOTE: SortTable is designed for small tables where the sortkey and the biggest field can sit in memory (RAM). This is around 500m records on a 64gb machine.

See SortTable for more details.

Improved response when User-Defined Field code contains errors

Improvements in error Handling:

Returns faster and with greater detail if invalid data supplied
Returns faster and with more information if there is an error in the UDF code

See User Defined Fields for more information

Viewing Results of Script Execution

Script results can be viewed from:

Script Editor | Right-Click | View Result (*NEW)
Script Editor | File | Execution History
Scripts | History
Scripts | Local History (Updated to include line numbers)

For more information, see Introduction to Scripting

View stored script option on JSON right-click for "ExecuteStoredScript"

Right-clicking in the JSON window of the ExecuteStoredScript command opens a read-only copy of the stored script in a viewer:

Scripts | Local History | Current shows currently executing line in script

From Scripts | Local History | Current

Script is still running: current refers to currently executing line in the script
Script is complete with errors: current refers to line with error
Script is complete with no errors: current refers to total executed lines in the script