Release Note: v6.02.20

7 Minutes to read

Print
Dark
Light
PDF

Release Note: v6.02.20

7 Minutes to read

Print
Dark
Light
PDF

Article summary

Did you find this summary helpful?

Thank you for your feedback

Key Features

The key features of this release are:

Overview

Administration

API

Modified

Engineering and Analytics

Scripting

AMP-43: Change "Save/Run" and "Push/Run" option in ScriptEditor | Run to be "Cancel/Save/Run"
AMP-24: Make script history dialogs more consistent - add line numbers and ability to view JSON

Reporting

Add Export To Table to Query matrix

Database Build, Export and Management

General Desktop UI

BUG Fixes

ExecuteCustomReport returns errors when filters applied

Resolves issues with filters being applied to standardized web reports. Previously when an error occurred there was no indication of this returned by the report.

AMP-46: Changing user roles requires server to be restarted before changes take effect

Fixed - user role changes are now propagated without restarting the server.

AMP-52: Package created on realm and in storage hub - needs to be only on storage hub

If the storage hub has been configured, packages will now only be created on the hub.

AMP-57: Category option does not appear in a project after applying metadata (via InjectPackage) until project is manually closed and opened.

The category option will automatically appear in the project explorer if includeProjectMetadata=true has been specified as part of InjectPackage and the source project contains metadata for the injected tables.

AMP-44: Dataview grid displaying rounded value for some doubles

Display issues when display doubles caused some numbers to be rounded up in the dataview grid:

20231211103440255 -> 20231211103440256

20231211103448215 -> 20231211103448216

20231211120002109 -> 20231211120002108

All numbers are now accurately displayed.

AMP-41: issues with MongoDB driver

MongoDB driver has been updated to version 7.0

AMP-39: Log files only being written to default location. Needs to honour path provided in StartErrorLog

{
  "method": "StartErrorLog",
  "filename": "ScriptLog_%DSTAMP%",
  "serverPath": "",
  "mailTo": [
    "me@email.com"
  ],
  "project": "Analyst Project"
}

AMP-72: Command Line GetOrphs API call added to help identify orphan data files

Open Admin | Runner and execute the following method:

{
  "method": "GetOrphs"
}

A list of orphan files (if there are any) will be returned:

{
  "list": [],
  "executed": true,
  "errors": [],
  "millis": 6,
  "projectEpoch": 638436807569543878,
  "method": "GetOrphs",
  "exmillis": 864
}

API Methods

NEW

Method	Description
GetOrphs	Returns a list of orphan data files. Use only in Admin Runner

Modified

Method	Description
InjectPackage	renames[] array added to allow tables to be renamed when injected. If homepath is empty, value is read from package metadata file homepath is set as empty if user chooses "Use Package homepath"
CopySourceTable	allfields:true is now the default. NOTE: accelerator files and metadata for the source table will be copied across by default.
StartErrorLog	serverPath added

Feature Details

Improvements to Package creation and deployment

The following changes have been implemented:

Second Tier Injection (injection of injected data)
New Configuration Setting to support cross-realm data sharing
InjectPackage behaviour changed to read "homepath" from Package metadata File by default if it is present
CreatePackage stores package metadata file just on Script Hub if Script Hub has been configured.
InjectPackage allows source tables to be given new names in target project

Second Tier Injection (injection of injected data) (REF: AMP-49,51)

Second Tier injection allows a project that is itself built from injected data to be injected:

The primary advantage of second tier injection is that it allows joins and accelerators between first tier projects to be built ahead of time. In the example above, links between DataSource1 and CustomerSource1 are built as part of the Package1 build process. When Package1 is injected into AnalystProject, the links between DataSource1 and CustomerSource1 are already present.

Network Path Server Configuration to support cross-realm data sharing

A Data Package (e.g., Package1) has a package metadata file (e.g., Package1Metadata.json) that provides the following:

Object Details (schema, counts, object metadata etc)
Path to Data Files (record level data, indexes, accelerators etc)

When a package is created, the PackageDataPath value is stored in the package metadata file. This will be the DataJet server's home directory plus the path to the source project folder.

So for example, if DataJet server's home folder is /home/engine/ and the repository folder is "repos", if the data for the package is in "Project1" then the full PackageDataPath would be:

"/home/engine/[REALM-NAME]/repos/Project1"

(Note: paths are case-sensitive)

When distributing Data Packages across multiple realms, it may be necessary for the PackageDataPath to provide an absolute path to a shared data location. For example:

"/mnt/shared/engine/[REALM-NAME]/repos/Project1"

This can be achieved by adding the "pathSubs" array pair to the Client Configuration File:

"pathSubs": [
{
"from": "/home/engine/"
"to":"/mnt/shared/"
}

If pathSubs is present in the configuration file then when reading PackageDataPath from the PackageXMetadata.json file, all instances of "from" will be replaced with the value of "to".

For example:

"/home/engine/dj-localhost/repos/Project1" -> "/mnt/shared/dj-localhost/repos/Project1"

InjectPackage behaviour changed to read "homepath" from Package metadata File by default if it is present (REF: AMP-49)

The InjectPackage method has a "homePath" property which provides the location for the data files used by the package referenced by the "name" property.

The homePath property can now be blank, in which case the value stored in [PackageMetadataFile].json will be used.

The purpose of this is to allow scripts to be developed that reference packages by name. For example, InjectPackage could be configured in script to inject the package "pkgCurrentVendor1". Operational processes can then manage the updating of the "pgkCurrentVendor1.json" metadata file, changing the homepath property in the metadata file to point to the latest version of the data without requiring alteration of any scripts.

To set the InjectPackage method to use the homePath from the metadata file, set "homepath" to empty:

{
  "method": "InjectPackage",
  "project": "",
  "homePath": "",
  "includeProjectMetaData": true,
  "errorOnMissingDataFiles": false,
  "name": "Current Package1",
  "pull": true,
  "tables": [
    "DataSource1",
    "CS1"
  ]
}

CreatePackage stores package metadata file just on Script Hub if Script Hub has been configured.

Previous versions of CreatePackage stored 2 copies of the metadata file if the Script Hub was configured:

In DataSource folder on the source realm
In the Script Hub

From version 6.02.09 onwards, if the Script Hub has been configured, metadata files will ONLY be created on the Script Hub, and not in the realm DataSource folder.

InjectPackage allows source tables to be given new names in target project (REF:AMP-45,47)

"renames[]" array added to InjectPackage

{
  "method": "InjectPackage",
  "project": "",
  "homePath": "",
  "includeProjectMetaData": true,
  "errorOnMissingDataFiles": false,
  "name": "Demo2023-11-09-16-18-17",
  "pull": true,
  "tables": [
    "sales",
    "customer"
  ],
  "renames": [
    {
      "from": "sales",
      "to": "isales"
    }
    {
      "from": "customer",
      "to": "icustomer"
    }
  ]
}

AMP-62: Implement CLIENT_ANALYST Role and Permissions - Viewer and Client_Analyst

2 new user roles have been added:

Viewer
Client_Analyst

See User Permissions for more details.

Viewer Role: Has access to reporting and shared reports within configured projects

Client Analyst Role: Has access to analytics, engineering, modelling and scripting within public projects

AMP-37,38: CopySourceTable now defaults to "allfields"=true

CopySourceTable has been modified so that allFields defaults to true.

{
  "method": "CopySourceTable",
  "project": "",
  "sourceProject": "",
  "table": "",
  "targetName": "",
  "fields": [],
  "allFields": true,
  "moveFiles": false
}

AMP-22: New DateTime Formats: (M/D/Y HH:MM:SS and D/M/Y HH:MM:SS )

Two new DateTime formats have been added:

M/D/Y HH:MM:SS
D/M/Y HH:MM:SS

These formats support missing leading zeros for M and D.

5/2/2023 00:00:00 will be loaded as 05/02/2023 00:00:00

Support for continuous and discrete string fields added to fmax() and fmin()

fmin() and fmax() functions (available in Expression BuilderEx) have been extended to work with string fields.

Data Audit shows if accelerator files have been added to a join

Join accelerators can be built using the ExtendIndexing method with type "PrimaryKey". These accelerators support fast joins and joinlayers, as well as fast discrete on variable length continuous strings.

To see if a join has an accelerator file attached to it, use the Join Tab on the Data Audit Report. This has been extended to show accelerator file status: