--- title: "Release Note: v6.02.09" slug: "release-note-v60220" status: "update" updated: 2024-02-21T16:58:31Z published: 2024-02-21T16:58:31Z canonical: "docs.datajet.app/release-note-v60220" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.datajet.app/llms.txt > Use this file to discover all available pages before exploring further. # Release Note: v6.02.20 ## Key Features The key features of this release are: - [Improvements to package creation and deployment](/v1/docs/release-note-v60209#improvements-to-package-creation-and-deployment): - [Second Tier Injection (injection of injected data)](/v1/docs/release-note-v60209#second-tier-injection-injection-of-injected-data) - [New Configuration Setting to support cross-realm data sharing](/v1/docs/release-note-v60209#network-path-server-configuration-to-support-crossrealm-data-sharing) - [InjectPackage behaviour changed to read "homepath" from Package metadata File by default if it is present](/v1/docs/release-note-v60209#injectpackage-behaviour-changed-to-read-homepath-from-package-metadata-file-by-default-if-it-is-present) - [CreatePackage stores package metadata file only on Script Hub if Script Hub has been configured.](/v1/docs/release-note-v60209#createpackage-stores-package-metadata-file-just-on-script-hub-if-script-hub-has-been-configured) - [InjectPackage allows source tables to be given new names in target project](/v1/docs/release-note-v60209#injectpackage-allows-source-tables-to-be-given-new-names-in-target-project-refamp4547) - [New User Roles and Permissions](/v1/docs/release-note-v60209#amp62-implement-clientanalyst-role-and-permissions-viewer-and-clientanalyst) - [CLIENT_ANALYST and VIEWER Role added](/v1/docs/release-note-v60209#amp62-implement-clientanalyst-role-and-permissions-viewer-and-clientanalyst) - Enhanced TechOps Usability, including: - [Visibility of join accelerators](/v1/docs/release-note-v60209#data-audit-shows-if-accelerator-files-have-been-added-to-a-join) - [Support for additional DateTime formats](/v1/docs/release-note-v60209#amp22-new-datetime-formats-mdy-hhmmss-and-dmy-hhmmss) - [Phase 1 HASH functions](/v1/docs/release-note-v60209#amp60-add-hash-function-to-strip-out-pii-from-email-fields-etc-and-still-allow-field-to-be-used-in-a-link) - [General Bug Fixes](/v1/docs/release-note-v60209#bug-fixes) ## Overview #### Administration - [AMP-62: Implement CLIENT_ANALYST Role and Permissions - Viewer and Client_Analyst](/v1/docs/release-note-v60209#amp62-implement-clientanalyst-role-and-permissions-viewer-and-clientanalyst) - [FIX: AMP-41: issues with MongoDB driver](/v1/docs/release-note-v60209#amp41-issues-with-mongodb-driver) - [AMP-50: New Config setting required for network path](/v1/docs/release-note-v60209#network-path-server-configuration-to-support-crossrealm-data-sharing) - [FIX: AMP-46: Changing user roles requires server to be restarted before changes take effect](/v1/docs/release-note-v60209#bug-fixes) - [AMP-52: Package created on realm and in storage hub - needs to be only on storage hub](/v1/docs/release-note-v60209#createpackage-stores-package-metadata-file-just-on-script-hub-if-script-hub-has-been-configured) - [AMP-72: Command Line GetOrphs API call added to help identify orphan data files](/v1/docs/release-note-v60209#bug-fixes) #### API Modified - [Modified: InjectPackage](/v1/docs/release-note-v60209#api-methods) - [Modified: CopySourceTable](/v1/docs/release-note-v60209#api-methods) #### Engineering and Analytics - [AMP-37,38: CopySourceTable : Defaults: allfields=true.](/v1/docs/release-note-v60209#amp3738-copysourcetable-now-defaults-to-allfieldstrue) - [AMP-22: DateTime Format: Add support for M/D/Y HH:MM:SS](/v1/docs/release-note-v60209#amp22-new-datetime-formats-mdy-hhmmss-and-dmy-hhmmss) - [AMP-22: Add support for D/M/Y HH:MM:SS DateTime format](/v1/docs/release-note-v60209#amp22-new-datetime-formats-mdy-hhmmss-and-dmy-hhmmss) - [AMP-XX: Add support for string fields to fmax() and fmin()](/v1/docs/release-note-v60209#support-for-continuous-and-discrete-string-fields-added-to-fmax-and-fmin) - [Make it possible to work out if a join has an accelerator file attached to it.](/v1/docs/release-note-v60209#data-audit-shows-if-accelerator-files-have-been-added-to-a-join) #### Scripting - [AMP-43: Change "Save/Run" and "Push/Run" option in ScriptEditor | Run to be "Cancel/Save/Run"](/v1/docs/release-note-v60209#amp43-saverun-option-in-scripteditor-run-changed-to-be-cancelsaverun) - [AMP-24: Make script history dialogs more consistent - add line numbers and ability to view JSON](/v1/docs/release-note-v60209#amp24-line-numbers-added-to-script-history-dialogs) #### Reporting - [Add Export To Table to Query matrix](/v1/docs/release-note-v60209#allow-query-matrix-to-export-to-table) #### Database Build, Export and Management - [FIX: AMP-57: Category option does not appear in a project after applying metadata \(via InjectPackage\) until project is manually closed and opened.](/v1/docs/release-note-v60209#bug-fixes) - [AMP-49: InjectPackage - omit explicit "homepath" property, and read package data location from package metadata file](/v1/docs/release-note-v60209#injectpackage-behaviour-changed-to-read-homepath-from-package-metadata-file-by-default-if-it-is-present-ref-amp49) - [AMP-49,51: Second tier injection - ability to inject a package that contains injected tables and cross-package joins](/v1/docs/release-note-v60209#second-tier-injection-injection-of-injected-data-ref-amp4951) - [FIX: AMP-39: Log files only being written to default location. Needs to honour path provided in StartErrorLog](/v1/docs/release-note-v60209#amp39-log-files-only-being-written-to-default-location-needs-to-honour-path-provided-in-starterrorlog) - [AMP-60: Add HASH function to strip out PII from email fields etc and still allow field to be used in a link.](/v1/docs/release-note-v60209#amp60-add-hash-function-to-strip-out-pii-from-email-fields-etc-and-still-allow-field-to-be-used-in-a-link) - [AMP-45,47: InjectPackage, ability to rename tables as in InjectTables](/v1/docs/release-note-v60209#injectpackage-allows-source-tables-to-be-given-new-names-in-target-project-refamp4547) #### General Desktop UI - [FIX: AMP-44: Dataview grid displaying rounded value for some doubles](/v1/docs/release-note-v60209#amp44-dataview-grid-displaying-rounded-value-for-some-doubles) - [Overwrite implemented for DataSet Rename](/v1/docs/release-note-v60209#overwrite-implemented-for-dataset-rename) #### BUG Fixes #### ExecuteCustomReport returns errors when filters applied Resolves issues with filters being applied to [standardized](/v1/docs/static-standardized-report-quick-reference) web reports. Previously when an error occurred there was no indication of this returned by the report. #### AMP-46: Changing user roles requires server to be restarted before changes take effect Fixed - user role changes are now propagated without restarting the server. #### AMP-52: Package created on realm and in storage hub - needs to be only on storage hub If the [storage hub](/v1/docs/storage-hub) has been [configured](/v1/docs/client-configuration-file), packages will now only be created on the hub. #### AMP-57: Category option does not appear in a project after applying metadata (via InjectPackage) until project is manually closed and opened. The [category](/v1/docs/getting-started-categories-and-metadata) option will automatically appear in the [project explorer](/v1/docs/database-categories) if **includeProjectMetadata=true** has been specified as part of [InjectPackage](/v1/docs/injectpackage) and the source project contains metadata for the [injected tables](/v1/docs/deep-dive-sharing-data-by-injection). #### AMP-44: Dataview grid displaying rounded value for some doubles Display issues when display doubles caused some numbers to be rounded up in the [dataview](/v1/docs/dataset-scratch) grid: 2023121110344025**5** -> 2023121110344025**6** 2023121110344821**5** -> 2023121110344821**6** 2023121112000210**9** -> 2023121112000210**8** All numbers are now accurately displayed. #### AMP-41: issues with MongoDB driver MongoDB driver has been updated to version 7.0 #### AMP-39: Log files only being written to default location. Needs to honour path provided in StartErrorLog ```json { "method": "StartErrorLog", "filename": "ScriptLog_%DSTAMP%", "serverPath": "", "mailTo": [ "me@email.com" ], "project": "Analyst Project" } ``` #### AMP-72: Command Line GetOrphs API call added to help identify orphan data files Open Admin | Runner and execute the following method: ```json { "method": "GetOrphs" } ``` A list of orphan files (if there are any) will be returned: ```json { "list": [], "executed": true, "errors": [], "millis": 6, "projectEpoch": 638436807569543878, "method": "GetOrphs", "exmillis": 864 } ``` ### API Methods NEW | Method | Description | | --- | --- | | GetOrphs | Returns a list of orphan data files. Use only in Admin Runner | Modified | Method | Description | | --- | --- | | InjectPackage | **renames[]** array added to allow tables to be renamed when injected. If **homepath** is empty, value is read from package metadata file **homepath** is set as empty if user chooses "Use Package homepath" | | CopySourceTable | **allfields:true** is now the default. NOTE: accelerator files and metadata for the source table will be copied across by default. | | StartErrorLog | **serverPath** added | ## Feature Details ### Improvements to Package creation and deployment The following changes have been implemented: - Second Tier Injection (injection of injected data) - New Configuration Setting to support cross-realm data sharing - InjectPackage behaviour changed to read "homepath" from Package metadata File by default if it is present - CreatePackage stores package metadata file just on Script Hub if Script Hub has been configured. - InjectPackage allows source tables to be given new names in target project #### **Second Tier Injection (injection of injected data) (REF:** AMP-49,51) Second Tier injection allows a project that is itself built from injected data to be injected: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707475700580.png) The primary advantage of second tier injection is that it allows joins and accelerators between first tier projects to be built ahead of time. In the example above, links between **DataSource1** and **CustomerSource1** are built as part of the **Package1** build process. When **Package1** is injected into **AnalystProject**, the links between **DataSource1** and **CustomerSource1** are already present. #### **Network Path Server Configuration to support cross-realm data sharing** A Data Package (e.g., **Package1**) has a package metadata file (e.g., Package1Metadata.json) that provides the following: - Object Details (schema, counts, object metadata etc) - Path to Data Files (record level data, indexes, accelerators etc) When a package is created, the *PackageDataPath* value is stored in the package metadata file. This will be the DataJet server's **home directory** plus the **path to the source project folder**. So for example, if DataJet server's home folder is **/home/engine/** and the repository folder is "**repos**", if the *data* for the package is in "**Project1**" then the full *PackageDataPath* would be: **"/home/engine/[REALM-NAME]/repos/Project1"** (Note: paths are case-sensitive) When distributing Data Packages across multiple realms, it may be necessary for the *PackageDataPath* to provide an **absolute** path to a shared data location. For example: **"/mnt/shared/engine/**[REALM-NAME]**/repos/Project1"** This can be achieved by adding the "pathSubs" array pair to the [Client Configuration File](/v1/docs/client-configuration-file): ```json "pathSubs": [ { "from": "/home/engine/" "to":"/mnt/shared/" } ``` If pathSubs is present in the configuration file then when reading PackageDataPath from the PackageXMetadata.json file, all instances of "from" will be replaced with the value of "to". For example: "**/home/engine/**dj-localhost/repos/Project1" -> "**/mnt/shared/**dj-localhost/repos/Project1" #### **InjectPackage behaviour changed to read "homepath" from Package metadata File by default if it is present (REF:** AMP-49) The [InjectPackage](/v1/docs/injectpackage) method has a "**homePath**" property which provides the location for the data files used by the package referenced by the "**name**" property. The homePath property can now be blank, in which case the value stored in [PackageMetadataFile].json will be used. The purpose of this is to allow scripts to be developed that **reference packages by name**. For example, InjectPackage could be configured in script to inject the package "pkgCurrentVendor1". Operational processes can then manage the updating of the "pgkCurrentVendor1.json" metadata file, changing the homepath property in the metadata file to point to the latest version of the data without requiring alteration of any scripts. To set the InjectPackage method to use the homePath from the metadata file, set "homepath" to empty: ```json { "method": "InjectPackage", "project": "", "homePath": "", "includeProjectMetaData": true, "errorOnMissingDataFiles": false, "name": "Current Package1", "pull": true, "tables": [ "DataSource1", "CS1" ] } ``` ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707506020264.png) #### CreatePackage stores package metadata file just on Script Hub if Script Hub has been configured. Previous versions of CreatePackage stored 2 copies of the metadata file if the [Script Hub](/v1/docs/script-hub-overview) was configured: 1. In DataSource folder on the source realm 2. In the Script Hub From version 6.02.09 onwards, if the Script Hub has been [configured](/v1/docs/client-configuration-file), metadata files will **ONLY** be created on the Script Hub, and **not** in the realm DataSource folder. #### InjectPackage allows source tables to be given new names in target project (REF:AMP-45,47) "renames[]" array added to [InjectPackage](/v1/docs/injectpackage) ```json { "method": "InjectPackage", "project": "", "homePath": "", "includeProjectMetaData": true, "errorOnMissingDataFiles": false, "name": "Demo2023-11-09-16-18-17", "pull": true, "tables": [ "sales", "customer" ], "renames": [ { "from": "sales", "to": "isales" } { "from": "customer", "to": "icustomer" } ] } ``` ### AMP-62: Implement CLIENT_ANALYST Role and Permissions - Viewer and Client_Analyst 2 new user roles have been added: - Viewer - Client_Analyst See [User Permissions](/v1/docs/user-permissions) for more details. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707747184500.png)Viewer Role: Has access to reporting and shared reports within configured projects ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707747413588.png)Client Analyst Role: Has access to analytics, engineering, modelling and scripting within public projects ### AMP-37,38: CopySourceTable now defaults to "allfields"=true [CopySourceTable](/v1/docs/copysourcetable) has been modified so that allFields defaults to true. ```json { "method": "CopySourceTable", "project": "", "sourceProject": "", "table": "", "targetName": "", "fields": [], "allFields": true, "moveFiles": false } ``` ### AMP-22: New DateTime Formats: (M/D/Y HH:MM:SS and D/M/Y HH:MM:SS ) Two new [DateTime](/v1/docs/getting-started-data-types) formats have been added: - M/D/Y HH:MM:SS - D/M/Y HH:MM:SS These formats support missing leading zeros for M and D. - 5/2/2023 00:00:00 will be loaded as 05/02/2023 00:00:00 ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707912731869.png) ### Support for continuous and discrete string fields added to fmax() and fmin() fmin() and fmax() functions (available in [Expression BuilderEx](/v1/docs/expression-field)) have been extended to work with string fields. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707916773500.png) ### Data Audit shows if accelerator files have been added to a join [Join](/v1/docs/joins) accelerators can be built using the [ExtendIndexing](/v1/docs/extendindexing) method with type "PrimaryKey". These accelerators support fast joins and [joinlayers](/v1/docs/count), as well as fast [discrete](/v1/docs/qq-how-do-i-find-out-the-cardinality-of-a-continuous-field) on variable length continuous strings. To see if a join has an accelerator file attached to it, use the Join Tab on the [Data Audit](/v1/docs/data-audit) Report. This has been extended to show accelerator file status: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707847861453.png) ### AMP-43: "Save/Run" option in ScriptEditor | Run changed to be "Cancel/Save/Run" The Script Editor "Run" option has been modified to give the option to cancel. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707848122079.png) ### AMP-24: Line numbers added to Script history dialogs - SCRIPT HISTORY: Script | History | Execution History | Right-Click | View - LOCAL HISTORY: Script | Local History | Right-Click | View See [History](/v1/docs/engineering-history) for further details. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707916887248.png)Script | History | Execution History | Right-Click | View ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1707916976833.png)Script | Local History | Right-Click | View ### Allow Query Matrix to Export To Table The [Query Matrix](/v1/docs/query-matrix) has the ability to Export To Table: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1708534623258.png) ### AMP-60: Add HASH function to strip out PII from email fields etc and still allow field to be used in a link. The following string functions have been added: - SHA1 - SHA256 - MD5 ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1708008661718.png) NOTE: Only string fields can be used with these functions ### Overwrite implemented for DataSet Rename Calling SaveDataSet from the User interface will now over-write existing datasets of the same name. #### ###