--- title: "Release Note: DataJet v5.11.27 - 27/11/2023" slug: "release-note-datajet-v51125-25112023" updated: 2023-11-27T17:17:49Z published: 2023-11-27T17:17:49Z canonical: "docs.datajet.app/release-note-datajet-v51125-25112023" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.datajet.app/llms.txt > Use this file to discover all available pages before exploring further. # Release Note: DataJet v5.11.27 - 27/11/2023 ## Key Features The key features of this release are: - [Performance improvements to JoinLayers and extension of JoinLayers from 15 to 16 max layers](/v1/docs/release-note-datajet-v51109-09112023#accelerator-files) - [Usability enhancements to field templates](/v1/docs/release-note-datajet-v51109-09112023#extensions-to-field-template) - [CopyDownFromTemplate](/v1/docs/release-note-datajet-v51109-09112023#changes-to-api-methods) - [Extensions to statistics](/v1/docs/release-note-datajet-v51109-09112023#extended-core-and-multifield-statistics) - [Improvements to Script Debugging and Execution](/v1/docs/release-note-datajet-v51109-09112023#viewing-results-of-script-execution) - [Extended options for memory configuration](/v1/docs/release-note-datajet-v51109-09112023#configuration-options-for-increasing-available-memory) - [Mobile/small screen support for web reports](/v1/docs/release-note-datajet-v51109-09112023#web-reports-are-mobile-small-screen-compatible) ## Overview #### Administration - [Extended info in Admin | Status | Threads](/v1/docs/release-note-datajet-v51109-09112023#extended-info-in-admin-status-threads) - [Extended Info - Admin | System Information | Threads](/v1/docs/release-note-datajet-v51109-09112023#extended-info-in-admin-status-threads) - [Open project added to Manage Project Dialog](/v1/docs/release-note-datajet-v51109-09112023#open-project-added-to-manage-project-dialog) - [Job Window supports tracking of execution time for jobs](/v1/docs/release-note-datajet-v51109-09112023#job-window-execution-time) - [Configuration options for increasing available memory](/v1/docs/release-note-datajet-v51109-09112023#configuration-options-for-increasing-available-memory) - [Plugins added to Remote File Admin](/v1/docs/release-note-datajet-v51109-09112023#plugins-added-to-admin-remote-file) #### API - [API: ExportIntoTables creates sub-tables from a Dataset Collection](/v1/docs/exportintotables) (NEW) - [API: CopyDownFromTemplate](/v1/docs/copydownfromtemplate) (NEW) - [API: SortTable extended to allow sorting of tables that contain variable continuous string fields](/v1/docs/sorttable) (MODIFIED) - [API: ExportIntoTable default behaviour changed](/v1/docs/release-note-datajet-v51109-09112023#changes-to-api-methods) (MODIFIED) #### Engineering and Analytics - [Additional configuration options in Query Matrix](/v1/docs/release-note-datajet-v51109-09112023#additional-configuration-options-in-query-matrix) - [Extended core and multi-field statistics](/v1/docs/release-note-datajet-v51109-09112023#extended-core-and-multifield-statistics) - [New statistical functions](/v1/docs/release-note-datajet-v51109-09112023#new-statistical-functions) - [Extension to Query Rules: Support added to FirstDiscrete for variable continuous strings](/v1/docs/release-note-datajet-v51109-09112023#extension-to-query-rules-support-added-to-firstdiscrete-for-variable-continuous-strings) - [Improved response when User-Defined Field code contains errors](/v1/docs/release-note-datajet-v51109-09112023#improved-response-when-userdefined-field-code-contains-errors) #### Scripting - [New options for viewing script editor results and script execution](/v1/docs/release-note-datajet-v51109-09112023#viewing-results-of-script-execution) - [View stored script option on JSON right-click for "ExecuteStoredScript"](/v1/docs/release-note-datajet-v51109-09112023#view-stored-script-option-on-json-rightclick-for-executestoredscript) - [Scripts | Local History | Current shows currently executing line in script](/v1/docs/release-note-datajet-v51109-09112023#scripts-local-history-current-shows-currently-executing-line-in-script) **(REF: AMP+24)** - [PreFlightCheck supports allowEmptyFiles **(REF: AMP+ 34)**](/v1/docs/release-note-datajet-v51109-09112023#preflightcheck-supports-allowemptyfiles) - [Scripts | Editor | File | Pull Recent gives list of recently pulled files](/v1/docs/release-note-datajet-v51109-09112023#scripts-editor-file-pull-recent-gives-list-of-recently-pulled-files) #### Reporting - [Web reports are mobile / small screen compatible](/v1/docs/release-note-datajet-v51109-09112023#web-reports-are-mobile-small-screen-compatible) #### Database Build, Export and Management - [JoinLayer accelerator files](/v1/docs/release-note-datajet-v51109-09112023#accelerator-files) - [Max Join Layers extended from 15 to 16 layers](/v1/docs/release-note-datajet-v51109-09112023#max-join-layers-extended-from-15-to-16-layers) #### General Desktop UI - [Data and field type filtering available on field template dialog](/v1/docs/release-note-datajet-v51109-09112023#extensions-to-field-template) - [Field Template auto selects owner table when empty.](/v1/docs/release-note-datajet-v51109-09112023#field-template-auto-selects-owner-table-when-empty) - [Field Template dialog uses multi-column list view](/v1/docs/release-note-datajet-v51109-09112023#extensions-to-field-template) ### Breaking Changes #### JoinLayers JoinLayers built in earlier versions will need to be rebuilt. The JoinLayer function "Score" has been deprecated and should no longer be used. #### Statistics Spreadsheets or Dashboard Excel Templates that have embedded statistics in them may need alteration due to the addition of "variance", "skew", "kurtosis", "mode" and "range" to core statistics. ### BUG Fixes | FIX: Context panel needs to be refreshed when a new project is opened | Sometime the context panel does not respond to changes in the database explorer selection. Workaround was to select templates or reports tab and then go back to database explorer | | --- | --- | | FIX: SP Mass Profiler - allows a collection to be dropped on a crosstab, and menu comes up, but then doesn't do anything | Crosstabs can now be mass profiled. | | FIX: Issues with unreliable tree update/display | The tree would sometimes fail to update after engineering. It should now update automatically | | FIX: Scan Rank Module (REF: AMP+20) | Engineering \| Rank has been extended to implement "alwaysIncrement" - this should now be used in place of scanrank | | FIX: Tables disappear from project after updating Hidden Tables list (REF: AMP+28) | Updated to account for scenario where tables are deleted but hidden tables are in effect | ### ### Changes to API Methods | Category | Method | Description | | --- | --- | --- | | NEW | [CopyDownFromTemplate](/v1/docs/copydownfromtemplate) | Copies down all fields in a field template from a PRIMARY to a FOREIGN table. | | Modified | [SortTable](/v1/docs/sorttable) | Extended to allow sorting of tables that contain variable continuous string fields | | NEW | [ExportIntoTables](/v1/docs/exportintotables) | Allows a single table to be split into multiple tables, based on a dataset collection | | Modified | [ExportIntoTable](/v1/docs/exportintotable) | "optimization":true is now the default | ## Feature Details ### Accelerator Files Accelerator files which support the building of rapid [joins](/v1/docs/deep-dive-datajet-joins) and join layers can be added to suitable fields as part of the [database build process](/v1/docs/getting-started-database-build-and-preparation). Use [ExtendIndexing](/v1/docs/extendindexing) with a type of "PrimaryKey" to add an accelerator file to a field: ```json { "method": "ExtendIndexing", "project": "Demo", "fields": [ "DATA.dj_Key_1", "DATA.dj_Key_2" ], "type": "PrimaryKey", "description": "build accelerator files on join keys" } ``` Build an extended index on all fields on the primary table that will be used as part of either a join or a joinlayer. (NOTE: Extended Indexes may take a relatively long-time to build - they are generally only built on large tables (100million+ records) that will be linked to many times by many different data sources) ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700049540064.png) ### Extensions to Join Layers It is now possible to build the following: - Main Join (e.g., [AddJoin](/v1/docs/addjoin)) - 1-16 Join layers (e.g., [AddJoinLayer](/v1/docs/addjoinlayer)) (Previously a maximum of 15 JoinLayers could be added to a join) ### Extended info in Admin | Status | Threads Executing threads now displays the thread owner, as well as lifetime since the thread started. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699868969845.png) ### Job window Execution Time Execution Time (in seconds) is reported in the [Jobs](/v1/docs/jobs) window for core functions. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700051696635.png) ### Open project added to Manage Project Dialog A project can be opened directly from the Project Management Console: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699869297064.png) ### ### Configuration options for increasing available memory "Requestable Memory Size" can be configured as part of the server configuration. This option is stored in the encrypted djserver.cnfg file. If more memory is required (see [Command Line](/v1/docs/command-line) for options to test memory requirements) contact [DataJet Support](mailto:support@datajetsoftware.com) for a new version of this file. ### Plugins added to Admin | Remote File Plugin extensions can be configured via the Remote Files dialog. See [Introduction to Plugins and Extensions](/v1/docs/introduction-to-plugins-and-extensions) for further details. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700062235481.png) ### Additional configuration options in Query Matrix See [Query Matrix](/v1/docs/query-matrix) for more information - Favor Integer - displays decimal data as integer if there is no loss of information - No Errors - continues executing the matrix if an error is encountered ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699875054027.png) ### Extended core and multi-field statistics Variance is now part of core [statistics](/v1/docs/statistics-getting-started): ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699875380956.png) [Multi-Field Statistics](/v1/docs/multi-field-statistics) has the option to include Extended Stats: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699875397663.png) Multi-Field Statistics has "Table Export" option: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699876327614.png) ### New statistical functions Various statistical functions are available to work with Missing Data: - ZEROTONULL - converts fields with value 0 to null - NULLTOZERO - converts NULL records to 0 - NULLTOAVG - converts NULL records to average for the field - NULLTOMED - converts NULL records to median for the field ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699876714965.png) ### Extension to Query Rules: Support added to FirstDiscrete for variable continuous strings It is possible to apply an [extended index](/v1/docs/extendindexing) to a continuous variable string so that it can be used in the FirstDiscrete [intrinsic function](/v1/docs/intrinsic-functions-overview): ```json { "method": "ExtendIndexing", "field": "DATA.StringField", "type": "PrimaryKey", "project": "VSJoin" } ``` NOTE: If an attempt is made to do FirstDiscrete on a continuous variable string that has not had its index extended, the following error will be generated: ```none "unhandled intrinsic function" ``` ### API: SortTable extended to allow sorting of tables that contain variable continuous string fields It is possible to sort tables that contain variable length strings: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699882246734.png) NOTE: SortTable is designed for small tables where the sortkey and the biggest field can sit in memory (RAM). This is around 500m records on a 64gb machine. See [SortTable](/v1/docs/sorttable) for more details. ### Improved response when User-Defined Field code contains errors Improvements in error Handling: - Returns faster and with greater detail if invalid data supplied - Returns faster and with more information if there is an error in the UDF code See [User Defined Fields](/v1/docs/programmed-field) for more information ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700070213227.png) ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700073301641.png) ### Viewing Results of Script Execution Script results can be viewed from: - Script Editor | Right-Click | View Result (***NEW**) - Script Editor | File | Execution History - Scripts | History - Scripts | Local History (**Updated to include line numbers**) For more information, see [Introduction to Scripting](/v1/docs/introduction-to-scripting-1) ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700228599065.png) ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700228622468.png) ### View stored script option on JSON right-click for "ExecuteStoredScript" Right-clicking in the JSON window of the ExecuteStoredScript command opens a read-only copy of the stored script in a viewer:![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699885254558.png) #### #### ### Scripts | Local History | Current shows currently executing line in script From **Scripts | Local History | Current** - **Script is still running:** current refers to currently executing line in the script - **Script is complete with errors:** current refers to line with error - **Script is complete with no errors:** current refers to total executed lines in the script ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700227305582.png) ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700227326214.png) ### PreFlightCheck supports allowEmptyFiles If the flag "allowEmptyFiles" is provided in [PreFlightCheck](/v1/docs/preflightcheck), then the method will not error when a 0 length file is encountered. ### Scripts | Editor | File | Pull Recent gives list of recently pulled files Script Editor will pull the MOST RECENT VERSION of recently pulled scripts via the **File | Pull Recen**t list. ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699894085916.png) ### Web reports are mobile / small screen compatible ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699894603201.png) ### Extensions to Field Template Fields in the Available Fields list can be filtered by: - [Decode](/v1/docs/decode-field-custom) / [Banded](/v1/docs/banded-fields-custom) field - [Discrete](/v1/docs/getting-started-data-types) / [Continuous](/v1/docs/getting-started-data-types) Field - [Engineered](/v1/docs/baked-fields) / Imported Field It is also possible to sort and filter by FieldType and discrete count ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1699895075411.png) #### Field Template auto selects owner table when empty. When opening a field template for a dataset with an empty template, the owner table of the dataset will be automatically set in the field selector: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1700050336319.png) ###