Key Features
The focus of this release is on improving project wide search capabilities. Various Tech-ops bugs and minor feature requests are also addressed.
- Advanced Search - Now Includes: the ability to search across the entire project for Field Names, Discrete Values, Descriptions, Meta data (Note: to choose which metadata columns to include in the output grid, use "select metadata")
- Improvements to Signals - reductions in disk space, processing time and query times as well as additional metrics
- PreProcessSegments - pre-processes *.tsv files, after which tsv files can be removed. Output files are approximately 1/3 the size of input tsv files, and are processed by ProcessSegments in approx 1.2 the time.
- ProcessSegments - Performance improvements:
- Processing time is reduced if PreProcessSegments is run
- Segment query times are improved if segment accelerator files are created - once the first signal segment is calculated, query times take approx 1/2 the time to calculate.
- Additional metrics - basic metrics for records in the primary contact table are available:
- entries: total number of times hash code appeared in input files
- total_segments: total number of segments in which hash code appears
- unique_segments: unique number of segments in which hash code appears
- Primary Contact / Signal synchronisation check
- If "segmentMetafile" property is added to CreateTableFromFile when creating the Primary Contact Table, the unique GUID associated with the loaded maindatafile.txt will be stored with the Primary Contact Table. Signal (segment) queries will generate an error if the project signal data does not match the Primary Contact data.
- Analyse Segments Report - provides user interface for setting up AnalyseSegments calculations
- Contingency Management Layer - Phase 1 thread management to add queuing to engineering functions on large (100m+) tables.
- New Data Engineering Functions:
- DMATH()
- INLIER(A,B)
- OUTLIER(A,B)
- DEVIATIONS(A)
- DMATH()
- Various AMP+ Issues
API Calls
| Type | API | |
|---|---|---|
| NEW | DetachPackage | removes all objects injected from package with optional id 'injectionId' or else with name 'name' |
| MODIFIED | CreatePackage | if tables defined and table does not exist then error (AMP-287) |
| MODIFIED | InjectPackage | packages will have an injectionId assigned (but optionally passed as 'instanceId':string). This id is stored in the definition of all injected objects. |
| MODIFIED | AnalyseSegments | "target_nontarget" mode outputs universe as count of individual records when outputting to Excel |
| NEW | AddNullField | Allows a null field of specified data type to be created |
| NEW | AddNullFieldLike | Allows a null field to be created where type matches another field |
| MODIFIED | CopyDownFromTemplate | now collects subordinate errors. |
| MODIFIED | ExportIntoTable | existing optional 'autoUseExtended' flag defaults to true from false. |
| NEW | CopyDatasetCollection | copies a dataset collection to a linked table |
| NEW | PreProcessSegments | Pre-processes *.tsv files to produce *.tsv.hash and *.tsv.hash.seg files. Once processed, *.tsv files can be removed, reducing required disk space to ~1/3 original disk requirements |
| MODIFIED | ProcessSegments | Can now process *.tsv.hash and *.tsv.hash.seg files. Processing time is reduced by half. Query times are improved Additional metrics are generated - number of times a hash (unique SHA256) appears in the input data, total number of segments in which hash appears, unique number of segments in which hash appears. |
| MODIFIED | CreateTableFromFile | Provide segmentMetafile property when loading maindatafile.txt output from ProcessSegments to create Primary Contact Table for signals. Queries will then cross-check that Signals and Primary Contact Table match, and will generate an error if they are out of synch. |
AMP+ Issues Resolved in this release
- DM-447 / AMP-64 - tabs replaced with spaces in filestream when delimiter is not tab
- DM-487 / AMP-78 - Modify InjectPackage so that FieldTemplate objects can be imported and accessed as Template objects.
- DM-589 / AMP-81 - auto-guess program for CreateTableFromFile modified so it can be cancelled. Return automatically if line exceeds a specified length.
- DM-680 / AMP-216 - Issues with nulls in Decodes (nulls not being included in dialog)
- DM-840 / AMP-267 - ... #define MAX_HEADER_BUFFER_SIZE 131072ULL
- DM-918 / AMP-290 - CopyUpFromTemplate does not support datasets (Function does not support universal and individual datasets)
- DM-933 / AMP-295 - AnalyseSegments not displaying target amount properly in output
- DM-927 / AMP-296 - Add ability to verify entire project, not just single table or field (use command verify in command line to iterate through all project objects)
- DM-900 / AMP-287 - CreatePackage allows a non-existent table to be specified in the table list - it should fail instead
- DM-934 / AMP 308 - Allow text to be pasted in to Add Dataset Row dialog for IN operator when dataset is first created
- DM-930 / AMP-310 - Postman not sending emails after upgrading to unencrypted djserver.cnfg
- DM-943 / AMP-315 - Ability to create sha256 as base64
- DM-938 / AMP-316 - Eyeota-prd01 recursive fail in Export Method
- DM-939 / AMP-317 - yearmonth() expression - INCORRECT RESULTS when source field is DATETIME
- DM-956 / AMP 323 - error in stored script causes project to remain locked
Other Issues Resolved in this release
- DM 936 - Templates not refreshed on open/close project
- DM 925 - Function: CastNull
- DM 919 - Venn : Values in dragged segment do not always have the correct filter applied
- DM 917 - MULTI-FIELD STATS - Ability to reset report/clear all fields
- DM 916 - VENN - When dragging from Counts Grid, can drag source or resolution table records
- DM 914 - Project Explorer: Add ability to view selected dashboard in Discovery
- DM 904 - MULTI-FUNCTION PROFILE (MFP) - Counting string arguments in a multi-function profile (e.g, counting cities in a zip code)
- DM 895 - OVERLAP - Once a segment has been dragged from grid, can't select grid scroll bar without triggering select
- DM 893 - AnalyseSegments - When dragging a dataset from Data Table Viewer Grid, give the dataset a useful name
- DM 891 - DATASET COLLECTIONS - Individual datasets can be dragged from the collection dialog
- DM 868 - TEMPLATES - Fields that have been hidden using Project Explorer | Show Hide Fields will not be available in Template dialogs or Dataset Row dialogs
- DM 867 - DATASET TAB - Datasets and Collections can be searched for by name
- DM 855 - REPORTS TAB - Data Table mode output is accessible from the Reports | Models tab in Project Explorer