Introduction to Scripting

DataJet comes with a comprehensive and powerful scripting engine that can be used to record, replicate and automate user activity.    It uses a JSON scripting language to communicate with the processing engine, allowing for detailed control and customisation.

Scripting is accessed via Script Editor, an integrated Desktop application that allows for the creation, management and execution of JSON scripts.

DataJet scripts interact with all aspects of the Desktop Application:

Script Overview

DataJet scripts are stored in *.json files and consist of one or more JSON methods which are executed sequentially:

Scripts can be created, recorded, viewed, modified, saved, stored, distributed, embedded and automated.

Creating, saving and storing scripts

New Script

New Scripts are created via Script Editor | File | New Script.    

Storing Scripts

Scripts can be stored in one of 3 ways:

• locally on the client machine file system
• in the server database (mongoDB)
• in the server storage hub

A mixture of all 3 storage types can be used, depending on workflow and team structure.

Active Scripts will often contain a mix of saved and unsaved methods.  Saved methods are displayed in black, unsaved methods are displayed in purple.  

Script Editor will prompt to save changes before running.  Changing the project name (in the Project Box) will set all methods to unsaved.

Saving a script to local file system

To save a local copy of the active script, use:

• Script Editor | File | Save
• Script Editor | File | Save As
• CTRL S  (Save/Save As)

Once a script has been saved, the following information will be available in the Script Editor window:

• Location : LOCAL (Active script is saved on client machine)
• Scriptname : Filename of saved script
• Last Saved DateTimeStamp - time that active script was last saved

Pushing a script to server database

Scripts can be stored in the server database.   This is useful in cases where a Script Hub has not been configured, or when testing server execution of scripts before pushing scripts into the main hub.

Once a script has been pushed to the server, the following information will be available in the Script Editor window:

• Location : REMOTE (Active script is stored on DataJet server)
• Project: project in which the script is stored 
• Scriptname : Filename of stored script

All local scripts are stored in the server's mongo database.  They can be executed using ExecuteStoredScript and are accessible to anyone connected to the server.

Adding a script to Script Hub

The script storage system provides support for version control, change tracking and script sharing.   Assuming that a Script Hub has been configured (see Client Configuration File ), scripts can be added to and retrieved from the Hub using:

• Script Editor | File | Push  
• Script Editor | File | Pull

The following metadata is stored along with the script:

• Location - the realm for which the script is active.   There will be 2 options:
  • Current Realm (e.g., dj-localhost, mycompanyrealm)
  • global - in allows sharing of scripts between connected DataJet servers and realms
• Project - the default project for the script.   This can be used to organise scripts into groups.
• Name - the name of the script.  If a script of the specified realm:name already exists, a new version will be added.
• Status - In Test / In Development / Currently Active / Archived.
• Visibility - Published / Unpublished
• Description - Free form description of script.  Required.
• Commit Comment - Free form comment describing the version.  Required.
• Version - Auto-generated each time the same script is pushed to the hub.

Once a script has been pushed to the Script Hub, the ActiveScript details will be updated:

• Location : REMOTE (Active script is stored on DataJet server)
• Realm-name: The realm in which the script is stored (active realm, or global)
• Scriptname : Name of stored script
• Version : indicates the version that is being viewed

Scripting Methods

Scripts are developed by adding methods.    Each method consists of an entry in the Script Panel, and the corresponding JSON.   For example:

{
  "method": "CreateKeyTable",
  "key": "Sales.Customer ID",
  "targetTable": "Customer",
  "join": true,
  "project": "Demonstration"
}

The JSON can be directly edited in the Script Editor JSON Panel.   To commit a change made in the JSON panel, select Apply:

The Script Panel will be updated, and the method will be displayed in purple to show that there are unsaved changes in the script.

Adding Methods

Add methods (or actions) to the script using one of the following:

• Drag an object from the project explorer
• Drag an object from the workspace
• Drag an object from the hotbar
• Add an empty method from the Script Editor | Script | Methods menu
• Copy & Paste from the active script
• Copy & Paste from another script
• Call a stored script
• Insert a plug-in
• Use a scripting helper
• Record an action
• Import from Project History

Editing Methods

Methods can be edited by:

• Right-clicking the method and choosing Edit (if available)
• Right-clicking in the JSON panel and choosing from available options
• Editing JSON directly

Right-Click Menu

The Script Editor right-click menu contains many of the frequent operations that are used when setting up and testing scripting methods:

Goto - Jumps to a navigation remark in script Preview - Previews the method with variables resolved ----- New - Adds an empty new method Delete - Deletes selected method(s) ------- Cut (Ctrl-X) - Cuts selected method(s) from the script and adds to clipboard Copy (Ctrl-C) -Adds selected method(s) to the clipboard Paste (Ctrl-V) - Pastes clipboard contents into the script Undo Entry - ------ Select All Similar - Selects and highlight all entries with the same method type Select Connected - Selects and highlights all entries that are linked to the selected method Search Same - Applies a search filter of the selected method ------ Remark - Adds a remark to the script (prefix remark with *** for navigation remark) Toggle Comment - Sets selected method(s) as comment ------ Edit - Opens method helper (if available) ------ Run to Here - Runs from start to selected method Run From Here - Runs from selected method to end Run Selected - Executes selected method(s) Open in Runner - Copies selected method to Admin | Runner Auto Guess Types - Attempts to guess data types for CreateTableFromFile ------ Clear All Filters - Removes any search filters Navigate To - Navigates to selected object in Database Explorer Fold - Adds selected items to a folded (read-only) section

Toolbar

The toolbar provides shortcuts to some of the actions on the right-click menu:

Method Display

• ID - script line number
• Method - Method name (can be filtered)
• Details - Default description (usually object manipulated or created by the method), or description, if provided.  Right-click in the JSON window to edit the description, or edit the JSON directly.
• State - execution status.   Can be Complete / Running / Failed.   If status is failed, selecting the method entry will display error details in the log window.   State is cleared when the script is re-opened, or when RUN is selected.  (Note to see state for previous executions, use File | Execution History)
• Time - time taken to complete execution (in seconds).  Overall script execution time is displayed in the script timer:

Method Categories

Scripting Methods are grouped into categories

• Method Categories
  • Audit - Data verification
  • Config - Project wide configuration settings
  • Connections - Data source connections
  • Control - Script control
  • Data Model - Schema design, creation and modification
  • Data Test - Tools for creating test data
  • Engineering - Derived fields
  • Export - Export of data
  • General - Various
  • Joins - Table relationships
  • Load - Import of data and table creation
  • Logging - tracking and recording of script execution
  • Reporting - data visualisation and presentation
  • Shared Data - Injected data, packages
  • Tools - Project administration and build
• Tools Menu

API documentation for scripting methods can be found here

Script Execution

Whenever a script is executed,  each method that is executed returns a result, and a timing. Details of what has been executed are also recorded.

{TODO: Video Snippet - script execution}

Execution Methods

The following script execution approaches are available:

• Run-time execution
• Stored Script Execution
• Run as Job
• Remote Execution

Run-Time Execution

Run-time execution is where the user selects one of the Script Editor run-time options: 

• Run 
• Run Selected 
• Run to Here  
• Run from Here.   

These options give the user detailed control over which methods to execute, and are frequently used during the script development process.  They are also accessible from the Script Editor toolbar:

Selecting one of these methods will start the script execution clock, which shows how long it has been since the script started.

Every constructor action (i.e., an action that results in a permanent change in the project data) is recorded in the project history. This includes methods executed by script editor.  As each method executes, the Script Panel updates with the method Result State and Execution Time:

Once execution has completed, methods can be highlighted in the Method Panel, and error information (if applicable) viewed in the Error Panel:

In addition to this, an "Execution History" file will be created on the client machine.

• Project History can be viewed via File | Import Definitions
• Runtime History can be viewed via File| Execution History
• If any data was loaded by the script, details of the load will be available in the Load Logs.

Run as Job

Run as Job runs the currently displayed script as a SCRIPT Job using the ExecuteScript method.  The Job window will open and will display the status of the running script.

Stored-Script Execution

Stored scripts can be executed via ExecuteStoredScript in the following ways:

• Using Script | Execute
• Calling ExecuteStoredScript from within a script.   Add the method to script by one of the following:
  • Script Editor | Methods | General | ExecuteStoredScript
  • Script Editor | Script | Stored Scripts | Run

Stored Scripts are scripts that have been added to the Script Hub or the server's MongoDB, using the Push Script Feature or Script Editor |File |Local | Push feature.  ExecuteStoredScript executes a script by name

Stored Script Execution will update the following logs:

• Script | History
• Load Logs
• Admin | Management Console | Execution History

Remote Execution

Remote Execution is where an external process triggers a script to run.   Typically this would be some sort of workflow automation.

Scripts are executed remotely using:

• ExecuteStoredScript (recommended)
• ExecuteScript 

Script execution history and progress of remote execution can be tracked using:

• TrackScript
• GetScriptResult
• Admin | System Information | Threads

Script History & Logs

Exactly what has been recorded and where depends on the execution method:

• Run-time execution
• Stored Script Execution
• Run as Job
• Remote Execution

The following log files are produced, depending on the execution method:

• Local History - Created by File | Run as Job and ExecuteScript
• Script History - Created by ExecuteStoredScript and Script | Execute
• Execution History - Created by Run, Run Selected, Run to Here and Run from Here
• Load Logs - Created for all table loads, regardless of execution method
• Execution History - Created by Script | Execute, ExecuteStoredScript and Run As Job.