---
title: "Introduction to Scripting"
slug: "introduction-to-scripting"
updated: 2023-10-25T20:03:57Z
published: 2023-10-25T20:03:57Z
canonical: "docs.datajet.app/introduction-to-scripting"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datajet.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction to Scripting

DataJet comes with a comprehensive and powerful scripting engine that can be used to record, replicate and automate user activity. It uses a JSON scripting language to communicate with the processing engine, allowing for detailed control and customisation.

Scripting is accessed via [Script Editor](/v1/docs/script-editor), an integrated Desktop application that allows for the creation, management and execution of JSON scripts.

DataJet scripts interact with all aspects of the Desktop Application:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697115288214.png)

## Script Overview

DataJet scripts are stored in *.json files and consist of one or more JSON methods which are executed sequentially:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696346311411.png)

Scripts can be [created](/v1/docs/introduction-to-scripting#new-script), recorded, viewed, modified, [saved](/v1/docs/introduction-to-scripting#saving-a-script-locally), [stored](/v1/docs/introduction-to-scripting#storing-scripts), distributed, embedded and automated.

## Creating, saving and storing scripts

### New Script

New Scripts are created via **Script Editor | File | New Script**.

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696425484671.png)

New Script

- If an existing script is open, opening a new script will cause unsaved changes in the currently opened script to be lost.
- It is advised to save or store the current script before creating a new script
- Script Editor will prompt the user to save changes before opening a new script

### Storing Scripts

Scripts can be stored in one of 3 ways:

- [locally on the client machine file system](/v1/docs/introduction-to-scripting#saving-a-script-locally)
- in the server database (mongoDB)
- [in the server storage hub](/v1/docs/introduction-to-scripting#adding-a-script-to-script-hub)

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697189574940.png)

| Storage Type | Used for |
| --- | --- |
| File System | - Personal scripts that are still in development and not ready to be shared with team members |
| Local | - Systems that do not have a script hub configured - Sharing with a local team - Version information and history tracking not required - Server-side execution (e.g, ExecuteStoredScript) |
| Script Hub | - Version control and change tracking - Accessible from *Script Editor \| Scripts \| Stored Scripts* - Cross-realm, cross-team sharing - Execution monitoring - Server-side execution (e.g, ExecuteStoredScript) |

A mixture of all 3 storage types can be used, depending on workflow and team structure.

Active Scripts will often contain a mix of saved and unsaved methods. Saved methods are displayed in **black**, unsaved methods are displayed in **purple.**

Script Editor will prompt to save changes before running. Changing the project name (in the Project Box) will set all methods to unsaved.

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696423767433.png)

#### Saving a script to local file system

To save a local copy of the active script, use:

- **Script Editor | File | Save**
- **Script Editor | File | Save As**
- **CTRL S**  (Save/Save As)

Once a script has been saved, the following information will be available in the Script Editor window:![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697190632012.png)

- **Location :** **LOCAL** (Active script is saved on client machine)
- **Scriptname :** Filename of saved script
- **Last Saved DateTimeStamp** - time that active script was last saved

#### Pushing a script to server database

Scripts can be stored in the server database. This is useful in cases where a Script Hub has not been configured, or when testing server execution of scripts before pushing scripts into the main hub.

Once a script has been pushed to the server, the following information will be available in the Script Editor window:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697190688524.png)

- **Location : REMOTE** (Active script is stored on DataJet server)
- **Project:** project in which the script is stored
- **Scriptname :** Filename of stored script

All local scripts are stored in the server's mongo database. They can be executed using [ExecuteStoredScript](/v1/docs/executestoredscript) and are accessible to anyone connected to the server.

#### Adding a script to Script Hub

The script storage system provides support for version control, change tracking and script sharing. Assuming that a [Script Hub](/v1/docs/script-hub-overview) has been configured (see [Client Configuration File](/v1/docs/client-configuration-file) ), scripts can be added to and retrieved from the Hub using:

- **Script Editor | File | Push**
- **Script Editor | File | Pull**

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696434811518.png)

The following metadata is stored along with the script:

- **Location** - the realm for which the script is active.   There will be 2 options:
  - Current Realm (e.g., dj-localhost, mycompanyrealm)
  - global - in allows sharing of scripts between connected DataJet servers and realms
- **Project** - the default project for the script.   This can be used to organise scripts into groups.
- **Name** - the name of the script.  If a script of the specified realm:name already exists, a new version will be added.
- **Status** - In Test / In Development / Currently Active / Archived.
- **Visibility** - Published / Unpublished
- **Description** - Free form description of script.  Required.
- **Commit** Comment - Free form comment describing the version.  Required.
- **Version** - Auto-generated each time the same script is pushed to the hub.

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697191056392.png)

Once a script has been pushed to the Script Hub, the ActiveScript details will be updated:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697190800146.png)

- **Location : REMOTE** (Active script is stored on DataJet server)
- **Realm-name:** The realm in which the script is stored (active realm, or global)
- **Scriptname :** Name of stored script
- **Version :** indicates the version that is being viewed

## Scripting Methods

Scripts are developed by adding methods. Each method consists of an entry in the Script Panel, and the corresponding JSON. For example:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696438107540.png)

```json
{
  "method": "CreateKeyTable",
  "key": "Sales.Customer ID",
  "targetTable": "Customer",
  "join": true,
  "project": "Demonstration"
}
```

The JSON can be directly edited in the Script Editor JSON Panel. To commit a change made in the JSON panel, select Apply:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696438191756.png)

The Script Panel will be updated, and the method will be displayed in purple to show that there are unsaved changes in the script.

### Adding Methods

Add methods (or actions) to the script using one of the following:

- Drag an object from the [project explorer](/v1/docs/project-explorer-overview)
- Drag an object from the [workspace](/v1/docs/workspace-quick-reference)
- Drag an object from the [hotbar](/v1/docs/dataset-hotbar-quick-reference)
- Add an empty method from the [**Script Editor | Script | Methods**](/v1/docs/scripting-methods-overview) menu
- Copy & Paste from the active script
- Copy & Paste from [another script](/v1/docs/supplementary-viewer)
- Call a [stored script](/v1/docs/stored-scripts-1)
- Insert a [plug-in](/v1/docs/plugins-1)
- Use a [scripting helper](/v1/docs/scripting-wizards)
- [Record](/v1/docs/script-editor) an action
- [Import](/v1/docs/import-definitions) from Project History

### Editing Methods

Methods can be edited by:

- Right-clicking the method and choosing **Edit** (if available)
- Right-clicking in the JSON panel and choosing from available options
- Editing JSON directly

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697191444120.png)

### Right-Click Menu

The Script Editor right-click menu contains many of the frequent operations that are used when setting up and testing scripting methods:

| ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696439597404.png) | Goto - Jumps to a navigation remark in script Preview - Previews the method with variables resolved ----- New - Adds an empty new method Delete - Deletes selected method(s) ------- Cut (Ctrl-X) - Cuts selected method(s) from the script and adds to clipboard Copy (Ctrl-C) -Adds selected method(s) to the clipboard Paste (Ctrl-V) - Pastes clipboard contents into the script Undo Entry - ------ Select All Similar - Selects and highlight all entries with the same method type Select Connected - Selects and highlights all entries that are linked to the selected method Search Same - Applies a search filter of the selected method ------ Remark - Adds a remark to the script (prefix remark with *** for navigation remark) Toggle Comment - Sets selected method(s) as comment ------ Edit - Opens method helper (if available) ------ Run to Here - Runs from start to selected method Run From Here - Runs from selected method to end Run Selected - Executes selected method(s) Open in Runner - Copies selected method to Admin \| Runner Auto Guess Types - Attempts to guess data types for CreateTableFromFile ------ Clear All Filters - Removes any search filters Navigate To - Navigates to selected object in Database Explorer Fold - Adds selected items to a folded (read-only) section |
| --- | --- |

### Toolbar

The toolbar provides shortcuts to some of the actions on the right-click menu: ![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696927403836.png)

### Method Display

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696927484112.png)

- **ID** - script line number
- **Method** - Method name (can be filtered)
- **Details** - Default description (usually object manipulated or created by the method), or description, if provided.  Right-click in the JSON window to edit the description, or edit the JSON directly.
- **State** - execution status.   Can be Complete / Running / Failed.   If status is failed, selecting the method entry will display error details in the log window.   State is cleared when the script is re-opened, or when RUN is selected.  (Note to see state for previous executions, use **File | Execution History**)
- **Time** - time taken to complete execution (in seconds).  Overall script execution time is displayed in the script timer:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696927870786.png)

### Method Categories

Scripting Methods are grouped into categories

- Method Categories
  - Audit - Data verification
  - Config - Project wide configuration settings
  - Connections - Data source connections
  - Control - Script control
  - Data Model - Schema design, creation and modification
  - Data Test - Tools for creating test data
  - Engineering - Derived fields
  - Export - Export of data
  - General - Various
  - Joins - Table relationships
  - Load - Import of data and table creation
  - Logging - tracking and recording of script execution
  - Reporting - data visualisation and presentation
  - Shared Data - Injected data, packages
  - Tools - Project administration and build
- Tools Menu

API documentation for scripting methods can be found [here](/v1/docs/scripting-methods-overview)

### 

## Script Execution

Run-time requirements

- Every script is run against an **active open project**
- The project must **already exist**, or the script must **explictly create the project**
- The project must be **open**, or the script must **explicitly open the project**
- The active open project must match the name in the CurrentProject

Whenever a script is executed, each method that is executed returns a result, and a timing. Details of what has been executed are also recorded.

{TODO: Video Snippet - script execution}

### Execution Methods

The following script execution approaches are available:

- [Run-time execution](/v1/docs/introduction-to-scripting#runtime-execution)
- [Stored Script Execution](/v1/docs/introduction-to-scripting#storedscript-execution)
- [Run as Job](/v1/docs/introduction-to-scripting#run-as-job)
- [Remote Execution](/v1/docs/introduction-to-scripting#remote-execution)

#### Run-Time Execution

Run-time execution is where the user selects one of the Script Editor run-time options:

- [**Run**](/v1/docs/script-editor)
- **Run Selected**
- **Run to Here**
- **Run from Here.**

These options give the user detailed control over which methods to execute, and are frequently used during the script development process. They are also accessible from the Script Editor toolbar:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696961682449.png)

Selecting one of these methods will start the script execution clock, which shows how long it has been since the script started.

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1696961704468.png)

Every constructor action (i.e., an action that results in a permanent change in the project data) is recorded in the [project history](/v1/docs/import-definitions). This includes methods executed by script editor. As each method executes, the Script Panel updates with the method Result State and Execution Time:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697193652080.png)

Once execution has completed, methods can be highlighted in the Method Panel, and error information (if applicable) viewed in the Error Panel:

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697193926977.png)

In addition to this, an "Execution History" file will be created on the client machine.

- Project History can be viewed via [File | Import Definitions](/v1/docs/import-definitions)
- Runtime History can be viewed via [File| Execution History](/v1/docs/execution-history)
- If any data was loaded by the script, details of the load will be available in the [Load Logs](/v1/docs/load-logs).

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697194009438.png)

#### Run as Job

**Run as Job** runs the currently displayed script as a SCRIPT [Job](/v1/docs/jobs) using the [ExecuteScript](/v1/docs/executescript) method. The Job window will open and will display the status of the running script.

Script Results

- [Run as Job](/v1/docs/run-as-job) will **not** update the Script Panel with script results - see [Local History](/v1/docs/local-history) for script results.
- Use this method of running a script to execute a script and store execution details in the **Local History** archive.

![](https://cdn.document360.io/73b6469b-c1e4-47e9-aa65-986c8ebd17ca/Images/Documentation/image-1697194886826.png)

![](blob:https://preview.portal.document360.io/88b85497-ab13-4e64-b12d-f42b90f37b03)

#### Stored-Script Execution

Stored scripts can be executed via [ExecuteStoredScript](/v1/docs/executestoredscript) in the following ways:

- Using **Script | Execute**
- Calling **ExecuteStoredScript** from within a script.   Add the method to script by one of the following:
  - **Script Editor | Methods | General | ExecuteStoredScript**
  - **Script Editor | Script | Stored Scripts | Run**

Stored Scripts are scripts that have been added to the [Script Hub](/v1/docs/script-hub-overview) or the server's MongoDB, using the [Push Script Feature](/v1/docs/push-pull-scripts) or **Script Editor |File |Local | Push** feature. [ExecuteStoredScript](/v1/docs/executestoredscript) executes a script by name

Stored Script Execution will update the following logs:

- [Script | History](/v1/docs/history)
- [Load Logs](/v1/docs/load-logs)
- [Admin | Management Console | Execution History](/v1/docs/execution-history)

#### Remote Execution

Remote Execution is where an external process triggers a script to run. Typically this would be some sort of workflow automation.

Scripts are executed remotely using:

- ExecuteStoredScript (recommended)
- ExecuteScript

Script execution history and progress of remote execution can be tracked using:

- [TrackScript](/v1/docs/trackscript)
- [GetScriptResult](/v1/docs/getscriptresult)
- Admin | System Information | Threads

### Script History & Logs

Exactly what has been recorded and where depends on the execution method:

- [Run-time execution](/v1/docs/introduction-to-scripting#runtime-execution)
- [Stored Script Execution](/v1/docs/introduction-to-scripting#storedscript-execution)
- [Run as Job](/v1/docs/introduction-to-scripting#run-as-job)
- [Remote Execution](/v1/docs/introduction-to-scripting#remote-execution)

The following log files are produced, depending on the execution method:

- [**Local History**](/v1/docs/local-history) - Created by *File |* *Run as Job* and *ExecuteScript*
- [**Script History**](/v1/docs/history) - Created by *ExecuteStoredScript* and *Script | Execute*
- [Execution History](/v1/docs/execution-history-1) - Created by *Run*, *Run Selected*, *Run to Here* and *Run from Here*
- [**Load Logs**](/v1/docs/load-logs) - Created for all table loads, regardless of execution method
- [**Execution History**](/v1/docs/execution-history) - Created by *Script | Execute*, *ExecuteStoredScript* and *Run As Job.*

##