Deep Dive - Sharing Data by Injection

Article In Progress

PROTOTYPE

Overview

What is injected data?

Injecting data allows data to be shared across projects and realms.    Depending on system architecture, data may be injected across:

• projects, 
• multiple realms on the same server, 
• multiple DJ servers.

Injecting data avoids loading multiple instances of the same dataset.  Using this technique, multiple projects can use the same data source. 

Features

Once the injection has completed, the injected table will appear in the target project.  Injected tables are green, and are read-only within the target project.

Example: sales table injected from Demonstration project into BikeData2 project:

Notes:

• Deleting the injected table in the source project will also remove it from all target projects
• To remove an injected table, right-click the table in Database Explorer and choose "detach".   Alternatively use the DetachTable API.
• It is possible to create derived data on an injected table within a target project.
• Derived data that has been created on an injected table will remain on the table even after the table has been detached
• Additions and changes to the source table will NOT be reflected in the injected table.    It is important to have a process in place that prevents source data from being modified once it has been injected in another project.

Approach

Whichever injection paradigm is used, the overall approach will be the same:

1. Prepare Source Data 
   1. Create, Load, Engineer
2. Create Injection Packages (if injecting across realm or server)
   1. Create a source project - make sure data files are accessible from Target machines.
   2. Backup method
   3. CreatePackage method
3. Prepare Target Project to receive data
   1. Create Project
4. Inject Data into Target Project
   1. InjectTable - Injects a single table into a project on the source realm
   2. InjectTables - Injects a set of tables into a project on the source realm
   3. InjectPackage - Injects a set of tables, joins and reports into a project on source realm or other connected realms
   4. Coming soon...  InjectFromWorkbook
   5. Coming soon...  InjectReports

Scripting Methods

CreatePackage

This will create a package file that can be injected into a target project

{
  "method": "CreatePackage",
  "path": "%DATAPATH%shareddata/Package_05_withmetadata.zip",
  "system": false,
  "includeProjectMetaData": true,
  "targetProject": "Package_05",
  "tables": [
    "DATA_S1",
    "DATA_S2",
    "DATA_S3"
  ],
  "project": "CreatePackage_05"
}

InjectTable

This will inject a single table from a source project into a target project.   Both source and target projects must be on the same realm.  See InjectTable for further details.

{
  "method": "InjectTable",
  "description": "Inject Data From Demonstration into BikeData2",
  "sourceProject": "Demonstration",
  "sourceTable": "Sales",
  "targetProject": "BikeData2",
  "targetTable": "sales",
  "project": "BikeData2"
}

InjectTables

This will inject a set of tables from a source project into a target project.  Both source and target projects must be on the same realm.  See InjectTables for further details.

{
  "method": "InjectTables",
  "project": "test",
  "sourceProject": "DataSource_01",
  "sourceTables": [
            "Customers", 
            "Transactions" 
            ],
  "targetTables": [
            "Cust_DS01", 
            "Trans_DS01" 
            ],
  "targetProject": "AllDataSource_01",
  "includeJoins": true
}

InjectPackage

InjectPackage will inject a static set of Data Objects and Data into one or more projects.   InjectPackage works across:

• Projects on the same realm
• Projects on the same server, but on different realms
• Projects on different servers (and hence different realms)

The process is as follows:

1. Prepare Data -create a project containing only the data that is to be injected
2. Create the package and make sur the project data is accessible from all target realms
3. Run InjectPackage on the target realm

See InjectPackage for more details

{
  "method": "InjectPackage",
  "project": "activeproject",
  "path": "%DATAPATH%sharedbackup/realm1_packageABC.zip",
  "homePath": "%DATAPATH%shareddata",
  "targetProject": "ClientA"
}

Methods

• Injecting tables within a realm
• Injecting a package within a realm
• Injecting Packages across realms.

Injecting tables within a realm

To inject a table from one project to another, where both projects are within the same realm, the InjectTable or InjectTables methods can be used.  Note that this method is injecting "live" project data into a target project - it is possible for the source data to be modified if processes are not in place to prevent this.

Injecting a package within a realm

To inject a table across realms on the same server, use the InjectPackage methos.

Injecting Packages across realms

To inject a package across realms , first make sure that the project data that is to be injected (i.e., the loaded project files *.uvl, *.dat) are accessible from all realms which will be using the package.    This may involve moving or copying data, depending on the various realm configurations.    Then use InjectPackage on the target realm.

Step 1: CreatePackage

Create package on the source machine/realm:

{
  "method": "CreatePackage",
  "path": "%DATAPATH%shareddata/Package_05_withmetadata.zip",
  "system": false,
  "includeProjectMetaData": true,
  "targetProject": "Package_05",
  "tables": [
    "DATA_S1",
    "DATA_S2",
    "DATA_S3"
  ],
  "project": "CreatePackage_05"
}

Step 2: Make sure data is accessible.

The data in project "Package_05" must be visible from the target machine, and have all the necessary permissions.  In this instance, the data will be in 

• /home/repos/realm-name/Package_05

And the package file will be at:

• "%DATAPATH%shareddata/Package_05_withmetadata.zip"

Let's assume that a drive on the target machine has been mapped to this location, and is configured as:

• /home/datasources/shareddata/Package_05

These will give the following parameters for InjectPackage:

• "path" : "%DATAPATH%shareddata/Package_05_withmetadata.zip"
• "homepath" : "%DATAPATH%shareddata/Package_05"

Step 3: InjectPackage

Run the following on the target machine/realm:

{
  "method": "InjectPackage",
  "path": "%DATAPATH%shareddata/Package_05_withmetadata.zip",
  "homePath": "%DATAPATH%shareddata/Package_05",
  "targetProject": "Inj_Package05",
  "description": "Inject Package_05",
  "sourceName": "dev01-Package_05",
  "includeProjectMetaData": true,
  "project": "inj_Package05"
}