From DataSource
  • 1 Minute to read
  • Dark
    Light
  • PDF

From DataSource

  • Dark
    Light
  • PDF

Article summary

In Progress...

There are 3 mechanisms available for importing data directly from a DataSource:

  1. Single Table import (Script | Editor | Import Data | Data Source)
  2. Bulk Import (Script | Editor | Import Data | Bulk Import | Entire DataSource) 
  3. Custom Importer (ProcessEx method)

The single table import dialog allows each table to be inspected individually, the bulk importer will create a definition for each table in the data source

Data Source Configuration
Before Data can be imported from a Data Source, the Data Source needs to be configured in Admin | Data Source Management.

Before Loading Data From a DataSource

Field Definitions
When importing from a DataSource, it's important to check field definitions before the import begins, otherwise load times can be very slow, or even fail.  

In particular, for large tables, check:
  • Very high cardinality fields (i.e., IDs, addresses, emails etc) are configured as CONTINUOUS
  • CONTINUOUS strings must be FIXED WIDTH - i.e., provide a field width that is sufficient to load the data
  • Use Import Data | Data Source to inspect individual table definitions and obtain string length information (if available for the Data Source)


Field Width for Continuous Strings
  • Continuous strings must be a fixed width - no variable length
  • Length is in bytes:   "NAMESURNAME|CONTINUOUS|STRING|100",
  • bytes = characters for ansii
  • bytes != characters for unicode

See Script | Methods | RDBImporterEx for full details of how to import continuous strings from a Data Source


Samples
When loading data from a DataSource, do a SAMPLE LOAD and then run a Data Audit once the load has completed. This will allow the resulting data to be inspected and field definitions to be modified if necessary before loading the full data source.


Import Data | Data Source





Note: When inspecting large tables, it may take up to a minute for the sample data to load

Import Data | Bulk Loader | Entire DataSource



Was this article helpful?

What's Next