Getting Started

The most common integration use case is to import data from another system, like a PLM. We refer to this as "loading data". With our powerful Loader framework, you can load complex data into VibeIQ with ease. This guide will walk you through the basics of that process.

The Loader currently supports:

The Loader framework simplifies integrations from your systems to VibeIQ. It abstracts away create-or-update logic within VibeIQ, offloading that work to the VibeIQ platform and reducing network traffic between systems. All data loads run within an asynchronous Loader Process that can be monitored and managed through our API, SDK, or the Contrail CLI.

Key Terminology and Concepts¶

Loader Process¶

A Loader Process is an entity within VibeIQ that contains information about the load process that was invoked. This includes information like:

A status indicating whether the load is pending execution, currently running, has completed successfully, or has failed.
The parameters provided to the Loader Process that were used to perform transformations on the input data.
A download link to Loader Process's logs.
Additional metadata about the process.

When a Loader Process is created, the VibeIQ platform enqueues it for execution. When it runs, the provided data in the load file will be inserted into the platform. If desired, this input data can also be manipulated with data transformations during the process. We call this the "Load Config", which allows you to do things like re-map fields, set or re-set values based on other values, and much more.

The full schema for a Loader Process is available in the Admin Console API documentation section.

A Loader Process can be created with the API, SDK, or Contrail CLI. A Loader Process requires:

A data load file in CSV format. This is the data you will insert into VibeIQ.
A Loader Process Config object. This will instruct the Loader to load the data into specific entity types within VibeIQ, as well as perform specific transformations on the CSV load file. The latter is not required.

Please see the below sections for more explanation on these requirements.

Loader Process Data File¶

CSV Load File¶

Load data must be provided to the Loader framework in tabular CSV format. Generally speaking, each row in this CSV (except the column header row) represents a single instance of the entity type being loaded into. For example, if you are creating Item entities, each row in the CSV would represent a single Item. Columns in the CSV must map to property keys or property labels on the entity type. If the column does not match a key or label on the specified entity type, the row column will be ignored. The Loader logs will indicate this. Please see the Loader Config section to learn more about how you can use it to dynamically map columns to entity properties.

Federated IDs¶

A federatedId is a value you provide which uniquely identifies the entity within an org. Every row in your data must have a federatedId. If an entity with that federatedId does not exist in VibeIQ, it will be created. If it does exist, it will be updated. This should be a unique identifier sourced from the system that originated the data. If a system did not generate it yourself, it's typically best to select an existing column to be your federatedId.

It is critical to ensure that federatedId values do not change between loads. Doing so will risk the Loader creating duplicate data. If you need to change a federatedId on existing data within VibeIQ, please contact the support team for assistance or guidance.

Example CSV Load File¶

Simple example for a common entity type Color:

federatedId	name	hexCode	red	blue	green
001	Red	#FF0000	255	0	0
002	Blue	#0000FF	0	255	0
003	Green	#00FF00	0	0	255

This load file format would successfully load into a Color entity with these properties. Properties have a key and label. Keys are used to match columns in the CSV to properties on the entity type. Labels are used to display the property in the VibeIQ app UIs.

federatedId - The external unique identifier for the color
key: "name", label: "Color Name" - The name of the color onto property with key name.
key: "hexCode", label: "Hex Code" - The hex code for the color onto property with key hexCode.
key: "red", label: "Red" - The red value of the color onto property with key red.
key: "blue", label: "Blue" - The blue value of the color onto property with key blue.
key: "green", label: "Green" - The green value of the color onto property with key green.

Loader Process Config¶

The Loader Process Config is a JSON object that instructs the Loader on how to load the data in the CSV load file. Using the below properties within it, you can specify how the Loader should handle the data in the CSV file.

loadType
federatedMappings
conditionalColumns
fileLocation
loaderConfigurationId

`loadType`¶

Specifies the entity type(s) to load the data into. This is an array of strings that can contain one of the following values unless that specific type supports additional types. Currently, only Item supports additional types.

ITEM: Load data into the Item entity type.
- PROJECT_ITEM: Load data into the ProjectItem entity type. Must be paired with ITEM. Selecting this will load Item and ProjectItems instances in the same load.
- ASSORTMENT: Load data into the Assortment entity type. Must be paired with both ITEM. Can also be paired with PROJECT_ITEM. Selecting this will load Item, ProjectItem and AssortmentItem instances in the same load.
COLOR: Load data into the Color entity type.
CUSTOM_ENTITY: Load data into the Custom Entity type for a specific typePath.
SIZE_RANGE_TEMPLATE: Load data into the Size Range Template entity type.

`federatedMappings`¶

A key-value object that allows you re-map data in the CSV into new columns. This is especially useful when you have a CSV file generated by another system that does not have the same column names as the entity type's property keys or labels.

Create new columns by copying all values from an existing column. This can be done by referencing the copied column by its column name. The Loader will create a new column in the CSV with the new name you specify and copy-over the data from the column you are referencing.
Create new columns with a static value for all rows.

Copy Column Example:¶

Load File:

federatedId	Color_Name	Hex_Code	RED	GREEN	GREEN
001	Red	#FF0000	255	0	0
002	Blue	#0000FF	0	255	0
003	Green	#00FF00	0	0	255

Config:

{
  "federatedMappings": {
    "name": "Color_Name",
    "hexCode": "Hex_Code",
    "red": "RED",
    "blue": "BLUE", 
    "green": "GREEN"
  },
  ...
}

Will make the Loader see the file as:

federatedId	name	hexCode	red	blue	green	Color_Name	Hex_Code	RED	BLUE	GREEN
001	Red	#FF0000	255	0	0	Red	#FF0000	255	0	0
002	Blue	#0000FF	0	255	0	Blue	#0000FF	0	255	0
003	Green	#00FF00	0	0	255	Green	#00FF00	0	0	255

Create New Static Column Example:¶

Load File:

federatedId	name	hexCode
001	Red	#FF0000
002	Blue	#0000FF
003	Green	#00FF00

Config:

{
  "federatedMappings": {
    "originatingSystem": "My ERP System"
  },
  ...
}

Will make the Loader see the file as:

federatedId	name	hexCode	originatingSystem
001	Red	#FF0000	My ERP System
002	Blue	#0000FF	My ERP System
003	Green	#00FF00	My ERP System

`conditionalColumns`¶

An array that allows you to create new columns based on conditions. This is especially useful when you need to set a value based on the value of another column or to set a static value. This is a lot like federatedMappings, but provides conditional logic support. Conditions are evaluated by the Loader as JavaScript expressions. If the expression evaluates to true, the value is set. If the expression evaluates to false or is invalid syntax, the Loader moves to the next condition. If no conditions evaluate to true, the default value is set, if provided.

Schema:

interface ConditionalColumns {
    toProperty: string;
    fromProperty?: string;
    conditions?: ConditionalValues[];
    default?: any;  
}

toProperty: string: The new column that will be written to. If this column already exists, it will be overwritten.
fromProperty?: string: An optional property used to copy the value of one column to another. This will be ignored if conditions are provided.
conditions?: ConditionalValues[]: An array of conditions to check and apply their value to the toProperty. The first condition that evaluates to true will be applied. If a default is provided, that value will be used.
- conditional: A JavaScript expression that will be evaluated. If it evaluates to true, the corresponding value will be set. Within the conditional's expression, you can reference other columns in the load file using curly braces { }.
- value: The value to set if the condition evaluates to true.
default?: string: The value to set if no conditions evaluate to true.

Set Column Value Based on Condition:¶

This example checks if column value trackConditions is equal to 'slippery'. If it is, the value of the new column treadType will be set to 'wet'. If not, the value of the new column treadType will be set to the value of the column treadTypeLegacy.

{
  "conditionalColumns": [
    {
      "conditions": [
        {
          "conditional": "{trackConditions} === 'slippery'",
          "value": "wet"
        }
      ],
      "toProperty": "treadType",
      "default": "{treadTypeLegacy}"
    }
  ]
}

Remember, you can provide multiple conditions to the conditions array.

`fileLocation`¶

Used when the CSV file you want to load is already in VibeIQ. This is commonly used for programmatic use cases (as opposed to manual, CLI use cases). Please see the Files API section of the Admin Console for more information on how to upload files.

`loaderConfigurationId`¶

An optional property that allows you to specify a pre-uploaded YAML or JSON file representing the Loader Process Config. This is useful when you have a common Config that you want to re-use across multiple Loader Process instances so you do not need to repeatedly define it. Please see the Files API section of the Admin Console for more information on how to upload files.

`propertiesToRemove`¶

An optional array property that allows you to specify which columns to remove from the CSV file before loading.

Creating a Loader Process¶

Loading With the CLI¶

If you have a CSV data file on and would like to load it from your local machine, you can use the Contrail CLI to create a Loader Process. This requires the Contrail CLI, so please install it before proceeding.

You must have both your data file as CSV and your Config file as YAML available locally. Start a Loader Process with the following command:

contrail loader-process upload-and-load <File Path> <Config Path>

<File Path>: specifies the file path to your CSV load file.

<Config Path>: specifies the file path to your YAML Config file.

Note: loaderConfigurationId cannot be used from the CLI. You must provide the Config through the CLI every time.

contrail loader-process upload-and-load ./path/to/data.csv ./path/to/config.yml

Loading With the API or SDK¶

If you would like to programmatically create a Loader Process, you can do so with the API or SDK. Like the CLI, both a data CSV file and a Config are required. The methodology to provide the CSV file and Config to the API/SDK differs slightly from the CLI, so please take note of the following steps.

Authentication¶

Please use the Contrail CLI to retrieve an API key or reach out to the support team for assistance.

If you are authenticating with the SDK, please see our SDK reference here: Contrail SDK Login. Your runtime must authenticate with login() before making any other requests.

If you are using the API, you will need to provide an API key via the X-Api-Key header as well as an X-Api-Org header with your VibeIQ environment's identifier key.

-- header 'X-Api-Key: {YOUR-API-KEY}' \
-- header 'X-Api-Org: {YOUR-ORG-SLUG}' \

Upload Your CSV Data File¶

To create Loader Process with the SDK or API, you must first upload the CSV load file to the system. This can be done with the Files API. A full reference for this API is available in the Admin Console.

This will look something like this:

JScURL

const csvFile = Buffer.from(csvContent, "utf-8");
const createdFile = await new Files().createAndUploadFileFromBuffer(
  csvFile,
  "text/csv",
  fileName,
  null,
  5 * 60 * 60 * 1000 // ttl: Time to Live in Miliseconds
);
console.log("createdFile: ", JSON.stringify(createdFile));
console.log('VibeIQ file ID: ', createdFile.id);

# 1. Create a new `File` entity via the Files API.
curl --location 'https://admin.vibeiq.com/api/files' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--data '{
      "contentType": "text/csv",
      "fileName": "my-load-file.csv"
  }'

# 2. Upload the file binary to the `File` entity.
# The request from step 1 will return an S3 bucket upload link. 
# Use that link to upload the file content to S3. The download URL
# from step 1 will then contain the file’s content.
curl --location 'https://prod-contrail-file-content-bucket.s3.amazonaws.com' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--form 'Content-Type="text/csv"' \
--form 'file=@"/path/to/file"'

Pass Config Parameters & Create a Loader Process¶

Using the parameters explained in the above sections, you will need to specify:

An entityName to indicate that you are creating a new entity type instance of type 'loader-process'.
An object with the Loader Process Config parameters in the request body in JSON format.