Getting Started
The most common integration use case is to import data from another system, like a PLM. We refer to this as "loading data". With our powerful Loader framework, you can load complex data into VibeIQ with ease. This guide will walk you through the basics of that process.
The Loader currently supports:
- Line Sheet Data: Global, Seasonal, and Assortment Level Item Data
- Custom Entity Data
- Color Data
- Size Range Template Data
The Loader framework simplifies integrations from your systems to VibeIQ. It abstracts away create-or-update logic within VibeIQ, offloading that work to the VibeIQ platform and reducing network traffic between systems. All data loads run within an asynchronous Loader Process
that can be monitored and managed through our API, SDK, or the Contrail CLI.
Key Terminology and Concepts¶
Loader Process¶
A Loader Process
is an entity within VibeIQ that contains information about the load process that was invoked. This includes information like:
- A status indicating whether the load is pending execution, currently running, has completed successfully, or has failed.
- The parameters provided to the
Loader Process
that were used to perform transformations on the input data. - A download link to
Loader Process
's logs. - Additional metadata about the process.
When a Loader Process
is created, the VibeIQ platform enqueues it for execution. When it runs, the provided data in the load file will be inserted into the platform. If desired, this input data can also be manipulated with data transformations during the process. We call this the "Load Config", which allows you to do things like re-map fields, set or re-set values based on other values, and much more.
The full schema for a Loader Process
is available in the Admin Console API documentation section.
A Loader Process
can be created with the API, SDK, or Contrail CLI. A Loader Process
requires:
- A data load file in CSV format. This is the data you will insert into VibeIQ.
- A
Loader Process
Config object. This will instruct the Loader to load the data into specific entity types within VibeIQ, as well as perform specific transformations on the CSV load file. The latter is not required.
Please see the below sections for more explanation on these requirements.
Loader Process Data File¶
CSV Load File¶
Load data must be provided to the Loader framework in tabular CSV format. Generally speaking, each row in this CSV (except the column header row) represents a single instance of the entity type being loaded into. For example, if you are creating Item entities, each row in the CSV would represent a single Item. Columns in the CSV must map to property keys or property labels on the entity type. If the column does not match a key or label on the specified entity type, the row column will be ignored. The Loader logs will indicate this. Please see the Loader Config section to learn more about how you can use it to dynamically map columns to entity properties.
Federated IDs¶
A federatedId
is a value you provide which uniquely identifies the entity within an org. Every row in your data must have a federatedId
. If an entity with that federatedId
does not exist in VibeIQ, it will be created. If it does exist, it will be updated. This should be a unique identifier sourced from the system that originated the data. If a system did not generate it yourself, it's typically best to select an existing column to be your federatedId
.
It is critical to ensure that federatedId
values do not change between loads. Doing so will risk the Loader creating duplicate data. If you need to change a federatedId
on existing data within VibeIQ, please contact the support team for assistance or guidance.
Example CSV Load File¶
Simple example for a common entity type Color
:
federatedId | name | hexCode | red | blue | green |
---|---|---|---|---|---|
001 | Red | #FF0000 | 255 | 0 | 0 |
002 | Blue | #0000FF | 0 | 255 | 0 |
003 | Green | #00FF00 | 0 | 0 | 255 |
This load file format would successfully load into a Color
entity with these properties. Properties have a key and label.
Keys are used to match columns in the CSV to properties on the entity type. Labels are used to display the property in the VibeIQ app UIs.
federatedId
- The external unique identifier for the colorkey: "name"
,label: "Color Name"
- The name of the color onto property with keyname
.key: "hexCode"
,label: "Hex Code"
- The hex code for the color onto property with keyhexCode
.key: "red"
,label: "Red"
- The red value of the color onto property with keyred
.key: "blue"
,label: "Blue"
- The blue value of the color onto property with keyblue
.key: "green"
,label: "Green"
- The green value of the color onto property with keygreen
.
Loader Process Config¶
The Loader Process
Config is a JSON object that instructs the Loader on how to load the data in the CSV load file.
Using the below properties within it, you can specify how the Loader should handle the data in the CSV file.
loadType
federatedMappings
conditionalColumns
fileLocation
loaderConfigurationId
loadType
¶
Specifies the entity type(s) to load the data into. This is an array of strings that can contain one of the following values
unless that specific type supports additional types. Currently, only Item
supports additional types.
ITEM
: Load data into theItem
entity type.PROJECT_ITEM
: Load data into theProjectItem
entity type. Must be paired withITEM
. Selecting this will loadItem
andProjectItems
instances in the same load.ASSORTMENT
: Load data into theAssortment
entity type. Must be paired with bothITEM
. Can also be paired withPROJECT_ITEM
. Selecting this will loadItem
,ProjectItem
andAssortmentItem
instances in the same load.
COLOR
: Load data into theColor
entity type.CUSTOM_ENTITY
: Load data into theCustom Entity
type for a specifictypePath
.SIZE_RANGE_TEMPLATE
: Load data into theSize Range Template
entity type.
federatedMappings
¶
A key-value object that allows you re-map data in the CSV into new columns. This is especially useful when you have a CSV file generated by another system that does not have the same column names as the entity type's property keys or labels.
-
Create new columns by copying all values from an existing column. This can be done by referencing the copied column by its column name. The Loader will create a new column in the CSV with the new name you specify and copy-over the data from the column you are referencing.
-
Create new columns with a static value for all rows.
Copy Column Example:¶
Load File:
federatedId | Color_Name | Hex_Code | RED | GREEN | GREEN |
---|---|---|---|---|---|
001 | Red | #FF0000 | 255 | 0 | 0 |
002 | Blue | #0000FF | 0 | 255 | 0 |
003 | Green | #00FF00 | 0 | 0 | 255 |
Config:
{
"federatedMappings": {
"name": "Color_Name",
"hexCode": "Hex_Code",
"red": "RED",
"blue": "BLUE",
"green": "GREEN"
},
...
}
federatedId | name | hexCode | red | blue | green | Color_Name | Hex_Code | RED | BLUE | GREEN |
---|---|---|---|---|---|---|---|---|---|---|
001 | Red | #FF0000 | 255 | 0 | 0 | Red | #FF0000 | 255 | 0 | 0 |
002 | Blue | #0000FF | 0 | 255 | 0 | Blue | #0000FF | 0 | 255 | 0 |
003 | Green | #00FF00 | 0 | 0 | 255 | Green | #00FF00 | 0 | 0 | 255 |
Create New Static Column Example:¶
Load File:
federatedId | name | hexCode |
---|---|---|
001 | Red | #FF0000 |
002 | Blue | #0000FF |
003 | Green | #00FF00 |
Config:
{
"federatedMappings": {
"originatingSystem": "My ERP System"
},
...
}
federatedId | name | hexCode | originatingSystem |
---|---|---|---|
001 | Red | #FF0000 | My ERP System |
002 | Blue | #0000FF | My ERP System |
003 | Green | #00FF00 | My ERP System |
conditionalColumns
¶
An array that allows you to create new columns based on conditions. This is especially useful when you need to set a value based on the value of another column or to set a static value. This is a lot like federatedMappings
, but provides conditional logic support. Conditions are evaluated by the Loader as JavaScript expressions. If the expression evaluates to true
, the value is set. If the expression evaluates to false
or is invalid syntax, the Loader moves to the next condition. If no conditions evaluate to true
, the default value is set, if provided.
Schema:
interface ConditionalColumns {
toProperty: string;
fromProperty?: string;
conditions?: ConditionalValues[];
default?: any;
}
toProperty: string
: The new column that will be written to. If this column already exists, it will be overwritten.fromProperty?: string
: An optional property used to copy the value of one column to another. This will be ignored ifconditions
are provided.conditions?: ConditionalValues[]
: An array of conditions to check and apply theirvalue
to thetoProperty
. The first condition that evaluates totrue
will be applied. If adefault
is provided, that value will be used.conditional
: A JavaScript expression that will be evaluated. If it evaluates totrue
, the correspondingvalue
will be set. Within the conditional's expression, you can reference other columns in the load file using curly braces{ }
.value
: The value to set if the condition evaluates totrue
.
default?: string
: The value to set if no conditions evaluate totrue
.
Set Column Value Based on Condition:¶
This example checks if column value trackConditions
is equal to 'slippery'
. If it is, the value of the new column treadType
will be set to 'wet'
. If not, the value of the new column treadType
will be set to the value of the column treadTypeLegacy
.
{
"conditionalColumns": [
{
"conditions": [
{
"conditional": "{trackConditions} === 'slippery'",
"value": "wet"
}
],
"toProperty": "treadType",
"default": "{treadTypeLegacy}"
}
]
}
conditions
array.
fileLocation
¶
Used when the CSV file you want to load is already in VibeIQ. This is commonly used for programmatic use cases (as opposed to manual, CLI use cases). Please see the Files API section of the Admin Console for more information on how to upload files.
loaderConfigurationId
¶
An optional property that allows you to specify a pre-uploaded YAML or JSON file representing the Loader Process
Config. This is useful when you have a common Config that you want to re-use across multiple Loader Process
instances so you do not need to repeatedly define it. Please see the Files API section of the Admin Console for more information on how to upload files.
propertiesToRemove
¶
An optional array property that allows you to specify which columns to remove from the CSV file before loading.
Creating a Loader Process¶
Loading With the CLI¶
If you have a CSV data file on and would like to load it from your local machine, you can use the Contrail CLI to create a Loader Process
.
This requires the Contrail CLI, so please install it before proceeding.
You must have both your data file as CSV and your Config file as YAML available locally. Start a Loader Process
with the following command:
contrail loader-process upload-and-load <File Path> <Config Path>
<File Path>
: specifies the file path to your CSV load file.
<Config Path>
: specifies the file path to your YAML Config file.
Note: loaderConfigurationId
cannot be used from the CLI. You must provide the Config through the CLI every time.
contrail loader-process upload-and-load ./path/to/data.csv ./path/to/config.yml
Loading With the API or SDK¶
If you would like to programmatically create a Loader Process
, you can do so with the API or SDK. Like the CLI, both a
data CSV file and a Config are required. The methodology to provide the CSV file and Config to the API/SDK differs slightly
from the CLI, so please take note of the following steps.
Authentication¶
Please use the Contrail CLI to retrieve an API key or reach out to the support team for assistance.
If you are authenticating with the SDK, please see our SDK reference here: Contrail SDK Login. Your runtime must authenticate with login()
before making any other requests.
If you are using the API, you will need to provide an API key via the X-Api-Key
header as well as an X-Api-Org
header with your VibeIQ environment's identifier key.
-- header 'X-Api-Key: {YOUR-API-KEY}' \
-- header 'X-Api-Org: {YOUR-ORG-SLUG}' \
Upload Your CSV Data File¶
To create Loader Process
with the SDK or API, you must first upload the CSV load file to the system.
This can be done with the Files API. A full reference for this API is available in the Admin Console.
This will look something like this:
const csvFile = Buffer.from(csvContent, "utf-8");
const createdFile = await new Files().createAndUploadFileFromBuffer(
csvFile,
"text/csv",
fileName,
null,
5 * 60 * 60 * 1000 // ttl: Time to Live in Miliseconds
);
console.log("createdFile: ", JSON.stringify(createdFile));
console.log('VibeIQ file ID: ', createdFile.id);
# 1. Create a new `File` entity via the Files API.
curl --location 'https://admin.vibeiq.com/api/files' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--data '{
"contentType": "text/csv",
"fileName": "my-load-file.csv"
}'
# 2. Upload the file binary to the `File` entity.
# The request from step 1 will return an S3 bucket upload link.
# Use that link to upload the file content to S3. The download URL
# from step 1 will then contain the file’s content.
curl --location 'https://prod-contrail-file-content-bucket.s3.amazonaws.com' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--form 'Content-Type="text/csv"' \
--form 'file=@"/path/to/file"'
Pass Config Parameters & Create a Loader Process¶
Using the parameters explained in the above sections, you will need to specify:
- An
entityName
to indicate that you are creating a new entity type instance of type'loader-process'
. - An
object
with theLoader Process
Config parameters in the request body in JSON format.
const process = await entities.create(
{
entityName: 'loader-processe',
object: {
fileLocation: createdFile.id,
loadType: ['ITEM'],
federatedMappings: { ... },
conditionalColumns: { ... },
}
});
console.log('Load has been kicked off. loader-process ID: ', process.id);
curl --location 'https://api.vibeiq.com/prod/api/loader-processes' \
--header 'X-Api-Key: {your-api-key}' \
--header 'X-Api-Org: {your-org-slug}' \
--header 'Content-Type: application/json' \
--data '{
"fileLocation": "someId",
"loadType": ["ITEM"],
"federatedMappings": { ... },
"conditionalColumns": { ... }
}'