Getting Started

The most common integration use case is to import data from another system, like a PLM. We refer to this as "loading data". With our powerful Loader framework, you can load complex data into VibeIQ with ease. This guide will walk you through the basics of that process.

The Loader currently supports:

• Item Data: Global, Seasonal, and Assortment Level
• Project Item Data
• Assortment Data
• Custom Entity Data
• Color Data
• Size Range Template Data

The Loader framework simplifies integrations from your systems to VibeIQ. It abstracts away create-or-update logic within VibeIQ, offloading that work to the VibeIQ platform and reducing network traffic between systems. All data loads run within an asynchronous Loader Process that can be monitored and managed through our API, SDK, or the Contrail CLI.

What Happens When You Load Data¶

When a Loader Process runs, your data goes through a multi-phase pipeline:

1. CSV Parsing: Your CSV file is downloaded and parsed into rows. Column headers become property keys.
2. Validation: The configuration and data are validated against entity-type-specific rules. Invalid configurations stop the load immediately.
3. Preprocessing: A 6-step transformation pipeline runs on your data -- converting booleans, applying federated mappings, removing unwanted columns, converting labels to slugs, evaluating conditional columns, and converting option set display values to keys.
4. Entity Construction: Transformed rows are built into entity objects with proper types. See Property Type Handling for how CSV values are coerced.
5. API Upsert: Entities are looked up by federated ID. Existing entities are updated; new ones are created.

For more detail, see the Preprocessing Pipeline page. For error handling during any phase, see Validation Rules & Error Handling.

Key Terminology and Concepts¶

Loader Process¶

A Loader Process is an entity within VibeIQ that contains information about the load process that was invoked. This includes information like:

• A status indicating whether the load is pending execution, currently running, has completed successfully, or has failed.
• The parameters provided to the Loader Process that were used to perform transformations on the input data.
• A download link to Loader Process's logs.
• Additional metadata about the process.

When a Loader Process is created, the VibeIQ platform enqueues it for execution. When it runs, the provided data in the load file will be inserted into the platform. If desired, this input data can also be manipulated with data transformations during the process. We call this the "Load Config", which allows you to do things like re-map fields, set or re-set values based on other values, and much more.

The full schema for a Loader Process is available in the Admin Console API documentation section.

A Loader Process can be created with the API, SDK, or Contrail CLI. A Loader Process requires:

1. A data load file in CSV format. This is the data you will insert into VibeIQ.
2. A Loader Process Config object. This will instruct the Loader to load the data into specific entity types within VibeIQ, as well as perform specific transformations on the CSV load file. The latter is not required.

Please see the below sections for more explanation on these requirements.

Loader Process Data File¶

CSV Load File¶

Load data must be provided to the Loader framework in tabular CSV format. Generally speaking, each row in this CSV (except the column header row) represents a single instance of the entity type being loaded into. For example, if you are creating Item entities, each row in the CSV would represent a single Item. Columns in the CSV must map to property keys or property labels on the entity type. If the column does not match a key or label on the specified entity type, the row column will be ignored. The Loader logs will indicate this. Please see the Loader Config section to learn more about how you can use it to dynamically map columns to entity properties.

Federated IDs¶

A federatedId is a value you provide which uniquely identifies the entity within an org. Every row in your data must have a federatedId. If an entity with that federatedId does not exist in VibeIQ, it will be created. If it does exist, it will be updated. This should be a unique identifier sourced from the system that originated the data. If a system did not generate it yourself, it's typically best to select an existing column to be your federatedId in concept.

Example CSV Load File¶

Simple example for a common entity type Color:

This load file format would successfully load into a Color entity with these properties. Properties have a key and label. Keys are used to match columns in the CSV to properties on the entity type. Labels are used to display the property in the VibeIQ app UIs.

• federatedId - The external unique identifier for the color
• key: "name", label: "Color Name" - The name of the color onto property with key name.
• key: "hexCode", label: "Hex Code" - The hex code for the color onto property with key hexCode.
• key: "red", label: "Red" - The red value of the color onto property with key red.
• key: "blue", label: "Blue" - The blue value of the color onto property with key blue.
• key: "green", label: "Green" - The green value of the color onto property with key green.

Loading into Specific Property Types¶

For a complete guide on how each property type is parsed from CSV values — including text escaping, number formatting, dates, select properties, object references, formulas, and property levels (Family, Option, All, Override) — see Property Type Handling.

Loader Process Config¶

The Loader Process Config is a JSON object that instructs the Loader on how to load the data in the CSV load file. Using the below properties within it, you can specify how the Loader should handle the data in the CSV file.

For a complete reference of all configuration properties, see the Loader Configuration Reference.

• loadType
• federatedMappings
• conditionalColumns
• fileLocation
• loaderConfigurationId

loadType¶

Specifies the entity type(s) to load the data into. This is an array of strings that can contain one of the following values unless that specific type supports additional types. Currently, only Item supports additional types.

• ITEM: Load data into the Item entity type.
  • PROJECT_ITEM: Load data into the ProjectItem entity type. Must be paired with ITEM. Selecting this will load Item and ProjectItems instances in the same load.
  • ASSORTMENT: Load data into the Assortment entity type. Must be paired with both ITEM. Can also be paired with PROJECT_ITEM. Selecting this will load Item, ProjectItem and AssortmentItem instances in the same load.
• COLOR: Load data into the Color entity type.
• CUSTOM_ENTITY: Load data into the Custom Entity type for a specific typePath.
• SIZE_RANGE_TEMPLATE: Load data into the Size Range Template entity type.

federatedMappings¶

A key-value object that allows you re-map data in the CSV into new columns. This is especially useful when you have a CSV file generated by another system that does not have the same column names as the entity type's property keys or labels.

1. Create new columns by copying all values from an existing column. This can be done by referencing the copied column by its column name. The Loader will create a new column in the CSV with the new name you specify and copy-over the data from the column you are referencing.

2. Create new columns with a static value for all rows.

Copy Column Example:¶

Load File:

Config:

{
  "federatedMappings": {
    "name": "Color_Name",
    "hexCode": "Hex_Code",
    "red": "RED",
    "blue": "BLUE", 
    "green": "GREEN"
  },
  ...
}

Create New Static Column Example:¶

Load File:

Config:

{
  "federatedMappings": {
    "originatingSystem": "My ERP System"
  },
  ...
}

conditionalColumns¶

An array that allows you to create new columns based on conditions. This is especially useful when you need to set a value based on the value of another column or to set a static value. This is a lot like federatedMappings, but provides conditional logic support. Conditions are evaluated by the Loader as JavaScript expressions. If the expression evaluates to true, the value is set. If the expression evaluates to false or is invalid syntax, the Loader moves to the next condition. If no conditions evaluate to true, the default value is set, if provided.

Schema:

interface ConditionalColumns {
    toProperty: string;
    fromProperty?: string;
    conditions?: ConditionalValues[];
    default?: any;  
}

• toProperty: string: The new column that will be written to. If this column already exists, it will be overwritten.
• fromProperty?: string: An optional property used to copy the value of one column to another. This will be ignored if conditions are provided.
• conditions?: ConditionalValues[]: An array of conditions to check and apply their value to the toProperty. The first condition that evaluates to true will be applied. If a default is provided, that value will be used.
  • conditional: A JavaScript expression that will be evaluated. If it evaluates to true, the corresponding value will be set. Within the conditional's expression, you can reference other columns in the load file using curly braces { }.
  • value: The value to set if the condition evaluates to true.
• default?: string: The value to set if no conditions evaluate to true.

Set Column Value Based on Condition:¶

This example checks if column value trackConditions is equal to 'slippery'. If it is, the value of the new column treadType will be set to 'wet'. If not, the value of the new column treadType will be set to the value of the column treadTypeLegacy.

{
  "conditionalColumns": [
    {
      "conditions": [
        {
          "conditional": "{trackConditions} === 'slippery'",
          "value": "wet"
        }
      ],
      "toProperty": "treadType",
      "default": "{treadTypeLegacy}"
    }
  ]
}

fileLocation¶

Used when the CSV file you want to load is already in VibeIQ. This is commonly used for programmatic use cases (as opposed to manual, CLI use cases). Please see the Files API section of the Admin Console for more information on how to upload files.

loaderConfigurationId¶

An optional property that allows you to specify a pre-uploaded YAML or JSON file representing the Loader Process Config. This is useful when you have a common Config that you want to re-use across multiple Loader Process instances so you do not need to repeatedly define it. Please see the Files API section of the Admin Console for more information on how to upload files.

propertiesToRemove¶

An optional array property that allows you to specify which columns to remove from the CSV file before loading.

Creating a Loader Process¶

Loading With the CLI¶

If you have a CSV data file on and would like to load it from your local machine, you can use the Contrail CLI to create a Loader Process. This requires the Contrail CLI, so please install it before proceeding.

You must have both your data file as CSV and your Config file as YAML available locally. Start a Loader Process with the following command:

contrail load upload-and-load <File Path> <Config Path>

<File Path>: specifies the file path to your CSV load file.

<Config Path>: specifies the file path to your YAML Config file.

Note: loaderConfigurationId cannot be used from the CLI. You must provide the Config through the CLI every time.

contrail load upload-and-load ./path/to/data.csv ./path/to/config.yml

Loading With the API or SDK¶

If you would like to programmatically create a Loader Process, you can do so with the API or SDK. Like the CLI, both a data CSV file and a Config are required. The methodology to provide the CSV file and Config to the API/SDK differs slightly from the CLI, so please take note of the following steps.

If you prefer following along with a Postman collection, you may download one here: Create-Load-Process-Example.postman_collection.json

Authentication¶

Please use the Contrail CLI to retrieve an API key or reach out to the support team for assistance.

If you are authenticating with the SDK, please see our SDK reference here: Contrail SDK Login. Your runtime must authenticate with login() before making any other requests.

If you are using the API, you will need to provide an API key via the X-Api-Key header as well as an X-Api-Org header with your VibeIQ environment's identifier key.

-- header 'X-Api-Key: {YOUR-API-KEY}' \
-- header 'X-Api-Org: {YOUR-ORG-SLUG}' \

Upload Your CSV Data File¶

To create Loader Process with the SDK or API, you must first upload the CSV load file to the system. This can be done with the Files API. A full reference for this API is available in the Admin Console.

This will look something like this:

const csvFile = Buffer.from(csvContent, "utf-8");
const createdFile = await new Files().createAndUploadFileFromBuffer(
  csvFile,
  "text/csv",
  fileName,
  null,
  5 * 60 * 60 * 1000 // ttl: Time to Live in Miliseconds
);
console.log("createdFile: ", JSON.stringify(createdFile));
console.log('VibeIQ file ID: ', createdFile.id);

# 1. Create a new `File` entity via the Files API. This is a VibeIQ entity that stores information about a File, including its location.
curl --location 'https://api.vibeiq.com/api/files' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--data '{
      "contentType": "text/csv",
      "fileName": "my-load-file.csv",
      "ownedByReference": null,
      "ttl": 86400000
  }'

# 2. Upload the file binary to the `File` entity's location.
# The request from step 1 will return a JSON response with a property `uploadPost`. This contains the URL and form
# input parameters and values for this request.

# Make a POST request to the URL stored in `uploadPost.url` and use values in `fields` as the form parameters.
curl --location 'https://s3.amazonaws.com/prod-contrail-file-content-bucket' \
--header 'X-Api-Key: {{your-api-key}}' \
--header 'X-Api-Org: {{your-org-slug}}' \
--form 'Content-Type="text/csv"' \
--form 'key="some-key"' \
--form 'bucket="prod-contrail-file-content-bucket"' \
--form 'X-Amz-Algorithm="AWS4-HMAC-SHA256"' \
--form 'X-Amz-Credential="ASIAXGPNJ353B6QX5X3J/20250527/us-east-1/s3/aws4_request"' \
--form 'X-Amz-Date="20250527T150153Z"' \
--form 'X-Amz-Security-Token="some-security-token' \
--form 'Policy="some-policty"' \
--form 'x-amz-meta-fileName="my-load-file2.csv"' \
--form 'x-amz-meta-uploaderId="RknIRsdH5kt0RmMB"' \
--form 'x-amz-meta-orgId="7jyoFN2gzgzoom34"' \
--form 'x-amz-meta-contentType="text/csv"' \
--form 'x-amz-meta-fileId="sd2Og8OYbNVPvJyJ"' \
--form 'file=@"/path/to/your/local/file"'   

Pass Config Parameters & Create a Loader Process¶

Using the parameters explained in the above sections, you will need to specify:

• An entityName to indicate that you are creating a new entity type instance of type 'loader-process'.
• An object with the Loader Process Config parameters in the request body in JSON format.
• A fileLocation with the ID of the File entity you created in the previous step. This tells the Loader Process where to find the data CSV data file to load.
  • If using the SDK, this is the id property of the File entity returned by createAndUploadFileFromBuffer().
  • If using the API, this is the id property of the File entity returned by POST /api/files.

const process = await entities.create(
  {
    entityName: 'loader-processe',
    object: {
      fileLocation: createdFile.id,
      loadType: ['ITEM'],
      federatedMappings: { ... },
      conditionalColumns: { ... },
    }
});

console.log('Load has been kicked off. loader-process ID: ', process.id);

curl --location 'https://api.vibeiq.com/prod/api/loader-processes' \
--header 'X-Api-Key: {your-api-key}' \
--header 'X-Api-Org: {your-org-slug}' \
--header 'Content-Type: application/json' \
--data '{
  "fileLocation": "{id of File entity created in step 1}",
  "loadType": ["ITEM"],
  "federatedMappings": { ... },
  "conditionalColumns": { ... }
}'