Image Loading

Introduction

The SDK supports creating script and applications to efficiently load large amounts of images from cloud or local directories into the VibeIQ platform.

This process encompasses tasks such as managing file uploads, creating or updating assets, and handling content associations. We can even link assets to specific items!

Covered Concepts

• Authenticating using Contrail SDK.
• Storing a list of existing images in a JSON file.
• Uploading files with Contrail SDK.
• Entity management (Assets, Items, Content-Holders) with Contrail SDK.
• Updating and linking entities based on specific criteria.

End Product of this Tutorial

Successful execution of the script will result in the images being uploaded to the VibeIQ platform, associated with relevant assets, and linked to corresponding items.

Authentication

App Authentication

If your image load is running as part of an app action, you do not need to explicitly authenticate, and can skip this section. App actions are authorized automatically during runtime.

To authenticate with Contrail, you can use API key and organization slug.

image_load_script.ts

import { login } from "@contrail/sdk";
await login({ orgSlug: process.env.ORG_SLUG, apiKey: process.env.API_KEY });

Skipping Processed Images

In case there's a need to keep track of previously-processed images and skip them in subsequent runs, we can incorporate a history log within the loadedfiles.json file.

image_load_script.ts

const loadedJSONFileName = path.resolve(
  path.join(__dirname, "loadedfiles.json")
);
let loadedJSONFileData: any = await fs.readFile(loadedJSONFileName);
let loadedJSONFileParsedData = JSON.parse(loadedJSONFileData);

const imagesToUpload = [];
const directoryPath = path.resolve("<PATH>");

//passing directoryPath and callback function
const files = await fs.readdir(directoryPath);
files.forEach((file) {
  if (file == ".DS_Store") return;

  if (!loadedJSONFileParsedData[file]?.image_loaded?.id) {
    imagesToUpload.push({
      name: file,
    });
  } else {
    console.log(`Skipping ${file} image is already processed.`);
  }
});

Loading New Images

After filtering out non-processed images in the imagesToUpload array, we can now import these images as files into the VibeIQ platform using the following logic. Note the usage of Files.createAndUploadFileFromBuffer method.

image_load_script.ts

let allFiles = [];
let i = 0;

let promises = [];
for (const item of imagesToUpload) {
  const promise = limit(async () {
    let done = false;
    let retry = 0;
    while (!done && retry < 3) {
      try {
        const arrayBuffer = await fs.readFile(`<PATH>/${item.name}`);

        const imageBuffer = Buffer.from(new Uint8Array(arrayBuffer));

        const fileName = item.name;
        const contentType = getContentType(fileName);
        const createdFile = await new Files().createAndUploadFileFromBuffer(
          imageBuffer,
          contentType,
          fileName,
          null,
          24 * 60 * 60 * 1000
        );

        if (!loadedJSONFileParsedData[fileName]) {
          loadedJSONFileParsedData[fileName] = {};
        }
        console.log(`${i}/${imagesToUpload.length}`);
        i += 1;
        loadedJSONFileParsedData[fileName].image_loaded = {
          id: createdFile.id,
          fileName,
        };
        fs.writeFile(
          loadedJSONFileName,
          JSON.stringify(loadedJSONFileParsedData, null, 2)
        );
        allFiles.push(createdFile);
        done = true;
      } catch (e) {
        console.log(
          `Failed to download image from Box and Upload to file ${__filename}: `,
          e
        );
        retry++;
        await new Promise((resolve) => setTimeout(resolve, 1000));
      }
    }
  });
  promises.push(promise);
}
await Promise.all(promises);

console.log("Uploading files to content.");

Managing Assets

Following the file upload, we'll need to create or update assets for each image based on the federatedId, which is derived from the filename, using the provided logic. Without associating a file with an asset, it will be difficult to include it in as part of an item, or on a board or showcase.

image_load_script.ts

i = 0;
promises = [];
for (const file of itemsToUpdate) {
  const promise = limit(async () {
    try {
      const fileId = file["id"];
      const fileName = file["fileName"];

      let asset;

      const assetArr = await new Entities().get({
        entityName: "asset",
        criteria: { federatedId: fileName },
      });

      if (assetArr.length) {
        asset = assetArr[0];
        console.log("Existing asset found");

        await new Entities().update({
          entityName: "asset",
          id: asset.id,
          object: {
            copyFileId: fileId,
          },
        });

        console.log(`Updated asset ${asset.id} with copyFileId ${fileId}`);
      } else {
        console.log("Creating asset...");
        asset = await new Entities().create({
          entityName: "asset",
          object: {
            copyFileId: fileId,
            federatedId: fileName,
          },
        });
        console.log("Yay! Asset Created", fileId, fileName);
      }
    } catch (e) {
      console.log(`Failed to upload content for ${file?.fileName}: `, e);
    }
  });
  promises.push(promise);
}

Linking to Items

Optionally, we can link an asset to an item. If the file name includes item information, we can proceed to link assets to existing items or create new contents, adhering to specific criteria, as outlined in the provided sample code.

image_load_script.ts

const arrFileNameSplit = fileName.split("_");
const itemReferenceNo = arrFileNameSplit[0];
const identifier = arrFileNameSplit[1];

let criteria = {};
if (!parseInt(identifier)) {
  criteria = {
    workNumber: itemReferenceNo,
    colorwayLetter: identifier,
  };
} else {
  criteria = { articleNumber: itemReferenceNo };
}

const itemResult = await new Entities().get({
  entityName: "item",
  criteria,
});

if (!itemResult.length) {
  console.log("Item Not found for file name: ", fileName);
  loadedJSONFileParsedData[fileName].item_not_found = true;
  console.log(`${i}/${itemsToUpdate.length}`);
  i += 1;
  await writeFile(
    loadedJSONFileName,
    JSON.stringify(loadedJSONFileParsedData, null, 2)
  );
  return;
}

const itemId = itemResult[0].id;
let itemRef = `item:${itemId}`;
let contentholdercontent = await new Entities().get({
  entityName: "content-holder-content",
  criteria: { contentId: asset.contentId },
});

if (!contentholdercontent.length) {
  console.log("Creating content-holder-content");

  await new Entities().create({
    entityName: "content-holder-content",
    object: {
      contentHolderReference: itemRef,
      contentId: asset.contentId,
    },
  });

  console.log(`Linked content ${asset.contentId} with item id ${itemId}`);
} else {
  console.log("Existing content holder content found");

  if (contentholdercontent[0].contentHolderReference !== itemRef) {
    await new Entities().update({
      entityName: "content-holder-content",
      id: contentholdercontent[0].id,
      object: {
        contentHolderReference: itemRef,
      },
    });

    console.log(
      `Updated content holder content ${asset.contentId} with contentHolderReference ${itemRef}`
    );
  }
}

const primaryContent = itemResult[0].primaryViewableId
  ? await new Entities().get({
      entityName: "content",
      id: itemResult[0].primaryViewableId,
    })
  : undefined;

const existingIdentifier = primaryContent
  ? primaryContent.fileName?.split("_")[1]
  : undefined;
if (isItemUpdateAllowed(existingIdentifier, identifier)) {
  console.log("Update Item");
  await new Entities().update({
    entityName: "item",
    id: itemId,
    object: {
      primaryViewableId: asset.contentId,
    },
  });

  console.log(
    `Updated item ${itemId} with primaryViewableId ${asset.contentId}`
  );
} else {
  console.log(
    `Skipping item primaryViewableId update existing file: ${primaryContent?.fileName} and new file: ${fileName} .`
  );
}
if (!loadedJSONFileParsedData[fileName]) {
  loadedJSONFileParsedData[fileName] = {};
}
loadedJSONFileParsedData[fileName].item_updated = true;
console.log(`${i}/${itemsToUpdate.length}`);
i += 1;
await writeFile(
  loadedJSONFileName,
  JSON.stringify(loadedJSONFileParsedData, null, 2)
);

Tutorial Files

The complete file created in this tutorial can be downloaded here.