Learning › transformer
Edit this page

iModel Transformation and Data Exchange

The @itwin/imodel-transformer package provides some classes that implement Extract, Transform, and Load (ETL) functionality:

The above classes contain the lower-level functionality required to implement transformation and data exchange services. These classes should be considered a framework and not confused with the actual packaged and deployed services that use the framework.

IModelExporter

The IModelExporter and IModelExportHandler base classes are used when the source data in an ETL workflow is contained within an iModel.

While it is possible to export data from an iModel using the standard IModelDb API, the IModelExporter and IModelExportHandler base classes offer the following capabilities:

Below is an example of using IModelExporter and IModelExportHandler to export all Code values from an iModel:


import { Code, CodeSpec } from "@itwin/core-common";
import { Element, IModelJsFs as fs, IModelDb, SnapshotDb } from "@itwin/core-backend";
process.env.TRANSFORMER_NO_STRICT_DEP_CHECK = "1"; // allow this monorepo's dev versions of core libs in transformer
import { IModelExporter, IModelExportHandler } from "@itwin/imodel-transformer";

/** CodeExporter creates a CSV output file containing all Codes from the specified iModel. */
class CodeExporter extends IModelExportHandler {
  public outputFileName: string;

  /** Initiate the export of codes. */
  public static async exportCodes(iModelDb: IModelDb, outputFileName: string): Promise<void> {
    const exporter = new IModelExporter(iModelDb);
    const exportHandler = new CodeExporter(outputFileName);
    exporter.registerHandler(exportHandler);
    await exporter.exportAll();
  }

  /** Construct a new CodeExporter */
  private constructor(outputFileName: string) {
    super();
    this.outputFileName = outputFileName;
  }

  /** Override of IModelExportHandler.onExportElement that outputs a line of a CSV file when the Element has a Code. */
  public override onExportElement(element: Element, isUpdate: boolean | undefined): void {
    if (!Code.isEmpty(element.code)) { // only output when Element has a Code
      const codeSpec: CodeSpec = element.iModel.codeSpecs.getById(element.code.spec);
      fs.appendFileSync(this.outputFileName, `${element.id}, ${codeSpec.name}, ${element.code.value}\n`);
    }
    super.onExportElement(element, isUpdate);
  }
}

IModelImporter

The IModelImporter base class is used when the target in an ETL workflow is an iModel.

While it is possible to import data into an iModel using the standard IModelDb API, the IModelImporter class offers the following capabilities:

IModelImportOptions.autoExtendProjectExtents

IModelImportOptions.autoExtendProjectExtents provides different options for handling the projectExtents of the target iModel. See the following for more information about projectExtents:

autoExtendProjectExtents = false

This setting should be used when the target iModel projectExtents are being set directly. For example:

  • If the target iModel projectExtents will be the same as the source iModel, then it can just be copied over.
  • If the target iModel projectExtents are known ahead of time, then it can be directly set.

autoExtendProjectExtents = true

This setting causes the target iModel projectExtents to be extended to include the range box of every element that is imported. This includes potential outliers (one/few elements that are located far away from the main collection of elements). Outliers tend to suggest a user modeling problem or a Connector problem, but it is difficult for a program to know for sure what the intent was. This setting assumes every Element is there for a reason.

autoExtendProjectExtents = { excludeOutliers: true }

This setting causes the projectExtents to be extended to include the range box of every element that is imported except for outliers. In this case, outliers are assumed to be a mistake and IModelImporter tries to detect them using fuzzy logic from the IModelDb.computeProjectExtents method in order to exclude them from the projectExtents calculation.

Either of the non-false autoExtendProjectExtents options are useful for consolidation cases or filtering cases where the target iModel will have different optimal projectExtents than the source iModel(s).

IModelCloneContext

The IModelCloneContext class provides the core cloning capability required for iModel transformation. It also maintains the sourceId --> targetId mapping which is required to successfully clone Entity instances from the source iModel into the target iModel. iModel entities are highly related to each other. Therefore, cloning an entity means copying a graph of objects and remapping their source references (Ids) to other target entities.

IModelTransformer

The IModelTransformer base class is used when the source and target in an ETL workflow are both/different iModels and some sort of data transformation is needed in the middle. An instance of IModelTransformer holds instances of IModelExporter, IModelImporter, and IModelCloneContext. This means that customization is possible at the export stage, the transformation stage, and the import stage of the overall ETL process.

Potential transformations include:

  • Cloning - copying an entity from the source and remapping its internal Ids for the target
  • Filtering - excluding data from the target that is contained in the source
  • Augmenting - generating data during transformation for the target that is not part of the source
  • Schema Mapping - mapping classes and properties to a new schema during transformation
  • Change Squashing - each iModel has its own change ledger, so multiple changesets from the source could be squashed into a single changeset to the target

Logging

With batch processes like iModel transformation and data exchange, logging is often the only way to figure out what is actually happening. The following logger categories are provided for use with the Logger:

Implementing iModel branching workflows through the transformer

Using the transformer as a framework, it is possible to implement the necessary operations for a branching workflow, where branch iModels from a master iModel form a tree-like change history as branches synchronize at distinct points to transfer data to the master. The implementation with examples and terminology by the transformer is elaborated in the branching iModels article.

ETL examples

More samples of using the transformer to export iModels or subsets of iModels to different formats can be found in the ETL (Extract-Transform-Load) samples repository

Last Updated: 04 April, 2023