HPE Shadowbase Extract, Transform, and Load (ETL) Toolkit

Extract Database Changes or Initial Load Data into Flat Files

The HPE Shadowbase ETL Toolkit provides online and offline loading (or unloading) of a source database into a target database.

Adobe

1. Choose a supported format

HPE Shadowbase Application Integration provides a suite of adapters to easily integrate the source data with the target environment.  Available adapters include:

  • Integrating with KAFKA
  • Integrating with IBM MQ environments
  • JSON (JavaScript Object Notation) output format
  • Flat-files (CSV, fixed position, tab-delimited)
Shutterstock/Miles Holenstein

2. Feed high-volume data quantities into a data warehouse, and feed row-based data into columnar databases

Using the Toolkit, source database file and table data can be extracted and stored in a flat-file.

Optionally, change data can also be periodically extracted from the audit trail.

The extracted data is processed into a flat file format for incremental loading into the target environment using a bulk or micro-batch update approach (provided by the target environment) to keep the target synchronized with the source.

The Toolkit can also be used to extend the SOLV loading capabilities to allow for reading and injecting events from flat files into the Shadowbase replication engine for processing.

This ability is useful, for example, to ingest into Shadowbase application-generated logs of events that otherwise are not readily available via an audit trail.

iStock

3. Extend SOLV using the Toolkit

The Toolkit can also be used to extend the SOLV loading capabilities to allow for reading and injecting events from flat files into the Shadowbase replication engine for processing:

  • Extract select data from a source database
  • Transform/cleanse the data into the proper format
  • Save the data into flat files understood by an ETL utility
  • Load the files into a target database environment not directly supported by Shadowbase replication

Key Features

Snapshot point-in-time bulk loading and unloading utility
Trickle-feed replicated events loading and unloading utility
Extract, transform, filter, and cleanse the data to be loaded or unloaded
Read or generate comma-separated value (CSV) and fixed-position flat files for the data to be loaded or unloaded
Keep the source or target open for read-write activity while the load or unload occurs