HPE Shadowbase Online Loader (SOLV) and HPE Shadowbase ETL Overview
Before a database is to be of any use, it must first be loaded with data. HPE Shadowbase Online Loader (SOLV) and HPE Shadowbase ETL are utility products designed to perform this task, providing online and offline loading (or unloading) of a source database into a target database. Both of these products are included with each Shadowbase license. Shadowbase SOLV loading and Shadowbase ETL may also be licensed separately if the other data replication features of Shadowbase are not required.
HPE Shadowbase (SOLV)
The HPE Shadowbase Online Loading (SOLV) utility provides online loading of a source database into a target database, meaning that the source and/or target databases can be open for reading and updating while the load (or copy) occurs. In contrast to other loading products, there is no need to take either the source or target databases offline, making them inaccessible to applications, while the load (or unload) occurs.
The HPE Shadowbase SOLV utility can load audited and non-audited HPE NonStop Enscribe source files and HPE NonStop SQL tables into any target environment and database combination supported by the Shadowbase line of data replication products (e.g., HPE NonStop Enscribe or HPE NonStop SQL targets, or other server targets such as Oracle, SQL Server, DB2, and MySQL). The Shadowbase SOLV utility can perform the load while regular replication is also taking place, consuming the source database changes and merging them with the data being loaded to keep the target fully synchronized with the source while the load takes place.
Key Features
- Initial and refresh target data-loading utility
- Supports point-in-time bulk loading of source to target
- Allows for extract, transform, filter, and cleanse of the data being loaded
- Integrates with the Shadowbase replication engine to keep target synchronized during and after the load completes
- Can use the same (or different) conversion routines for replication as are used for loading
- Replicated events are consumed/merged with the loading activity, so there is no queue of changes to drain after the load completes
- Dynamically adjusts loading activity to efficiently utilize available resources
- Source and target can be opened for read-write activity while the load or refresh occurs
- SOLV does not write data to interim disk queues during the load, making the loading process much more efficient than other loading methods
HPE Shadowbase ETL Toolkit
The HPE Shadowbase ETL toolkit uses and extends the SOLV loading capabilities to allow for reading and injecting events from flat files into the Shadowbase replication engine for processing, as well as producing flat files of database data or database change events that can then be subsequently processed by an ETL tool. The HPE Shadowbase ETL Toolkit is used to extract database changes or initial load data into flat files for subsequent ETL loading into a data warehouse using comma-separated value (CSV), fixed-position, and tab-delimited formats. It is particularly useful for feeding high-volume quantities of loading data into a data warehouse, and when feeding row-based data into columnar databases.
More specifically, for certain applications, you may want to use an existing vendor’s ETL loading utility to load either initial data, or change data, into a data warehouse. The Shadowbase ETL toolkit allows a Shadowbase user to perform these tasks. For example, when coupled with the Shadowbase SOLV loader, the Shadowbase ETL toolkit can be used to extract select data from a source database, transform/cleanse it into the proper format, and save it into flat files that your ETL loading utility can understand and load into a target database environment not directly supported by Shadowbase replication. Using the Shadowbase ETL Toolkit, you can also periodically extract source database change data from the source database’s audit trail (change log), and process it into a flat file format that can be incrementally loaded into the target environment using a micro-batch incremental update approach to keep the target environment synchronized with the source.
Key Features
- Trickle-feed replicated-events loading utility
- Trickle-feed replicated-events UN-loading utility
- Snapshot point-in-time bulk loading utility
- Snapshot point-in-time bulk UN-loading utility
- Allows for extract, transform, filter, and cleanse of the data to be loaded or unloaded
- Reads or generates comma-separated-value (CSV) and fixed-position flat files for the data to be loaded or unloaded
- Source or target can be open for read-write activity while the load or unload occurs
- Leverages the Shadowbase Online Loader (SOLV) features for ETL processing
Related Pages
Related Case Studies:
- Vodacom’s One Year Recovery
- A Major ISP Migrates from Sybase to HPE NonStop with No Downtime
- Prescription Drug Fraud Prevention — An HPE Shadowbase Data Integration Case Study