COVID-19 Update — We are available to assist you during these trying times. Please contact us for assistance.

HPE Shadowbase Enabling Technology

Stock photo of cogsSections:


Audited and Non-Audited Environments

For HPE NonStop environments, HPE Shadowbase replicates both audited (TMF-protected), as well as non-audited, Enscribe and SQL data.

There are two audited and non-audited replication methods:

  1. Load
    • Use HPE Shadowbase SOLV to read the source data and create a point-in-time “snap-shot” load of that data into the target database. By default, SOLV replicates the entire record (for an Enscribe file), row (for an SQL a table), or a record/row key-range.
    • Note: This method will not include subsequent changes made to the source data (however, real-time replication and/or file chasing is available to replicate subsequent changes).
  2. Real-time Replication
    • Use HPE Shadowbase Replication to read the source system’s transactional Data Manipulation Language (DML) events (an insert, update, or delete) from the TMF Audit Trail, and replicate the changes into the target database as a transaction.
    • Note: Enscribe file and SQL table DML changes are only captured in the TMF Audit Trail when the application has started or is running under a TMF transaction (e.g., a BEGINTRANSACTION) call or a BEGIN WORK statement has been successfully executed), and the file or table’s AUDIT flag is set to “ON.”

For Audited Environments

Either method can be used to replicate the source data to the target database. For example, SOLV can copy the source data into the target database to create an initial copy of the source data in the target database. Subsequently, Shadowbase Real-time Replication can read the audit trail for the source data changes and replicate them into the target database to keep the target database synchronized.

For Non-Audited Environments

Load

Shadowbase SOLV supports snap-shot replication for loading non-audited Enscribe file and SQL table data into a target database.

Real-time Replication

Shadowbase software cannot replicate individual DML changes made to a non-audited file or table (with one notable exception, see below). In order to replicate the DML changes for non-audited files or tables, the HPE NonStop AutoTMF product is required (and must be properly configured for the file or table).

Exception: For certain application-maintained non-audited “log” files or tables, where the DML changes are always INSERTs (appends) at the current end-of-file (EOF), Shadowbase SOLV can periodically “chase” the file or table’s EOF and replicate the records/rows that are appended into the source file (or table) to the target database. In this specific situation, HPE NonStop AutoTMF is not needed for Shadowbase software to replicate the inserted/appended data.

For more information, see HPE Shadowbase SOLV Manager (below).


HPE Shadowbase SOLV Snapshot Loading (SOLV)

Audited Enscribe Files and SQL Tables

HPE Shadowbase SOLV Snapshot Loading, also known as HPE Shadowbase point-in-time loading, is shown in Figure 1. This technique reads and replicates TMF-audited or non-audited source Enscribe files and SQL tables into a target database. When the source file/table is audited, Shadowbase software uses real-time replication to replicate subsequent changes made to the source file/table to the target database to keep it synchronized.

Diagram of Sample SOLV snapshot loading architecture (please see the paragraph that starts with "HPE Shadowbase SOLV Snapshot Loading as" for a full image description).

Figure 1 — HPE Shadowbase SOLV Snapshot Loading

Non-audited Enscribe Files and SQL Tables

When the source file/table is non-audited, HPE Shadowbase SOLV Snapshot Loading can be similarly used to copy the source file/table data into the target database. However, since the source file/table is non-audited, Shadowbase software will not see the subsequent changes made to the source file or table (as they do not appear in the TMF audit trail). To pick up subsequent changes made to the non-audited source file or table, another SOLV Snapshot Load would need to be run (see below for alternative methods to replicate the subsequent changes).


HPE Shadowbase SOLV Manager (SOLVMGR)

HPE Shadowbase SOLV Manager (SOLVMGR, previously called “File Chaser”) can be used to replicate non-audited file/table changes. This approach works for application environments that create and maintain a log file containing the change events to be replicated (similar to an audit trail of the change events).

For Non-audited Enscribe Files

SOLVMGR and SOLV can be configured to “chase” the EOF of a file, and replicate the newly appended data into the target database in near real-time. SOLVMGR acts as the manager for SOLV and restarts SOLV after it stops for any reason. For example, SOLVMGR completes a load by hitting EOF or when the SOLV process prematurely stops due to system issues. When run in this mode, SOLV replicates all the data in the file up to EOF and stops; SOLVMGR sets a delay timer, delays, and then re-runs SOLV to pick up and replicate any newly-appended data in the file using an “EOF chaser” cycle. (the delay timer is configurable and can be very short.)

For Applications that Create a Sequence of Files to be Replicated

SOLVMGR and SOLV can replicate the file sequence. SOLVMGR uses special logic that looks at the file creation information for the sequence of files to know when to roll from one file to the next in sequence. For example, if the application has a fileset called FILEnnnn, where the application fills FILE0000 first, then it fills FILE0001, followed by filling FILE0002, etc., SOLVMGR will watch for the application rolling to the next file in sequence, and then instruct SOLV to chase the newly appended data in that file.

SOLVMGR (SOLV Manager) Drill Down

Figure 2 illustrates SOLVMGR. An application makes database changes to non-audited Enscribe files and SQL tables, and generates log files of these changes (business transaction data). SOLVMGR periodically launches SOLV loaders, which read the log files in sequence from a saved “restart point” as each successive SOLV loader runs. This step allows for optional and periodical transformation and maintenance of the application-generated log files/tables. The SOLV loaders inject the log data into the replication stream, which is made available to any Shadowbase target environment/data replication system for processing.

Diagram of Shadowbase file chaser architecture (please see the paragraph that starts with "Figure 1 illustrates" for a full image description).

Figure 2 — HPE Shadowbase SOLVMGR Architecture

For applications that create and maintain their own log files/tables, SOLVMGR can be configured to read these application-generated log files/tables, optionally transform the events, and inject the events into the replication stream to the target environment, as the application appends data to the log files/tables. This technique avoids the need for the source database to be audited, while helping to ensure that such changes are also replicated.

During configuration, SOLVMGR can be given a range of sub-volumes and application-maintained files/tables to monitor for changes. This is useful, for example, when the application creates a sequenced list of log files and periodically rolls through them as it fills them. As the application appends change-data into these log files/tables, SOLVMGR will see those changes, and periodically read/inject them into the Shadowbase data replication engine for processing.


HPE Shadowbase Online Loading and Verification* Utility for OSS Regular Files (SOLVUTIL)

Loading, Copying, Validating, and File Chasing OSS Regular Files

SOLVUTIL, also known as Shadowbase SOLV for MX (or SOLVMX), is available for loading, copying, validating, and file chasing Open System Services (OSS) Regular Files (or “Flat Files”) that are added in append mode, with no subsequent updates to the added data written into them. (Note: the OSS Regular Files are never TMF-audited.)

Figure 3 shows an application in “append mode” (data is periodically added to the EOF) writing data to an OSS Regular File. When SOLVUTIL sees that the file has changed, it then chases the EOF of the file, and in near-real time replicates that data to the target OSS Regular File. SOLVUTIL replicates all the data in the file up to the EOF, sets a delay timer, delays, and then checks for and replicates all the newly appended data (since the old EOF). The delay timer is configurable.

SOLVUTIL Diagram

Figure 3 — HPE Shadowbase SOLVUTIL Snapshot Loading

SOLVUTIL requires Expand when replicating between systems. (Contact Gravic if you are interested in TCP/IP support.) SOLVUTIL runs in the Guardian space, and periodically checks if any of the files it is monitoring have been modified. When it finds a modified file, it performs end-of-file (EOF) chasing to retrieve the OSS data. As SOLVUTIL performs its periodic chasing, it recognizes new data and replicates it to the target file. SOLVUTIL can track updates to multiple files and maintain their sequence. SOLVUTIL has numerous parameters, is configured with TACL environment variables, and can handle massive data volumes.

Compare Two OSS Regular Files

SOLVUTIL can also compare two OSS Regular Files. It confirms that the files match or lists the offset of the first byte of the mismatch.

*Note: SOLVUTIL Verification requires an HPE Shadowbase Compare license.

Please reference: SOLVUTIL Manual


HPE NonStop AutoTMF: TMF Transaction Protection

HPE NonStop AutoTMF is a product available from HPE which utilizes non-intrusive intercept technology to automatically invoke TMF transaction protection for applications that do not use TMF when updating data files. This technology enables non-audited applications running on a NonStop system to access and create audited Enscribe files and to perform I/O against those files without program modification or recompilation.

AutoTMF software dynamically determines effective and efficient TMF transaction boundaries and automatically consolidates multiple database updates, ensuring low overhead. There is no need to make any source code changes to programs, or even to recompile them, since AutoTMF works with existing application object files. AutoTMF–enabled applications use audited operations, which adds performance optimization benefits, such as advanced file buffering and elimination of the block-splitting overhead associated with insertions to a file. Another benefit is improved data consistency, which keeps the base file and its alternate key files consistent with each other regardless of the outcome of the transaction or if there is a file system failure.

Want to use Shadowbase software with your non-audited application?
Then follow these three easy steps to replicate your application’s individual DML activity:

  1. Use AutoTMF to quickly and easily enable non-audited data to be TMF-audited.
  2. Once AutoTMF is installed and configured for your applications, turn the TMF file/table AUDIT attribute to ON, and all subsequent file/table operations will be captured by TMF.
  3. Shadowbase software can then be rapidly implemented to identify and replicate the changes from the TMF audit trails in real-time.

HPE NonStop AutoSYNC

Non-database Files

HPE NonStop AutoSYNC is a product available from HPE which replicates non-database files such as program object code files, command scripts, OBEY files, etc., to a target environment, usually for NonStop to NonStop business continuity purposes. It can be configured to periodically check for files that need to be replicated so that the target will always have an up-to-date copy in case of failover. AutoSYNC is a powerful addition to a business continuity solution when non-database files need to be replicated.