HPE Shadowbase Enabling Technology

Audited and Non-audited Environments

For HPE NonStop environments, HPE Shadowbase replicates both audited (TMF-protected), as well as non-audited, Enscribe and SQL data.

There are two audited and non-audited replication methods:

  1. Load

    • Use HPE Shadowbase SOLV to read the source data and create a point-in-time “snap-shot” load of that data into the target database. By default, SOLV replicates the entire record (for an Enscribe file), row (for an SQL a table), or a record/row key-range.
    • Note: This method will not include subsequent changes made to the source data (however, real-time replication and/or file chasing is available to replicate subsequent changes).
  2. Real-time Replication

    • Use HPE Shadowbase Replication to read the source system’s transactional Data Manipulation Language (DML) events (an insert, update, or delete) from the TMF Audit Trail, and replicate the changes into the target database as a transaction.
    • Note: Enscribe file and SQL table DML changes are only captured in the TMF Audit Trail when the application has started or is running under a TMF transaction (e.g., a BEGINTRANSACTION) call or a BEGIN WORK statement has been successfully executed), and the file or table’s AUDIT flag is set to “ON.”
    • The HPE AutoTMF facility is used to turn non-audited applications/data into audited applications/data. This allows Shadowbase to then read the DML changes from the TMF audit trail and replicate them. Please read this important article on the why TMF protection is so much more efficient and effective than not using TMF protection, and how it actually improves overall file system performance and efficiency: Only the Truth: Debunking TMF NonStop Data Protection Myths.

For Audited Environments

Either method can be used to replicate the source data to the target database. For example, SOLV can copy the source data into the target database to create an initial copy of the source data in the target database. Subsequently, Shadowbase Real-time Replication can read the audit trail for the source data changes and replicate them into the target database to keep the target database synchronized.

For Non-audited Environments


Shadowbase SOLV supports snap-shot replication for loading non-audited Enscribe file and SQL table data into a target database.

Real-time Replication

Shadowbase software cannot replicate individual DML changes made to a non-audited file or table (with one notable exception, see below). In order to replicate the DML changes for non-audited files or tables, the HPE NonStop AutoTMF product is required (and must be properly configured for the file or table).

Exception: For certain application-maintained non-audited “log” files or tables, where the DML changes are always INSERTs (appends) at the current end-of-file (EOF), Shadowbase SOLV can periodically “chase” the file or table’s EOF and replicate the records/rows that are appended into the source file (or table) to the target database. In this specific situation, HPE NonStop AutoTMF is not needed for Shadowbase software to replicate the inserted/appended data.

For more information, see HPE Shadowbase SOLV Manager (below).


HPE Shadowbase SOLV Snapshot Loading (SOLV)

Audited Enscribe Files and SQL Tables

HPE Shadowbase SOLV Snapshot Loading, also known as HPE Shadowbase point-in-time loading, is shown in Figure 1.

Non-audited Enscribe Files and SQL Tables

When the source file/table is non-audited, HPE Shadowbase SOLV Snapshot Loading can be similarly used to copy the source file/table data into the target database. However, since the source file/table is non-audited, Shadowbase software will not see the subsequent changes made to the source file or table (as they do not appear in the TMF audit trail). To pick up subsequent changes made to the non-audited source file or table, another SOLV Snapshot Load would need to be run (see below for alternative methods to replicate the subsequent changes).

Discover SOLV

SOLV Illustration

Figure 1 illustrates SOLV. This technique reads and replicates TMF-audited or non-audited source Enscribe files and SQL tables into a target database. When the source file/table is audited, Shadowbase software uses real-time replication to replicate subsequent changes made to the source file/table to the target database to keep it synchronized.

Figure 1 — HPE Shadowbase SOLV Snapshot Loading


HPE Shadowbase SOLV Manager (SOLVMGR)

HPE Shadowbase SOLV Manager (SOLVMGR, previously called “File Chaser”) can be used to replicate non-audited file/table changes. This approach works for application environments that create and maintain a log file containing the change events to be replicated (similar to an audit trail of the change events).

For Non-audited Enscribe Files

SOLVMGR and SOLV can be configured to “chase” the EOF of a file, and replicate the newly appended data into the target database in near real-time. SOLVMGR acts as the manager for SOLV and restarts SOLV after it stops for any reason. For example, SOLVMGR completes a load by hitting EOF or when the SOLV process prematurely stops due to system issues. When run in this mode, SOLV replicates all the data in the file up to EOF and stops; SOLVMGR sets a delay timer, delays, and then re-runs SOLV to pick up and replicate any newly-appended data in the file using an “EOF chaser” cycle. (The delay timer is configurable and can be very short.)

For Applications that Create a Sequence of Files to be Replicated

SOLVMGR and SOLV can replicate the file sequence. SOLVMGR uses special logic that looks at the file creation information for the sequence of files to know when to roll from one file to the next in sequence. For example, if the application has a fileset called FILEnnnn, where the application fills FILE0000 first, then it fills FILE0001, followed by filling FILE0002, etc., SOLVMGR will watch for the application rolling to the next file in sequence, and then instruct SOLV to chase the newly appended data in that file.

SOLVMGR (SOLV Manager) Drill Down

Figure 2 illustrates SOLVMGR.

For applications that create and maintain their own log files/tables, SOLVMGR can be configured to read these application-generated log files/tables, optionally transform the events, and inject the events into the replication stream to the target environment, as the application appends data to the log files/tables. This technique avoids the need for the source database to be audited, while helping to ensure that such changes are also replicated.

During configuration, SOLVMGR can be given a range of sub-volumes and application-maintained files/tables to monitor for changes. This is useful, for example, when the application creates a sequenced list of log files and periodically rolls through them as it fills them. As the application appends change-data into these log files/tables, SOLVMGR will see those changes, and periodically read/inject them into the Shadowbase data replication engine for processing.

SOLVMGR Illustration

Figure 2 illustrates SOLVMGR. An application makes database changes to non-audited Enscribe files and SQL tables, and generates log files of these changes (business transaction data). SOLVMGR periodically launches SOLV loaders, which read the log files in sequence from a saved “restart point” as each successive SOLV loader runs. This step allows for optional and periodical transformation and maintenance of the application-generated log files/tables. The SOLV loaders inject the log data into the replication stream, which is made available to any Shadowbase target environment/data replication system for processing.

Figure 2 — HPE Shadowbase SOLVMGR Architecture


HPE Shadowbase Online Loading and Verification* Utility for OSS Regular Files (SOLVUTIL)

Loading, Copying, Validating, and File Chasing OSS Regular Files

SOLVUTIL is available for loading, copying, validating, and file chasing Open System Services (OSS) Regular Files (or “Flat Files”) that have data inserted at the EOF in append mode.  (Note: the OSS Regular Files are never TMF-audited.)  SOLVUTIL does not replicate subsequent updates (to the previously inserted data).

SOLVUTIL is illustrated in figure 3. SOLVUTIL currently requires Expand when replicating between systems. (Contact Gravic if you are interested in TCP/IP support.) SOLVUTIL can track updates to multiple source files and maintain their sequence when replicating into similar target files. SOLVUTIL has numerous parameters, is configured with TACL environment variables, and can handle massive data volumes.

Compare Two OSS Regular Files

SOLVUTIL can also compare two OSS Regular Files. It confirms that the files match or lists the offset of the first byte of the mismatch. SOLVUTIL can also be used to correct mismatched data in the target file by overlaying the mismatched target data with the correct data from the source file.

*Note: SOLVUTIL Verification requires an HPE Shadowbase Compare license.

Please reference: SOLVUTIL Manual

SOLVUTIL Illustration

Figure 3 shows an application inserting data to an OSS Regular File in “append mode” (data is periodically added to the EOF) . When SOLVUTIL sees that the source file has changed, it then chases the EOF of the source file, and in near-real time replicates the newly inserted data to the target OSS Regular File. SOLVUTIL inserts the data into the same relative position as it was on the source file. SOLVUTIL replicates all the data in the file up to the EOF, sets a delay timer, delays, and then checks for and replicates all newly appended data (since the old EOF). The delay timer is configurable.


Figure 3 — HPE Shadowbase SOLVUTIL Snapshot Loading


HPE NonStop AutoTMF: TMF Transaction Protection

HPE NonStop AutoTMF is a product available from HPE which utilizes non-intrusive intercept technology to automatically invoke TMF transaction protection for applications that do not use TMF when updating data files. This technology enables non-audited applications running on a NonStop system to access and create audited Enscribe files and to perform I/O against those files without program modification or recompilation.

AutoTMF software dynamically determines effective and efficient TMF transaction boundaries and automatically consolidates multiple database updates, ensuring low overhead. There is no need to make any source code changes to programs, or even to recompile them, since AutoTMF works with existing application object files. AutoTMF–enabled applications use audited operations, which adds performance optimization benefits, such as advanced file buffering and elimination of the block-splitting overhead associated with insertions to a file. Another benefit is improved data consistency, which keeps the base file and its alternate key files consistent with each other regardless of the outcome of the transaction or if there is a file system failure.

Want to use Shadowbase software with your non-audited application?
Then follow these three easy steps to replicate your application’s individual DML activity:

  1. Use AutoTMF to quickly and easily enable non-audited data to be TMF-audited.
  2. Once AutoTMF is installed and configured for your applications, turn the TMF file/table AUDIT attribute to ON, and all subsequent file/table operations will be captured by TMF.
  3. Shadowbase software can then be rapidly implemented to identify and replicate the changes from the TMF audit trails in real-time.

For more information, read the FAQ.


HPE NonStop AutoSYNC

Non-database Files

HPE NonStop AutoSYNC is a product available from HPE which replicates non-database files such as program object code files, command scripts, OBEY files, etc., to a target environment, usually for NonStop to NonStop business continuity purposes. It can be configured to periodically check for files that need to be replicated so that the target will always have an up-to-date copy in case of failover. AutoSYNC is a powerful addition to a business continuity solution when non-database files need to be replicated.

For more information, read the FAQ.