Shadowbase Support Tips

Photo of Bill G. Holenstein

William Holenstein
Senior Manager of Product Delivery

HP Shadowbase Other Servers Batch Transaction Feature

Effective with release 6.100 of HP Shadowbase Other Servers, a new feature was added to enable DOC WRITER/TRS process configuration pairs to batch multiple source transactions into a single target transaction.  When enabled, the default is to batch up to 1000 source tx’s into a single target tx (a single BEGIN WORK/COMMIT WORK).  This parameter (SHAD_BATCHTX_SIZE=1000) is configurable.

We already know that if the customer’s Service Level Agreement (SLA) requires it to only replay COMMITted transactions into the target database, then a Shadowbase DOC/TRS is the configuration option to use. This architecture organizes the source transaction data that it reads into COMMIT timestamp order, so that the source transactions will be replayed one at a time into the target database in the order that the source transactions called COMMIT against the source database. Furthermore, the DOC/TRS configuration can also be used to guarantee that any single transaction is only ever replayed once against the target database through the use of a SHADTRS processing table.

Batch Transaction processing provides a significant performance improvement to DOC/TRS environments, particularly when the source transactions are small (have very few individual insert, update, or delete (IUD) events in them). The reason is that COMMIT processing is often the most expensive and ‘slowest’ operation (in terms of CPU and disk I/O) for the transaction, and tx batching reduces the number of target-side COMMITs that must be applied to replay the source data.

Those familiar with database I/O understand that the COMMIT can be the most expensive resource call applied to a database because a COMMIT will usually force cache data (typically redo logs) to be flushed to disk (e.g., for durability), restart points to be rewritten, locks to be released, etc., each time one is executed. Therefore, by limiting the number of COMMITs being performed on the target database, one can reduce target system and database resource allocations, and significantly improve replication throughput.

The HP Shadowbase Other Server (SBOS) parameters related to BATCH TRANSACTION processing are (with their defaults listed):

In order to turn transaction batching off, set SHAD_BATCHTX_MODE=0. This means that each source tx that is replicated will be replayed with a single target tx.

Setting it to 1 (default) will count the number of IUDs that are being processed, ignoring any commits that it sees as it replays the source transactions in commit order. When the number of IUDs hits the SHAD_BATCHTX_SIZE value, Shadowbase replication will continue to apply IUD events until the next COMMIT is processed, at which time it will then call commit against the target database.

Note, if a very long source transaction is being replayed, you can get many more IUD events inside the target tx beyond the SHAD_BATCHTX_SIZE setting than you would otherwise expect.

SHAD_BATCHTX_MODE can also be set to 2 which applies the same logic, but instead of counting source IUD events, it will count the number of source transactions it is replaying instead. When the number of TXs reaches SHAD_BATCHTX_SIZE, it will apply the commit at that time.

SHAD_BATCHTX_SIZE can be set between 1 and 2147483647. Its default value is 1000.

SHAD_BATCHTX_TIME_LIMIT is used to manage the TRS processing when it hits EOF on the DOC (EOF means that no more source data or tx’s exist to be replayed from the DOC).  The parameter can be set to a number of seconds that the TRS will delay after reaching EOF of the current DOC file before it calls COMMIT. If this parameter is set to anything greater than 0, then while the TRS is delaying, it will check again for new events that may have arrived during the delay. If it finds any, the TRS will continue counting IUDs/TXs from where it left off. If the timer expires without the TRS seeing new transactions, then the TRS will apply a COMMIT to the transaction in process, even if the SHAD_BATCHTX_SIZE limit has not been reached. By setting SHAD_BATCHTX_TIME_LIMIT to 0, it will not delay when the TRS reaches DOC EOF, but it will simply apply a COMMIT immediately.

Here is an example of the TRS logfile output when these parameters are set to their DEFAULTS:

2015-03-26 12:25:53 -[3048] Info: ********** BatchTx Configuration Settings ************************************

2015-03-26 12:25:53 -[3048] Info: ** Parameter value Description

2015-03-26 12:25:53 -[3048] Info: ** ———————– ——— —————————————–

2015-03-26 12:25:53 -[3048] Info: ** SHAD_BATCHTX_MODE [ 1] [BatchTx counting IUD events]

2015-03-26 12:25:53 -[3048] Info: ** SHAD_BATCHTX_SIZE [ 1000] [Number of events before Commit issued]

2015-03-26 12:25:53 -[3048] Info: ** SHAD_BATCHTX_TIME_LIMIT [ 0] [Wait seconds after DOC EOF reached]

2015-03-26 12:25:53 -[3048] Info: ******************************************************************************

Therefore, when the SLA is of top priority, and the delivery guarantee of the data to the target database is critical, consider using a DOC/TRS configuration to take advantage of this feature. For more information, questions, or concerns with the new Shadowbase Knowledgebase, or to receive a login as a customer, please contact Shadowbase Product Delivery at [email protected]

Please reference our Newsletter Disclaimer.