HPE Shadowbase Queue Manager (QMGR)

long queue of impatient people waitingWhat Is It?

  • The HPE Shadowbase QMGR is an optional process that sits between the Collector and a Consumer that safely (persistently) stores the replication data for that Consumer on disk. The data is either stored on the sending side, or the receiving side.
  • Configured on a Consumer by Consumer basis, each QMGR manages data for a single target Consumer. Each Consumer is fed by at most one QMGR. Each Consumer that a Collector feeds can independently, optionally have its own QMGR.

Why Use It?

  • The HPE Shadowbase QMGR allows the Collector and Consumer to independently process data at their own pace. As shown in Figure 1, it decouples the need for the network or target environment to be available and accessible for the Collector to continue processing source data. If configured on the source system, it provides network and Consumer independence. A slow (or inaccessible) Consumer’s data is queued for processing, allowing the Collector to continue reading the audit trail and transmitting to all Consumers.
  • If configured on the target system, it reduces the RPO (data loss) in the event of a catastrophe at the source system. When this event occurs, the target system has the data queued, even if the Consumer is behind in its processing or cannot access the target database.
  • In either configuration, the Consumer can be taken offline (for example, to allow database maintenance), while audit trail data is continually sent and accumulated.
QMGR Flow Chart - from application to target database, the QMGR sits between the collector and consumer

Figure 1 — HPE Shadowbase Queue Manager (QMGR) Flow Chart

Key HPE Shadowbase QMGR Features Include:

  • Minimal Impact to the Replication Lag Time – The lag time is minimal from when the data is written on the source system to when it is applied at the target. Adding a QMGR into a Collector/Consumer replication thread does not increase the lag time from when the event occurred on the source to when it is replayed on the target (assuming the Consumer can keep up with the data generation rate). The QMGR forwards data to the target Consumer as soon as it is received without waiting for it to write to disk.
  • Minimal I/O Impact – Data is buffered and written (in streaming append mode) to the disk in large blocks, reducing the number of write operations to the disk. Data is only read from disk when it is not available in the QMGR’s internal queue of messages. When required, data is read sequentially in large blocks, again minimizing the number of read operations against a disk.
  • Extensive Configuration Options – All of the major parameters affecting performance are configurable, allowing the QMGR to be tuned for a specific transaction profile.
  • Multi-Threaded – All I/Os are performed using “no-waited” I/O and are non-blocking, improving thruput and reducing replication lag time.
  • Less Data is Sent on a Restart – When replication is restarted, the data stored in the queue is used. The Collector needs to re-read and re-send less source audit trail data.