The Folly of Using Non-audited Data

Even today, we still come across mission-critical applications with non-audited data. These architectures do not follow best practices regarding transaction semantics and protection.

There are various reasons cited why, but the most common are comments along the lines of it being unnecessary, difficult, and/or inefficient by increasing overhead, and slowing down the application.

Another excuse is that the application was never written to use transactions, and the effort to rewrite it is not worth the trouble or perhaps not even possible if the application is old and the source code is poorly understood or has been lost.

None of these reasons stand up to scrutiny

In addition, myths continue to circulate within the community, perpetuating these misconceptions.

Let’s Delve into Why

First, a brief background on the HPE NonStop Transaction Management Facility (TMF)

On HPE NonStop systems, the HPE NonStop Transaction Management Facility, also known as the TMF Audit Trail (TMF AT) subsystem is embedded in the HPE NonStop Operating System (OS). Enscribe files along with SQL/MP and SQL/MX tables can be protected by the TMF AT. Doing so enables changes to these files and tables to be automatically intercepted and logged into the TMF AT as a transaction.

Data managed in this way by TMF are known as audited files or audited data. If you want your application to perform well, then you absolutely should be using TMF-audited files and tables.

Unraveling the myth: using the TMF AT should be avoided at all costs

In the early days of Tandem Computers and TMF, this TMF auditing function was not particularly efficient, and could negatively impact application performance. In addition, TMF and the audit trail files were considered to be difficult to operate and manage, requiring large amounts of disk storage capacity.

So, a myth grew: using TMF audited files and tables should be avoided at all costs. This myth took root and is still prevalent today, even though technology has dramatically changed to the point that this myth and its origins are now manifestly untrue.

Exposing the truth: far from creating performance issues, using the TMF AT will almost always result in improved application performance

This myth spread before significant enhancements were made to TMF and DP2 software.[1]

These enhancements include:

• Most importantly, the database disk processes (DP2) can eliminate physical I/O operations by caching if updates are audited[2], with no possibility of data loss. If a system fails, “lost” updates are reapplied from the audit trail.
• TMF overhead is minimal. Since it is implemented as part of the NonStop OS kernel and disk process, transaction processing requires a low level of resources.
• Audit records sent from the database disk process to the audit trail disk process are blocked together, using a technique called boxcarring. A few messages can support a large number of transactions.
• Audit trail writes are also boxcarred. Audit for many transactions from many disks is collected and written to the end of the audit trail with a single I/O.

These enhancements expose the truth that using the TMF AT is possible and uses relatively low system overhead.

Performance comparison

Since some users may still be skeptical, a performance analysis was undertaken[3], comparing the transaction rates and response times for a sample application using audited and non-audited files.

Transactions per second (TPS)

• The absolute best transaction rate that could be achieved using non-audited files was 960 transactions per second (TPS),
• whereas using audited files reached 2,450 TPS.
  • This best non-audited transaction rate was achieved when using database buffering, which comes at the expense of possible data loss in the event of a failure, something which is not possible when using audited files.
  • Only 150 TPS was achieved for the same risk of data loss, using an unaudited, unbuffered database.

Response Times

The same story is true for transaction response times.

For the same load on the system:

• when using unaudited files the transaction response time was as much as 10x greater (> 500 ms)
• than when using audited files (< 50 ms).

The net result of these previously mentioned TMF/DP2 enhancements is confirmed by the performance analysis, turning this myth on its head.

---

Another myth: adding TMF AT protection to existing HPE NonStop environments will require an application rewrite

The second objection about audited data is that use of TMF will require an application rewrite, which is also false.

Exposing the truth: applications with non-audited data can easily utilize HPE NonStop AutoTMF without rewriting the application

HPE NonStop AutoTMF software automatically provides TMF transaction protection without requiring any change to the application code or logic. In addition, AutoTMF works just as effectively and efficiently as explicitly using TMF.

In terms of performance, online backup and recovery, and capturing database updates, AutoTMF is currently in use in hundreds of customer applications. Some of these customers adopted AutoTMF to support TMF-based data replication such as Shadowbase software, and many of them are also using AutoTMF to improve application performance.

Another myth: other methods of applying transaction protection are good enough

Another reason given is that other methods of applying “transactionality” are performed by the application to help prevent data and state inconsistencies and facilitate recovery (e.g., application logging), therefore, the TMF AT is not needed.

Exposing the truth: these alternative approaches cannot provide the same level of protection as the TMF AT

However, these alternative approaches do not provide the same levels of protection as does the use of full transaction semantics, while also introducing other problems.[4]

---

Summary

TMF-audited databases offer significant operational, reliability, and integrity advantages, improving overall system performance. Additionally, using TMF enables ACID transaction semantics[4]. Far from being an impediment, using a TMF-audited database offers significant operational, reliability, integrity, and performance advantages, which improves the application and overall system performance.

Upgrading a non-audited application to an audited one is made by simply using facilities provided by AutoTMF software. Therefore, there is good reason to migrate to an audited solution for any existing non-audited application. In fact, doing so will improve performance, enable transaction protection, and help future-proof the application, exposing it to new capabilities and architectures such as those provided by HPE Shadowbase data replication software.[4]

---

Footnotes

[1] DP2 = Disk Process 2, an HPE NonStop software component that provides the interface between TMF and the database and file subsystems.

[2] To achieve optimal performance, ensure that the DP2 disk cache size is set to a sufficiently large value. As a rough guide, the DP2 disk cache should be at least 10-20% of the total amount of main memory available. DP2 disk cache size can be checked for each data volume by using the SCF INFO <volume>, CACHE, and SCF STATS <volume> commands.

[3] Best Practices: Using TMF to Implement Business Continuity/Disaster Recovery, Richard Carr, Carr Scott Software Inc. The Connection Sept – Oct 2013 | Volume 34, No. 5.

[4] This subject is too detailed to be discussed fully in this newsletter, see Gravic Publishes New White Paper: The Advantages of Transaction Protection for more information.

---

Specifications are subject to change without notice. Trademarks mentioned are the property of their respective owners.

Please reference our Newsletter Disclaimer.

Back to: Spring 2023 Newsletter