JCC LogMiner Loader V2.2 Features

Overview

The JCC LogMiner Loader™ is a fast, powerful, flexible tool to reflect data changes from a source Rdb™ database to multiple targets. Targets can be other Rdb databases, Oracle® databases, a customer supplied API in XML format, or a Tuxedo application.  The Loader has been exhaustively tested and tuned for all recent versions of these companion products.

The JCC LogMiner Loader has been used to replicate databases (or subsets of a database) for use at remote locations or for reporting databases with alternate tuning, to support database reorganization with downtime of minutes instead of hours, to regression test the performance of Rdb databases after physical tuning, to regression test an application, to audit changes to data, to rollup data from multiple source databases into a single target, and to provide data for alternate tools.

Once an organization has started to use the Loader, they quickly discover many additional uses for replicated data and often run multiple Loader sessions propagating data from a single source database.  The licensing model for the Loader permits this at no incremental cost.

The JCC LogMiner Loader works with Oracle’s Rdb LogMiner to provide either off-line (static) or near real-time (continuous) support. Recent versions of the Loader use multi-threading and other advanced performance features to support the workloads implied by update-intensive databases. Monitoring and logging tools are available and are configurable and provide extensive support for performance analysis of Loader sessions and the architecture the Loader is used in.  Additionally one style of monitoring conforms to the T4 performance measurement methodology supplied with version 7.3-2 of OpenVMS. 

Advanced features provide for very flexible architectures.  The Loader can be configured as an ETL tool in data warehouse projects to take changes to one row and direct them to several different tables in the target.  Advanced filtering using SQL can restrict which rows are sent to the target.  A Data Pump can be configured using SQL statements and a driver file to send selected rows to populate the target or correct issues that occur in the target because of other processing. 

The Loader kit includes tools to assist the DBA and the Loader is driven by DBA input via a Control File and logical names.  The Loader kit includes tools to assist the DBA in the creation of the control files and to monitor Loader performance while it is operating.

Architecture

An Oracle Rdb database may be configured such that the database engine logs all changes made to a production database into a special file called the after image journal. The after image journal (or AIJ) contains records that indicate the status of each and every database object at the completion of a transaction. The Oracle Rdb LogMiner tool uses the after image journal to extract the final copies of data rows as they appear at the end of each transaction.

The LogMiner output represents the changes to the source database. The JCC LogMiner Loader enables a database administrator to apply the contents of the LogMiner output to one or more targets.

When run in continuous mode, the Loader coordinates the actions. In continuous mode, the LogMiner Loader updates the target in near real-time. (The results cannot be quite real-time since the information is not available from the LogMiner until the database commit.)

Features

The DBA specifies the results required of the Loader through the Loader’s Control File. Loader options and features include:

·       Easy configuration

o      All of the Loader’s actions are prescribed by a Control File that can be easily changed with a text editor.

o      The Control File for the Loader enables DBA control of which tables and which columns within those tables are to be sent to the target. (Others may be updated in the source, but the changes will not be passed on to the target.) The DBA can also indicate that table and column names will be different in the source and target.

o      The Control File for the Loader enables DBA control of how source database actions affect the target. Inserts, updates, and deletes can each be included or excluded to support replication, rollup, auditing, or other combinations.

o      Rows can be excluded or included with filters on values for specific columns.

o      Rows can be identified with their primary keys. When primary keys are not available in the source, the DBA can use a Loader supplied mechanism based on dbkeys.

·       Robustness

o      The Loader provides transaction consistency.

o      The Loader provides restart capabilities, even across AIJ backups and after lengthy interruptions.

o      The Loader ensures delivery of all changes to the target.

·       Monitoring

o      The Loader supports on-line, real-time monitoring that the DBA can control.

o      The Loader provides extensive logs. The DBA can also direct what will be included in these logs.

·       Performance

o      The Loader supports dynamically adjusting parallel threads.

o      The Loader provides commit intervals to group source transactions into a single target transaction that improves system throughput. Timeouts ensure timely delivery of all data to the target.

o      Many Loader families can run simultaneously from the same source database with differing targets.

·       Additional DBA Support

o      Values can be materialized to provide virtual columns that identify the commit time, the TSN, or other variables.

o      Indexing can be different on the source and the target.

o      A maintenance facility for the logical names is part of the Loader kit.

o      Examples are included in the kit.

Prior Version Support

Since Rdb environments generally strive for the utmost stability, many customers are running older versions of VMS.  Release 2.2 of the Loader has been rigorously regression tested against a variety of  OpenVMS versions  including 7.3-x, 7.1 and 7.2.  A single kit suffices for all of these versions.

The Loader kit has been linked with the SQL*net libraries for Oracle 8.1.7, 9.0 and 9.2.0.4 on VMS.  Each of these interfaces has likewise been aggressively tested, though newer versions of the SQL*net libraries are not supported on earlier versions of VMS.

The Loader itself supports running a standard version and several varianted versions concurrently on the same system.  This allows customers to install version 2.2 while continuing to run version 2.1 or 2.0 Loader sessions and even version 1.x sessions, although the version 2.21 kit includes sufficient fixes to merit an upgrade from earlier versions.

Success Stories

The JCC LogMiner Loader is a general-purpose tool and can assist a database administrator with a variety of functions. The following examples demonstrate a selection of the varied uses of the Loader:

Case 1

A company did not provide routine maintenance for the Rdb database because the company could not afford to be without it for long enough to accomplish the maintenance. In time, the company “hit the wall” as mixed areas filled up. They exceeded the limits of the database physical design. This was a 60 Gb database and traditional reorganization techniques would require 24 hours of down time. Business growth was healthy, but the computer support was already not adequate to handle past growth. Database reorganization could be deferred no longer. 

A copy of the production database was made. The reorganization was accomplished on the copy and was extensively tested. Meanwhile, business continued on the production database. When the reorganized database was ready, the Loader was used to apply the data changes that had occurred in the production database since the copy was made. The total downtime for user support was thirty-five minutes, including testing.

Case 2

A company wanted to begin using a wider range of products and applications. All of these were to be dependent on the data in the production Rdb databases. The original application had divided the data into almost thirty related Rdb databases because the data volumes, update rates, and query demands were so large that partitioning was deemed necessary to successfully manage the huge data volumes.

The Loader provides XML to a customer-defined API and also provides data to a Tuxedo application. A set of Loaders handle all the transactions for all the source databases while maintaining transaction consistency and interacting with the targets. The Loaders process the data at a peak rate of  2,400 rows per second.

Case 3

The primary Rdb database resides at the central office. Other offices are around the world. The critical challenge came when the production database could not offer information quickly enough to satisfy queries in the remote offices and to capture the business that was available through the internet.

Using the Loader has permitted distributing multiple copies of the database with up-to-the-second accuracy. The query databases can be tuned differently from the source database to get the best query responses. Bookings are rolling in.

Support

A Bronze support contract for one year is provided as part of the software purchase. Bronze support includes the right to new releases (during the term of the support contract), plus call in and e-mail support during normal JCC business hours.

An upgrade to Gold support includes all bronze support, plus 24 X 7 coverage.

Bronze or gold contracts can be renewed for subsequent years.

JCC Consultants are also available to work with you on-site on a consulting basis.

For further information about the JCC LogMiner Loader, the current documentation is available at the JCC FTP site FTP.JCC.COM. The JCC LogMiner Loader supports either the static or continuous LogMiner. 

The distribution kit may be obtained from the  JCC FTP site  via anonymous ftp..  This path also includes the documentation and release notes for version 2.2 together with training seminar materials and a presentation on the Loader.