Troubleshooting high-availability cluster environments

A high-availability cluster environment requires little or no additional troubleshooting when compared with a stand-alone server environment. This topic explains the terminology used to describe high-availability cluster environments and provides some common troubleshooting procedures.

You use the diagnostic tools to display the configuration of a high-availability environment and to verify that your secondary servers are set up correctly to update data.

Transactions are processed by the servers very quickly. The onstat commands display status information only for the instant the command was run.

To update data on secondary servers, HCL OneDB™ creates proxy distributors on both the primary and the secondary database servers. Each proxy distributor is assigned an ID that is unique within the cluster. The proxy distributor is responsible for sending DML update requests from secondary servers to the primary server. Secondary servers determine how many instances of the proxy distributors to create based on the UPDATABLE_SECONDARY setting in the secondary server's onconfig file.

For updatable secondary servers in a high-availability cluster environment, encryption from the updatable secondary server to primary server requires SMX encryption. To encrypt data sent from an updatable secondary server to the primary server, set the ENCRYPT_SMX configuration parameter on the secondary server. See ENCRYPT_SMX configuration parameter for more information.

When initializing an updatable secondary server in a high-availability cluster, the server remains in fast recovery mode until all open transactions, including open and prepared XA transactions, are complete. Applications cannot connect to the server while it is in fast recovery mode. The server remains in fast recovery mode until all open transactions are either committed or rolled back.

Use the onstat -g proxy command on the primary server to view information about all proxy distributors in the high-availability cluster.

onstat -g proxy

Secondary    Proxy      Reference     Transaction  Hot Row
Node         ID         Count         Count        Total
serv2        392        0             2            112
serv2        393        0             2            150

Examining the output from the onstat command in the previous example, there are two proxy distributors whose IDs are 392 and 393. The Transaction Count indicates the number of transactions currently being processed by each proxy distributor.

You run onstat -g proxy on a secondary server to view information about the proxy distributors that are able to service update requests from the secondary server.

onstat -g proxy

Primary    Proxy     Reference     Transaction  Hot Row
Node       ID        Count         Count        Total
serv1      392       0             2            112
serv1      393       0             2            150

In this example, the server is a shared disk (SD) secondary server, and is configured to update data. In addition, the server is connected to the primary server named serv1, and there are two proxy distributors, each with a transaction count of 2.

Use onstat -g proxy all on the primary server to display information about proxy distributors and proxy agent threads. One or more proxy agent threads are created by the proxy distributor to handle data updates from the secondary server.

onstat -g proxy all

Secondary  Proxy      Reference     Transaction  Hot Row
Node       ID         Count         Count        Total
serv2      392        0             2            1
serv2      393        0             2            0

TID  Flags       Proxy  Source  Proxy   Current  sqlerrno iserrno
                 ID     SessID  TxnID   Seq
63   0x00000024  392    22      1       5        0        0
64   0x00000024  392    19      2       5        0        0
62   0x00000024  393    23      1       5        0        0
65   0x00000024  393    21      2       5        0        0

In the output of the onstat -g proxy all command, TID represents the ID of the proxy agent thread that is running on the primary server. Source SessID represents the ID of the user's session on the secondary server. Proxy TxnID displays the sequence number of the current transaction. Each Proxy TxnID is unique to the proxy distributor. Current Seq represents the sequence number of the current operation in the transaction being processed. Each database transaction sent to a secondary server is separated internally into one or more operations that are then sent to the primary server. The last two fields, sqlerrno and iserrno, display any SQL or ISAM/RSAM errors encountered in the transaction. An error number of 0 indicates completion with no errors.

Running onstat -g proxy all on the secondary server displays information about all of the sessions that are currently able to update data on secondary servers.

onstat -g proxy all

Primary   Proxy      Reference     Transaction  Hot Row
Node      ID         Count         Count        Total
serv1     3466       0             0            1
serv1     3465       0             1            0

Session  Proxy  Proxy  Proxy  Current  Pending  Reference
         ID     TID    TxnID  Seq      Ops      Count
19       3465   67     1      23       0        1

In the output from the onstat -g proxy all command run on the secondary server, Session represents the ID of a user's session on the secondary server. The Proxy ID and Proxy TID are the same as those displayed on the primary server. Pending Ops displays the number of operations that are waiting to be transferred to the primary server. Reference Count displays the number of threads in use for the transaction. When Reference Count displays 0 the transaction processing is complete.

To display detailed information about the current work being performed by a given distributor, use:

onstat -g proxy <proxy id> [proxy transaction id] [operation number]

The proxy transaction ID and operation number are both optional parameters. When supplied, the first number is considered the proxy transaction ID. If a secondary number follows it is interpreted as the operation number. If no proxy transaction ID exists, the command performs the same as: onstat -. Similarly, if no such operation number for the given proxy transaction ID exists, the command performs the same as: onstat -.

Use the following command to display information about whether a server is configured to allow updates to data. The command can be run either on the primary or secondary server:

onstat -g <server_type>

Examples:

onstat -g rss
onstat -g sds
onstat -g dri