IBM-AUSTRIA - PC-HW-Support    30 Aug 1999

Servicer Assistance Information for Shared Disk Clusters

Abstract: PROBLEM DIAGNOSTICS IN A SHARED DISK CLUSTER ENVIRONMENT_V02

Summary: Servicer Assistance Information for Shared Disk Clusters.

Shared Cluster - Problem Diagnostic 

The intent of the document is to assist with the diagnosis of problems running the following on PC Servers and Netfinity Servers that support each:

•  Novell NetWare 3.12 / 4.1 with IBM Cluster Pack for NetWare.
•  Novell IntraNetware 4.11 with IBM Cluster Pack for Netware.
•  Microsoft Windows NT 4.0 Enterprise Edition (Service Pack III required) with Microsoft Cluster Server.
•  OS/2 Warp Server with IBM Cluster Pack for OS/2.
•  OS/2 Warp Server SMP with IBM Cluster Pack for OS/2.

•  PC Server 325 type 8639 model RB0/PT0/PTW/PB0
•  PC Server 330 type 8640 model PB0/PT0/PM0
•  PC Server 704 (all models).
•  Netfinity 7000

The IBM Storage Enclosures certified by IBM for clustering are:

•  3518 Enclosure.
•  3519 Rack Storage Enclosure.
•  MetaSTOR DS-20E/RM-20E (Refer to Authorized MetaSTOR Retailer if a problem is isolated to the MetaStor disk enclosure).

This document should be used in conjunction with the Servicer Training Video Series Volume 19, Shared Disk Clustering.

This document will be updated as necessary. The latest version of this document can be found at:

http://www.us.pc.ibm.com/support (You may search on the title of this document)

Maintaining Cluster High Availability requires the Cluster is always available for LAN attached users to access. With this in mind, always consider keeping at least one node supporting a cluster online while performing problem isolation/diagnostics on other nodes that are part of the cluster.

1.0 Shared disk cluster identification: 

Provided below are ways to identify shared disk cluster configurations. Positive identification to one of the below tasks does not ensure that you will be working with an active shared disk cluster but should allow you to take appropriate steps as necessary. Problem determination steps outlined in this document are safe to use with stand-alone server configurations.

1.  Ask the customer if they are using shared disk clustering.
2.  Ask the customer if they are running one of the following packages:
   •  Microsoft Windows NT.
   •  IBM Cluster Pack for NetWare.
   •  IBM Cluster Pack for OS/2.
   •  Microsoft Cluster Server.

3.  Look for cluster identifiers such as:
   •  External disk enclosures cabled to two or more servers.
   •  Disk enclosures maybe dual connected.
   •  Disk enclosures may be daisy-chained.
   •  Ethernet cables directly connecting two servers.
   •  Close proximity of multiple servers.

2.0 Strategies to prioritize where to begin problem diagnosis: 

When performing problem diagnosis on a node in a cluster or on a cluster, maintaining cluster high availability should be the greatest priority.

  Remember:

Maintaining Cluster High Availability requires the Cluster is always available for LAN attached users to access.
With this in mind, always consider keeping at least one node supporting a cluster online while performing problem isolation/diagnostics on other nodes that are part of the cluster.

1.  Try to get cluster operations running on at least one node if multiple nodes are down.
2.  Do problem determination that allows the cluster to continue to run.
3.  Only if absolutely necessary should problem determination be implemented that requires the  entire cluster to be shut down.

2.1 Problem determination guidelines: 

The guidelines below can assist in maintaining cluster high availability and quick problem resolution. Problem determination applied to a node that requires the node is NOT available for cluster operation:

•  Running diagnostics on a cluster node
  •  Except for host adapter diagnostics, all cluster nodes should be shut down before running  system diagnostics on host adapters which are attached to shared disk subsystems.
  •  Looped diagnostics may run on a system but may invoke host adapter diagnostics.
•  Running diagnostics on the communications adapter.
•  Inspecting cables and connections of cluster nodes and  shared disk components.

Problem determination requiring the shutdown of an entire Cluster:

•  Running diagnostics on shared disk subsystems.
•  Testing to determine if shared disk subsystem cables function.
•  Running diagnostics on shared disk subsystem host adapters.

2.2 Recovering a down cluster before problem determination 

•  Request the customer shut down all applications and operating system activities for both nodes, if   applicable.
•  Power off cluster nodes and shared disk subsystems.
•  Power on one cluster node and shared disk subsystem.
•  Request customer validate cluster operations are functioning shortly after operating system   initialization.
•  If cluster operations are still not functioning, request customer shut down the node's operating   system.
•  Power off cluster node and shared disk subsystems.
•  Power on the other node and shared disk subsystems.
•  Have the customer validate cluster operations.
•  If the cluster still does not function, use problem determination strategy for down clusters.

2.3 Specific Cluster/Node Strategies: 

If a cluster is down and each node is in a different node state use the strategy of the highest ranking node state:

       Rank              Node State
        1                Node failure
        2                Hang/Trap condition
        3                Running with errors
        4                Running without errors


The following Problem Determination Guideline Matrix references Cluster states in conjunction with Node states that reference their respective document sections that follow.

  +------------------------------------------------------------------------------+
  | Cluster      |   Node(s)     |   Node(s)     |   Node(s)     |   Node(s)     |
  | State        |   State       |   State       |   State       |   State       |
  |--------------|---------------|---------------|---------------|---------------|
  | Cluster      |  Node failure |   Node(s) in  |  Node(s) run  | Node(s) run   |
  | Down         |               | a Hang/Trap   |   with errors |   w/o errors  |
  |              |        2.3.1  |       2.3.2   |       2.3.3   |       2.3.4   |
  |--------------|---------------|---------------|---------------|---------------|
  | Cluster      |  Node failure |   Node in a   | Node running  |  Node(s) run  |
  | in           |               |   Hang/Trap   |   with errors |    w/o errors |
  | failover     |       2.3.5   |       2.3.6   |       2.3.7   |       2.3.8   |
  | mode         |               |               |               |               |
  +------------------------------------------------------------------------------+


The following strategies should be used in conjunction with the Problem Determination Guideline Matrix above.

Note: Items are rated "High" (most probable failure) to "LOW" (least probable failure).

2.3.1 Cluster down and node failure. 

               * Shared Disk Subsystem -------------------------------   High
               * Cables to shared disk systems -----------------------   High
               * Shared disk subsystem host adapter ------------------   Medium
               * Node failure ----------------------------------------   Low
               * Software errors or configuration --------------------   Low
               * Communication Link ----------------------------------   Low

2.3.2 Cluster down and node(s) in a hang trap condition. 

               * Software errors or configuration --------------------   High
               * Cables to shared disk systems (termination) ---------   High
               * Subsystem host adapters (SCSI ID) -------------------   High
               * Shared disk subsystem -------------------------------   Medium
               * Communication Link ----------------------------------   Low
               * Node failure ----------------------------------------   Low

2.3.3 Cluster down and node(s) running with errors. 

               * Shared disk subsystem -------------------------------   High
               * Subsystem host adapters -----------------------------   High
               * Cables to shared disk systems -----------------------   High
               * Software errors or configuration --------------------   Medium
               * Node failure ----------------------------------------   Low
               * Communication link ----------------------------------   Low

2.3.4 Cluster down and node(s) running without errors. 

               * Communication link ----------------------------------   High
               * Software errors or configuration --------------------   High
               * Shared disk subsystem -------------------------------   Low
               * Cables to shared disk systems -----------------------   Low
               * Subsystem host adapters -----------------------------   Low
               * Node failure ----------------------------------------   Low

2.3.5 Cluster in failover mode and node failure. 

               * Node failure ----------------------------------------   High
               * Cables to shared disk systems -----------------------   Medium
               * Subsystem host adapters -----------------------------   Medium
               * Communication line ----------------------------------   Medium
               * Software error or configuration ---------------------   Low
               * Shared disk subsystems ------------------------------   Low

2.3.6 Cluster in failover mode and a node in a hang trap condition. 

               * Software errors or configuration --------------------   High
               * Node failure ----------------------------------------   High
               * Subsystem host adapter ------------------------------   Medium
               * Cables to shared disk systems (termination) ---------   Medium
               * Shared disk subsystems ------------------------------   Low
               * Communication link ----------------------------------   Low

2.3.7 Cluster in failover mode and a node running with errors. 

               * Communication link ----------------------------------   High
               * Subsystem host adapters -----------------------------   High
               * Cables to shared disk systems -----------------------   High
               * Node failure ----------------------------------------   Low
               * Software errors or configuration --------------------   Low
               * Shared disk subsystems ------------------------------   Low

2.3.8 Cluster in failover mode and node(s) running without errors. 

               * Communication link ----------------------------------   High
               * Cables to shared disk systems -----------------------   High
               * Subsystem host adapters -----------------------------   Medium
               * Software errors or configuration --------------------   Medium
               * Shared disk subsystem -------------------------------   Low
               * Node failure ----------------------------------------   Low

3.0 Replacing a failed ServeRAID II adapter in a High-Availability configuration. 

NOTE: The following procedure requires that specific configuration settings on the ServeRAID II adapter can be obtained form the adapter that is being replaced or were noted when the adapter was previously configured and are available for reconfiguring the new adapter.

NOTE: Obtaining the correct information for these settings is the responsibility of the user and is required to accomplish this procedure.

Step 1:

•  Backup the system configuration using the system's SCU or System utility Diskette(whichever  applies to the specific system) for restoring the configuration IF required.

•  Depending upon the operational condition of the adapter being replaced, it may or may not be  possible to obtain the following information by booting the ServeRAID configuration diskette and  choosing the "Display/Change Adapter Params" item from the "Advanced Functions" menu. If the  adapter is non-functional, the following information should have been noted when the adapter was  originally configured:
  •  SCSI Bus Initiator_Ids
  •  Adapter Host_Id
  •  Cluster Partner's Host_Id

•  IF the original settings were never noted and the adapter being replaced in non-functional,  following tips MAY help:

Tip:

SCSI Bus Initiator_Ids for non-shared SCSI channels will normally be set to 7. However, for shared SCSI channels the ID's will usually be 7 or 6 and must be different than the SCSI Bus Initiator_Ids for the corresponding SCSI channels of the cluster partner adapter. You may obtain the SCSI Bus Initiator_Ids from the corresponding cluster partner adapter by booting the ServeRAID Configuration Diskette on the cluster partner system and selecting the "Display/Change Adapter Params" option from the "Advanced Functions" menu. From this information, the correct settings for the replacement adapter can be determined. For example, if the cluster partner's shared SCSI bus Initiator_Ids were set to 7, then the replacement adapter would typically need to be set to 6.

The proper settings for the Host_Id and Cluster Partner's Host_ID of the adapter being replaced may be determined by reading the settings from the cluster partner system by using the "Display/Change Adapter Params" option. In this case, the adapter being replaced should have it's Host_Id set to the same value as is defined for the Cluster Partner's Host_Id on the corresponding adapter in the cluster partner system. The Cluster Partner's Host Id of the replacement adapter should be set to the same value as is defined in the Host_Id of the corresponding adapter in the cluster partner system.

Example:

                                          Node A          Node B
    SCSI Bus Initiator_Ids                   7               6
    Adapter Host_Id                      Server001       Server002
    Cluster Partner's Host_Id            Server002       Server001


Step 3:

•  Power down the system, note which SCSI cables are connected to the SCSI channel connectors on  the adapter, note which PCI slot the adapter is in, remove the cables from the adapter, then remove  the ServeRAID II adapter from the system.

•  Install the New ServeRAID II adapter into the same PCI slot. DO NOT reconnect SCSI channel  cables to the adapter at this time.

•  Boot the system to the "IBM ServeRAID, ServRAID II, and onboard ServeRAID configuration   diskette" version 2.40 or higher. Initialize the adapter configuration as follows:
  •  Select "Advanced Functions" from the main menu.
  •  Select "Init/View/Synch Config"
  •  Select "Initialize Config"

•  Ensure that the adapter is at the latest BIOS/Firmware levels. The latest BIOS/Firmware levels are  available from the IBM Website at URL: http://www.pc.ibm.com/us/files.html

•  Search on: RAID BIOS
•  Download the .txt to determine the latest level available, and apply them if the adapter is  downlevel (the actual BIOS/Firmware level of the adapter is displayed after system post when the  adapter BIOS loads).

•  Update the configuration parameters as follows:
  •  Boot to the "IBM ServeRAID, ServRAID II, and onboard ServeRAID configuration  diskette" version 2.40 or higher.
  •  Select the "Advanced Functions" from the main menu.
  •  Select "Display/Change Adapter Params".
  •  Select and configure each of the following with the original settings that were noted  when the original adapter was last configured (see step 2 above):
    •  SCSI Bus Initiator_Ids
    •  Adapter Host_Id
    •  Cluster Partner's Host_Id

•  Shutdown the system and reconnect the SCSI channel cables to the adapter. Be sure to connect  the cables to the correct SCSI channels as noted in step 3.

  NOTE: If the adapter being replaced is not the adapter which attaches the server's boot disk array or  other non-shared disk arrays, then the following steps may not apply and the system may now be  restarted normally.

•  If the adapter that was replaced was the adapter that attached the operating system boot disk array  for the system or if other non-shared disk arrays are attached to this adapter, then boot the system  using the "IBM ServeRAID, ServRAID II, and onboard ServRAID onboard controller diskette"  version 2.40 or higher.
  •  Select "Advanced Functions" from the main menu.
  •  Select "Merge Group Management"
  •  Follow the steps below to restore the adapter's disk array configuration:
  •  To restore non-shared disk array configurations, select the "Merge Own Non-Shared Drives"  option from the merge group management menu. Enter the "merge group ID" for the array as 20X  (normally 206 or 207) where X is the value of the shared SCSI Bus Initiator_Id that was used  to configure the replacement adapter in step 7 above.
     If this was the boot adapter, the system should now be able to boot to the operating system.
  •  Select the Merge Own Shared Drive option for each shared Array (merge group Id's  in the range from 1 to 8) which has not failed over to the cluster partner system  (e.g. RAID-5 Arrays in critical/degraded state) in order to restore the  configuration of these shared Arrays. Typically all shared Arrays have failed over  and should not be merged now.

•  Repeat Merging shared drives which have not failed over if  multiple shared drives with different Group_Ids exist that  have not failed over to the cluster partner system.

•  Once all array configurations have been restored, the server may be restarted.

_____________________________________________________________________________________________________

4.0 Contacting support

When it is necessary to contact IBM support for assistance in resolving the cluster problem, the below information will help support more quickly understand the cluster environment and problem.

•  Machine type & model of all nodes
•  Shared disk subsystems connected and how they are attached.
•  Cluster software and operating system platform
•  Private interconnect
•  State of the cluster

5.0 Service tips

5.1 Diagnostics

•  Running host adapter or shared disk diagnostics while the shared disk host adapter is still  connected to the shared disk enclosure may interrupt the operation of the surviving node or cause  the system running diagnostics to report false error conditions.

•  Looped diagnostics should only be run on one component at a time when a cluster is running in  failover state. Looped diagnostics for more than one test will run all tests, including shared disk  subsystem tests.

5.2 Shared Disk Subsystems

•  Booting system diagnostics hangs at the screen:  "Booting PC DOS" If the MetaStor host adapter BIOS is not disabled when attempting to run  diagnostics, some diagnostic programs will hang while booting. Request the customer verify  that the MetaStor host adapters do not have BIOS enabled.

•  Fault Isolation on cluster with connected MetaStor disk subsystems.
•  If the MetaStor unit exhibits blinking and / or amber colored lights, have the customer contact  MetaStor service or support for assistance.
•  Errors reported in MetaStor's SYMplicity program should be handled by authorized MetaStor  service personnel.
•  Host adapters that are connected to the same bus should not have the same SCSI ID. One adapter  should be set to ID 6 and the other to ID 7.

5.3 Communication Link/LAN Adapters

•  LAN Adapter diagnostics and LAN Adapter diagnostics for IBM EtherJet 10/100 adapters may be  run on a node while the surviving node is operating in failover mode.

6.0 Glossary of terms

Cluster:
A collection of interconnected whole computers utilized as a single unified computing resource.

Node:
A server participating in a cluster.

Communication Link:
The link between the nodes of the cluster used for Cluster communication. This is usually an Ethernet link.

Failover:
The action where processes and applications are stopped on one node and restarted on the other node.

Failback:
The action where processes and/or applications return back to the node they are configured to run on during normal cluster operation.

Cluster down:
A state where either multiple nodes are physically not functioning or clients cannot access the cluster or virtual servers configured on the cluster.

Failover mode:
A state where one node in the cluster is handling cluster activity while the other node is offline or not functioning.

Node failure:
A state where a node hardware failure is exhibited by one of the following attributes:

•  Post errors
•  Failure to power on
•  Video failure

Hang trap condition:
A state where a software failure has halted operation of a node exhibited by any of the following:

•  Blue screen in Windows NT
•  ABEND (Abnormal END)
•  OS/2 Trap
•  NMI (Non Maskable Interrupt)
•  System unresponsive to Keyboard and / or Mouse

Running with errors:
The operating system on the node is capable of running and is reporting errors. Cluster activity on this node has ceased to function.

Running without errors:
The operating system on the node is capable of running and is NOT reporting errors. Cluster activity on this node is not functioning.

_____________________________________________________________________________________________________

Windows NT, and Microsoft Cluster Server are trademarks of Microsoft Corporation.
Microsoft is a registered trademark of Microsoft Corporation.
NetWare and IntranetWare are trademarks of Novell, Inc.
SYMplicity is a trademark of Symbios Logic, Inc.
MetaStor is a trademark of Symbios Logic, Inc.

_____________________________________________________________________________________________________

Please see the LEGAL  -  Trademark notice.
Feel free - send a   for any BUG on this page found - Thank you.