Achieving high availability for the PMC through active-passive configuration (v5.2 and later)

Note: This article applies to PeerGFS v5.2 and later. If you are using an earlier version of PeerGFS, see Achieving high availability for the PMC through active-passive configuration (v4.7.0 - v5.1.1).

This article describes how to achieve high availability for your PeerGFS environment by setting up an active-passive configuration. It covers:

An overview of active-passive configuration.
Setting up active-passive configuration.
Performing manual failover to the passive PMC.
Configuring automatic failover to the passive PMC.

Overview

With an active-passive configuration, you run an active PMC server (physical or virtual) with a second passive PMC server (physical or virtual) configured as a backup. When a failure occurs on the active PMC server, the passive PMC server is ready to take over the work of the active server. Once configured, minimal user interaction is required when failing over between PMCs. Agents will automatically failover from the active PMC server to to the passive PMC server. Jobs set to auto-start will automatically start on the passive server.

Benefits

The benefits of an active-passive configuration include the absence of a need for additional hardware and software, except for provisioning a new physical or virtual server. Furthermore, it is a simpler and more cost-effective solution to implement across multiple locations, particularly for covering PMC disaster recovery scenarios.

Licensing

Since only one PMC server is active at a time, only one PeerGFS license is needed. You are allowed to use the same license on the active and passive servers but only one server can be active at a time. If you try to start a second PMC when there is a running PMC, then the second one will not be allowed to start. When this is attempted, an alert will be sent to the running PMC before shutting down the second PMC.

Prerequisites

If you are using a network of brokers, set up the network before setting up active-passive configuration.
Ensure that the identical version of PeerGFS is already installed on both the active and passive servers.
License and set up one PMC to function as the active one while leaving the other PMC unlicensed and unconfigured.

Setting Up an Active-Passive Configuration

The setup consists of two tasks:

Task 1. Adding sites to the topology - Establish connectivity within the topology to enable mutual awareness between the two PMCs for seamless failover operations and to guarantee synchronization of their configurations.

Task 2. Configure Agents to failover to the passive server – Set up each Agent to automatically failover to the passive PMC when the active PMC is inaccessible, and failback to the original PMC when it is available again.

The next section, Configuration Tasks, provide step-by-step instructions for both tasks.

Configuration Tasks

This section describes how to configure each PMC host so that they are connected together. Start with the currently active PMC followed by the passive one.

Task 1. Adding sites to the topology

To add the active and passive PMC servers to the topology perspective:

Open the PMC client (rich or web) on the active PMC server.
Click the Open Perspective icon in the upper right corner of the PMC client.
Select Topology, and then click Open.
The Topology Perspective appears.
Click the Add Site button to open the Add Site wizard.
Name the site, drag the site icon to the desired location on the topology map, and then click Next.
(Optional) Enable TLS to encrypt all broker connections to and from this site.
Configure the broker on this PMC host:
1. Enter a host name in the Hostname field.
2. In the Failover Method section, keep the default failover method of All Primary.
3. In the Network Interfaces section, enter one or more IP addresses for the broker.
4. Keep the default value for the available bandwidth.
Note: Ensure that network routing policies are set correctly to achieve the desired behavior. Note that PeerGFS has no control over these policies, so please contact your network administrator for assistance.
Click Finish.
Repeat Steps 4-8 for the passive PMC.
After adding the second (passive) PMC, the new site will momentarily have a status of Unconfigured before being automatically configured with the same settings as the active PMC. After this, the second PMC should have a status of Online but will be in a passive state.

Task 2. Configure Agents to Failover to the Passive Server

This section outlines the steps needed to configure all Agents for failover to or from the passive server. This process entails making modifications to the Broker configuration for each individual Agent.

Within the Peer Management Center client, select all Agents, right-click and select the Edit Agent Configuration on Selected Agents option, and then click OK in the message that appears.
In the Failover Brokers section, add the broker IP address for the passive server in Broker Hosts field (if there is an address already present, then add the new address separated by a comma).
Note: Do not change the primary broker, even if there are multiple values in Broker Hosts. If a network of brokers is deployed, then ensure that both the active and passive PMC are included in one of the Broker Hosts fields.
Set the connection type appropriately for your environment, and then click OK.
Restart all Agents by selecting all Agents in the Agents view, and then right-clicking and selecting the Restart Agent Service option.

Performing Manual Failover to the Passive PMC Server

This section describes the actions to take when failing over between the active and passive PMC servers. There are two scenarios covered in this section:

Performing a planned failover between PMC servers
Manually enabling a passive PMC after failure

Scenario 1: Performing a planned failover between PMC servers

Occasionally, deliberate control transfer between PMC servers may be necessary, often as part of routine maintenance procedures.

To activate a passive PMC server:

Open the PMC on the currently active PMC server.
Open the Topology perspective.
Select the passive PMC site in the Sites panel.
Click the Activate PMC button.
Click OK in the message that appears.
Once this has been done, the two PMCs will negotiate the handover of control. Any jobs that are configured to automatically start will do so.

Scenario 2: Manually enabling a passive PMC after failure

In the event of a genuine failure necessitating environment recovery, this section describes the process to disable the active PMC and manually start the passive PMC. Even if connectivity between the PMCs is lost, it remains necessary to deactivate the currently active PMC.

Disable the active PMC

To disable the active PMC, follow these steps if the original active PMC server is still available:

Launch the PMC on the active PMC server.
Open the Topology perspective.
The passive PMC will appear to be offline as there is no network connectivity between them.
Select the active PMC in the Sites view, and then click the Deactivate PMC button in the Details view.

The PMC will automatically close and all Peer Management services will be stopped.

Enable the passive PMC

To enable the passive PMC, you must manually start the Peer Management service:

Log in to the passive PMC server.
Open the Windows Services application and find the Peer Management Services.
The number of Peer Management services listed will vary, depending on your installation.
Edit the properties of each disabled service:
1. Change Startup type from Disabled to Automatic.
2. Click OK to save the changes.
Start the services that were just enabled.

The PMC will start and become active. Any jobs marked to automatically start will do so.

Configuring Automatic Failover

The preceding sections have explained the manual execution of failover operations. However, depending on your license, you can configure two PMCs to automatically switch over between each other in the event of a failure.

To enable automatic failover:

Verify that the environment is licensed for automatic failover.
Install the watch node broker.
Configure automatic failover.

Note: In automatic failover scenarios, no PMC is designated as more important or given higher priority. As a result, automatic failback is not inherent. If there is a preference for a specific PMC to be active, it must be manually accomplished by following the steps outlined in the planned manual failover process.

Verify that the environment is licensed for automatic failover

Open the PMC client (rich or web)
From the Help menu, select Licenses.
Verify that Automatic Failover is enabled in the Licensed Features section.

Install the watch node broker

For automatic failover to function, each PMC server must be aware of the connected and disconnected status of other PMC servers in the environment. This awareness is facilitated through the establishment of a reliable third point of view. To achieve this, a broker known as a watch node is introduced to the environment, positioned independently and not affiliated with either PMC site.

To install a watch node broker:

Download the broker installer.
Run the installer and navigate through the installation wizard.
Read and accept the license agreement.
Confirm the location of the installation directory.
Confirm the location of the Start Menu folder.
Finish the installation. If this is a new installation, add the recommended firewall rules if needed.
All brokers connect to each other on TCP ports 61616 (TCP) or 61617 (TLS); therefore access to these ports is required.
Once the watch node broker has been installed, add the new site to the topology perspective in the same way as described in Task 1. Adding sites to the topology.

Configure automatic failover

To configure automatic failover:

Navigate to the topology perspective within the PeerGFS client.
Click the Failover Options button.

The Failover Options dialog appears.
Select Enable Auto-Failover.
Select a watch node broker. If your environment utilizes a Network of Brokers, all configured brokers will be available as options.
Click OK.

Best Practices

Considerations during a failover event:

PeerGFS web server configuration:
- Take note of the hostname used for your PeerGFS web service configuration if you plan to utilize web access during a failover event.
- The web service port (default 8443) binds to the hostname specified in the configuration.
- Verify that the hostname is set to the default of "0.0.0.0" to guarantee access to either PMC's web server using its NetBIOS/FQDN/IP address in the web server URL during normal operation or in the event of a failover.
- During a failover, point your browsers at the secondary PMC.
Network connectivity:
- All PMCs and the watch node broker should be capable of receiving incoming network connections.
Location planning:
- Pay attention to the positioning of PMCs and the watch node broker.
- In both local and distributed environments, place the watch node broker in a neutral location relative to both PMCs.
- Strategically position PMCs to prevent a single point of failure, ensuring robustness in the event of failovers.

Troubleshooting

PMC startup failure:
- Verify that another PMC instance is not already running.
- If automatic failover is active, ensure that the watch node or passive PMC is online.
Lack of passive PMC or watch node:
If automatic failover was enabled and there is no passive PMC or watch node available:
- Note that a PMC running with automatic failover requires the presence of the watch node or passive PMC to remain active.
- If these components are unavailable, automatic failover needs to be disabled for the PMC to continue running.
To manually disable automatic failover:
1. Edit the SPGlobal.prefs file located in the workspace/prefs directory.
2. Change the value of autoFailover from true to false.