This article applies to PeerGFS v4.7.0 and later.

Overview

The knowledge base article High Availability Options for Peer Management Center describes four options for achieving high availability for your PeerGFS environment. This article describes how to implement the fourth option, setting up an active-passive configuration. As of PeerGFS v4.7.0, you can configure Agents to failover from one PMC server to a backup PMC server.

With an active-passive configuration, you run a primary active PMC server (physical or virtual) with a second passive PMC server (physical or virtual) configured as a backup. When a failure occurs on the active PMC server, the passive PMC server is ready to take over the work of the active server. Once configured, no user interaction is required when failing over to or from the backup PMC other than starting the Peer services on the passive server. Jobs set to autostart will start automatically on the passive server.

Important

The PMC workspace folder (a subfolder in the PMC installation folder) on the passive server must be kept up-to-date so that it is ready to take over if the active server goes down. Although any third-party replication tool can be used to keep the PMC configuration on the passive server up-to-date, with PeerGFS v4.5.2 and later, the Task Scheduler includes a Back up PMC Configuration task that can be scheduled to regularly back up the configuration of the active server directly to the passive server. If you use a third-party replication tool, it will require VSS support to ensure it gets a clean snapshot of the PMC's configuration database.

Note, If a Network of Brokers topology is being used then setup all PMCs & brokers using this method prior to following this guide.

Benefits

The benefits of an active-passive configuration are that it does not require any extra hardware and software (aside from provisioning a new physical or virtual server); and it is easier (and less expensive) to use across multiple locations to cover PMC disaster recovery-type scenarios.

Licensing

Since only one PMC server is active at a time, only one PeerGFS license is needed. You are allowed to use the same license on the primary and backup servers but only one server can be active at a time. Having both the active and backup servers active at the same time would require an additional PeerGFS license.

Configuration Process Overview

Ideally, the active-passive configuration should be set up before the first production job is configured. As soon as PeerGFS jobs nears production-ready status, the PMC configuration should be backed up somewhere, regardless of whether there is a server designated as a backup PMC device. Backing up the PMC configuration takes seconds and could help to avoid hours of job reconfiguration in the event of system failure.

Once you have backed up the PMC configuration, you are ready to set up the active-passive configuration. The setup consists of four tasks:

  1. Disable the Peer services on the passive server – All Peer services should be set to a manual start to ensure this server is in a passive/standby role while the primary server is active.

  2. Give the active server permission to write to the passive server – The active server needs permission to back up the PMC configuration to the passive server.

  3. Create a backup task on the active server – The backup task should regularly back up the PMC configuration on the active server to the PMC workspace folder on the passive (backup) PMC server.

  4. Configure Agents to failover to the passive server – Set up each Agent to automatically failover to the backup PMC when the primary is inaccessible, and failback to the original PMC when it is available again.

The next section, Configuration Tasks, provide step-by-step instructions for these four tasks. The Recovery section describes how to failover to the backup PMC server and then how to failback to the original PMC server.

Configuration Tasks

Prerequisites

  • PeerGFS should be already installed on both the active and passive servers.
  • Active and passive servers must be part of a Windows Domain.

Task 1. Disable the Peer services on the passive server

This task is necessary to prevent Peer services running on the passive server while the active server is running.

  1. Login to the passive PMC server.
  2. Open the Windows Services application and stop the following Peer services:
    • Peer Management API Service - Stop this service first. (Note: The Peer Management API Service is available only in PeerGFS 4.6 and later.)
    • Peer Management Broker Service - Stopping this service will cause the remaining two services to be stopped.
    • Peer Management Center Service
    • Peer Management Center Web Service

Note, if Network of Brokers is being utilized then the Peer Management Broker Service and Peer Management Topology Service should be left running.

  1. Edit the properties of each service:
    a. Set Startup type to Manual.
    b. Click OK to save the changes.


    Note, if Network of Brokers is being utilized then the Peer Management Broker Service and Peer Management Topology Service should be left with Startup type set to Automatic.

Task 2. Give the active server permission to write to the passive server

Before you can back up the active server's PMC configuration to the passive server, you must authorize the active server to write to the passive server.

To add the active server's computer account to the local Administrators group on the passive server:

  1. Open the Windows Computer Management application.
  2. In the console tree, click Local Users and Groups, and then open Groups.
  3. Right-click the Administrators group, and then click Add to Group.
  4. Click Add in the Administrators Properties dialog.
  5. Click Object Types.
  6. Click the Computers checkbox and then click OK.
  7. Type the name of the primary PMC server into the Enter the object names to select field.
  8. Click Check Names to confirm the computer is found on the network.
  9. Click OK to add it as a member of the local Administrators group.
  10. Click OK to save the Group change.
  11. Close Computer Management.
  12. Log off from this server.

Task 3. Create a backup task on the active server

The backup task will back up the configuration of the active PMC server to the passive PMC server on scheduled basis.

  1. Log in to the active PMC server.
  2. Open the PMC.
  3. Select Preferences from the Window menu.
  4. Select Task Scheduler in the navigation tree.

  5. Click the Create button.

  6. Name the backup task (e.g., PMC Configuration Backup).

  7. Select Peer Management Center from the Solution dropdown list.

  8. Select Back up PMC Configuration for Job/Task Types.

  9. Click Next.

  10. Select the backup frequency from the Settings options (One Time, Daily, Weekly, or Monthly); at least weekly is recommended).

    DailyAllows selecting a recurring frequency of every 1, 2, 3, 5, 10, or 15 days.
    WeeklyRequires selecting at least one day of the week to run on.
    MonthlyAllows selecting All or specific month(s) in which to run, as well as any numbered day of the month or the Last day of selected month(s).

  11. Select the start date and time of the first backup.

  12. Click Next.
  13. In the Target Path field, enter the UNC path to the PMC installation folder on the passive PMC server.
    Use the admin path rather than a share path (e.g. \\secondaryPMC-DNSname\c$\Program Files\Peer Software\Peer Management Center).

  14. Click Finish to save the backup task.

    The backup task appears in the Task Scheduler's list of tasks.

  15. (Optional) To create the initial backup, select the backup task, and then click the Run Now button.


The Task History view displays the details for past task runs (e.g. Start Time, End Time, Elapsed Time, any associated errors ).
To display the Task History view, select Show View from the Window menu, and then select View Task History.

Task 4. Configure Agents to Failover to the Passive Server

This section describes the steps required to configure all Agents to failing over to or from the backup server. It involves modifying the Broker configuration for each Agent.

  1. PeerGFS v4.7 users only: To gain access to the Agent broker failover options, insert the following line into the workspace\hub.ini file in the PMC installation folder. Create the file if necessary.

    Agent Broker UI enable

    hub.agent.enable.broker.secondary.ui=true
    JAVA
  2. PeerGFS v4.7 users only: Restart the Peer Management Center Client application.
  3. Within the Peer Management Center Client, select all Agents, right-click and select the Edit Agent Configuration on Selected Agents option.

  4. In the Broker Configuration section, add the failover broker IP address in the Failover Broker section (if there is an address already present, then add the new address separated by a comma). Set the other failover broker fields appropriately for your environment.
    Note: Do not change the primary broker, even if there are multiple values.
    Note: If a Network of Brokers is deployed then ensure that both the primary and secondary PMC is included in one of the Broker Hosts parameters

  5. Restart all Agents by selecting all Agents and right-clicking the Restart Agent Service option.

Recovering from a failure

This section describes the actions to take when the active PMC server fails. Recovering from a failure on the active PMC server involves:

Failover to the backup PMC server

If the primary (active) PMC server fails, follow these steps to make sure the passive PMC server takes on the primary role:

  1. Ensure that the active PMC server is completely stopped. If the server has partially failed but is still running, disable the Peer Management Services:
    a. Log in to the primary (active) PMC server.
    b. Open the Windows Services application and stop the following Peer services:
    • Peer Management API Service - This service must be stopped first. (Note: The Peer Management API Service is is applicable only to PeerGFS in 4.6 and later.)
    • Peer Management Broker Service - Stopping this service will cause the remaining two services to be stopped.
    • Peer Management Center Service
    • Peer Management Center Web Service
    c. Edit the properties of each of the Peer services: Set Startup type to Manual, and then click OK to save.
    This prevents the problems that occur if both the active and passive servers are running simultaneously.
  2. Log in to the backup PMC server.
  3. Open the Windows Services application and start the Peer services on the backup server.
  4. Start the job(s).

Note: If a Network of Brokers is being used then the Peer Management Broker Service and Peer Management Topology Service can be left running on all PMC servers

Failback to the primary PMC server

Once the failure event has been resolved and you have access to the former primary PMC:

  1. Follow the recommendations in Best Practices prior to failing back.
  2. Return the primary server back to its active role:
    1. Stop and disable the Peer services on the backup PMC server.
    2. Start the Peer services on the primary PMC server.
    3. Start the job(s) that aren't set to autostart.

Note: If a Network of Brokers is being used then the Peer Management Broker Service and Peer Management Topology Service can be left running on all PMC servers.

Best Practices

Things to consider during a failover event:

  • When the backup task runs from your primary PMC server, the entire current configuration, including the backup task itself, will be written to your secondary server. This means in the event of a failover, while you are running on your secondary PMC, the next time the backup task runs on this server, it will be trying to back up to itself (using the path that was configured on primary PMC server).

  • It is not recommended to run in a failover mode for a prolonged period of time. If you have to run on the secondary PMC for a prolonged period where configuration changes were made that need to be synced back to the primary PMC Server:

    1. Repeat Task 2 in the reverse direction, granting your secondary PMC server the rights to write to the primary PMC server.
    2. You must also edit your task on the secondary PMC server. Repeat Step 13 in Task 2 but modify the path of the backup task to include the primary server in the UNC path, rather than the secondary.
    3. Run the backup task one time prior to failing back to the primary PMC server.
    4. After failback to the primary PMC server, again edit the backup task to point to the UNC path of the secondary PMC server.
  • It is recommended that you also create a secondary backup task that targets a central network share to store another backup. This will ensure you always have a backup you can restore manually to either PMC server in the event you run into any issues backing up directly between your PMC servers at any point.
  • You will need to consider the hostname you use for your PeerGFS web server configuration if you intend to use the web access during a failover event. The web server port (default 8443) binds to the hostname specified in the configuration; you must ensure that you use a hostname that resolves to either PMC depending on which is the active one or use the default setting of "0.0.0.0", which should ensure access to either PMC web server using its NetBIOS/FQDN/IP address in the web server URL during normal operation or in the event of a failover event.