Use High Availability with U-Series Appliances

High availability (HA) is designed to work in an active / passive configuration. At any time, one of your two servers has the role of the active node, while the other is the passive node. When the passive server detects that the active server has failed, then the passive is promoted to active, and the active is demoted.

Turn on High Availability Pairing

Before setting up high availability, you must turn on the High Availability role in the Roles Editor for both the active and passive U-Series Appliances. For more information, please see High Availability Role.

  1. Log in to the U-Series Appliance web site on the primary server.
  2. From the menu, select Roles Editor.
  3. Click High Availability, then select a mirroring option:
    • HA will mirror both Server and Database
    • HA mirroring for services only

To save resources, you can turn off services that are not required to run on any secondary U-Series Appliances. Check the Standalone Password Safe Worker Node box. Check the corresponding boxes to turn off services: Disable BeyondInsight UI or Disable Password Safe UI.

  1. Click Apply Changes.
  2. On the main Roles Editor page, click Apply Pending Changes.
  3. Repeat these steps for the secondary server.

Configure High Availability

  1. Log in to the U-Series Appliance, and then select High Availability. For a first-time configuration, the Initial Setup page displays. Certificates must be set up between the U-Series Appliances for secure communication.
  2. Click Go to the API Key Maintenance Page.
  3. Copy the API registration keys between the partner U-Series Appliances. Registering the API key with the partner U-Series Appliance permits secure communication between the U-Series Appliances.
  4. Enter the host name of the passive U-Series Appliance, then click Apply.
  5. A message displays that the exchange is in progress. If an error occurs during the certificate exchange, a Show/Hide Results button displays. Exchanging certificates can take up to approximately five minutes. After the certificates are exchanged with no errors, the configuration settings display.

 

Toggle High Availability

  1. Toggle the High Availability switch on to turn on the feature.
  2. Enter the mirroring port number. The default port is 5022.
  3. Click Set High Availability.

 

High Availability Settings

  1. For Partner Contact Timeout, enter the number of minutes that pass with no contact between the active server and passive server. When the active server receives no response from the passive server, then the active continues to start. If the passive server has no contact with the active, the passive server starts up as the active one.
  2. For Partner Failover Timeout, enter the number of minutes that pass with no ping received from the primary server. After this time, the passive server switches to the active one.
  3. For Reboot Blackout Window, enter the number of minutes that should pass before the passive server takes control. On graceful shutdown, the passive server switches to the active one after no response for this length of time.

    This is useful when you want to shut down the active U-Series Appliance but do not want the passive U-Series Appliance to take control. For example, you might want to move the active U-Series Appliance and know this will take about thirty minutes. To be sure the passive U-Series Appliance does not take control while the active U-Series Appliance is offline, set this value to sixty minutes.

You must shut down the primary U-Series Appliance from the Maintenance > Schedule a Reboot page.

  1. We recommend that you enable Attempt Auto-Resync only for testing scenarios.
  2. Synchronize Session Archving Files synchronizes local session recording files from Password Safe with the partner U-Series Appliance. This allows you to replay the session recordings from within Password Safe if a failover occurs and the passive U-Series Appliance is made active.
  3. You can select Send Alerts on Failover to send either an email or events to BeyondInsight.
  4. If you select Medium Failover Mode, then when communication between the pairs is lost, the passive U-Series Appliance is in a failover-pending state only. Action is required on your part to start a failover process.
  5. In Background Settings Update Rate, enter the number of minutes that pass before a file synchronization occurs. Files copied to the passive server are configuration files, certificates, and registry files.
  6. Set the Failed Notification Rate to provide notification after your active U-Series Appliance has failed over. If you are using medium failover mode, the email indicates that action is required on your part. The default value is fifteen minutes.
  7. You can click Queue File Synchronization to start a file synchronization.
  8. Click Update Settings.

For more information, please see the following:

Use a Load Balancer in an Active / Passive Configuration

When setting up an active / passive pair, you might want to configure a load balancer that acts as a DNS redirector. Configure the load balancer between two U-Series Appliances so that it can determine which U-Series Appliance is active and which is passive. The load balancer then sends the traffic to the active U-Series Appliance.

You can use the following endpoint API to configure the load balancer. Refer to your load balancer documentation to ensure that it is configured to use the endpoints.

GET https://<ApplianceAddress>/UVMInterface/api/HighAvailability

The code above returns an object with one member:

{
string Role;
}

You can set the formatting of the requested return value in the Content-Type request header.

To return a value in JSON format, you can specify:
Content-Type: application/json;charset=UTF-8

The available values for Role are:

  • Off: High Availability is not turned on.
  • Active: The U-Series Appliance is in active mode.
  • Passive: The U-Series Appliance is in passive mode.

Test High-Availability Failover

You can use Attempt Auto-Resync as a quick way to restore high availability in a scenario where databases on the active and passive servers are synchronized. We do not recommend a production failover scenario. Data loss can occur if databases are not synchronized.

  1. Select Attempt Auto Resync of database when connecting after failover.
  2. Unplug or power off the active server.
  3. Wait for failover. Ensure that the passive is now the active.
  4. Restore the active server (turn on or plug in).
  5. The auto re-sync restores the high-availability configuration.
  6. The passive server is now acting as the active server. Click Switch Roles to restore the server partners to their original roles.

Use Medium Failover Mode

Use medium failover mode when you do not want the services on the passive U-Series Appliance to start automatically when the communication between pairs is lost.

The passive U-Series Appliance waits in a pending state until you manually start the failover process. When the active U-Series Appliance fails, you must log in to the U-Series Appliance software to start the failover process to the passive U-Series Appliance.

  1. Log in to the U-Series Appliance, and then select High Availability.
  2. In the High Availability Maintenance section, click Failover to this U-Series Appliance to start the services and database.

This button is active only when the primary U-Series Appliance is down.

Resume and Suspend SQL Mirroring

You might want to pause mirroring if you want to take care of maintenance tasks on the database server. A failover cannot occur when the database is in a suspended state.

  1. Log in to the U-Series Appliance, and then select High Availability.
  2. Click Suspend to pause mirroring.
  3. Click Resume to start mirroring again.

If the U-Series Appliance is in a failover state and mirroring is suspended, you can click Resume to start mirroring.

Discard High-Availability Configuration Settings

To reset the U-Series Appliances to the initial setup state, you can remove all high-availability configuration settings established between U-Series Appliances. You might want to do this if you want to set up new high-availability pairs.

  1. Log in to the U-Series Appliance, and then select High Availability.
  2. Click Abandon Configuration.

Recognize a Failover

Review the following to help you determine if a failover has occurred.

  • If you are using a U-Series Appliance version 1.5.4 or later, an email is sent to the address set in the Configuration Wizard. If you are using a U-Series Appliance version earlier than 1.5.4, you can contact BeyondTrust Technical Support to activate the email feature.
  • If you are not using a load balancer, you might notice that BeyondInsight is no longer responsive on the active server.
  • On the Diagnostics web site (for the primary), only two tabs are displayed. This indicates that the server is in passive mode.
  • Confirm that the passive server is in active mode.

Prepare for Disaster Recovery

If you are using high availability as a disaster recovery solution, review the following points as a guide to restoring roles.

  • Determine if the active server has failed. Confirm the role of the live server (the primary).
  • If a failure has occurred on the primary, investigate and resolve issues on the primary.
  • After a failover to the disaster recovery server (the secondary), you can restore roles on the active server's web site.

Verify Connectivity between Servers

On the High Availability Configuration page, verify that the communication between U-Series Appliances is active. The Last Heartbeat indicates the last ping to the passive server and the return response to the active server.

Check the Database Status after a Failover

 

In all scenarios, we strongly recommend investigating the cause of the failure. We do not recommend resuming database mirroring until issues are resolved.

The following database status indicators might display after a failover:

  • DISCONNECTED: Failover was catastrophic, and the server is completely unavailable or unreachable. Turn off high availability and investigate the issues with the failed server. After the failed server is cleared for use, turn on high availability and synchronize the databases.
  • EXPOSED: The other server is still available and possibly still healthy, but the failover was serious or lengthy enough to disable high availability. After the failed server is cleared for use, turn on high availability and synchronize the databases.
  • SUSPENDED: The interruption was of a minor or transient nature. While it may be possible to restore connectivity without disabling high availability, we recommend that you turn off high availability and investigate the issues with the server. After the failed server is cleared for use, turn on high availability and synchronize the databases. Optionally, contact BeyondTrust Technical Support to see if mirroring can be restored.

Restore Roles After a Failover

After a failure has been identified and resolved on a U-Series Appliance, you can restore the roles to the initial state. Log in to the U-Series Appliance, and then select High Availability. Then click Switch Roles.

Review Database Metrics

Historical Sync Rate and BeyondInsight Database Size are shown

On the High Availability Settings page, review information about earlier database synchronizations and the size of the current database.

You can then determine from these values how long a synchronization between servers might take.

Check the status of the BeyondInsight mirror state on the High Availability tab to ensure that synchronizations are occurring between the active and passive servers.

State Description

EXPOSED

Databases are not mirrored.

SYNC PENDING: INITIAL DB SYNC STARTED

The process of backing up and transferring the database to the passive server has begun.

SYNC PENDING: SET MIRROR CALLED

The database has been transferred and restored to the passive server. Mirroring is being turned on.

SYNCHRONIZING

The server is actively transmitting transaction logs to the other database to apply changes.

EXPOSED: MAX SYNC ATTEMPTS REACHED

Five consecutive attempts were made and failed to establish mirroring. Mirroring was not established and is no longer trying. To troubleshoot, check for connectivity issues and ensure the database mirror port is set to 5022.

SYNCHRONIZED

Databases are actively mirrored. High availability is considered to be working.