BeyondTrust Appliances are available in virtual, hardware, and Cloud versions in addition to hosted sites, which run on shared appliances in BeyondTrust's data centers. When any of these go offline unexpectedly, the process necessary to repair and/or replace the failed site or appliance varies depending on the appliance or site in question. The various repair/replacement scenarios are described below so an effective strategy can be developed to prepare for them in advance.
Failover and Spare Appliances
BeyondTrust recommends using a preconfigured failover relationship between a "primary" and a "backup" BeyondTrust Appliance. This ensures that the BeyondTrust software is available in the event either BeyondTrust Appliance should fail. BeyondTrust customer clients and representative consoles are built to attempt connection with the primary BeyondTrust Appliance at a specific address. In the event of a primary appliance failure, this address is used to redirect clients from the failed appliance to the backup appliance. This can done using one of three network routing methods: shared IP, DNS swing, or NAT swing. For more information, please see Configuring Failover .
Though client traffic is redirected to the backup appliance, this appliance does not accept connections until it takes the primary role. Once a backup appliance takes the primary role, it begins accepting client connections and provides all the same services the failed appliance did. This role change can be triggered manually or automatically.
Given the above information, here are the basic steps to take in the event of a primary appliance failure in a failover pair:
- Redirect network traffic from the primary to the backup appliance. If the appliances are configured with:
- Shared IP: The backup appliance automatically takes over the IP address of the failed appliance.
- DNS swing: Update the DNS A-record of the primary appliance to resolve the IP address of the backup appliance.
- NAT swing: Update the firewall NAT rule(s) to resolve the client-facing / public IP of the failed appliance to the private IP of the backup appliance.
- Make the backup appliance take over the primary role. If Enable Automatic Failover is:
- Enabled: If the backup appliance can reach the Network Connectivity Test IPs and cannot reach the primary appliance during the Primary Site Instance Timeout period, the backup appliance automatically takes the primary role.
- Disabled: Use the Become Primary button or the API command: set_failover_role. To use the button, log into the backup appliance's /login administrative web interface. Browse to Management > Failover. Click Become Primary, leaving the adjacent box not checked.
- Confirm the clients are working, and proceed to perform maintenance on the failed appliance.
In the event that there is a cold spare instead of a failover appliance, begin the recovery process by restoring settings and data from the backup(s) to the spare appliance. Once the data is restored, redirect the client traffic to the spare appliance using DNS or NAT swing. If the spare appliance is on the same local network as the failed appliance, attempt to assign the IP of the failed appliance to the spare appliance. However, if the spare appliance is on the same switch as the failed appliance, this switch must be rebooted for the change to take effect.
BeyondTrust's virtual appliances are certified for VMware vCenter 5.0+ and Hyper-V 2012 R2. These appliances support virtual machine snapshots (VMware) and checkpoints (Hyper-V). A checkpoint or snapshot represents the state of a virtual machine at the time it was taken and includes the following:
- Files and memory state of the virtual machine's guest operating system
- Settings and configuration of the virtual machine and its virtual hardware.
- If the BeyondTrust Virtual Appliance experiences a failure and there is a recent snapshot or checkpoint, try restoring it first. This is often the fastest way to restore functionality.
If the BeyondTrust Virtual Appliance is under an active support maintenance contract, BeyondTrust Support sends an up-to-date VMware or Hyper-V deployment file for the appliance upon request the event of a failure and/or loss of the virtual appliance. To receive a copy, contact BeyondTrust Support with company information from an authorized email address. This address would normally be the same used to communicate with BeyondTrust during the initial deployment of the appliance and/or subsequent administrative-level incident management. A local copy of the virtual appliance file should be saved in case the appliance needs to be restored outside of BeyondTrust Support's normal business hours.
To re-install the virtual appliance, follow the procedures outlined in the BeyondTrust Virtual Appliance Installation Guide . Access to the VMware or Hyper-V administrative management tool is needed to complete this process.
- Log into the Hyper-V Manager or VMware infrastructure client.
- Deploy the BeyondTrust OVA (VMware) or EXE (Hyper-V) file.
- Use the Hyper-V Manager or VMware client to power on the virtual appliance.
- Open the virtual console.
- Enter the IP address, subnet mask, and default gateway of the appliance.
The network settings of the appliance should already be saved from previous configuration. Otherwise, contact the company network administrator for the appropriate settings. Once the appliance is accessible on the network, log into the /appliance administrative web interface, update the appliance, and restore settings, as needed.
In the event of hardware failure, the first task is to restore the BeyondTrust service on the network. If there is a backup or spare hardware appliance, bring it online. If this is not an option, it may be possible to repair the failed appliance remotely with assistance from BeyondTrust Support. In the event that this is not possible, BeyondTrust Support can provide access to a temporary BeyondTrust site hosted on BeyondTrust's servers while new hardware is being shipped. Shipment of new hardware is covered under the maintenance contract with BeyondTrust, and BeyondTrust Support remains actively involved throughout the the process of shipment, installation, and return of the failed hardware. BeyondTrust makes all reasonable attempts to help retain data but cannot guarantee that any or all data will remain intact through this process.
Temporary Hosted Site
As mentioned above, BeyondTrust can provide access to temporary hosted sites on shared appliances in BeyondTrust's data centers. However, hosted sites have limitations. In most cases, client software from the original site (e.g., Jump Clients, Jumpoints, and representative consoles) do not transfer to the hosted site, and new clients deployed from the hosted site are not normally transferred back to the original site, once it is back online.
Because hosted sites exist outside of the network, endpoints planning to be supported from a hosted site must be able to access these data centers over the public internet. If there is an up-to-date backup, user data can be uploaded and configured in the hosted site. If these users and/or configurations rely on resources internal to the network (e.g., Active Directory, RADIUS, and/or Kerberos servers), these resources are usually inaccessible from the hosted site. Traffic to and from the hosted site is encrypted with BeyondTrust's SSL certificates and connects with a domain name in BeyondTrust's namespace (e.g., "tempsite.beyondtrust.com"). The original company certificate(s) and hostname(s) are not used.
If a temporary hosted site would be helpful, follow these steps to obtain a hosted site and to upload the settings of a backup:
- Contact BeyondTrust Technical Support at help.bomgar.com.
- Wait for credentials.
- Log into the new site.
- Go to Management > Software Management.
- Use Restore Settings to browse for the software backup.
- Enter the backup password created, if applicable.
- Click Upload Backup.
Return Materials Authorization (RMA)
As outlined in the BeyondTrust Maintenance Service Agreement, BeyondTrust Support provides support for BeyondTrust hardware. Every reasonable attempt is made to restore BeyondTrust Appliances to full operation while still at the location. This typically involves a technician going to wherever the appliance is located and connecting a PS/2 keyboard and VGA monitor to the back of the appliance. To do this, a torx 8 or 10 screwdriver is required to remove the backplate. It is best practice to have all this hardware on-site prior to a failure event. If the appliance is racked in a data center with a KVM switch, this can be used instead. It is also best practice to allow the appliance outbound access to the internet over TCP 443. This allows BeyondTrust Support to establish a secure connection with the appliance for low-level troubleshooting and recovery. If BeyondTrust Support determines that at-location repair is not possible, the RMA process is initiated.
Hardware RMAs may include one or more completely new appliances and/or Field Replaceable Units (FRU). An FRU can be issued only for B300 and B400 appliances. Replaceable items include hard disk drives and power supplies. Any other hardware problems result in a full RMA of the entire appliance. To process an RMA, BeyondTrust Support needs the following information:
- Contact name, email, and phone
- Shipping address
- Appliance serial number
- Value Added Tax (VAT) number, if necessary
When shipping BeyondTrust hardware internationally, BeyondTrust Support must begin by confirming whether the appliance is to be returned to BeyondTrust temporarily and shipped back after repair or returned to BeyondTrust permanently and replaced with new hardware.The former option can take up to four weeks. Because the latter option may incur VAT fees for international shipments, it is important to send documented consent to BeyondTrust Support for the payment of any fees incurred by this process. Replacements for appliances located in Ireland or the European Union are usually shipped from BeyondTrust's inventory in Ireland. No VAT fees are expected for these shipments.
Once the RMA request has been opened, a printable packing slip for the old appliance is emailed along with tracking and setup instructions for the new or repaired hardware. For RMAs in the United States, a return shipping label is sent. For appliances outside the United States, either BeyondTrust's DHL account number is given to schedule a DHL pickup, or BeyondTrust Support arranges for the Waste Electrical and Electronic Equipment (WEEE) scrapping of the appliance. In any of these cases, BeyondTrust Support follows up to ensure the replacement hardware is installed and functional before closing the RMA incident.
If replacement hardware was sent prior to the receipt of the failed hardware, BeyondTrust Support must verify that the old appliance is returned. If a replacement appliance is received, it is a requirement that the defective appliance be returned to BeyondTrust within two weeks of the new appliance becoming operational. Outstanding appliances are invoiced at list price. The packing materials of a new appliance, the above-mentioned DHL number, and the return shipping label can be used to return the old appliance.
Once the replacement appliance has been received and powered on, the settings can be restored using either the local console or appliance administrative web interfaces. It is usually more convenient to use the web interfaces. The /appliance administrative web interfaces of a hardware appliance can be accessed locally using either of its NIC ports. These are provisioned with non-routable IP address(es) on the 169.254.1.0/16 network. To gain access, follow these steps:
- Rack the appliance.
- Plug the power cable into a safe power source.
- Use a patch cable to connect a computer to NIC1 or NIC2 on the rear of the BeyondTrust Appliance.
- Press and release the Power button on the front of the appliance.
- Edit the IP configuration of the connected computer:
- IP address: 169.254.1.5
- Subnet mask: 255.255.0.0
- Default gateway: none
- DNS server: none
- Wait for the appliance to finish booting.
- Launch a web browser.
- Enter the address https://169.254.1.1/appliance/login.ns in the URL address field.
The /appliance login page should load. If not, try alternately substituting ".2", ".3" and ".4" for the last decimal in the address above. Load each of these addresses separately until one responds. If none of these responds, try all four addresses using the other NIC of the appliance.
Once the /appliance login page has loaded, log in and enter the IP network settings of the appliance. Replacement appliances typically ship with default login credentials. For IP settings, reference a saved copy of the IP settings from the previous appliance. To restore IP configuration, follow these steps:
- Log in with the default credentials, admin and password.
- Enter a new password.
- Save this password in a secure location. It is difficult to recover if lost.
- Browse to Networking > IP Configuration.
- Click Add New IP.
- Fill out the resulting page with settings appropriate for the network.
- Save the new IP address.
- Add the default gateway in the Global Configuration section.
These are the basic settings required to bring an appliance online in most situations; however, the network may have additional DNS, NTP, syslog, SMTP, and/or other servers and settings. Ideally, these have been saved in backup files. Like IP configuration, these other settings cannot be restored automatically. Instead, they must be manually entered. Once all the /appliance settings and/or IP configuration settings are restored, proceed to restore the certificates and /login administrative interface.
BeyondTrust Software-as-a-Service (SaaS)
BeyondTrust's SaaS products include Starter Service Licenses and BeyondTrust Cloud Appliances. Both of these are hosted by BeyondTrust, but Starter Service Licenses do not provide all of the isolation or functionality provided by Cloud Appliances. BeyondTrust's Starter Service is limited to five total licenses by default, and their functionality is more limited in some ways than regular licenses. In contrast, each BeyondTrust Cloud instance is a single-tenant virtual appliance and has none of the Starter Service licensing limitations.
In the event that the Production Starter Service or Cloud Appliance site goes down and becomes inoperable unexpectedly, follow these steps:
- Log into the /login administrative web interface and attempt a restart.
- If a Cloud Appliance is owned, log into the /appliance administrative web interface. Check the settings to ensure there are no errors, warnings, or anomalies.
- If all this fails, try to find a bypass or workaround solution (such as an alternative site or remote access product).
- Contact BeyondTrust Support with a description of the situation and the steps taken thus far. In cases where the production site is offline and there is no workaround or alternative solution, BeyondTrust Support's maximum target time for First Response (start of resolution) is 30 minutes within normal business hours.