BeyondTrust Appliance B Series Recovery
B Series Appliances are available in virtual, hardware, and Cloud versions in addition to hosted sites, which run on shared B Series Appliances in BeyondTrust's data centers. When any of these go offline unexpectedly, the process necessary to repair and/or replace the failed site or B Series Appliance varies depending on the B Series Appliance or site in question. The various repair/replacement scenarios are described below so an effective strategy can be developed to prepare for them in advance.
Failover and Spare B Series Appliances
BeyondTrust recommends using a preconfigured failover relationship between a "primary" and a "backup" B Series Appliance. This ensures that the BeyondTrust software is available in the event either B Series Appliance should fail. BeyondTrust customer clients and representative consoles are built to attempt connection with the primary B Series Appliance at a specific address. In the event of a primary B Series Appliance failure, this address is used to redirect clients from the failed B Series Appliance to the backup B Series Appliance. This can done using one of three network routing methods: shared IP, DNS swing, or NAT swing. For more information, please see Configuring Failover .
Though client traffic is redirected to the backup B Series Appliance, this B Series Appliance does not accept connections until it takes the primary role. Once a backup B Series Appliance takes the primary role, it begins accepting client connections and provides all the same services the failed B Series Appliance did. This role change can be triggered manually or automatically.
Given the above information, here are the basic steps to take in the event of a primary B Series Appliance failure in a failover pair:
- Redirect network traffic from the primary to the backup B Series Appliance. If the B Series Appliances are configured with:
- Shared IP: The backup B Series Appliance automatically takes over the IP address of the failed B Series Appliance.
- DNS swing: Update the DNS A-record of the primary B Series Appliance to resolve the IP address of the backup B Series Appliance.
- NAT swing: Update the firewall NAT rule(s) to resolve the client-facing / public IP of the failed B Series Appliance to the private IP of the backup B Series Appliance.
- Make the backup B Series Appliance take over the primary role. If Enable Automatic Failover is:
- Enabled: If the backup B Series Appliance can reach the Network Connectivity Test IPs and cannot reach the primary B Series Appliance during the Primary Site Instance Timeout period, the backup B Series Appliance automatically takes the primary role.
- Disabled: Use the Become Primary button or the API command: set_failover_role. To use the button, log in to the backup B Series Appliance's /login administrative web interface. Browse to Management > Failover. Click Become Primary, leaving the adjacent box not checked.
- Confirm the clients are working, and proceed to perform maintenance on the failed B Series Appliance.
In the event that there is a cold spare instead of a failover B Series Appliance, begin the recovery process by restoring settings and data from the backup(s) to the spare B Series Appliance. Once the data is restored, redirect the client traffic to the spare B Series Appliance using DNS or NAT swing. If the spare B Series Appliance is on the same local network as the failed B Series Appliance, attempt to assign the IP of the failed B Series Appliance to the spare B Series Appliance. However, if the spare B Series Appliance is on the same switch as the failed B Series Appliance, this switch must be rebooted for the change to take effect.
RS Virtual Appliances
BeyondTrust's RS Virtual Appliances are certified for VMware vCenter 5.0+ and Hyper-V 2012 R2. These RS Virtual Appliance support virtual machine snapshots (VMware) and checkpoints (Hyper-V). A checkpoint or snapshot represents the state of a virtual machine at the time it was taken and includes the following:
- Files and memory state of the virtual machine's guest operating system
- Settings and configuration of the virtual machine and its virtual hardware.
BeyondTrust does not recommend or support creating snapshots (VMware) or checkpoints (Hyper-V) of actively running RS Virtual Appliances.
If the BeyondTrust RS Virtual Appliance experiences a failure and there is a recent snapshot or checkpoint, try restoring it first. This is often the fastest way to restore functionality.
If the BeyondTrust RS Virtual Appliance is under an active support maintenance contract, BeyondTrust Technical Support sends an up-to-date VMware or Hyper-V deployment file for the RS Virtual Appliance upon request the event of a failure and/or loss of the RS Virtual Appliance. To receive a copy, contact BeyondTrust Support with company information from an authorized email address. This address would normally be the same used to communicate with BeyondTrust during the initial deployment of the B Series Appliance and/or subsequent administrative-level incident management. A local copy of the RS Virtual Appliance file should be saved in case the RS Virtual Appliance needs to be restored outside of BeyondTrust Support's normal business hours.
To re-install the RS Virtual Appliance, follow the procedures outlined in the BeyondTrust RS Virtual Appliance Installation Guide . Access to the VMware or Hyper-V administrative management tool is needed to complete this process.
- Log in to the Hyper-V Manager or VMware infrastructure client.
- Deploy the BeyondTrust OVA (VMware) or EXE (Hyper-V) file.
- Use the Hyper-V Manager or VMware client to power on the RS Virtual Appliance.
- Open the virtual console.
- Enter the IP address, subnet mask, and default gateway of the RS Virtual Appliance.
The network settings of the RS Virtual Appliance should already be saved from previous configuration. Otherwise, contact the company network administrator for the appropriate settings. Once the RS Virtual Appliance is accessible on the network, log in to the /appliance administrative web interface, update the RS Virtual Appliance, and restore settings, as needed.
Hardware B Series Appliances
In the event of hardware failure, the first task is to restore the BeyondTrust service on the network. If there is a backup or spare hardware B Series Appliance, bring it online. If this is not an option, it may be possible to repair the failed B Series Appliance remotely with assistance from BeyondTrust Technical Support. In the event that this is not possible, BeyondTrust Technical Support can provide access to a temporary BeyondTrust site hosted on BeyondTrust's servers while new hardware is being shipped. Shipment of new hardware is covered under the maintenance contract with BeyondTrust, and BeyondTrust Support remains actively involved throughout the process of shipment, installation, and return of the failed hardware. BeyondTrust makes all reasonable attempts to help retain data but cannot guarantee that any or all data will remain intact through this process.
Temporary Hosted Site
As mentioned above, BeyondTrust can provide access to temporary hosted sites on shared B Series Appliances in BeyondTrust's data centers. However, hosted sites have limitations. In most cases, client software from the original site (e.g., Jump Clients, Jumpoints, and representative consoles) do not transfer to the hosted site, and new clients deployed from the hosted site are not normally transferred back to the original site, once it is back online.
Because hosted sites exist outside of the network, endpoints planning to be supported from a hosted site must be able to access these data centers over the public internet. If there is an up-to-date backup, user data can be uploaded and configured in the hosted site. If these users and/or configurations rely on resources internal to the network (e.g., Active Directory, RADIUS, and/or Kerberos servers), these resources are usually inaccessible from the hosted site. Traffic to and from the hosted site is encrypted with BeyondTrust's SSL certificates and connects with a domain name in BeyondTrust's namespace (e.g., "tempsite.beyondtrust.com"). The original company certificate(s) and hostname(s) are not used.
If a temporary hosted site would be helpful, follow these steps to obtain a hosted site and to upload the settings of a backup:
- Contact BeyondTrust Technical Support at www.beyondtrust.com/docs/index.htm#support.
- Wait for credentials.
- Log in to the new site.
- Go to Management > Software Management.
- Use Restore Settings to browse for the software backup.
- Enter the backup password created, if applicable.
- Click Upload Backup.
Return Materials Authorization (RMA)
As outlined in the BeyondTrust Maintenance Service Agreement, BeyondTrust Technical Support provides support for BeyondTrust hardware. Every reasonable attempt is made to restore the BeyondTrust Appliance B Series to full operation while still at the location. This typically involves a technician going to wherever the B Series Appliance is located and connecting a PS/2 keyboard and VGA monitor to the back of the B Series Appliance. To do this, a torx 8 or 10 screwdriver is required to remove the backplate. It is best practice to have all this hardware on-site prior to a failure event. If the B Series Appliance is racked in a data center with a KVM switch, this can be used instead. It is also best practice to allow the B Series Appliance outbound access to the internet over TCP 443. This allows BeyondTrust Technical Support to establish a secure connection with the B Series Appliance for low-level troubleshooting and recovery. If BeyondTrust Technical Support determines that at-location repair is not possible, the RMA process is initiated.
Hardware RMAs may include one or more completely new B Series Appliances and/or Field Replaceable Units (FRU). An FRU can be issued only for B300 and B400 B Series Appliances. Replaceable items include hard disk drives and power supplies. Any other hardware problems result in a full RMA of the entire B Series Appliance. To process an RMA, BeyondTrust Support needs the following information:
- Contact name, email, and phone
- Shipping address
- B Series Appliance serial number
- Value Added Tax (VAT) number, if necessary
When shipping BeyondTrust hardware internationally, BeyondTrust Technical Support must begin by confirming whether the B Series Appliance is to be returned to BeyondTrust temporarily and shipped back after repair or returned to BeyondTrust permanently and replaced with new hardware. The former option can take up to four weeks. Because the latter option may incur VAT fees for international shipments, it is important to send documented consent to BeyondTrust Technical Support for the payment of any fees incurred by this process. Replacements for B Series Appliances located in Ireland or the European Union are usually shipped from BeyondTrust's inventory in Ireland. No VAT fees are expected for these shipments.
Once the RMA request has been opened, a printable packing slip for the old B Series Appliance is emailed along with tracking and setup instructions for the new or repaired hardware. For RMAs in the United States, a return shipping label is sent. For B Series Appliances outside the United States, either BeyondTrust's DHL account number is given to schedule a DHL pickup, or BeyondTrust Technical Support arranges for the Waste Electrical and Electronic Equipment (WEEE) scrapping of the B Series Appliance. In any of these cases, BeyondTrust Technical Support follows up to ensure the replacement hardware is installed and functional before closing the RMA incident.
If replacement hardware was sent prior to the receipt of the failed hardware, BeyondTrust Technical Support must verify that the old B Series Appliance is returned. If a replacement B Series Appliance is received, it is a requirement that the defective B Series Appliance be returned to BeyondTrust within two weeks of the new B Series Appliance becoming operational. Outstanding B Series Appliances are invoiced at list price. The packing materials of a new B Series Appliance, the above-mentioned DHL number, and the return shipping label can be used to return the old B Series Appliance.
Once the replacement B Series Appliance has been received and powered on, the settings can be restored using either the local console or /appliance administrative web interfaces. It is usually more convenient to use the web interfaces. The /appliance administrative web interface of a hardware B Series Appliance can be accessed locally using either of its NIC ports. These are provisioned with non-routable IP address(es) on the 169.254.1.0/16 network. To gain access, follow these steps:
- Rack the B Series Appliance.
- Plug the power cable into a safe power source.
- Use a patch cable to connect a computer to NIC1 or NIC2 on the rear of the BeyondTrust Appliance B Series.
- Press and release the Power button on the front of the B Series Appliance.
- Edit the IP configuration of the connected computer:
- IP address: 169.254.1.5
- Subnet mask: 255.255.0.0
- Default gateway: none
- DNS server: none
- Wait for the B Series Appliance to finish booting.
- Launch a web browser.
- Enter the address https://169.254.1.1/appliance/login.ns in the URL address field.
The /appliance login page should load. If not, try alternately substituting ".2", ".3" and ".4" for the last decimal in the address above. Load each of these addresses separately until one responds. If none of these responds, try all four addresses using the other NIC of the B Series Appliance.
Once the /appliance login page has loaded, log in and enter the IP network settings of the B Series Appliance. Replacement B Series Appliances typically ship with default login credentials. For IP settings, reference a saved copy of the IP settings from the previous B Series Appliance. To restore IP configuration, follow these steps:
- Log in with the default credentials, admin and password.
- Enter a new password.
- Save this password in a secure location. It is difficult to recover if lost.
- Browse to Networking > IP Configuration.
- Click Add New IP.
- Fill out the resulting page with settings appropriate for the network.
- Save the new IP address.
- Add the default gateway in the Global Configuration section.
These are the basic settings required to bring a B Series Appliance online in most situations; however, the network may have additional DNS, NTP, syslog, SMTP, and/or other servers and settings. Ideally, these have been saved in backup files. Like IP configuration, these other settings cannot be restored automatically. Instead, they must be manually entered. Once all the /appliance settings and/or IP configuration settings are restored, proceed to restore the certificates and /login administrative interface.
BeyondTrust Software-as-a-Service (SaaS)
BeyondTrust's SaaS products include Starter Service Licenses and BeyondTrust Cloud Appliances. Both of these are hosted by BeyondTrust, but Starter Service Licenses do not provide all of the isolation or functionality provided by Cloud Appliances. BeyondTrust's Starter Service is limited to five total licenses by default, and their functionality is more limited in some ways than regular licenses. In contrast, each BeyondTrust Cloud instance is a single-tenant RS Virtual Appliance and has none of the Starter Service licensing limitations.
In the event that the Production Starter Service or Cloud Appliance site goes down and becomes inoperable unexpectedly, follow these steps:
- Log in to the /login administrative web interface and attempt a restart.
- If a Cloud Appliance is owned, log in to the /appliance administrative web interface. Check the settings to ensure there are no errors, warnings, or anomalies.
- If all this fails, try to find a bypass or workaround solution (such as an alternative site or remote access product).
- Contact BeyondTrust Technical Support with a description of the situation and the steps taken thus far. In cases where the production site is offline and there is no workaround or alternative solution, BeyondTrust Technical Support's maximum target time for First Response (start of resolution) is 30 minutes within normal business hours.