Upgrade an Azure Deployment

There are several steps you need to go through for the Azure deployments. Be sure to download the latest build for the version of PMC that you are upgrading to. It is in the File Downloads area of the Customer Support Portal.

 

PMC 2.4 SR1 is compatible only with Reporting database 5.5. If you do not intend to upgrade your Reporting database to 5.5, please do not proceed with the upgrade of PMC.

Turn on your Jump Box

You need to use your jump box to upgrade your Azure deployment. You should have disabled this after you deployed PMC. To re-enable it:

  1. Go to the Azure Portal.
  2. Navigate to the resource group for this PMC installation.
  3. Click the jump box virtual machine (VM).
  4. Click Start.
  5. The machine will now start up. Ensure you turn your jump box off when you have finished the upgrade.

Upgrade the Management Database

Prior to upgrading your application, you need to ensure your database is up-to-date, as this process is not managed with the upgrade scripts.

Prerequisites

You need to upgrade the Avecto.IC3.Database.Management database before you upgrade the application.

For more information, please see Upgrade the Application.

Upgrade the Management Database

  1. Go to the Azure Portal.
  2. Navigate to the Resource Group for this PMC installation.
  3. Click the Type column header to order the list by type.
  4. Click the SQL database called Avecto.IC3.Management (with some characters post-fixed).

Locate the PMC server name.

  1. Locate the Server name. This is shown on the top right:

 

  1. Using SQL Server Management Studio, log in to the database using the Server name from Azure and the SQL administration credentials you created when you deployed PMC.
  2. After you have successfully connected, expand the Databases node under Object Explorer, right-click on the Avecto.IC3.Database.Management database, and click New Query.
  3. Select File > Open > File and navigate to the AzurePaaS\DeployDatabases\SQL folder for the version you are upgrading to.
  4. Locate the Avecto.IC3.Database.Management.sql script. This contains all the database migrations required to perform an upgrade.
  5. Run the script by pressing F5, or click Execute.

Copy and execute the following query to confirm that your upgrade was successful: 

Select Top (1000) [MigrationID]
   ,[ContextKey]
   ,[Model]
   ,[ProductVersion] 
FROM [dbo].[__MigrationHistory]

Upgrade the Application

Enable WinRM with SSL on the Portal VM

  1. Connect to your Portal VM and copy the Enable-WinRMWithSSL.ps1 script from the AzurePaas folder to the Portal VM.
  2. Run PowerShell as an administrator and navigate to the location of Enable-WinRMWithSSL.ps1
  3. Type .\Enable-WinRMWithSSL -SubjectName portalVm -ForceNewSSLCert.

Perform Upgrade on the Jump Box VM

You need the AzurePaaS folder for the version of PMC that you are upgrading to.

  1. On the jump box VM that you turned on, copy the Upgrades folder from the build you wish to upgrade to onto the jump box. This contains all the files needed to prepare and upgrade your environment.

If you need to change any values in the configuration (for example, the location of the portal and connection strings), you must provide them as arguments to the PrepUpgradeConfig.ps1 script before you run it. For more information, please see Change Application Parameters Before Upgrade.

  1. Copy your ClusterAdminCert.pfx file to the jump box. This certificate should have been placed in a secure location after the deployment and removed from the jump box.
  1. Import the certificate into your Current User > Personal location.

If you changed the default location of the portal when you installed PMC, you need to provide the following argument to the upgrade script before you run it:

-PortalWebsiteVmLocation "C:\MyFolder\PMC"
  1. You are now ready to run the PrepUpgradeConfig.ps1 script. If you changed the location of the portal from the default value, you need to supply it as an optional argument.
  2. For example, in an elevated PowerShell window, type PrepUpgradeConfig.ps1 -PortalWebsiteVmLocation "C:\MyFolder\PMC". When you press Enter, you will be prompted for the mandatory parameters listed below. If you did not change the location and do not need to change any other parameters, type PrepUpgradeConfig.ps1 and press Enter.

    • ClusterEndpoint: Your DNS with :19000 postfixed. For example, PMCtest.example.com:19000 (HTTPS:// is not required).
    • ClusterAdminThumbprint: The thumbprint output during initial deployment for the PMC Cluster Admin certificate.
    • ServerCertThumbprint: The thumbprint output during the deployment for the PMC Cluster Admin certificate (same as the ClusterAdminThumbprint).
    • PortalVmAdminUsername: The administrator username for the portal machine that was entered in the initial deployment.
    • PortalVmPassword: The password for the portal machine that was entered in the initial deployment.
    • PortalVmIpAddress: The IP address of the portal machine.
    • ParametersConfigFilePath: The full file path of the parameter config file in the Upgrades folder. For example, C:\Users\myuser\Desktop\Upgrades\Production.5node.xml
    • WebConfigFilePath: The full file path of the web config file in the Upgrades folder. For example, C:\Users\myuser\Desktop\Upgrades\Web.Production.config

When this script is executed, a text file containing all of the original values is output to the location in which the script is run. This will need to be saved to a secure location in case these values are needed. In the event that they are needed, the required value will need to be copied from this text file into the config file.

  1. Use Remote Desktop (RDP) to connect to the jump box (ensure you have the Cluster Administration *.pfx certificate portion installed on the machine before continuing).
  2. Open Powershell as admin and run the UpdateServiceFabricAppSetting.ps1 (in the Upgrades folder) script with the following parameters:
    • ClusterAddress: The DNS Name of your cluster postfixed with :19000. For example, PMCcert.PMC:19000.
    • ServerCertThumbprint: The thumbprint of the ClusterAdminCertificate.
    • ClusterAdminThumbprint: The thumbprint of the ClusterAdminCertificate (same as ServerCertThumbprint).
    • UpdateConfigParameters: The event pump service Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled set to true.
    .\UpdateServiceFabricAppSetting.ps1 -ClusterAddress "pmc.domain.com:19000" -ServerCertThumbprint "54761d496fe75fd4fe81a488fa709e4e79613385" -ClusterAdminThumbprint "54761d496fe75fd4fe81a488fa709e4e79613385 " -UpdateConfigParameters @{"Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled"  = "true";”Avecto.IC3.JobAgent.DeploymentType” = “0”;}
  3. The update will apply to each node one at a time. You can check update status through Service Fabric Manager.
  4. Once the update is complete, run the following command in PowerShell to check if the setting is applied.
    Get-ServiceFabricApplication -ApplicationName fabric:/IC3.Fabric

    This will output the application configuration.

    • The Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled parameter should now be set to true.
    • The Avecto.IC3.JobAgent.DeploymentType should be set to 0.
  5. Through SSMS, check to make sure the CopyFromStaging job has finished running.
  6. Copy the Package.zip folder from the AzurePaaS folder (the version you are upgrading to) to your jump box and unzip it.
  1. From your PowerShell instance, navigate to the UpgradeApp.ps1 script in the Upgrades folder and provide the following parameters:
    • PackagePath: The path to the unzipped Package folder you copied over. For example C:\Users\myuser\Desktop\Package
    • AppParamsPath: The location of the Production.5Node.xml file in the Upgrades folder. For example, C:\Users\myuser\Desktop\Upgrades\Production.5node.xml.
    • ClusterAddress: Your DNS with :19000 postfixed. For example, PMCtest.example.com:19000 (HTTPS:// is not required).
    • ClusterAdminThumbprint: The thumbprint output during the deployment for the PMC Cluster Admin certificate.
    • ServerCertThumbprint: The thumbprint output during the deployment for the PMC Cluster Admin certificate (same as the ClusterAdminThumbprint).
  2. The script will run and begin the upgrade process. To check the progress, navigate to Service Fabric explorer, expand the cluster and select Applications from the tree view. In the right-hand work pane, you will see Upgrades in progress text. Click on this to see the progress for each node. It shows the current version and the target version you are upgrading to. During the upgrade, Service Fabric will display several warnings as each domain is taken down. Upon completion of an upgrade, these warnings should be removed. During the upgrade, the policy on endpoints is still be applied and the policy will remain functional.

For more information, please see Privilege Management Console Post-Deployment Steps.

Additional Upgrade Scripts

Security improvements have been made in PMC 2.4. SR1. The following scripts are available to apply those security improvements to your existing environment.

Before running any of the post-upgrade scripts, you need to install AzureRm: install-module -name azurerm –allowclobber.

If an error message displays similar to Unable to download the list of available providers, check your internet connection. Run the following command to enforce the TLS 1.2 protocol:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

This command only affects the current PowerShell session and does not persist.

PMC 2.4 SR1 does not support communication over TLS 1.1 or older due to security vulnerabilities. Run the following script to harden your TLS infrastructure.

…\AzurePaaS\Upgrades\PostUpgradeHardenTLSSecurity.ps1

 

Do not run this script against a dual load-balancer configuration. This script is only intended to be run against the internal load-balancer configuration (default) of PMC.

For PMC 2.4 SR1, improvements on node security have been implemented. Run the following script to apply those security updates to your environment.

…\AzurePaaS\Upgrades\PostDeploymentInfrastructureHardening.ps1

Check for Successful Application Upgrade

You can check whether your upgrade was successful by navigating to ClusterApplications in Service Fabric. The application shown on the right should match the version you have upgraded to.

Upgrade the Service Fabric Cluster Durability Level

This section only applies if you are upgrading from 2.x to 2.4 SR1.

The current Service Fabric Cluster Durability Level is set to Bronze. We strongly recommend running the PowerShell script UpgradeClusterDurabilityTier.ps1 to upgrade the cluster to the Silver level, which reduces the possibility of quorum loss in the stateful services.

The script is located in the following directory:

\v2xxx\AzurePaaS\Upgrades

Script Prerequisites

  • AzureRM PowerShell module
  • Azure login credentials

To run the script from PowerShell:

.\UpgradeClusterDurabilityTier.ps1 -resourcegroupname “<Name of the Azure Resource Group for the Service Fabric Cluster>”

The process can take up to an hour to complete. Status can be viewed in the Azure Portal under the Service Fabric Cluster Node Types blade.

Application Upgrade Issues

Should an upgrade run and fail, it will automatically rollback once it detects errors in Service Fabric. After a period of 30 minutes, these errors should be removed and another attempt at an upgrade can begin.

Error on subsequent application upgrade after failed upgrade

When the UpgradeApp script is run again, there may be an error in PowerShell (see below); however, the script will continue to run and begin the upgrade process and (assuming all parameters are correct) finish successfully.

If you receive an error that states Application type and version already exists at <path>, then the error is due to the previous failed run leaving the application type and version provisioned in Service Fabric. Running the script again will clash, as it is the same version. The script will continue and overwrite this version. To avoid seeing this error, you can navigate to Service Fabric explorer and manually unprovision the new version of the application before rerunning the script. However, you cannot roll back to previous versions if you unprovision the application. You can do this by navigating to the Cluster > Applications > IC3.FabricType node and clicking Unprovision.

Upgrade the Portal

Lastly you need to upgrade the portal. Please follow the steps below.

  1. Log on to the jump box and then remote onto your portal VM.
  2. Create a new folder under C:\inetpub\wwwroot named with the new version number.
  3. Open the zip file you downloaded from the Customer Support Portal, navigate to the \Azure Paas\DeployPortal\SupportFiles\ folder, and copy the contents of the Portal.zip file into the folder you just created.
  4. Rename the Web.production.config file that was created previously by the PrepUpgradeConfig.ps1 script to web.config and copy into the new portal folder with the version you just created. This will overwrite the existing one.

In IIS, navigate to Sites > your PMC portal

  1. Open Internet Information Services (IIS) and navigate to Sites, then to your PMC portal.
  2. In the Actions menu under Basic Settings, select the new physical path you have created and click OK.

 

Disable the IIS Logging Setting

In PMC 2.4 and earlier, IIS logging is enabled on the portal VM. This can fill the hard drive with log files.

You can run the following script to turn off logging. The script is available with PMC 2.4 SR1.

DisableIISLogging.ps1

The script is located in a folder called Upgrades in the AzurePaaS folder. For example:

…PMC\v2.4-1581\AzurePaaS\Upgrades

The deployment tool turns off IIS logging on the portal VM. When turned off, then logging is off for all IIS sites on the portal VM. If you require logging to be enabled for any other sites, then you must enable logging at the site level for those specific sites.

PMC Portal Security Upgrade

For PMC 2.4 SR1, we have hardened the security of the portal instance. Run the following script to update the portal security for your environment.

…\AzurePaaS\Upgrades\PostUpgradeHardenPortalSecurity.ps1

You must restart the portal VM after running the script for the changes to take effect.

Upgrade Privilege Management Reporting Database

 

You must upgrade your reporting database to 5.5 in order to use PMC 2.4.

Prerequisites

Log on to the customer portal to download the scripts from the following location: EnterpriseReporting\5.5\5.5.40\Enterprise Reporting\SQL.

  1. Ensure the event pump is turned off as outlined in the procedure Perform Upgrade on the Jump Box VM.
  1. Wait for any CopyFromStaging job to finish.

Upgrade Steps

To upgrade a Privilege Management database using SQL scripts:

  1. The SQL scripts are provided as part of the Privilege Management installers, located in the Privilege Management Reporting release folder, which can be found in the BeyondTrust portal. Alternatively, you can contact BeyondTrust Technical Support.

There is a README file provided in this directory to assist you.

  1. Run the following SQL query to return the version of the database.

  2. select * from DatabaseVersion
  3. Execute the upgrade script where the name is the next version number and carry on applying these until the desired version is reached.

  4. For example, if your current database version is 4.3.16 and you want to upgrade to version 5.0.0, run the following scripts in order:
    1. Script_4.5.0_Updates.sql
    2. Script_5.0.0_Updates.sql

    Please check the SQL log for any errors and contact BeyondTrust Technical Support if necessary.

  1. Run and execute the following SQL query against the reporting database to return the versions in the InstallShield table:
  2. SELECT * FROM [dbo].[InstallShield]
  3. Open the InstallShield query file. This is available in the SQL folder, and is a Privilege Management Reporting artifact.
  4. Copy the relevant INSERT lines from this query file that are not included in the database table.
    For example, if the upgrade is from 5.1.1 to 5.4, you need to copy these lines:
    INSERT [dbo].[InstallShield] ([ISSchema]) VALUES (N'5.3.0          ')
    INSERT [dbo].[InstallShield] ([ISSchema]) VALUES (N'5.4.0          ')
  5. Copy these into a query against the Reporting Database and execute it.
  6. View the InstallShield table by running the query below. These values are added.
  7. SELECT * FROM [dbo].[InstallShield]

Turn on Service Fabric Components

You need to turn the Service Fabric settings back on for incoming events.

  1. Remote Desktop onto the jump box (ensure you have the cluster administration *.pfx certificate portion installed on the machine before continuing).
  2. Open PowerShell as admin and run the UpdateServiceFabricAppSetting.ps1 (in the upgrades folder) script with the following parameters:
    • ClusterAddress: The DNS Name of the cluster postfixed with :19000. For example, PMCcert.PMC:19000.
    • ServerCertThumbprint: The thumbprint of the ClusterAdminCertificate.
    • ClusterAdminThumbprint: The thumbprint of the ClusterAdminCertificate (same as ServerCertThumbprint).
    • UpdateConfigParameters: The event pump service Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled set to true
    .\UpdateServiceFabricAppSetting.ps1 -ClusterAddress "pmc.domain.com:19000" -ServerCertThumbprint "54761d496fe75fd4fe81a488fa709e4e79613385" -ClusterAdminThumbprint "54761d496fe75fd4fe81a488fa709e4e79613385 " -UpdateConfigParameters @{"Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled"  = "false";”Avecto.IC3.JobAgent.DeploymentType” = “1”;}
  1. The update will apply to each node one at a time. You can check update status through Service Fabric Manager.
  2. Once the update is complete, run the following command in PowerShell to check if the setting is applied:
  3. Get-ServiceFabricApplication -ApplicationName fabric:/IC3.Fabric

    This will output the application configuration.

    • The Avecto.IC3.Fabric.EndpointEventPump.EventProcessingDisabled parameter should be set to false.
    • The Avecto.IC3.JobAgent.DeploymentType should be set to 1.
  4. Check Reporting in PMC to confirm events are flowing through to the database.

Change Application Parameters Before Upgrade

You can use the script to update values in both the Production.5Node.xml or the Web.config file that are provided as part of the upgrade in the \AzurePaaS\Upgrade folder, if required. You need to use the script to do this rather than edit the files directly, otherwise any changes will be overwritten by the script.

  1. Run PowerShell as an administrator and navigate to the location of the PrepUpgradeConfig.ps1 script in the Upgrades folder.
  2. To change values in the Production.5Node.xml file, use the following command:
PrepUpgradeConfig.ps1 -UpdateApplicationParameters @{"String.Name.One" = "argument"; "String.Name.Two" = "argument";}
PrepUpgradeConfig.ps1 -UpdateApplicationParameters @{"Avecto.IC3.Authentication.Domain" "https://login.microsoftonline.com/53c8dbb9-fb9b-467a-8930-f23d8e0199c9";}
  1. To change values in the Web.config file, use the following command:
PrepUpgradeConfig.ps1 -UpdateWebConfigParameters @{"String.Name.One" = "argument}
PrepUpgradeConfig.ps1 -UpdateWebConfigParameters @{"Avecto.IC3.Log.Seq.Host" = "https://localhost:5391"}