Perform Other Smart Rule Actions

The below sections detail how to perform various actions with Smart Rules, such as:

• Viewing the results of processed Smart Rules from the grid
• Editing, cloning, deactivating, deleting, and auditing Smart Rules
• Configuring Smart Rules Options
• Viewing Smart Rules statistics

View Results of a Smart Rule from the Grid

From the Smart Rules grid, you can view the results of any processed Smart Rule using the View Results row action, as follows:

1. From the left menu in BeyondInsight, click Smart Rules.

2. Click the vertical ellipsis to the right of the Smart Rule.
3. Select View Results to go to the associated grid where the contents of the Smart Rule are listed.

 

Edit a Smart Rule

1. From the left menu in BeyondInsight, click Smart Rules.

2. Click the vertical ellipsis to the right of the Smart Rule.
3. Select Edit Smart Rule.
4. Make the necessary changes and then click Save Changes.

 

Clone a Smart Rule

You can clone custom or predefined Smart Rules.

1. From the left menu in BeyondInsight, click Smart Rules.

2. Click the vertical ellipsis button for the Smart Rule you wish to clone, and then select Clone.
3. If you are using the multi-tenant feature, select the organization from the list, and then click Clone Smart Rule.
4. Select the newly cloned Smart Rule from the grid, click the vertical ellipsis button, select View Details, and then edit the Smart Rule filters as needed.
5. Click Save Changes.

Cloned Smart Rules have full (read/write) user group permissions.

Deactivate a Smart Rule

You cannot delete predefined Smart Rules. However, if you have several smart groups, you can mark unused Smart Rules as inactive.

A Smart Rule that is used in another Smart Rule cannot be deleted or marked as inactive.

An inactive Smart Group is no longer displayed in the Smart Group browser pane until marked active again.

To deactivate a Smart Rule:

1. From the left menu in BeyondInsight, click Smart Rules.

2. Select the Smart Group or multiple Smart Groups, and then click Deactivate above the grid.

Delete a Smart Rule

1. From the left menu in BeyondInsight, click Smart Rules.

2. Select one or more Smart Rules.
3. Click the Trash Can icon above the grid. You can also click the vertical ellipsis to the right of a single Smart Rule and select Delete.

Built in Smart Rules cannot be deleted. These are identified by the Lock icon.

A Smart Rule that is used in another Smart Rule cannot be deleted or marked as inactive.

Audit Smart Rules

To audit new or edited Smart Rules:

1. Go to Configuration > General > User Audits.
2. Select Section from the Filter by dropdown.
3. Select Smart Rule from the Section dropdown.
4. Click the information icon to the right of the Smart Rule.
   • If a Smart Rule is added, the Add Details pane displays with all added information.
   • If a Smart Rule is edited, the Edit Details pane displays with all edited information.

Configure Smart Rule Options

From Configuration > General > Smart Rule Options, you can configure multi-worker node usage, the number of Smart Rule threads per type, and the failure thresholds using the Smart Rule Omni Worker Options.

Multi-Node Processing is turned off by default. Enable this to allow assignment of Smart Rules to process specific worker nodes. Choosing a worker node for a Smart Rule to process is accomplished by setting the Target Processing to Workgroup action on the Smart Rule in question. When enabled, this allows multiple Omni Workers to process Smart Rules.

 

For the following options to be available, you must enable Multi-Node Processing. An all Omni Worker restart is required to enable this processing.

• Asset Threads: (Default 5) Choose a number of threads to use for processing asset based Smart Rules.
• Managed Account Threads: (Default 5) Choose a number of threads to use for processing managed account based Smart Rules.
• Managed System Threads: (Default 5) Choose a number of threads to use for processing managed system based Smart Rules.
• Policy User Threads: (Default 5) Choose a number of threads to use for processing policy based Smart Rules.
• Force Re-queued if stale: (Default 12) Choose a number of hours after which an unprocessed Smart Rule is considered stale and re-queued for processing.
• Failure cool off threshold: (Default 5) Choose a number of times to let a Smart Rule process fail after which a cool-off period is observed.
• Failure cool off skip time: (Default 60) Choose a number of minutes to wait before trying to process the Smart Rule again after reaching the failure cool off threshold.

Click Update Smart Rule Omni Worker Options when you have finished setting the options.

Additional Multi-Node Processing Information

The Multi-Node Processing feature was added to allow more granular control over the performance of smart rule processing.

Impact of Multi-Node Processing

Multi-node processing is a combination of features:

• Controls the number of nodes and threads per node that are used for processing different types of Smart Rules.
• Restricts the processing of certain Smart Rules to specific nodes if required. This might come into play if the Smart Rule is built on a directory query that only one worker node has access to. Trying to process a Smart Rule like this across all Omni Workers would result in occasional failures if the node doing the processing lacks the necessary access to run the directory query.
• Controls certain behaviors in failure scenarios. The defaults should be sufficient, but are adjustable to give more control to support assisting customers in this area.
• When multi-node processing is turned off, then Smart Rule processing occurs on a single node using N threads, where N is configurable per Smart Rule TYPE in the configuration user interface (Asset Threads, Managed Account Threads, Managed System Threads, and Policy User Threads). While better than the historical single-threaded model, this can still be a lot of work for the Omni Worker and might cause poor performance in other areas (password rotations, event forwarding, etc.).
• When multi-node processing is turned on, then Smart Rule processing is shared across ALL worker nodes, using N threads per worker node, where N is configurable per Smart Rule TYPE in the configuration user interface (Asset Threads, Managed Account Threads, Managed System Threads, and Policy User Threads).
• The default setting for each Smart Rule type is 5 threads. The valid range is between 1 and 20 threads.
• Changes to the multi-node processing settings, as well as changes to thread counts and changes to failure scenario handling, can be made anytime but do not take effect until all Omni Worker services are restarted. This restart is a manual step. There is no risk to enabling or disabling these settings during production times, but you will not see any change in processing until Omni Worker services are restarted.

Overall Best Practices

The Multi-Node Processing setting is turned off by default. Turning it on is beneficial if multiple worker nodes or Omni Workers are available, and if the existing Omni Workers are running at full capacity. If turning this feature on doesn’t help Omni Worker performance, support should be contacted.

The lower the thread count, the less benefit you may get from turning this setting on. However, setting the thread count too high can also result in problems if your Omni Worker or worker nodes are not powerful enough to handle the load. Start with the default and adjust up or down as necessary.

Reason for Multi-Node Processing

Before this feature was added, Smart Rule processing was only supported in a single-threaded model running in RemManagerService. Moving it to Omni Worker allows it to be multi-threaded on a single node. Adding the multi-node option allows Smart Rule processing to be scaled out even further.

Multi-Node Processing Environment

This feature is used in an environment with multiple worker nodes or Omni Workers, where an Omni Worker is taxed by Smart Rule processing.

Assign a Rule to a Node

If multi-node processing is turned on and a Smart Rule contains a specific criteria or action that only works if executed on a particular worker node, then that Smart Rule is expected to get an action of Targeted to Workgroup set. The Omni Worker or worker node that executes this Smart Rule should be manually set to the same work group under Worker Nodes. Some examples of criteria or actions that only work on a particular node are directory queries that run on a specific network, or database account onboarding that runs on a specific network. Any network-specific Smart Rules are likely candidates to target a specific worker node.

Troubleshooting Methods

• Smart Rule Grid

  Three optional columns have been added to the Smart Rule grid to give some extra visibility into Smart Rule processing: Processed Date (checks to see if any rules were not processed recently), Successful Attempts, and Failed Attempts. Other columns that are helpful are Reprocessing Limit, Average Time, Last Attempt, and Processing Status.

• Dynamic Dashboard

  Troubleshooting also includes checking the Omni Worker Dynamic dashboard in the user interface (administrators only). There you can see the Omni Worker agents, queued messages, messages sent to dead-letter (undeliverable letters, reached the limit of processing attempts), and messages actively being processed.

• Health Dashboard

  This dashboard shows stats regarding issues on worker nodes, slowest Smart Rules, failed Smart Rules, and errors in the system.

• Logfiles

  There is one log file per Omni Worker. Because this can be hard to read across environments, we have added the System Event Viewer and System Event Settings features. Enabling System Event Database Recording logs error or warning messages from across the system into the BeyondInsight database so they can be viewed and searched using the System Event Viewer. Purging these events from the database is configurable. The default is 5 days.

Issues with Feature

The feature has been developed to avoid deadlocks, race conditions, memory leaks, etc., as part of our development and QA process. However, it is possible that some issues still exist. Contact BeyondTrust Support with any issues that arise for resolution.

Changed Behaviors in the Database

On its own, multi-node processing does not make changes in the database. Any database changes to schemas, tables, views, procedures, etc., that are required for this and other features in BeyondInsight are made during an upgrade, whether this feature is enabled or not. If the Enable System Event Database Recording setting is turned on, then database entries are made for warnings or errors in the system. Purging is enabled for this data, and the time frame is configurable.

Logged Nodes

Each Omni Worker has its own logs. Logging takes place across multiple nodes when this setting is turned on. The System Event Viewer shows any issues that are occurring.

Failover Processing

Existing support for worker node or Omni Worker service failover also encompasses the Smart Rule processing function. In the event of a failover situation, the secondary node picks up where the primary node leaves off.