Troubleshoot Event Collections

If the events are not appearing on the event collector perform the following troubleshooting steps:

Test Event Forwarding

If all of the event forwarding components are functioning (and there's minimal network latency), a test event created on the source computer should arrive in the event collector's Forwarded Events log within 60 seconds.

On the source computer create an Endpoint Privilege Management event. Alternatively, if you configured the subscription to capture all events from the application log you can use the following command line to create a test event.

  1. On the source computer, open a command prompt.
  2. Type:
    eventcreate /id 999 /t error /l application /d "Test event."

Test event forwarding from the command prompt.

  1. This event should appear on the event collector.

 

The syntax above creates an event which may not match the criteria within the previously created subscription. To use this test feature, you must ensure your subscription will forward this event.

Troubleshoot Log Locations

Event forwarding and WinRM have operational logs that can be viewed in the Event Viewer or by using the command line tool wevtutil.exe. The following Windows logs provide information on any errors that may occur:

  • Down-level clients
    Windows Forwarding/Operational
  • Vista Upwards
    Application and Services Logs > Microsoft > Windows
    • EventLog-Forwarding Plugin (log)
    • Windows Remote Management (log)
    • Event Collector (log)

The EventLog-ForwardPlugin and Windows Remote Management operational logs are the locations to which the local WinRM service logs. WinRM logs all activities to Microsoft-Windows-Forwarding/Operational in the Event Viewer on Windows XP.

Query the Event Forwarding log using the Microsoft-Windows-Forwarding publisher with the command line tool wevtutil.

 
wevtutil qe "<PATH_TO_LOG>" /c:1 /rd:true /q:"<XPATH_QUERY>"

If PATH_TO_LOG is not within %SYSROOT%\system32\Winevt\Logs\, the /lf option must be used with the true argument. The /rd option cannot be used on .EVT files. The help documentation of the wevutil tool provides more information about the tool. Run the following command to view the documentation:

wevutil/?

Check You Can Ping the Event Collector’s FQDN

Ensure you can ping the FQDN of the event collector from the source computer:

Ping Server1.BeyondTrustlab.com

Check Policy is Applied to the Source Computer

This can be forced by running the following command on the source computer:

gpupdate /force

Check Windows Remote Management Service on the Source Computer

On the source computer, navigate to the services.msc and check the WinRM service is running and set to automatically.

Check the Collector Can Reach the Source Computer Using WinRM

Run the following command on the collector:

winrm id /r:<Source Computer> /a:none

Check the Source Computer has successfully Subscribed

From the event collector, you can check whether the source computer has subscribed by viewing the subscription status.

Check the Collector is Using the Right Credentials (Collector Initiated Only)

Run the following command on the collector:

winrm id /r:<Source Computer> /u:<username> /p:<password>

These are the credentials defined in the subscription on the event collector. The credentials do not need to be in the local administrators group on the source computer, as long as they are in the Event Log Readers group on the source computer (local administrators group will also work).

Check the Source Computer has Registered with the Collector

Run the following command on the collector:

wecutil gr <subscription name>

This lists all the registered source somputers (if the subscription is Collector Initiated, then this lists all configured source computers), their state (from the collector's perspective), and their last heartbeat time.

Check the Windows Forwarding/Operational Event Log on the Source Computer for Errors

Event ID 105 The forwarder is having a problem communicating with the subscription manager address is often a result of the Windows Firewall on the event collector blocking communication.

Ensure the following rules are accepting incoming connections:

  • Windows Firewall ports Windows Remote Management (HTTP-In) Port 5985 configured for inbound communication
  • Windows Firewall ports Windows Remote Management (HTTP-In) – Compatibility Mode - Port 80 configured for inbound communication
  • Windows Firewall ports Windows Remote Management (HTTPs-In) configured for inbound communication

Enumerate the Active WinRM Listener

The command below provides the syntax required to enumerate the active WinRM listeners on the event collector:

enumerate winrm/config/listener

When compatibility mode is enabled, WinRM creates a second port (80) to access its services. The approach to test if WinRM is listening on port 80 is to enumerate the listeners.

View the WinRM config

The command line below provides syntax to view the WinRM configuration on the event collector:

winrm get winrm/config

These two commands display the configuration for both WinRM client and service. Viewing configuration settings can help identify any possible incorrect configuration settings.

winrm get winrm/config/client/auth
winrm get winrm/config/service/auth

View Remote Machine Details

winrm id –remote:TARGET

The above command identifies (id) the remote machine (TARGET) by asking the remote machine its operating system version and WinRM version. The TARGET can be a NetBIOS name, Domain name, or FQDN.

Alternatively, using the–auth:none option forces WinRM to not use authentication when requesting information from the remote machine. Using this option only provides a minimal set of details (version of WinRM only).

View WinRM Communication Information

The identify option provides insight if communication between two WinRM parties are correct and not interrupted. This interruption can be the result of a firewall blocking WinRM or WinRM not running.

winrm get wmi/root/cimv2/Win32_Service?Name=WinRM

This command provides useful information about the WinRM service running on the local machine (for example, ProcessID and Context WinRM runs in).

Restore WinRM Defaults

WinRM allows the restoration of default settings using the command:

winrm invoke restore winrm/config @{}

View Error Code Help

WinRM error messages display the description of the error and an error code. The definition behind the error code can be shown by executing the command:

winrm helpmsg ERRORCODE

The ERRORCODE needs to be supplied exactly as it was displayed in the original error message (for example, 0x80070005 means Access Denied). These errors are Win32 error codes.

View Authentication Help

Generally, WinRM produces an error message when authentication fails. The service provides a second option to help the authentication process. A detailed explanation of different authentication methods used by WinRM can be viewed using the following command.

winrm help auth

Authentication Error Example: The WinRM client cannot process the request. Negotiate authentication is currently disabled in the client configuration. Change the client configuration and try the request again. If this is a request for the local configuration, use one of the enabled authentication mechanisms still enabled. To use Kerberos, specify the local computer name as the remote destination. To use basic, specify the local computer name as the remote destination, specify basic authentication, and provide user name and password.

The recommended method to satisfy WinRM is to supply the –remote option with the target host name (local or remote). If the source is part of a domain, then executing this command requires an uninterrupted connection to the Domain Controller. Assume the command is executed on a computer whose host name is ABCD.

winrm get winrm/config –remote:ABCD

Access Denied Errors

Certain operations of the WinRM command may result in access denied errors. There are multiple reasons for the following error:

WSManFault

Message = Access is denied.
Error number: -2147024891 0x80070005
Access is denied.
  • User needs to be part of local administration group, WinRMRemoteWMIUsers__, or domain administrator.

    The administrator password cannot be blank.

  • Incorrect user name or password.
  • WMI operations need permissions to allow secure connections.
  • Windows Firewall service needs to be running (this will result in the subscription set to inactive).

Event Collector Subscription is Inactive

The Event Collector Subscription status is Inactive when a retry is initiated. You may receive an access denied error.

The root cause of this problem is related to an unspecified dependency on the Windows Firewall Service. Please ensure the service is installed and started, you will then be able to start the subscription.

Ensure the WinRM Firewall Ports are Open

When using third-party firewalls, you need to ensure the following ports are open on the event collector’s firewall:

  • Windows Remote Management v2.0 over HTTP = Port 5985 #
  • Windows Remote Management v2.0 over HTTPS = Port 5986
  • Windows Remote Management v1.1 over HTTP = Port 80
  • Windows Remote Management v1.1 over HTTPS = Port 443

If using the Windows Firewall, the above ports can be configured using the GUI. Alternatively, the following command line options can be used to configure the firewall:

netsh advfirewall firewall set rule name="Windows Remote Management – Compatibility Mod (HTTP-In)" new enable=yes
netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" newenable=yes

Large Kerberos Token Sizes May Cause Event Forwarding to Fail

If your organization has large Kerberos token sizes, you may experience issues with event forwarding.

How to Check the WinRM Version You Are Running

For more information, please see Versions of Windows Remote Management.

Errors When Creating Subscription

Common subscription creation errors:

wecutil cs Subscriptions\Logons.xml

Subscription Error Example 1

Error = 0x3ae8. The subscription fails to activate.

The subscription is saved successfully, but it cannot be activated at this time. Use retry-subscription command to retry the subscription. If the subscription is running, you can also use getsubscriptionruntimestatus command to get extended error status.

This error may be caused by the WinRM Firewall exception rule being disabled. The error code that is displayed is a Win32 error code. The error code’s message is shown beneath it.

Subscription Error Example 2

Failed to open subscription.

Error = 0x6b5. The interface is unknown.

This error may be caused by the Windows Event Collector not running.

Sources will create subscriptions locally after receiving a list of subscriptions applicable to them. Certain subscriptions may not be created on the sources due to permissions issues or non-existing logs. WinRM will raise an Event ID 102 with a Win32 error code of 5004 in the EventLog-ForwardingPlugin/Operational log. The error code states that a cluster resource is not available. This error code may be a result of the subscription attempting to access a log file that it does not have permissions to access.

Verify the channel’s (log file) permissions by navigating to: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels and locating the channel of interest. Within the registry key of the desired channel, view the contents of the registry value named ChannelAccess to identify the permissions of the channel. This previous error is applicable to Windows Vista and later.

XPath Query Diagnostic

XPath queries used in subscriptions do not display errors to the user who created them when deployed to sources. Query errors are shown in the Applications and Services Logs > Microsoft > Windows > EventLog-ForwardingPlugin > Operational log on Windows Vista and later sources. Event ID 101 raised by the Event Forwarding plug-in is to notify the user an XPath query was incorrect.

ID Level Event Log Event Source OS Version
101 Warning (3) EventLog- ForwardingPlugin/Operational EventLog- ForwardingPlugin Windows 7+

The human-readable details of the event do not clearly indicate the reason why the event was raised. The specific reason can be identified by viewing the XML details of the event. An error code of the XPath query is hidden as part of the event data. The error code can be viewed by:

  1. Locating event ID 101 under the event log-ForwardingPlugin > Operational log.
  2. Select the Details tab, and then select the XML view.
  3. Under the EventData node, a data node named Status exists that shows the decimal value of a Win32 error code.

A Win32 error code of 15001 indicates an invalid query of ERROR_EVT_INVALID_QUERY.