In 1993, I had the privilege of working with a brilliant engineer who developed and patented a technology called Above Real Time Training (ARTT). The concept was applied to military flight simulators to train pilots how to deal with high speed, stressful engagements, by making the simulators operate at speeds faster than real time. It was like playing a video game at 2x or 3x speed but also requiring the pilot to manage the controls at the same accelerated rate. The theory suggested that if the pilot could manage situations at high speed (above real time), then when an actual situation occurs in normal time, they would be calmer and make the best decisions without the additional stress. This worked amazingly well to teach the correct course of action and have pilots maintain their demeanor during flight. Tasks and responses became second nature. When dealing with overwhelming information technology security problems, we all could learn a few lessons from ARTT.
Everything is Not Equal
When dealing with any situation, you need to triage the incoming data and prioritize which needs to be reviewed first and which can wait. If security information is being presented all equally weighted (i.e. every message is critical), there is no way to prioritize. It does not matter if the events trickle in or flood a system (ARTT). Without knowing what to look at first, the user ultimately becomes overwhelmed and complacent to look at the results or any new information. This is commonly leads to burn out among security professionals. Every situation is not a fire (or fire drill) and every event should not be treated the same. To manage the fatigue of security professionals, the information they process must be properly prioritized by risk, presented in a consumable fashion and speed (not ARTT), and require “all hands on deck” only if absolutely needed.
Many of the parameters for risk scoring are well established, but some are not. Log on and log off events for Microsoft Windows based assets are a good example. The Windows Event Logs themselves classify these events as information but depending on the identity, the time of day, and the location the event could be deemed critical. The risk rating an organization chooses for these parameters could lead to false positives, event floods when management tools access the system, or even dismissal of the events because they occur with a high or even low frequency. Much like evacuating a building because of the fire alarm, if the alarm goes off every night (like many of us probably experienced in college dorms) we stop caring and just go back to sleep. A string of false positives can lead to ignoring the plain and simple warning signs something is wrong.
To Problem Solve - Think ARTT
Businesses need to operate calmly, rationally, and make good decisions when a crisis hits. Red team and blue team security testing is critical for any business or government entity protecting sensitive information. Responses should be practiced for when a situation occurs. Like ARTT, if you can demonstrate the security event, with the proper risk classification, and an accelerated timeline for response you can train teams to respond in a meeting. Like a flight simulator, this is pure role playing with a timeline as your accelerated guide. As crazy as it sounds, it is an effective way to teach rapid response and make the results learned reactionary conditioning verses panic and denial.
Consider the auditing security tools you need to be successful. They should have the capability to classify data and report important exceptions verses a massive data dump. The output needs to be humanly readable and report critical windows events that require attention accurately. While this may sound like the holy grail, if we reacted to every event where a password was miss-typed, we would drive the wrong behavior from our teams. To determine critical risks better, select tools that generate events with a strong likelihood of being accurate. Log every operational event for forensics, but to avoid fatigue in response, alarms need to go off only when there is a real problem.
Explore more on how to pinpoint what’s most important in this on-demand webinar, with Security Expert, Nick Cavalancia, Not All Threats are Created Equal. Ready to take the next steps in avoiding false alarm fatigue? Request a customized demo or contact us. We can help.
Morey J. Haber, Chief Security Officer, BeyondTrust
Morey J. Haber is the Chief Security Officer at BeyondTrust. He has more than 25 years of IT industry experience and has authored three books: Privileged Attack Vectors, Asset Attack Vectors, and Identity Attack Vectors. He is a founding member of the industry group Transparency in Cyber, and in 2020 was elected to the Identity Defined Security Alliance (IDSA) Executive Advisory Board. Morey currently oversees BeyondTrust security and governance for corporate and cloud based solutions and regularly consults for global periodicals and media. He originally joined BeyondTrust in 2012 as a part of the eEye Digital Security acquisition where he served as a Product Owner and Solutions Engineer since 2004. Prior to eEye, he was Beta Development Manager for Computer Associates, Inc. He began his career as Reliability and Maintainability Engineer for a government contractor building flight and training simulators. He earned a Bachelor of Science degree in Electrical Engineering from the State University of New York at Stony Brook.