Everyone in information technology (IT) has ‘those’ user stories that get brought out at holiday parties. The folklore about that time someone got things spectacularly wrong.
“He ran shutdown to replace a disk on a server, and watched in horror as the LEDs on the wrong server turned off. Yup, that was the ESX box hosting the database server for that legacy payroll system we’re still using“.
“She was asked to take a dev instance offline for patching and had just finished. Then, Slack lit up with screams that the public website was now showing a holding page. Turns out, she was logged into production when the request came in, but forgot to switch to dev.”
The scale of the impact may vary, but the root cause in each case is painfully evident. Sometimes referred to as a ‘meatware issue’ (also facetiously called “layer 8” in the OSI model), the problem is not software or hardware, but rather sitting between chair and keyboard. Operator error is a less charged description, but ultimately, the factor is all too human: a simple mistake.
Context and IT Errors
If we look at the examples above, a common thread is context. A user might have been performing a correct operation, but in the wrong place at the wrong time. That missing context is crucial. So, how can we make users aware of it?
“Let’s use a desktop background that shows the server name and environment!“
“The users in an app, can’t see the desktop and no-one looks at that information anyway. Also, what if they’re working in a command-line interface?”
“Hmmm, good point. Ahhhh, they’re going to see User Account Control!”
“Nice try, but maybe it’s not a Windows system. And no one ever reads those messages anyway.”
“Hang on, why does no one look at that information or read those messages?”
Several factors are in play here. However, muscle memory is a major problem. If users must constantly click on multiple dialogs to accomplish regular, “correct” tasks, this becomes the routine. Eventually, dialog fatigue sets in and the user will automatically click on “Yes” when the identical message is displayed for the thirtieth time that day, just before they do the very, very wrong thing.
The same process kicks in if the user is constantly having to elevate privileges. There is no active thought process triggered when the user needs to use sudo to run the required operation, no prompting about “Am I really sure this is right?”.
3 ways to reduce operator error
So, how can we reduce the problem of human muscle memory that works against our security interests?
1. First, reduce the number of messages or prompts. When an operator is blocked from running an operation in Linux/Unix, the same message appears whether they’re doing something as mundane as installing a minor update, as sensitive as changing firewall configuration, or as flat-out wrong as deleting the contents of an entire volume. If we can stop the messages for innocuous, low-impact and commonly-occurring events, the prompts for riskier or more questionable operations will have a greater chance of triggering critical thought processes in the user.
2. Second, add contextual information. Dialogs and prompts are generic in nature. They typically don’t provide enough information to educate a user’s decision. For example, Windows User Account Control (UAC) prompts are intended to alert the user that a called process or application needs to run with elevated rights. However, Windows only displays the application path and details of digital signatures—there is no means to add more useful information (such as context about the machine the user is working on).
3. Finally, we could take the decision away from the user. Frequently, when the user is running as an administrator or as root, they are over-privileged. Rather than giving users broad entitlements, we would ideally like to provision just enough privilege to do their job, in effect, protecting them from inadvertent misconfiguration or other errors.
How privileged access management (PAM) prevents and mitigates IT errors
Most organizations focus the bulk of their effort and resources on securing their environment against malicious actors. While the information security team will typically be tasked with investigating these threats, the solutions the IT security team implement to mitigate those risks can also be used to protect against human operator error.
Privileged Access Management (PAM), more specifically, Endpoint Privilege Management capabilities, can be used to eliminate standing admin rights and replace them with a more focused set of granular permissions. This applies to both operating system functionality and applications.
So, for the low-trust junior user, we can set up a more restrictive policy, protecting the user from the functionality they might otherwise misuse to become corporate folklore, while still giving them access to the functions they need to complete their work.
Aside from explicitly allow-listed and block-listed operations, flexible exception handling can be used to easily process user requests for other tasks not yet handled by policy, while refining the policy by explicitly adding to allow- and block-lists.
Tiered workstyles can allow a greater degree of freedom for more senior users to whom we want to grant more autonomy. In effect, we can allow these users to make their own decisions to run a broader set of apps and tasks, but we need to consider what information they will need to make the right choices.
The ability to replace OS functionality, such as sudo, the UAC, or the macOS authorization process, with customized dialogs and prompts, means we can add the missing context from those default messages. For example, we can remind the user that they’re about to run an operation that’s part of a group of restricted operations on a production system, link them to an Acceptable Use Policy, and ultimately, get them to acknowledge the risk of the operation before proceeding.
This is starting to sound suspiciously like the status quo, with users still being deluged with a constant barrage of messages and prompts that they need to acknowledge. We can now consider how to prevent the development of unwanted muscle memory.
Through explicit allow-listing, Endpoint Privilege Management can automatically allow and seamlessly elevate preinstalled operations that are lower-risk, without users needing to constantly resort to sudo or click on OS elevation messages. This capability, coupled with block-listing, means we can reduce the number of decisions users must consciously make. Reducing decisions, in turn, prevents dialog fatigue and the ensuing automatic clicking on “Allow”. We can also add further context by setting up different messages and prompts that could require re-authentication, or similar steps, for higher-risk operations.
So, aside from its better-known security capabilities such as managing privileged accounts and enforcing least privilege, another of PAM’s many benefits is that it can actually reduce the number of dialogs a user will need to click through. Thus, PAM not only helps users work more effectively, but enables them to make better decisions.
