Tales from the Datacenter: Vulnerability Management Nightmares

Vulnerability management is a discipline that encompasses many varied activities in the realms of information security and IT operations. Vulnerability scanning, threat management, risk analysis, patching, and configuration management are some of the major activities usually associated with vulnerability management, and none of these are new…so why are we failing so badly at many of them? As a security consultant, I see a lot of bad behaviors and unsecured technology implementations. Over the years, some cases stand out as stark reminders of just how bad vulnerability management programs can really be. The first example of terrible security practices relates to scanning. Those of us who wield scanning tools know that they really need to be properly tuned and calibrated for the environment to be scanned, lest you find yourself wreaking havoc across the entire environment with scans gone awry, possibly saturating the network, disrupting services and systems, or worse. One organization I worked in several years ago had recently implemented a new vulnerability scanner within their environment, and had appointed a junior staff member to take over the scanning responsibilities. This erstwhile young man meant well, but failed to follow one of the principal tenets of scanning - find out what legacy systems and services are in use that may not react favorably to scanning, and scan them separately or not at all. This could only lead to disaster, and sure enough, it did. He scanned a subnet that contained critical legacy HP-UX servers running availability services that (ironically) crashed completely when scanned with a full throttle scan. Two of these machines did not come back up, and had to be rebuilt (which was incredibly difficult). Another scanning story takes us in a different direction - the case of the VNC suppression. A healthcare organization I did some work in had had a systems integrator in the datacenter for several months and had installed VNC on several important servers in a staging environment. These were scanned for vulnerabilities with a custom policy, and the security engineer was tired of seeing the VNC services come up as high risk, so manually suppressed this finding in the policy. This didn’t go so well when the system builds were replicated to the DMZ, where they happily ran VNC until another intrusion led to a rapid fire succession of VNC to VNC to VNC compromise that ended up causing them huge headaches…and they had been scanning the environment the whole time! There are too many patch management horror stories to tell here, but I’ve had way too many cases of stumbling across systems missing the most notorious patches - not too long ago I found MS08-067, and even MS03-026 (the notorious RPC DCOM flaw of many years ago)! Why, why, WHY do people not patch their systems in 2015?! I’d love to say that all the issues I come across are related to legitimate exceptions, but sometimes they’re not - they’re just patches that were missed, ignored, or overlooked. For more stories and examples of vulnerability management FAILURE, check out this webinar we recorded on May 26, 2015 - it’s fun, sad, and informative all at the same time!