Vulnerability Management Buyer's Guide

Quality Criteria for VM Solutions 

When evaluating VM solutions, buyers should appraise the performance of the technology and ensure it will allow them to quickly answer these critical questions: 

• Which areas of my network present the greatest risk right now?
• Is the most recent high-profile vulnerability present anywhere on my critical infrastructure?
• What are the most effective steps we can take immediately to reduce our security risks? 

The following capabilities are what you need to look for in order to find a comprehensive VM solution. 

Varied Assessment Methods 

Assessment depth can significantly impact the accuracy of results. Deeper assessments gather more detailed information, which the system can use to improve accuracy. There are four main types of vulnerability assessment methods to consider: 

Agentless discovery

Look for a solution with unlimited agentless discovery with comprehensive fingerprinting and application/service detection to accurately identify and profile the your assets. This allows you to inventory ports, services, and applications exposed on your network and identify each device type and operating platform. 

Agentless credentialed scanning

Credentialed assessments use administrative credentials to inspect file system, registry and configuration files. Credentialed assessments take longer to run, but the additional information gathered dramatically improves both discovery and assessment accuracy.

Non-credentialed scans

In contrast, assessments performed remotely or without credentials provide the same view an outside attacker would have. If agent-based or credentialed scanning are akin to white box testing, remote analysis would be black box testing. Less information is gathered about the application footprint of the asset, but more data is available regarding the protocols and services that can communicate with the asset. 

While white and black box assessments should be performed together for a holistic view of your security posture, it’s important you use accurate and reliable methods of testing remotely. In some cases, products rely on banner checks that can lead to inaccurate results. It’s better to look for a solution that relies on direct condition tests and inference when reporting vulnerabilities remotely on an asset. 

Agent-based scanning

Agent-based scanning can be conducted as a stand-alone process or in tandem with agentless scans to provide a more comprehensive view. A network scan should dynamically recognize when an asset has an agent and optimize the scan by using the data collected by the agent. 

Ideally, a VM product offers both methods so you can use the one that best balances your organization’s requirements for assessment speed versus depth. Combining the in-depth assessment provided by agent-based scanning with non-credentialed remote scanning can be a good strategy when credentialed access isn’t viable.

Accurate Detection 

Because many VM tools have significant accuracy problems, they deliver too much data and too many alerts. Massive reports that include a lot of undifferentiated data about possible changes or vulnerabilities make it nearly impossible to determine which issues need attention now and which can wait. As a result, valuable resources are wasted investigating events that aren’t “bad,” such as reporting false positive findings. 

While false positives are easy to track — as they’re reported by customers when they’re encountered — it’s much more difficult to generate a false negative rate. The definition of a false positive is as straightforward as “an incorrect result.” A false negative is the lack of a correct result, but not all missing results are false negatives. A false negative occurs when an application or vulnerability that should be found was not. Typically, reported false negatives are better classified as requests for coverage. Once you remove these coverage requests, Tripwire’s false negative rate quickly approaches zero. 

Sharing change and vulnerability data between IT operations and security teams makes it easy to optimize resources for specific business goals, yet most VM data isn’t easy to share. Many VM tools also waste valuable resources because they require manual effort to export and format data for consumption by other teams. 

Thorough Discovery Capabilities 

Asset discovery and inventory features and capabilities differ from product to product, but a worthwhile VM product should offer the following discovery capabilities: 

Continual device inventory

Your solution should be able to take a continuous inventory of all devices, including wired and wireless devices, virtual machines, cloud instances and containers. 

Continual software inventory

Continual inventory of all software applications and versions includes desktop applications, operating systems, ports and services, and protocols. 

Asset tagging

Your solution should provide the ability to tag assets by group, technical owner, regional location and criticality. 

The process of putting these capabilities in place helps align your organization with CIS Controls 1–3: the top three prioritized controls that create a system hardened against risk from vulnerabilities. CIS Control 3 is Continuous Vulnerability Management: “Continuously acquire, assess, and take action on new information in order to identify vulnerabilities, remediate, and minimize the window of opportunity for attackers.” 

Look for a solution that does device profiler (DP) pooling as well. DP pooling lets you group multiple device profilers into a pool of appliances that can be used to conduct scheduledscans, as opposed to depending on a single DP to do any given scan. 

This provides several advantages. First, it gives you the ability to scan a given network faster because the load is divided up among multiple scan appliances. Second, it adds resiliency in that if there’s a failure on the part of any particular scan appliance — or if an appliance’s connection to the network is lost or becomes degraded — then the other appliances in the pool will pick up the load for the lost or degraded appliance and ensure that the scan completes. Third, it allows you to simplify your scan schedules by allowing larger network blocks to be scanned by dynamically load balancing a scan job across the pool of appliances rather than having to break them up and schedule them manually. 

Intelligent Assessment Technology 

Intelligent assessment technology ensures frequent and accurate assessments for improved visibility and confidence in security posture assessments. 

Indiscriminate testing

In this older method, the solution scans through a defined range of asset IPs and indiscriminately checks each asset against a list of known vulnerabilities maintained by the VM vendor. This results in time-consuming checks that may not apply to the device being assessed. For example, this approach will result in checking a Linux machine for a Windows vulnerability. This scenario is also likely to occur when device and application inventory is inaccurate, such as when a NetApp filer running a UNIX-derivative OS is profiled as a Windows device because it’s running a Windows SMB/CIFS service. 

Targeted testing

This method first inventories and profiles each asset to determine the type of device, operating system, and applications present. It then uses that information to efficiently check for relevant vulnerabilities, skipping checks that don’t apply to a particular asset, OS or application version. 

Targeted testing offers several advantages: 

• Improved assessment speed and efficiency
• Custom tests for OSs, applications and services that exist on a device minimizes resource usage and improves network stability
• Improved accuracy, resulting in fewer false positives
• Ad-hoc assessments limited to vulnerabilities, configurations or software versions specified by the user for faster results 

Mobile Asset Management 

Many smartphones and tablets — like Apple iOS devices — require special tools for management and assessment. Because of this, many organizations run mobile device assessment separately from their vulnerability assessment and then aggregate the report data. At a minimum, a highquality VM product should be able to discover the exposed surface area of mobile devices connected to wired and/or wireless networks.

Virtual and Cloud Infrastructure 

Within large organizations, devices may be owned by different people or departments. They may also exist in many physical and virtual locations. Security teams need to be able to quickly identify the owner, location and criticality of an asset, as it’s important when assessing and responding to events on that device. 

Many VM solutions require humans to review and keep a current list of assets (as well as their types and configurations) in order to identify their criticality. An automated API-driven workflow for this time-intensive process allows an administrator to define a set of rules in which devices can be categorized based on the unique taxonomy of the organization. 

Real-time Data Navigation 

Reporting can transform volumes of security data into actionable information that can be used to reduce risk. This not only makes it easier to identify vulnerabilities that need patching, but also to detect misconfigurations that affect compliance or security posture and provide actionable remediation advice. 

Constantly-changing technology infrastructure combined with rapidly-emerging vulnerabilities requires organizations to strive for a comprehensive, accurate, real-time view of vulnerabilities — yet many VM solutions produce overly simple reports. These reports provide only a snapshot of vulnerabilities discovered within a specific time period. 

Advanced VM solutions offer real-time data navigation and synthesis. For example, they can produce a list of assets that share a specific vulnerability or compare two historical asset assessments to identify new vulnerabilities or applications that have changed. Indexing OS, application, and vulnerability results enables a real-time response to emerging threats by searching the conditions that would make assets vulnerable without having to wait for a signature or the need to run a scan. Advanced solutions offer benefits in the following areas: 

Audience-specific report filters

Look for a solution that provides reports with the appropriate level of detail for a variety of audiences, including auditors who may wish to see proof of compliance, and business executives, who want an overview of the organization’s risk posture, graphical views of risk trends and visualization of risk data by organizational hierarchies or geographical location. It must also allow all users to create, save, and share report filters and filter report content to include or exclude data based on asset score, vulnerability type or severity, operating system group, and other characteristics. Your solution should also automate report distribution to users based on their roles. 

Remediation advice

Effective VM solutions offer advice on correcting vulnerabilities, including accurate and complete remediation details, potential mitigations, links to patches, vendor advisories and relevant vulnerability information. 

Transparent vulnerability checks

These checks provide details about how vulnerabilities are detected so system administrators can manually verify them. A vulnerability may reappear in a report after a patch or other fix has been applied — or misapplied — because the machine is still vulnerable. This can also happen when the device requires a reboot for the patch to take effect. Information on how a vulnerability was discovered helps teams find underlying sources of additional risk to the organization. 

System Integrity Monitoring 

System integrity monitoring, which includes file integrity monitoring (FIM), is another foundational security control used by most enterprise security teams, but this data often exists in a separate silo from VM data. Correlating these two controls provides critical insights into attack methods and targets. However, it requires coordination across multiple teams, and access might be hindered by internal policies or business processes. 

Without “who” data, it’s more difficult to know if a change is good or bad. For example, an unauthorized or unrecognized user may be an indicator of “bad” change, but if you know who made the change, you can ask that person about it or investigate further to verify that the account hasn’t been compromised. Combining change data that contains “who” information with VM data allows security teams to quickly identify systems at high risk. 

Another example of the correlation between robust vulnerability data and detailed FIM data is the versioning and history of a specific file. Without this data, it’s hard to connect multiple pieces of information and then conclude whether a change was good or bad. 

To determine if a breach is in progress, security teams need to be able to quickly answer these questions: 

• When did this change occur?
• What did the configuration of this device look like before the change?
• Are there other changes that have happened in the past on this device? 

In order to identify the exact changes taking place, you need detailed information that’s only available by comparing reports from different security tools. Change data alone doesn’t identify the changes that indicate a potential breach. You need a known, “good” baseline state and a way to do a side-by-side comparison of the change data against that state. 

Detailed Risk Scoring 

Every organization is different, with different priorities, risk tolerance and unique threats to combat. Standard industry vulnerability scoring systems, such as CVSS, may rate a vulnerability as an 8 on a scale of 1–10, but for your specific organization, the same threat may present a higher or lower risk. CVSS is a good baseline scoring option that allows you to compare vulnerabilities across products. But because you can have 5,000 CVSS “8”s, you’re likely to need more granular scoring. 

Risk scoring is a good example of how customization can help organizations tailor VM data to their specific business requirements. Many VM systems offer a 1–10 or High/ Medium/Low scores. In organizations with thousands—or tens of thousands—of vulnerabilities, these rough scores lose meaning. Your VM solution should be supported by a worldclass research team that analyzes conditions and risks rather than relying on automated feeds and scoring metrics. 

Accurate, up-to-date vulnerability data combined with a variety of other security solutions provides valuable insights. However, combining data manually from multiple sources is costly and increases response time to breaches and incident detection. Many VM tools offer limited or API-only integration with other security tools. These solutions require expensive internal resources to build and keep necessary integrations current. To be an effective tool in breach detection and prevention, VM solutions should let users instantly view and manipulate the historical and current data they need in real time. 

Business Context 

You should also look for a solution that can organize hosts and networks in a business-aligned structure. For example, business unit categories like finance and sales, or geographies like North America or EMEA should be offered. This helps the solution apply business context when calculating and trending vulnerability scores.