Ensure Measurements of Your Intelligence Program Don’t Prioritize Vanity Metrics

Vanity metrics get a bad rap—and most of the time, deservedly so. While their usage is often relegated to data analytics and digital marketing teams, they can also infiltrate intelligence programs with measurements of things often purporting to be key performance indicators (KPIs) that instead lack a true context that would better give them value.

As their name implies, vanity metrics are measurements that look good on paper and appear to indicate solid performance or value, but are in reality, deceptive and trivial.

Many KPIs used to evaluate intelligence programs are vanity metrics masquerading as KPIs. Measurements such as the number of intelligence reports an organization produces, the number of indicators of compromise (IoCs) processed, data points collected, or keyword alerts received, don’t accurately indicate performance.

Relying on these types of vanity metrics may distort your perception of how well an intelligence operation is running. If a financial organization is hoping to reduce losses due to fraud, for example, it may measure its progress by tracking the number of data points collected to support the operation. While a large, or larger-than-average, number of data points collected would look good on paper and might even suggest increased collections efficiency, it would not definitively indicate that the operation was successful in achieving its objective of reducing fraud losses.

Rather, you would need to choose KPIs that 1) map to the operation’s requirements and objectives; 2) can feasibly measure at scale; and 3) will help identify any blind spots or areas for improvement that could arise. The right KPIs for this operation might include:

• Number of emerging fraud schemes identified

• Number of new anti-fraud measures informed

• Number of fraud attempts that were prevented versus the number that were successful

• Relative reduction in fraud losses

Vanity metrics may also distort an organization’s perception of value; this is especially applicable with respect to vendor offerings. Many intelligence programs obtain the data, intelligence, and tools on which their operations rely from third-party vendors, but choosing the right vendor may be difficult given the abundance of misleading claims in the market—many of which are fueled by vanity metrics.

Collection strategies are perhaps the biggest differentiator among intelligence vendors. But because many intelligence consumers and decision-makers aren’t always aware of exactly which types of data and sources are best suited for their operations, they often—and understandably—assume that more is better. This assumption is largely why many vendors choose to highlight the following vanity metrics in order to differentiate their offerings:

• Number of data sources

• Number of total data points

• Number of new data points points collected per day

• Number of customers

But similar to how web page views aren’t indicative of website performance (compared to other measurements such as active users and engagement), vanity metrics aren’t indicative of the extent to which an intelligence offering is suitable to a program’s needs and objectives. It’s crucial to remember that regardless of how they are marketed or perceived, the quantitative aspects of a vendor’s collection strategy do not reflect the qualitative aspects.

This same principle also applies to your operational KPIs. In other words, there’s no point in having billions of data points if those data points aren’t timely, accurate, actionable, and adequately map to your intelligence objectives and requirements.