Introduction:
This brief write-up is to discuss the features of Key Performance Indicators (KPIs) and Key Risk Indicators (KRIs) from a cybersecurity vantage point. Both essential tools for organizations, but they serve different purposes. In this paper we will highlight their relationship, how these are different and same in certain ways and their value-add from cybersecurity operations (CyOps) point of view.
KPI measures the performance and progress of specific business objectives associated with any operational function, cybersecurity (CyOps) or otherwise. Usually, KPIs help organizations track success in their operations and help enhance operational performance by way of improving or increasing the operational KPIs.
On the other hand, KRI identifies and assesses the potential risks that could impact that operation, which in turn is designed to achieve business objectives. KRI serves as early warning systems to prevent issues or risk before they escalate, or disaster strikes. Let us take two examples to understand these.
KPI and KRI example area, our health.
Goal: Improve personal fitness. | KPI: Number of steps taken daily.
How KPI works: Let us say we are using a fitness tracker to monitor our daily activity. Our target might be to take at least 10,000 steps each day. This KPI helps us measure how well we are meeting our fitness goals by tracking our daily step count.
Goal: Avoid health issues due to a sedentary lifestyle. | KRI: Number of hours spent sitting each day.
How KRI works: To mitigate the risk of health problems caused by prolonged sitting, we can start tracking the number of hours we spend sitting each day. If we notice that we are sitting for more than 8 hours a day, it serves as an early warning indicator (a KRI) that we need to incorporate more movement into our routine to reduce health risks.
In summary for our example KPIs help us track progress toward positive goals like increasing our daily step count. Whereas KRIs help us identify and manage potential risks like sitting for too long each day without moving.
KPI and KRI example area, CyOps.
Goal: Improve incident response (IR) time. | KPI: Mean Time to Detect (MTTD).
How KPI works: Your organization tracks the Mean Time to Detect (MTTD) cyberthreats and we can measure the average time taken to identify security incidents from the moment they occur. This KPI helps the organization evaluate the efficiency of its detection mechanisms processes and its ability to promptly identify and respond to threats. By aiming to reduce the MTTD, the organization can improve its overall cybersecurity posture and minimize the potential impact from security incidents.
Goal: Manage the risk of data breaches. | KRI: Number of detected malware infections per month.
How KRI works: Your organization tracks the number of detected malware infections each month as a Key Risk Indicator (KRI). If the number of detected infections increases, it serves as an early warning that the organization’s cybersecurity measures may need to be enhanced. By monitoring this KRI, the organization can proactively address potential upcoming risks and prevent major security incidents that could lead to data breaches.
Just to reinforce the idea again, KPI like MTTD helps the organization track progress towards positive goals such as improving incident response time. Whereas KRI, in our example of the number of detected malware infections, helps the organization identify and manage potential risks such as data breaches.
Captured KPIs establish and demonstrate how effective an organization’s function is performing.
KRI must be calculated to check if your aggregate KRI number is reaching near to the identified and agreed KRI threshold number. If reached or near to the threshold, it can increase the chances of the risk materialization.
Defining KPIs and KRIs:
Key Performance Indicators (KPIs) tell the behind-the-scenes story. Did the organization achieve its established goals, or exceed those, or missed it? Of course, to measure if something is missed, one must first establish a goal and associate it with a measurement scale. Then if missed or exceeded, KPIs highlights by how much, based upon a pre-determined measurement understanding, preferably based upon the possible risks, if the measurement goal is not achieved.
In short, KPIs are a quantitative measure of highlighting the success or failure of a target goal achievement.
For any function within our organization, we can identify a few critical KPIs for a specific project, task, or process. May it be finance or production or marketing or cybersecurity operations, or any other business function.
As a side note, you may even identify KPIs for task / projects at home or even for your own personal relationships, because KPIs will help you measure the success or failure of your accomplishments and / or their operational efficiency.
For example, say actual sales volume for a given product or service for a given quarter. Or say actual number of vulnerabilities identified and corrected, within a sub-group of your infrastructure including applications. These numbers a.k.a. KPIs are collected after the fact, for a specific period, usually weekly or monthly or quarterly. These numbers are collected from the sales of your product or services, or the actual number of vulnerabilities identified versus the vulnerabilities rectified (patches applied and vulnerabilities mitigated) for a specific period. These become your department’s / project’s / task’s established KPIs. At the backend there must be an understanding as to what you are required to accomplish for a given period. Based upon your predefined targets, or Service-Level Agreements (SLAs) or Operational-Level Agreements (OLAs) it could clearly be established if this pattern is in the uptick or downtick.
Management is constantly looking into these KPIs and tuning the strategy to keep them in the positive territory. It is worth noting that sometimes a downtick is good, for example, take a KPI of number of cyberattacks on your infrastructure; the lesser, the better! When analyzing KPIs, there are three high level possibilities.
First you just met the threshold. Second you are exceeding the agreed threshold (by a little or by a significant amount) or third and the final one is, you are missing the threshold by a significant amount or by a little amount. In all these three cases, the common denominator here is that you either met your performance goals or missed it.
So, KPIs are after the fact, pre-agreed performance measure artifacts that help you fine-tune your approach, for the next run-time-window.
In short, KPIs are quantifiable performance monitoring metrics. -Asad Syed
Key Risk Indicators (KRIs) on the other hand are a little different. If KPI is a noun, then KRI is a verb. KRIs are also associated with performance measurement but are a matrix that tells you when to initiate an action when your performance starting to move in the wrong direction.
KRIs are always calculated with a mathematical operation. Let us take an example again:
Example #1: Actual sales volume for a given product or service by month. A target sales number is defined, and the sales teams work to achieve that goal.- KPI: Collecting and making this analytical data available for every week, in-itself is the sales team’s KPI.
- KRI: If this number is below 80% of the target. (Management defined and communicated in advance.)
- KRI initiated actions: You must devise a series of actions associated with the sales team, helping them to improve the sales for the future time periods.
Example #2: Actual number of vulnerabilities identified verses the vulnerabilities rectified for last month.- KPI: Collecting and making this analytical data available for every month, in-itself is the security vulnerability team’s KPI. Three numbers are collected here. Total number of systems by departments, department’s vulnerable systems for that period, and number of systems whose vulnerabilities were patched for the same period.
- KRI: Defined by Management. KRI is less than (<) 95% of critical and high vulnerabilities not patched for a period, then one or more mitigation action(s) must trigger. Like…
- ⁂ Highlight identified vulnerabilities in the monthly management vulnerability report.
- ⁂ Or a management meeting to discuss the variance and/or find ways to mitigate the newly identified unmitigated risks.
- ⁂ Or perform an architectural / security audit of the IT systems / departments, where these vulnerabilities were not mitigated.
- ⁂ Or any possible other actions that management would like to initiate with a goal to get the risk under acceptable level.
KRI measures the risk associated in the accomplishment of a specific project, task, or process and is used during risk mitigation. KRIs are numbers derived by mathematical operations on the identified KPIs for a given project, task, or process. The job of KRIs is to highlight when “The Risk” of a project, task, or process has stepped out of our comfort zone, in technical terms this is referred to as “out of your risk-appetite” boundary.
Both (KPI and KRI) need to be identified before additional actions can be taken.
Key Dimensions of KPIs and KRIs like Focus, Timeframe, Similarities and Strategic Importance:
KPIs and KRIs Focus Dimension:
Focus with KPIs is on positive outcomes and achievements, such as revenue growth, customer satisfaction, CyOps Incidents addressed, productivity, etc. Whereas with KRIs, the focus is on potential threats and vulnerabilities tracking, financial risks, operational risks including CyOps risks, compliance risks, etc.
KPIs and KRIs Time-frame Dimension:
From a Time-frame point of view, KPIs often are forward-looking and measure ongoing performance. Whereas KRIs can be both forward-looking and backward-looking, as they assess potential risks and the risks of performance impact at any point of time for which you have KRI data. Hence KRI can be done for the future or for the past.
KPIs and KRIs Similarities:
Both are measurements of quantifiable metrics but from different perspectives. Used to monitor and evaluate specific aspects of an organization’s performance and risk management.
Both are Data-Driven, meaning they rely on data to provide insights and support decision-making processes but from different contexts. It is worth noting that if the captured data is incorrect, then the predictive analysis will be wrong!
KPIs and KRIs Strategic Importance:
Both KPIs and KRIs are critical for achieving strategic goals and ensuring the long-term success and stability of your organization’s mission and vision. Both provide insight from the collected data from different contexts and perspectives.
KPI and KRI Key Takeaways:
Below are thirteen paired KPI/KRI discussion points to highlight the respective concept in greater depth and the role each one of these plays within the cybersecurity (CyOps) context. A meticulously crafted KPI and KRI within your cybersecurity operations (CyOps) will illuminate a wealth of performance metrics and security risk insights. This, in turn, propels your organization’s CyOps to the forefront of innovative efficiency and effectiveness.
№1
- KPI: Need to be identified and then collected.
- KRI: Need to be identified and then calculated by some mathematical operation or referenced “industry benchmark”.
№2
- KPI: Collected from cybersecurity operations (CyOps) data.
- KRI: Calculated from the collected KPI data, with one to one or one to many relationships. Meaning every identified KPI will have one or more associated KRIs.
№3
- KPI: Health gauge for your CyOps. In short, it is your cybersecurity operation’s performance indicators.
- KRI: Security Risk gauge for your identified cybersecurity KPIs.
№4
- KPI: Helps in CyOps performance measurement, monitoring, and decision-making.
- KRI: Helps in CyOps Risk Management and Risk Mitigation process.
№5
- KPI: Quantifies CyOps performance.
- KRI: Quantifies and aligns CyOps Risks with local environmental perspective. Used as an early warning system before risk materialization.
№6
- KPI: Alerts in advance of CyOps performance failure. Used for anomaly detection.
- KRI: Alerts in advance of CyOps risks unfolding.
№7
- KPI: Helps avoid CyOps bottlenecks.
- KRI: Helps your CyOps to avoid entering the red zone.
№8
- KPI: Provides cybersecurity project’s, tasks, or process, comparative view over a linear period.
- KRIs: Are a guide to initiating actions solely based upon “The Risk” when it has reached out of your appetite.
№9
- KPI: Project, or task, or process performance gauge.
- KRI: Security Risk gauge for your collected cybersecurity KPIs.
№10
- KPI: Measurement of cybersecurity Control’s Performance. Meaning, how well a Security Control performs.
- KRI: Measurement of cybersecurity Control’s Risk. Meaning, possibility of a Security Control failure.
№11
- KPI: Can measure cybersecurity Control’s effectiveness, efficiency, and compliance.
- KRI: Focus will be on the control failure Risk and not Performance.
№12
- KPI: Best when designed as Specific, Measurable, Auditable, Risk Aware and Timely (a.k.a. SMART) and reported to management in a clear, easy to understand report or dashboard.
Specific (Design your GRC Processes specific to your Business & Infrastructure needs)
Measurable (Design your GRC Processes to measure your Infrastructure health and finetune them by elevating the standards one notch at a time)
Auditable (Design your Infrastructure to pass Audits by design)
Risk Aware (Design your Infrastructure to be Risk Aware from the Gate / Inception)
Timely (Design your GRC Processes to address Infrastructure Health in a timely fashion)
- KRI: Best when designed to generate automated alerts, upon a defined risk threshold break. Meaning KRI crossing agreed threshold boundary. Then immediately initiate Risk Mitigation action or actions.
Both KPI and KRI are trailing indicators, meaning by the time you saw those, the matrix event already happened.
№13
- KPI: Downside in KPIs means there are operational issues. Before it deviates too much into negative territory, corrections need to be applied by the security operation / CyOps team.
- KRI: The question here is… if KRI has crossed the agreed risk threshold, how would we mitigate the impending risk?
The mitigation is possible with a well thought out plan in place. The function of KRI is just to alert you that the risk threshold is out of your comfort zone for a given KPI. Now what you do to get it back on track is entirely up to you.
Summary and Conclusion:
Both KPI and KRI are metric measures. These measurements tell you two different things. KPI highlights the performance measurements or lack there off, for a given project, task, or process. Whereas KRI gets triggered as an early warning system when the risk for a KPI reached a threshold (as defined by you) into a risky area.
KRI can then take an action (like a verb, as stated earlier) by alerting you to the fact that your defined threshold was crossed and help you initiate a mitigation action to bring back your KRI into the normal range. How you design your “mitigation action” is entirely your prerogative.
Both KPI and KRI are “analytical trend tools” based upon collected and calculated data patterns with a goal to let you know a deviation in performance or risk has happened in your project, task, or process. This is only possible if you plan and collect the analytical data and decide the threshold, to trigger the alerts and define in advance the measures you must take, in either of these cases. Should you have difficulties in visualizing your project/task/process’s KPIs or KRIs, drop me an email, highlighting the context and I can help. Hope the write-up added some value to your understanding of KPIs and KRIs. Think of these as the “secret ingredients” to your project’s success. Just like in the culinary world, these indicators add the essential flavors to your project, task, or process that determine whether your initiative will be delectable, widely appreciated, and/or triumphant.
About the Author:
Asad Syed is a graduate of Mathematics, Applied Mathematics and Statistics. His experience spans in Security Architecture, Security Operation Management, Digital Investigations & Forensics, Crisis & Threat Simulation, GRC Management, Threat Hunting, Cybersecurity Emerging Trends & Threat Mitigation, Database Security, Identity & Access Management, and Identity Federation. His interests are in the application of newer technologies, to enhance the output performance of technologies with which he is working. He is a writer, teacher, and cybersecurity evangelist, who has worked for multiple fortune five hundred companies and currently providing cybersecurity consulting to the upstream operations of the oil and gas industry. Reach him via Asad at ASyed dot com. ■
