This brief write-up is to discuss the features of Key Performance Indicators (KPIs) and Key Risk Indicators (KRIs) from a cybersecurity vantage point. In this paper we will highlight the relationship between these, how these are different and same in certain ways and their value add from cybersecurity operations (CyOps) point of view.
KPI measures the performance of any operational function, cybersecurity, CyOps or otherwise.
Captured KPIs establish and demonstrate how effective an organization function is performing.
KPIs tell the behind story. Did the organization achieve its established goals, or exceed those, or missed it? Of course to measure if something is missed, one has to first establish a goal and associate measurement statistics. Then if missed or exceeded, KPIs highlights by how much.
In short, KPIs are a quantitative measure of highlighting the success or failure of a target goal achievement.
For any function within your organization, we can identify a few critical KPIs for a specific project, task, or process. May it be finance or production or marketing or cybersecurity operations, or any other business function.
As a side note, you may even identify KPIs for task / projects at home or even for your own personal relationships — because KPIs will help you measure the success or failure of your accomplishments and / or operational efficiency.
For example, say actual sales volume for a given product or service for a given quarter. Or say actual number of vulnerabilities identified and corrected, within a sub-group of your infrastructure including applications. These numbers a.k.a. KPIs are collected after the fact, for a specific period, usually weekly or monthly or quarterly. These numbers are collected from the sales of your product or services, or the actual number of vulnerabilities identified versus the vulnerabilities rectified (patches applied and vulnerabilities mitigated) for a specific period. These become your department’s / project’s / task’s established KPIs. At the backend there must be an understanding as to what you are required to accomplish for a given period. Based upon your predefined targets, or Service-Level Agreements (SLAs) or Operational-Level Agreements (OLAs) it could clearly be established if this pattern is in the uptick or downtick.
Management is constantly looking into these KPIs and tuning the strategy to keep them in the positive territory. It is worth noting that sometimes downtick is good, for example, take a KPI of number of cyberattacks on your infrastructure — the lesser, the better! When analyzing KPIs, there are three high level possibilities.
First you just met the threshold. Second you are exceeding the agreed threshold (by a little or by a significant amount) or third and the final one is you are missing the threshold by a significant amount or by a little amount. In all these three cases, the common denominator here is that you either met your performance goals or missed it.
So, KPIs are after the fact, pre-agreed performance measure artifacts that help you fine-tune your approach, for the next run-time-window.
In short, KPIs are quantifiable performance monitoring metrics.
Key Risk Indicators (KRIs) on the other hand are a little different. If KPI is a noun, then KRI is a verb. KRIs are also associated with performance measurement but are a matrix that tells you when to initiate an action when your performance starting to move in the wrong direction.
KRIs are always calculated with a mathematical operation. Let us take the same examples as of above:
Example #1: Actual sales volume for a given product or service by month. A target sales number is defined, and the sales teams work to achieve that goal.- KPI: Collecting and making this analytical data available for every week, in-itself is the sales team’s KPI.
- KRI: If this number is below 80% of the target. (Management defined and communicated in advance.)
- KRI initiated actions: You must devise a series of actions associated with the sales team, helping them to improve the sales for the future time periods.
Example #2: Actual number of vulnerabilities identified verses the vulnerabilities rectified for last month.- KPI: Collecting and making this analytical data available for every month, in-itself is the security vulnerability team’s KPI. Three numbers are collected here. Total number of systems by departments, department’s vulnerable systems for that period, and number of systems whose vulnerabilities were patched for the same period.
- KRI: Defined by Management. KRI is less than (<) 95% of critical and high vulnerabilities not patched for a period, then one or more mitigation action(s) must trigger. Like…
- ⁂ Highlight identified vulnerabilities in the monthly management vulnerability report.
- ⁂ Or a management meeting to discuss the variance and/or find ways to mitigate the newly identified unmitigated risks.
- ⁂ Or perform an architectural / security audit of the IT systems / departments, where these vulnerabilities were not mitigated.
- ⁂ Or any possible other actions that management would like to initiate with a goal to get the risk under acceptable level.
KRI measures the risk associated in the accomplishment of a specific project, task, or process and is used during risk mitigation. KRIs are numbers derived by mathematical operations on the identified KPIs for a given project, task, or process. The job of KRIs is to highlight when “The Risk” of a project, task, or process has stepped out of our comfort zone, in technical terms this is referred to as “out of your risk-appetite” boundary.
Both (KPI and KRI) need to be identified before additional actions can be taken. Below thirteen paired KPI/KRI features highlighted to explain the KPI and KRI concept in greater depth and the role each one of these play within the cybersecurity context. A well designed KPI and KRI from your cybersecurity operations (CyOps) will highlight a wealth of operations performance and associated security risk related information.
Key Takeaways
№1
- KPI: Need to be identified and then collected.
- KRI: Need to be identified and then calculated by some mathematical operation or referenced “industry benchmark”.
№2
- KPI: Collected from cybersecurity operations (CyOps) data.
- KRI: Calculated from the collected KPI data, with one to one or one to many relationships. Meaning every identified KPI will have one or more associated KRIs.
№3
- KPI: Health gauge for your CyOps. In short, it is your cybersecurity operation’s performance indicators.
- KRI: Security Risk gauge for your identified cybersecurity KPIs.
№4
- KPI: Helps in CyOps performance measurement, monitoring, and decision-making.
- KRI: Helps in CyOps Risk Management and Risk Mitigation process.
№5
- KPI: Quantifies CyOps performance.
- KRI: Quantifies and aligns CyOps Risks with local environmental perspective. Used as an early warning system before risk materialization.
№6
- KPI: Alerts in advance of CyOps performance failure. Used for anomaly detection.
- KRI: Alerts in advance of CyOps risks unfolding.
№7
- KPI: Helps avoid CyOps bottlenecks.
- KRI: Helps your CyOps to avoid entering the red zone.
№8
- KPI: Provides cybersecurity project’s, tasks, or process, comparative view over a linear period.
- KRIs: Are a guide to initiating actions solely based upon “The Risk” when it has reached out of your appetite.
№9
- KPI: Project, or task, or process performance gauge.
- KRI: Security Risk gauge for your collected cybersecurity KPIs.
№10
- KPI: Measurement of cybersecurity Control’s Performance. Meaning, how well a Security Control performs.
- KRI: Measurement of cybersecurity Control’s Risk. Meaning, possibility of a Security Control failure.
№11
- KPI: Can measure cybersecurity Control’s effectiveness, efficiency, and compliance.
- KRI: Focus will be on the control failure Risk and not Performance.
№12
- KPI: Best when designed as Specific, Measurable, Auditable, Risk Aware and Timely (a.k.a. SMART) and reported to management in a clear, easy to understand report or dashboard.
Specific (Design your GRC Processes specific to your Business & Infrastructure needs)
Measurable (Design your GRC Processes to measure your Infrastructure health and finetune them by elevating the standards one notch at a time)
Auditable (Design your Infrastructure to pass Audits by design)
Risk Aware (Design your Infrastructure to be Risk Aware from the Gate / Inception)
Timely (Design your GRC Processes to address Infrastructure Health in a timely fashion)
- KRI: Best when designed to generate automated alerts, upon a defined risk threshold break. Meaning KRI crossing agreed threshold boundary. Then immediately initiate Risk Mitigation action or actions.
Both KPI and KRI are trailing indicators, meaning by the time you saw those, the matrix event already happened.
№13
- KPI: Downside in KPIs means there are operational issues. Before it deviates too much into negative territory, corrections need to be applied by the security operation / CyOps team.
- KRI: The question here is… if KRI has crossed the agreed risk threshold, how would we mitigate the impending risk?
The mitigation is possible with a well thought out plan in place. The function of KRI is just to alert you that the risk threshold is out of your comfort zone for a given KPI. Now what you do to get it back on track is entirely up to you.
Summary and Conclusion:
Both KPI and KRI are metric measures. These measurements tell you two different things. KPI highlights the performance measurements or lack there off, for a given project, task, or process. Whereas KRI gets triggered as an early warning system when the risk for a KPI reached a threshold (as defined by you) into a risky area.
KRI can then take an action (like a verb, as stated earlier) by alerting you to the fact that your defined threshold was crossed and help you initiate a mitigation action to bring back your KRI into the normal range. How you design your “mitigation action” is entirely your prerogative.
Both KPI and KRI are “analytical trend tools” based upon collected and calculated data patterns with a goal to let you know a deviation in performance or risk has happened in your project, task, or process. This is only possible if you plan and collect the analytical data and decide the threshold, to trigger the alerts and define in advance the measures you must take, in either of these cases. Should you have difficulties in visualizing your project/task/process’s KPIs or KRIs, drop me an email, highlighting the context and I can help. Hope the write-up added some value to your understanding of KPIs and KRIs. The final thought on these is that you may consider KPIs and KRIs as “key ingredients” of your project, task, or process that can make or break your initiative. In culinary terms, KPIs and KRIs make your initiative palatable!
About the Author:
Asad Syed is a graduate of Mathematics, Applied Mathematics and Statistics. His experience spans in Security Architecture, Security Operation Management, Digital Investigations & Forensics, Crisis & Threat Simulation, GRC Management, Threat Hunting, Cybersecurity Emerging Trends & Threat Mitigation, Database Security, Identity & Access Management, and Identity Federation. His interests are in the application of newer technologies, to enhance the output performance of technologies with which he is working. He is a writer, teacher, and cybersecurity evangelist, who has worked for multiple fortune five hundred companies and currently providing cybersecurity consulting to the upstream operations of the oil and gas industry. Reach him via GRC at ASyed dot net. ■
