Due to the nature of the application, DarkHorse is not optimized for mobile devices. Please view it on a desktop or tablet for the best experience.
Blog
(words. probably too many of them.)Posts:
The Seven Tiers of Vulnerability Identification
They don't think it be like it is, but it do.
- Oscar Gamble
Being able to identify vulnerabilities is an essential part of any security program. However, there is often much confusion around (1) what the mechanisms are for identifying those vulnerabilities; (2) what they’re good for; and (3) when / where they’re most commonly used.
At DarkHorse, as part of our mission to make proactive security accessible and affordable to organizations of all sizes and budgets, it’s not uncommon for us to encounter some confusion around the different vulnerability identification options in the market - especially when working with earlier stage organizations who are just starting to build out their security program (though it should be noted that this confusion is certainly not limited to early-stage organizations).
In this longer-than-planned post, I’ll aim to cover what we at DarkHorse see as the seven tiers of vulnerability identification, and what they mean for you.
As touched on above, these offerings may vary widely in definition depending on who you’re talking to - which is why this guide exists - as it’s essential that there be a clear common language by which we can understand the differences and be able to speak confidently around what’s needed vs. what’s offered vs. what’s delivered. For the sake of this not becoming longer than it already is, this will only speak to vulnerability identification at the application and network/host levels (which is where the vast majority of vulnerabilities live).
These tiers are listed in the order that we recommend a maturing security program follow (e.g. starting at tier 0 comes before jumping to tier 5, and so on).
Without further ado, let's get started!
Tier 0: Vulnerability Disclosure Program (VDP)
Before we get too far down the rabbit hole, at a very basic level, every organization needs to have a VDP. What is a VDP? I’m glad you asked.
A VDP is a mechanism by which anyone who happens across a vulnerability within an asset of yours can responsibly report said vulnerability to your organization in a secure manner.
People will often passively find security issues, and it’s essential to have a way for them to responsibly report those issues to you. This isn't a theoretical use case either - for example, just the other day I was doing something online for my dog and stumbled across a high risk issue - which necessitated the need to have a way to securely share this information with the organization. The easier you make it, the more effective it will be. Of note, having a VDP is also part of the NIST Cybersecurity Framework (CSF), making it an essential (and easy-to-do) component of vuln identification within your security program.
A VDP can be as simple as a single line on the organization’s security or contact page saying “if you found a security issue, please report it to security@orgdomain.com” with a link to your public PGP key for the purposes of ensuring that sensitive vuln data over email is encrypted.
Or it can be equally as simple as using a purpose-built platform (such as DarkHorse) to intake the reports in a secure manner, and then putting a link to it on your site so that people can know where to responsibly disclose reports.
The advantages of using a platform vs. email are many; notably:
Easier to ingest report (vs. having to deal with PGP vs. using a platform that encrypts at rest and in transit by default)
Easier to track via a purpose built solution (vs. tracking via spreadsheets or similar mechanisms which can be slow and error-prone)
Formatted / better data. Using a platform allows you to ensure specific, essential details are included on every report, as opposed to everyone using their own format via free text in an email.
It’s free (with DarkHorse). Because we genuinely believe every organization should have a VDP, we’ve made that a reality with our free premium VDP offering on our platform - with no restriction around it being publicly listed, etc.
And more…
As it relates to VDPs, do note that:
A VDP is a passive way of learning about vulnerabilities. One should not expect a large number of reports via this mechanism, but reports do happen, and it’s important to have a means by which they can be responsible and securely reported to your organization.
VDPs do not offer any compensation. It’s fine to reciprocate if you find a report particularly insightful, but VDPs do not offer any sort of reward. Doing so turns it into a de facto bug bounty (which we’ll cover later).
Tier 1: Automated Vulnerability Scanning Assessment (AVS)
Note that this may also be marketed or sold as a “vulnerability assessment”, without clarification in the title as to whether it’s automated or not. And in unfortunate cases, some less scrupulous vendors may even try to sell an automated scan as a “pentest”. For this reason, it’s essential that you be able to know the difference, so you can ascertain if what someone is selling is truly a pentest, or if it’s just an automated scan masquerading as a pentest - as well as being able to confidently delineate between an automated assessment and a manual one.
Automated scanning is the cheapest and easiest way to set up any form of active vulnerability identification. Costs can range from free, all the way to tens (or hundreds) of thousands of dollars, depending on your scope and the solution you choose.
Automated scanning is near-always the first pass for securing a network or application. It’s quick, cost-efficient, and will (generally) pick up the extremely low hanging fruit, which then allows future testers to be more effective with their time - as opposed to spending their efforts identifying and reporting easy-to-find issues.
Automated scanners also have the benefit of being able to work nonstop. Once configured, you can run them daily, weekly, or 24/7, if you feel so inclined. They require little active maintenance and are a good way to provide low-cost continuous testing.
For reference, there are two types of application level testing:
DAST (Dynamic Application Security Testing) - this is where the application is tested as it is running and active. The scanner spiders the application and tests each individual input, parameter, and aspect of the target with a wide range of (sometimes contextual) payloads, and then evaluates the output to see if there’s a vulnerability. We’ll cover it more later when we talk about human testing, but it’s worth briefly highlighting that for lack of a better term, most scanners are “dumb” - meaning that they will throw the same thousands of payloads at just about everything, regardless of whether it makes sense to do so. Think of most scanners like an early-years Roomba. They just brute force their way around the room by bumping into things and seeing what falls out. Not horribly efficient, but easier than having a human have to do the same work, on average.
SAST (Static Application Security Testing) - this is where the application source code is reviewed by the scanner to identify vulnerabilities (1) earlier than them going into a live build; and / or (2) that may not or cannot be caught in dynamic testing, as the tester would likely never be exposed to that information unless they found another vulnerability that revealed it (such as hardcoded secrets, etc).
Neither solution is a replacement for the other, and doing both is the best and most comprehensive approach. At this time, no SAST tool will find everything a DAST tool would, and vice versa - so the best way forward is to use both.
One notable disadvantage to scanners is that they produce a high number of false positives. In fact, they create so many false positives that there are organizations whose entire business model is predicated on weeding them out, so that all the client sees is valid findings. It’s a lot of work to test each potential finding, only to find out that the scanner was wrong. Sometimes there can be hundreds, or even thousands of results, with only a few real (often low priority) issues. That said, scanning is absolutely still worth doing both as a first pass, and also as something that can run continuously.
That said, if you need help parsing through the results from a scanner, DarkHorse is happy to help connect you with a capable professional who can pull the signal from the noise.
It’s also worth highlighting that many scanners (and companies in general, for that matter) are incentivized to rate findings as high as possible, so as to give the illusion of providing significant value. For instance, they’ll commonly rate all sorts of low impact issues as “high” priority issues, giving a false sense of urgency to items that would otherwise be considered less important. For the avoidance of confusion, I’m 100% not saying to immediately downgrade anything that comes from a scanner. But I am saying that you should perform your own evaluation of risk when dealing with findings from any 3rd party - particularly so with scanners. If you want to learn more about how DarkHorse characterizes risk, we’ll be publishing a piece soon on the matter. It’s going to be another fun one!
Some free dynamic scanning tools include OpenVAS for network testing, and Zap for webapp scanning (those are the two most common free/open source ones). For SAST there are free/freemium solutions as well, but in the interest of avoiding making a recommendation, I’ll just say a Google search can get you started on that front. Additionally, there are a ton of paid options that go from relatively affordable, all the way to extremely unaffordable. Again, I won’t list them out here, but they’re not hard to find.
That said, if you happen to need someone to help you set up a scan with a quality solution, DarkHorse is, as always, happy to help at an affordable rate. And if you can’t afford it, we’re happy to help find a way to get you at least some minimal level of coverage. Let us know what you need, and we’re here to help. Again, our goal is to make proactive cybersecurity accessible and affordable for all organizations, regardless of their size or budget.
Tier 2: Manual Vulnerability Assessment (MVA)
This is where a human manually goes through the scope / targets and tests for vulnerabilities for anywhere from a half a day to as much as a day and a half (depending on the size of scope). When paired with automated scanning this approach can be highly effective and efficient - typically identifying as much as 80% of the findings that would be found in a full penetration test, in a fraction of the amount of time, and for significantly less cost.
What makes the scanner + human combo so effective is because while scanners are good at throwing high numbers of payloads at targets, they’re not so good (or good at all) at contextual intelligence. Where humans are markedly slower at shooting off payloads, they are incomparably better at trying to abuse any given functionality from a logical and contextual perspective.
For example, a scanner will throw thousands of payloads at a given parameter and then move on. However, an astute human will realize that the same parameter only has an impact on the application’s response when another contextual variable is checked - which would otherwise potentially be missed attack surface by the scanner. A scanner will spider a given application as best it can, while a human will recognize that certain actions need to be taken before other parts of the application become accessible (such as a step by step process, going through an onboarding flow, etc). A scanner will try all sorts of injections on a price parameter, while a human will realize that manipulating that value to the negative gives them that item for free. And so on. Despite all the hype around AI, there is still no replacement for human testing. And even if you only get a few hours of it, those few hours have asymmetrical returns relative to putting that time in elsewhere.
During a manual vulnerability assessment the tester will typically follow a stripped down version of a more verbose testing methodology (such as the OWASP testing guide) that’s focused on spending their time where scanners are less capable. Additionally, as they go through the target the tester will focus their time and efforts on where they feel the target has the highest probability of containing contextual or logical vulnerabilities. Over time, experienced testers are able to get a sense for where they think the highest value findings are located, and in cases such as with a manual assessment, will focus their time and attention there, so as to maximize the value and output from the limited hours that they’ll have to spend on the target.
Yet again, you may see this sold as a penetration test by some vendors. And while it’s a whole lot closer to a pentest than just running an automated scanner, we feel that it still doesn’t meet the bar for what a pentest should be. In our view, a pentest is a test that follows a comprehensive methodology across all aspects of the target(s) - budgeting enough time to be able to test each item and area to a reasonably thorough degree. To account for this more comprehensive approach, a pentest typically takes significantly longer than a manual vulnerability assessment (usually anywhere from 5-10x longer). Once more, we strongly recommend you be able to speak to this distinction when communicating with vendors who would otherwise sell you a “pentest”, and then perform a sub-pentest level of work.
At DarkHorse, we call this solution our “Essentials” - it’s our stripped back version of a pentest that can deliver a high amount of value for a low amount of time and money. If you need something done quickly, or done on an extremely tight budget, but don’t need it for compliance purposes, The Essentials may be right for you. Examples of when The Essentials could be ideal include if you don’t have the budget for a full pentest, but want to ensure you get at least some human testing in place, or if you want to perform human-driven testing on your scope more often, but only have the budget for two pentests a year. Instead of doing two pentests, you could do one pentest and then three manual vulnerability assessments, and so on.
Tier 3: Manual Penetration Test (“pentest”)
This is the gold standard as it relates to testing the security of an application, network, device, or just about anything (for instance, you could have a pentest done on a bank, physical location, etc).
In our view, a penetration test is a test that follows a comprehensive methodology specific to the scope being tested, and that allows sufficient time for the whole of the scope to be tested to a reasonable level by an expert in accordance with the defined methodology.
To ensure sufficient coverage, pentests are usually measured in terms of “days” of effort. After going through the scoping process with a given vendor, you’ll typically be advised of how many days of effort they estimate the test (and reporting) to be. Note that this estimate does not guarantee that testing will take place for X days - in some cases the test might conclude more quickly, and in others it may take longer.
NOTE: in cases where the test will take longer due to inaccurate or incomplete scoping information provided by the buyer / client, the client will typically be responsible for the cost of the overrun, since the original estimate was provided based on the expectation of having complete and accurate information from the client in the first place. So, when engaged in any sort of pentest scoping exercise, be sure to be as accurate and precise as possible.
Upon completion of the pentest, an executive summary as well as a full PDF report is usually provided to the client. There may be check-ins along the way, but that varies by provider. However, with the advent of Penetration Testing as a Service (PTaaS), organizations are now able to see their results, track the tester’s progress against the methodology, and communicate with the tester - all in the platform in real time. Note that not all PTaaS offerings provide these functionalities, but they are available in the market.
Modern PTaaS also enables organizations to launch their programs more quickly, with less effort. Historically, pentests have required long scoping processes, lengthy SoWs, and take weeks to months to get started. With PTaaS (and specifically DarkHorse), you can input your requirements and scope within minutes, and get started within days - not weeks-to-months.
Pentests (like all test types) can take place against static (source code) or dynamic (live) targets (as noted above, pentests can also take place against virtually any target type - including physical targets, etc).
Additionally, retesting findings is a common exercise for after items have been remediated, along with a re-issuing of an updated pentest report stating what has / hasn’t been fixed from the original test. This may or may not be included by default, depending on your vendor, so be sure to make sure it’s addressed if this is something you will want or need.
Depending on your specific need, the contents of the report can vary significantly from vendor to vendor. Some vendors use standardized text, while more bespoke pentest vendors may include narratives around the testing, and everything in-between. Do note that the more complex the report, on average, the more expensive it will be.
Finally, it's worth being aware that our definition of pentesting is based on how we see the market interpret and use the term today. In the same way that the meaning of words can and do change over time, the same has happened with pentesting. Some purists may insist that pentesting is testing that goes beyond vulnerability identification - which is semantically accurate. It makes sense that to “penetration test” something is to penetrate as far as one can go, so as to see the depth of the penetration. In this semantically-accurate paradigm, if a tester found a code execution vulnerability, they wouldn’t stop at the identification of the finding, but would instead use that vulnerability to see if they could elevate privileges on the machine, and then see about using that machine to gain access to other machines or sensitive information stored locally, and so on. Again, this view is definitionally accurate, but not necessarily what 95%+ of the market means when they say “pentest”. So, just like referring to someone from Canada as an “American” is technically correct, as they’re from one of the American continents, it’s neither useful nor helpful to do so. It may have been loosely true 300 years ago, but not now. The same holds true for how and why we call a pentest what we call it today. This current definition of a pentest (having a qualified tester follow a defined methodology, with enough time allocated to comprehensively follow said methodology in relation to the scope) is widely understood and presently accepted (by both auditors as well as purchasers) as the primary definition of what the thing is at this point in time. If it changes in the future, we’ll also update our perspective accordingly. That said, post-exploitation type testing absolutely has its place - and we’ll get to that as our final type of vulnerability identification under the term “red-teaming”.
Tier 4: Fractional / Objective-Based Testing.
But what if you just want someone to perform security testing against one specific thing? You may not need a full pentest, you just need effort on a specific thing, or in a specific area. That’s where fractional / objective-based testing comes in.
Fractional / objective-based testing is where one sets a specific task or objective for the test, as well as how much effort they expect to be expended in pursuit of the goal (in hours or days). The work then happens via a qualified tester, and upon completion each party goes their separate ways.
This is usually, but not always accompanied by a writeup / report on what was tried, as well as any findings that resulted from the test.
Fractional / objective-based testing is great for when you want to ensure something has been reviewed by a security expert, but you don’t want or need to pentest everything else around it. For instance, you just built a new feature, and want to make sure it’s secure, or you have a suspicion around a specific asset or configuration that doesn’t feel quite right. In those cases, fractional or objective based testing is a great solution. Note: it’s important to be aware that since this is a partial test, this will typically not be accepted as a valid penetration test by auditors, and is most commonly used for internal use / peace of mind.
This is not offered by all providers in the space, but is offered by DarkHorse. If this is something you feel you or your organization may need, we’d be happy to help.
Tier 5: Bug Bounty.
If you’ve already done all of the above, that’s fantastic! Now it’s time to kick things into high gear. If you’re serious about finding as many vulnerabilities as possible in whatever your scope is, there is no better active tool for doing so than bug bounty.
Bug bounty is the process by which you offer incentives to anyone in the world where, if they’re the first to identify a unique security vulnerability in your defined scope, you’ll reward them with some form of compensation - usually cash.
As you can imagine, offering money to find vulnerabilities brings a massive array of talent to bear on your scope - collectively referred to as “the crowd”. And each individual within the crowd has a slightly unique approach to how they perform testing, creating an incredibly diverse set of perspectives that are able to collectively identify more vulnerabilities than any one person or team ever could. The logic for the value in this approach is pretty straightforward: five testers are more likely to find more issues than a single tester, and fifty will find more than the five, and five-hundred more than the fifty, and so on. There are diminishing returns as the numbers get larger, but as a rule, there is no upper bound where more becomes a bad thing. It’s possible that the ten-thousand-and-first person to look at the target / scope has just the perspective that nobody else had, and so on.
I’ve written tens (possibly hundreds) of thousands of words on the nuance of bug bounty, so I’ll constrain myself here to just some of the most frequently asked questions; the largest of which is “but why would I want to invite someone to hack me?”
The short answer is: the bad guys are doing it already. Nefarious actors don’t need your permission to look for vulnerabilities and to exploit them. The good guys do need your permission, because they’re playing by the rules. So, in reality, by setting up a bug bounty program we’re merely leveling the playing field to allow the good guys to have as much of a shot at finding vulnerabilities as the bad guys (which is good for us).
As evidenced by the tiers here, bug bounties are something we recommend organizations grow into, as opposed to a starting point. But once you’ve done scanning and have run a pentest (or at least a manual assessment), and want to get truly serious about identifying as much risk as possible, there is no better way to expand your proactive security program than bug bounty.
As a way of allowing organizations to grow into bug bounties, it’s also possible to run a “private” bug bounty - which will allow you to have a level of control on how many testers (and thereby the number of reports) come in over time. The (general) goal for all bug bounties is to ideally become completely open to the entire public, but we recommend all organizations start small (private) and grow from there (crawl, walk, run), with a clear plan and path in place to be open to the public in the future. In some cases that takes weeks, in other cases years - it all depends on your appetite and capabilities to handle additional reports.
There are many more topics we won’t get into here, including scope, rewards, growth, crowd nuances, and more. But we will likely cover those topics in detail in the future.
If you’re interested in running a bug bounty program, DarkHorse can help. DarkHorse’s goal is to make proactive security (including bug bounties) accessible and affordable for organizations of all sizes and budgets. Where it would typically cost tens of thousands of dollars to run a bug bounty with one of the major providers today, DarkHorse enables every organization to affordably start their bug bounty journey for free - because we believe that security needs to be available to all organizations, not just those with deep wallets.
Tier 6: Red-Teaming.
Our final tier of vulnerability identification, red-teaming, is where the tester (or team) is given a specific goal to go after, as well as some rules of engagement (RoE), and then they do whatever is necessary to accomplish the goal within the constraints of the engagement. Note that this differs from our view on pentests - where pentests are methodology driven, red-teaming is goal based. Astute observers will note that this sounds a lot like “objective-based” testing, which is true. Both types of testing have set goals, but for the avoidance of confusion, objective-based testing is not always “red-teaming”, but red-teaming will always be objective based - even if that objective is to “do as much damage as possible”.
For instance, a red team’s rules of engagement may be to get as far and as deep as they possibly can into a given network or system - going well beyond initial vulnerability identification and exploiting the findings to “pivot” further and further. E.g. if a tester is able to get code execution on a machine during a pentest (as we’ve defined it above), they would typically stop there and write up the finding upon validating the presence of the vulnerability. On a red-team engagement, depending on the scope and objective, they may try to establish persistence on the machine, escalate privileges, access other machines or services on the same internal network as the compromised system, search the compromised system for sensitive information such as saved passwords, and then use that information to gain access to additional systems, and so on.
It’s incredibly important to point out here that with a red-team engagement the express goal is to accomplish the objective, not to cover the attack surface comprehensively, or to identify as many vulnerabilities as possible (unless that’s the stated goal). If the tester is able to find a path to get to the objective (say, Domain Admin) within five minutes, they’ll have accomplished the goal of the test - and as such, they’ll write up their findings and the engagement will be considered complete. There could be a number of other routes to accomplish the same objective, or other vulnerabilities along the way, but unless explicitly stated in the statement of work that the tester is to comprehensively test everything along the way, they won’t - because spending their time testing everything else would be antithetical to the goal of getting to the objective, which is the whole point of the exercise in the first place. To be hyper-clear - this isn’t a bad thing. It’s simply how it works - we don’t get mad if a race car can’t pull a travel trailer, because that’s not what it’s built for. Same applies here - each of these offerings are built for a specific use case and function. As in the rest of the world, no one thing does everything.
Other types of red-teaming tests include getting to or accomplishing a certain objective, like gaining access to a certain room or building, or capturing a flag that was planted by the person who set up the test. It may be getting Domain Admin, gaining access to a certain person’s machine, account, or any number of other outcomes.
These types of engagements can be broad, or highly specialized. They may have little-to-no-rules, or may have a complex set of requirements the testers have to abide by. For instance, in some cases red teamers are allowed to perform social engineering on employees as a way to establish a foothold in the organization. In other contexts, it’s limited to testing a finite set of systems, and so on.
The scope and nature of these engagements is limited only by your imagination and probably more immediately by your budget. As part of our commitment to making proactive security accessible and affordable for all organizations, regardless of their size or budget, at DarkHorse we offer affordable rates for objective-based testing. Though we don’t do any on-site or in-person testing, we are able to service just about any other need or requirement.
And there you have it! Those are the seven types of vulnerability identification as we (DarkHorse) see it today. You should now know:
What the different types / tiers are.
How to tell the difference between a pentest and other items masquerading as a pentest.
What the typical progression is between tiers.
How / when to use each solution.
Where to find help with each of the above (hint: DarkHorse)
Was this too long? Probably. Yes.
Could it be shorter? If I had more time.
Did you actually read this far? Congrats!
Did we miss something? It's possible. We’ll revisit over time and make updates as we see necessary.
Notice something in the above is factually incorrect or off? Please let us know, and we’ll fix it ASAP.
As with everything DarkHorse does, we hope this is useful for organizations of all sizes and natures, and if you’d like DarkHorse to help you with your proactive security / vulnerability identification program, let us know. Even if you’re not able to afford any of the above solutions today, if you’re looking to become more secure, we may be able to set you up with some pro bono help.
Here’s to making the world a better and more secure place,
Grant McCracken Founder & Head Rodeo Clown @ DarkHorse https://darkhorse.sh