Bug Bounties Versus Penetration Testing

For companies that are considering a bug bounty versus a penetration test, my opinion is that this is not an either-or question. Bug bounties are helpful for various reasons. There are also different reasons why you might want a penetration test.

First of all, what is a bug bounty and what is a penetration test?

Bug Bounty: Companies allow security researchers, penetration testers, and hackers around the world to submit security bugs they find. If a bug is deemed worthy, the company pays the person who submitted the security vulnerability some money, typically called a bug bounty.

Penetration Test: An organization hires a company like 2nd Sight Lab to perform a test of their environment and report back on any security problems.

A penetration test or bug bounty differs from a security audit or security assessment because the company performing the test will actively try to exploit vulnerabilities.

Why are bug bounties useful?

By creating an organized program to allow authorized hackers to break into systems, anyone on the Internet who finds a vulnerability will be motivated to report it and work with the organization’s security team to resolve the problem. Bug bounties incentivize security researchers and anyone who stumbles across a bug to report it.

When companies pay well, it gives security researchers a reason to spend more time and dig deeper into a bug than they might do otherwise. Instead of just reporting the issue and moving on, they may try to determine if the vulnerability leads to other problems and the severity of the finding.

Bug bounties are also useful because they are ongoing, not a snapshot in time. Bug bounty hunters will be constantly reviewing the company’s applications for new problems as the products change. I like to say each new line of code is a new potential bug.

The problem when organizations don’t offer a bug bounty

If an organization does not pay for time invested in reporting vulnerabilities, people who find problems may be less apt to want to spend a lot of time on resolving the matter if they report it at all. As I explained in another blog post, I find so many vulnerabilities just carrying out my day-to-day obligations and cannot afford to spend the time it would take working on all those things for free. I always report vulnerabilities, if I can find a way to do so on the organization’s website, but I stop there unless they are paying me to research and test the issue further.

For example, I spoke at RSA 2020 on security serverless cloud environments. I showed how I used a technique called fuzzing on a serverless application penetration test and uncovered a vulnerability that both revealed some identity information and also led to a cross-site scripting attack.

When I discovered that issue initially, I was not sure if the problem was an AWS problem or a bug in the software written by my customer. Before I reported the finding to my customer, I told the vendor (AWS) about it via their secure reporting channel. I did not want to reveal a critical issue on the AWS platform to anyone else until they had a chance to review it and fix it if they deemed that necessary. AWS replied that the bug was in a beta service and did not seem too concerned.

I dropped it at that point because they were not paying me, but my customer was. To my knowledge, Amazon did not have a bug bounty for AWS vulnerabilities at that time and still does not. Case in point, see James Kettle’s talk from DEFCON where he received $20,000 for a vulnerability from one company running systems on AWS but nothing from Amazon:

After the AWS security team did not seem to think the vulnerability was an issue since it was in a beta system that no one should be using for production, I did not spend more time with them. I had a penetration test to complete with a deadline. I worked with the company that was paying me to further report and resolve the issue. That company told me it was a bug in their code, not an AWS bug. I sent off my report and prepared a presentation for RSA which included some of the findings, without revealing the organization with whom I was working.

My customer’s application was still in development and not available to the public, so I felt the risk of revealing the vulnerability was low. After I gave my presentation at RSA and spoke about the bug, my customer came back and told me they thought the issue was an Amazon problem after all. I told them to talk to Amazon about the matter to resolve it. I asked them to let me know if they needed more information and directed them to my video presentation if they needed help explaining the issue to Amazon.

How Bug Bounties Help

Had Amazon offered a bug bounty and a history of paying well for security vulnerabilities I probably would have spent more time trying to convince them this was a serious problem. I may not have accepted my customer’s statement that it was their bug at face value and worked to verify exactly where the vulnerability existed. However, given that my company needs to make money, same as Amazon, I didn’t put a lot of unpaid time into that effort. Companies that pay bug bounties will get a different response from security researchers and pentesters than the one I just described.

I want to reemphasize the part about having a history of paying well for vulnerabilities. Google and Microsoft have such a history. Bug hunters that submit bugs to bounty programs talk about avoiding companies that tend to reject all bugs and don’t pay well. I’ve also written before about hacker competitions where security researchers stockpile zero-days to win hundreds of thousands of dollars. A bug bounty alone is not going to get researchers to submit bugs. Companies need to pay a fair amount.

Limitations of Bug Bounties

Bounty hunters tend to go for the things they know or what pays the most. they often find things by accident and sometimes while performing actions they would have performed either way. The best security researchers often do not perform a methodical test of all common types of attacks a website might face. Some types of tests take a lot of time and often provide little in terms of payout so they won’t bother.

Some bug hunters only do basic, automated testing. To get serious research, reverse engineering, logic analysis, and testing for complicated vulnerabilities takes time, so a lot of penetration testers have come out on Twitter saying they don’t bother with bug bounties. The fact that you may or may not get paid is not worth it. Some researchers told me they spend time surfing sites and testing in their free time for fun, but it’s not a serious effort.

Other researchers such as James Kettle, who spoke in the link I provided above, are often researching a specific topic, not spending time understanding all the ins and outs of a particular application to try to break into it. He can afford to do deep research since his company sells a penetration product, and that research goes into product improvements. However, it is typically a deep dive on one new type of vulnerability. That is much different than a penetration test designed to look at your web applications or cloud environment as a whole.

Where penetration tests prevail over bug bounties

One issue with bug bounties is that often the scope of the test restricts bounty hunters from using automated tools to perform their tests. As I explained above and in my talk at RSA, I tend to use a method called fuzzing. The bounty restrictions generally say that the security team does that, so don’t bother. The issue for the company is that if everyone is fuzzing their website at once, it will likely cause performance issues and possibly additional costs in a cloud environment.

The problem from a security perspective is that there is a huge variety of attacks using fuzzing that the security team may not have tested or know about. That means they can miss something even though they perform the generic scans provided by canned software products. In my case, I use large and varied fuzzing lists and am always searching for new things to test. Some fuzzers also randomize the inputs. A security team may not fuzz or scan all available attack vectors. Although I could manually do the same thing on a bug bounty it is just not worth the time and effort. I can perform the work much more quickly and get improved coverage using a fuzzer than via manual testing, though I do some of both on penetration tests.

Penetration tests also provide value because a company will perform a methodical and thorough battery of tests. A penetration test is not about spending as little amount of time as possible and moving on (a good one, that is). They have a method and a documented process for testing applications, systems, and networks that they use for every customer. They will also likely spend more quality time looking for logic flaws and reverse engineering to find problems unique to that particular application. They can do that because they are getting paid regardless of whether they discover something or not.

A penetration test report includes a list of findings categorized as high, medium, or low risk. Companies may need this report for compliance purposes. They can also use the report to show potential customers that their systems have been methodically tested by a known and reputable cybersecurity firm.

In the case of 2nd Sight Lab, we also try to provide some guidance towards fixing and preventing vulnerabilities in the future. Our tests also include 2nd Sight Lab’s overall assessment and security issues that go beyond a simple web application scan, application vulnerabilities, or cloud misconfiguration. We provide more holistic security operations, architectural, and software development recommendations in some cases to reduce future vulnerabilities, depending on the nature of the engagement.

Limitations of a penetration test

One of the limitations of a penetration test is that it is a point-in-time snapshot of an organization’s security. Some companies may know a penetration test is coming and take steps to secure the environment right before the engagement. That can give an organization and those reading the report a false sense of security. It’s advisable to perform penetration tests frequently — preferably a couple of times a year, but at least once per year.

Another limitation of a penetration test is time. Often attackers will take months and years to break into systems. Penetration tests are generally shorter in length. 2nd Sight Lab works with customers to set up a test in such a way that we can find the most vulnerabilities in the shortest amount of time for this reason.

Even with those additional efforts, I often hear stories about bug bounty hunters that find things by accident. In one case, a bounty hunter submitted something into a system and forgot about it. Months later, he went back to use that system in a new way and inadvertently uncovered a bug, which he then spent additional time nailing down and reporting.

So what’s the verdict? Bug Bounty or Penetration Test?

There is no one answer for every company, but large companies will get the most coverage if they do both. Allow hackers to constantly scan your applications for new vulnerabilities. When a new class of vulnerabilities becomes available through some published research, you can be sure that those bounty hunters are going to try it out to see if they can make some money.

Companies that have never even had a penetration test may want to start there. The time and cost to resolve numerous low-level bug reports will probably be greater than the cost of a basic penetration test. There are many different levels of what people call penetration tests. A simple web application scan is not a very good penetration test. Try to get something more detailed and thorough.

Organizations working on new technology and seeking first-mover competitive advantage may not want to expose systems to a bug bounty until they have gained market share. Once you expose your new idea to a bug bounty program, numerous people will see what you’re developing.

Additionally, organizations that don’t have the resources to evaluate how risky or damaging a bug bounty finding is may want to avoid bug bounty programs. An incorrect interpretation may lend itself to attacks rather than prevent them.

Additionally, organizations that do not have the staff and bandwidth to fix bug bounty findings may open themselves up to future compromise. One honest bug bounty hunter may find a problem, while criminal bug hunters use the bug bounty system to find vulnerable systems to attack for other purposes. If you open yourself up to a bug bounty program, make sure you have adequate monitoring in place to identify malicious activity on your network.

Make sure you can pay a fair amount if you have a bug bounty program. A disgruntled bug bounty hunter may complain about inadequate payments or worse. They may sell the vulnerabilities they find to bad actors.

Hopefully, these considerations will help organizations when trying to decide if they should or should not create a bug bounty program. I’ve explained some of the pros, cons, and caveats. Overall I think bug bounties are great! However, I also see ways in which penetration tests can be very beneficial for organizations. And if you need the latter, please consider 2nd Sight Lab! Contact me on LinkedIn for more information.

Teri Radichel — Follow me @teriradichel

© 2nd Sight Lab 2021

___________________________________________________

Want to learn more about Cybersecurity and Cloud Security? Check out: Cybersecurity for Executives in the Age of Cloud on Amazon

Need Cloud Security Training? 2nd Sight Lab Cloud Security Training

Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.

Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.

For a recap of cybersecurity news last week check out the 2nd Sight Lab Cybersecurity News Blog. Malware, vulnerabilities, data breaches, cost of a data breach, cybersecurity laws, and interesting cybersecurity developments.

Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts