- › A vulnerability scan lists known weaknesses system by system, and a penetration test shows what a skilled person can actually reach by chaining them together.
- › Contracts, insurers, and frameworks ask for a penetration test, and price-driven buying lets an automated scan with a new cover page fill that requirement.
- › A report with no attack narrative, no evidence of access gained, and findings that read like tool output is almost certainly a scan.
- › Before buying, ask for the objective, the number of manual testing days, a redacted sample report, and who will actually do the work.
- › Buy both and call each by its name: frequent scanning to keep the basics clean, and a scoped test with a goal to find the paths scanning cannot.
Picture the report arriving. Your organization paid for an annual penetration test because a customer contract and your cyber insurance application both ask for one, and two weeks later an eighty-page PDF lands in your inbox. It has an executive summary with a colored risk chart, then page after page of findings, each one with a severity score, a short description, and a paragraph of remediation advice. The conclusion says no critical exploitable issues were identified.
Look more closely and every finding reads the same way. The descriptions are generic, the advice is the vendor’s standard text, and nothing in the document describes anybody actually doing anything. Nobody logged into your application, nobody tried to move from the first weakness to the second, and nobody wrote down what they reached. What you bought was a vulnerability scan with a new cover page, and the box on the insurance form now says yes.
Two Different Tools That Share a Name
A vulnerability scan is automated. A tool looks at each system it can reach, compares what it sees against a large catalog of known problems, and reports every match. Scanning is fast, broad, and cheap, and it is very good at telling you which systems are missing patches, running outdated software, or exposing services they should not. It is also prone to false positives, and it is structurally blind to anything that is not in its catalog, including flaws in how your own application handles its business logic.
A penetration test is a person with a goal. The tester is given an objective, such as reaching the payment database, taking over an administrator account, or getting from the guest wireless network into the finance systems, and tries to achieve it using whatever they find. The value comes from the chaining. A medium-severity misconfiguration, a reused password, and an overly broad file share are three unremarkable lines in a scan report, and in the hands of a tester they can become a single path to the data that matters.
Both are worth paying for, and neither replaces the other. The problem is one being sold, bought, and reported as the other.
Why the Substitution Keeps Happening
The pressure starts with the requirement. Customer contracts, insurance applications, and compliance frameworks commonly ask whether you have had a penetration test in the last year, and the person who has to answer is rarely the person who can judge the work. To them, the requirement is a document with the right title and a date on it.
Procurement makes it worse. When three proposals arrive and one costs a fraction of the others, the cheap one looks like good negotiating, especially when all three use the same words. Pricing per IP address or per application makes the comparison feel objective, and it rewards whichever vendor can cover the most addresses in the least human time, which is the vendor running a scanner.
On the selling side, scanning scales and skilled testers do not. A firm can run scans for hundreds of clients with a small staff, and terms like “automated penetration testing” blur the line in the proposal before the buyer has read past the first page. Both sides end up with what they were measured on, a pattern we described on the questionnaire side of vendor risk in the security questionnaire nobody believes.
How to Tell From the Report
The report usually gives it away if you know what to look for, and four signs carry most of the weight. The first is the absence of a story. A real test report describes attack paths in sequence: what the tester found first, what that gave them, what they tried next, and where they ended up. If the document is only a list of individual findings with no narrative connecting any of them, nobody connected them.
The second is evidence of access. A tester who got somewhere shows it, with screenshots of a session they obtained, a sample of data they could read, or a description of the account they compromised. Findings that say a system “may be vulnerable” without anybody confirming it are scanner output.
The third is severity that ignores context. Scanners score each finding in isolation using a generic rating. A tester adjusts severity based on what the weakness actually allowed in your environment, so a low-rated issue that formed part of a path to sensitive data gets called out, and a high-rated issue on an isolated test box gets explained.
The fourth is missing application logic. If your web application was in scope and the report contains no findings about what an authenticated user could do, such as changing an order number in a request to see another customer’s order, then nobody tested the application as a user. That class of flaw is exactly what automated tools miss and attackers look for.
How to Tell Before You Buy
It is far cheaper to sort this out in the proposal than after the report arrives. Five questions do most of the work.
What objective will the tester pursue?
A real test has a goal stated in plain language. If the proposal only lists the IP ranges and applications to be “assessed,” ask what the tester will be trying to reach and how they will know they succeeded.
How many days of manual testing are included?
Ask for person-days of hands-on testing, separate from any scanning. A proposal that cannot give you that number, or gives a number that is implausibly small for the scope, is telling you how much human attention you are paying for.
Can we see a redacted sample report?
Read it for the four signs above. A firm that does real testing has reports full of attack narratives and evidence, and it can show you one with the client details removed.
Who will do the work?
Ask for the names and backgrounds of the people who will be on your engagement, and whether they are the same people named in the proposal. Some firms sell with senior staff and deliver with whoever is free.
What happens when they get in?
The rules of engagement should say whether testers stop at the first foothold or continue to show impact, and how they will handle sensitive data if they reach it. Also ask whether a retest is included, because a finding nobody rechecks after the fix is a finding you are trusting on faith.
Buy Both, and Call Each by Its Name
Scanning should be frequent and routine, internal and external, monthly at a minimum and continuous where you can manage it. It keeps the basics clean, and it should feed your patching process directly. Pairing it with external attack surface management tells you what you have exposed that nobody remembered owning.
A penetration test should be less frequent, scoped around an objective, and timed after major changes to the environment as well as on the annual cycle. Run your scans and fix the obvious issues first, so that the tester’s paid days go into finding the paths a scanner cannot see instead of rediscovering missing patches. And read the scope carefully, because what the assessment was not allowed to touch is often where the real risk sits.
When a contract or an insurer asks whether you have had a penetration test, answer about the work you actually received. A form that says yes on the strength of a relabeled scan is a statement somebody may examine closely after an incident, and insurers have become more specific about technical requirements in the last few years.
What the Wrong One Costs
A scan tells you what is wrong with each system on its own. Attackers do not work one system at a time. They take the small, unremarkable weaknesses a scan rates as medium and low, put them together, and walk a path nobody looked for, because the only test you bought was never designed to look for paths.
The money saved by buying the cheaper proposal is real and visible. The cost shows up later, when an incident investigation traces the route an attacker took and you find every step of it listed separately in last year’s report, with nobody ever having noticed that the steps connected.
Chris Armour is Director of Information Security at Grab The Axe.
Operating on the philosophy that 'you can't build a secure system if you don't know how to break it,' Chris leads our engineering division. A top 1% National Cyber League competitor, he hardens our digital infrastructure against the very exploits he has mastered.
View Author Page →
