Harness’s LLMScanBench Q3 2026 found that large language model (LLM) security scanners can return different results on unchanged code, and Harness lets customers configure an LLM scan to block a build. Harness is competing for the decision about what ships to production, and its benchmark defines what it must show before an AI scan can block a merge.
What Is Covered in This Article:
- What Harness’s LLMScanBench Q3 2026 found about LLM security scanning, and two related Harness posts on pipeline scanning and decision models.
- Why the decision about what ships to production is where Harness is competing.
- Why a merge gate needs verdict stability, and what Harness has and hasn’t shown.
- How GitHub, OpenAI, and Harness each treat AI security findings at the merge.
- What Harness, its competitors, and platform and security leaders should do next.
The News: On September 28, 2026, Harness published LLMScanBench Q3 2026. It ran GPT-5.5 and Claude Opus 4.8, both through OpenAI Codex Security’s security skills and with no human in the loop, against Harness AI SAST, its static application security testing (SAST) tool, on three intentionally vulnerable Open Worldwide Application Security Project (OWASP) applications. Harness reports that the LLM scanners posted 85% to 100% precision but low recall, with GPT-5.5 ahead of Opus 4.8 in most cases, and took 1.6 to 37 times longer and cost 6 to 39 times more than AI SAST. Three Opus 4.8 runs on unchanged WebGoat code returned recall of 16.2%, 19.7%, and 35.5%.
Harness concludes that LLM scanners are “a harder fit for scanning regular PRs or code commits today.” It recommends them for penetration testing and periodic deep scans, and in pipelines, using LLMs “to validate and reason on the results of traditional scanners and to identify novel vulnerabilities.”
Two related posts show how Harness applies those findings. A September 23 post describes Harness Security Testing Orchestration (STO), which is built on OpenAI Codex Security and runs an LLM scan on every pull request; the post does not state its availability or pricing. An October 5 post describes an application security (AppSec) classifier that flags customer API endpoints that call an LLM. It is built on Jev, a TypeSafe AI decision model that returns a probability rather than text. Harness routes the cases Jev is least sure about to an LLM. Details are in Harness’s LLMScanBench Q3 2026 benchmark post.
Harness Shows Why LLM Security Scanning Needs a Repeatable Verdict
Analyst Take: Harness lets customers configure an LLM security scan to stop a release. GitHub keeps its AI detections advisory during preview, OpenAI tells teams to start the advisory, and Harness’s LLM security scanning benchmark explains the caution.
As agents write more of the code, the decision about what ships to production becomes the control point that code hosts, model vendors, and delivery platforms compete for. In July, Futurum Research practice lead and analyst Mitch Ashley said Codex Security should earn a merge-gate role only after OpenAI shows that repeated scans of the same code produce consistent findings. Harness’s data shows why, and verdict stability, how often a scan’s pass or fail result changes on unchanged code, is the number that should settle it.
The Release Decision Is the New Control Point, and Harness Lets Customers Hand It to AI
The decision to ship enforces the customer’s policy, which makes it harder to displace than code generation, where every model vendor competes. Harness, which sells “AI for everything after code,” is positioning its pipeline to make that decision, and its STO step “can be configured to block the build” on an LLM scan’s findings.
For the CIO and head of Engineering, the release gate carries more than a tool’s verdict. It records that a human-owned policy was enforced before code shipped, which is the evidence an auditor asks for and the line a post-incident review traces back to. Hand that control to a nondeterministic system, and the record stops being defensible because a verdict no one can reproduce is difficult to stand behind. The engineering problem and the governance problem are one and the same. A gate that cannot repeat its answer cannot be owned.
Code hosts and model vendors are treating that decision more carefully than Harness is. GitHub’s AI security detections, in public preview, “are informational and won’t block pull request merges,” and OpenAI’s Codex Security CI guide tells teams to “start with advisory results.” That leaves Harness as the vendor in this group whose AI verdict can stop a release, so its evidence matters most.
Competing for the release decision suits an incumbent delivery platform, and Harness’s connector for ChatGPT and Codex, which lets OpenAI’s agents run their pipelines with the user’s approval, puts more agent-started changes in front of that gate. It pays off only if teams trust the AI verdict there.
LLM Security Scanning That Changes Its Answer Teaches Developers to Ignore It
A gate that flips its verdict on unchanged code teaches engineers to override it, and then they end up overriding real findings too. Google saw this with tests. In May 2016, Google’s John Micco reported that about 84% of the pass-to-fail changes in its continuous integration (CI) system involved flaky tests, which pass and fail on the same code, and that “it is quite common to ignore legitimate failures in flaky tests.”
Harness’s benchmark shows that LLM scanners can behave similarly, though the evidence is limited. In three runs of Opus 4.8 through Codex Security’s prompts on unchanged code, recall on Juice Shop went from 16.0% to 30.0% to 18.8%, while Benchmark for Java stayed within a point. Harness tested one model this way, not STO, and a change in recall becomes a pass-or-fail flip once a finding at the blocking severity appears or disappears.
At least one team has already seen the flip in a live pipeline, from the pattern Harness recommends. On October 5, 2026, an issue in Stacklok’s dockyard repository reported a blocking scan that “flips between pass and fail on byte-identical content” because an LLM step decided which rule-based findings were false positives. An LLM that only validates a traditional scanner’s results, as Harness advises for pipelines, can still flip a gate.
The cost of an unreliable gate reaches far beyond the pull request. Once a team learns that a check fails on code that already passed, the team stops trusting the check, and the habit of overriding spreads to every control in the pipeline. That is organizational debt, and it does not show up in a precision or recall score. It surfaces in audit findings and incident post-mortems, where the question is why a control that was in place did not hold. A benchmark measures the scanner. The CIO ends up accountable for what the benchmark cannot see.
Verdict stability is the measure that should decide whether an AI scanner can gate a release. Snyk published the most detailed measurement so far in a June 2026 benchmark, where 134 of 158 Claude findings that matched Snyk Code’s reference set appeared in all five runs and about half of Claude’s extra findings (80 of 161) appeared in only one. Harness, GitHub, OpenAI, and Anthropic haven’t published a verdict-stability figure for their scanners.
Most Buyers Lack Scanning Built for AI Code, the Gap AI Gates Are Sold to Fill
Demand for AI scanning at the gate is real because AI is writing code faster than organizations can check it. In The Futurum Group’s 2H 2026 Software Lifecycle Engineering Global Enterprise Decision Maker Survey (n=839, fielded June 2026), 40% of organizations said AI generated or substantially modified most of the production code they merged in the previous 90 days, yet only 39% require security scanning specific to AI-generated code.
Cost and accuracy also hold buyers back. The cost of tokens, compute, and licensing (44%) and code quality and accuracy concerns (43%) are the two most-cited limits on expanding AI use, so a vendor that shows a stable, affordable AI verdict answers both.
The difference between how much code AI writes and how little gets AI-specific scanning is an ownership gap before it is a tooling gap. Most organizations have not named who owns the risk in machine-generated code, so no one funds the control that would check it.
The cost figures point in the same direction. No CIO adds a line item that runs several times the cost of the existing scanner on a verdict that has not proven it repeats. The question a buyer actually asks is whether AI scanning folds into platform spend the organization already carries, which is the ground an incumbent delivery platform competes on and a point tool does not.
Harness’s Benchmark Sets a Bar STO Hasn’t Yet Shown It Clears
Harness has set a standard; STO hasn’t yet shown it meets it. LLMScanBench concludes that LLM scanners are a harder fit for regular pull requests, and Harness’s data supports that conclusion. The STO post itself says, “You cannot simply point a model at a repository and call the result a scanner,” and then describes an LLM scan that runs on every pull request and can block the build.
Of Harness’s three designs, only the Jev classifier keeps a generative model out of the final call by thresholding Jev’s probability in Harness’s code and sending only uncertain cases to an LLM. AI SAST filters a deterministic engine’s findings with AI reasoning, which Harness says yields “deterministic findings,” and STO’s verdict comes from the LLM itself; Harness says STO’s orchestration “does not make the underlying model deterministic.” Both need the test Stacklok’s flip points.
Only deterministic findings should block, and no LLM step should remove one from the blocking set. LLM discovery of vulnerabilities that no rule covers still belongs in the pipeline, with its findings posted as comments.
A stable verdict also isn’t enough on its own. Embedding security at the moment of creation, the pull request, needs a known rate of what the check misses, because a deterministic scanner can miss the same vulnerability on every run; Harness’s AI SAST recall on WebGoat ran from 40.3% to 54.8%, depending on how it filtered findings.
Harness deserves credit for disclosing the benchmark’s main weaknesses itself, including that the models are trained on these applications. Its AI SAST precision and recall also cover WebGoat only; two of the three answer keys were built partly from the scanners’ own findings, the cost comparison sets per-scan token charges against an amortized monthly fee, and the post links no scripts or raw outputs.
An AI Scan Should Not Block a Merge Until Its Verdict Repeats
- Harness: publish verdict stability for STO’s pipeline scan, plus LLMScanBench’s scripts and raw outputs.
- GitHub: set the evidence AI security detections must show before they can block a merge, before a delivery platform does.
- OpenAI and Anthropic: publish verdict stability for Codex Security and Claude’s security review alongside precision and recall, since every platform that wraps their scanners inherits that number.
- Platform engineering and AppSec leaders: before giving any LLM security scanning tool permission to block a merge, run it at least three times on unchanged code, as Harness and Stacklok each did. Let only deterministic findings block, and post AI-only findings as comments until they recur. Name who rules on a disputed finding and decide whether a scan that times out passes or fails.
- CIOs and CISOs: decide who owns a wrong verdict, a block that stops good code, and a pass that ships a real vulnerability, before any AI gate is allowed to block. Require the gate to produce a record that can be reproduced, so the control can be shown to an auditor.
- Vendors: Treat verdict stability as a procurement acceptance test, not an engineering preference, and weigh the concentration risk of customers letting the delivery vendor own the policy the organization is accountable for enforcing.
If Harness publishes verdict stability for STO, its benchmark becomes a reason to buy its pipeline. Until it does, the safe setting for LLM security scanning is the one GitHub chose and OpenAI recommends: show the finding and let something repeatable decide the merge.
What to Watch:
- Whether Harness publishes availability, pricing, and verdict-stability data for STO’s LLM pipeline scanning, and whether customers enable it to block builds.
- Whether GitHub lets AI security detections block merges when the feature leaves public preview.
- Whether OpenAI or Anthropic publishes verdict-stability figures for their code security scanners.
- Whether the next LLMScanBench edition tests code the models are unlikely to have seen and publishes its scripts and raw outputs.
- Whether calibrated decision models, such as TypeSafe’s Jev, in early access since September 15, 2026, move into the security pipeline gates.
See the complete benchmark write-up from Harness’s engineers on the Harness website.
Declaration of generative AI and AI-assisted technologies in the writing process: This content has been generated with the support of artificial intelligence technologies. Due to the fast pace of content creation and the continuous evolution of data and information, The Futurum Group and its analysts strive to ensure the accuracy and factual integrity of the information presented. However, the opinions and interpretations expressed in this content reflect those of the individual author/analyst. The Futurum Group makes no guarantees regarding the completeness, accuracy, or reliability of any information contained herein. Readers are encouraged to verify facts independently and consult relevant sources for further clarification.
Disclosure: Futurum is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of Futurum as a whole.
Read the full Futurum Group Disclosure.
Other Insights From Futurum:
Anthropic Glasswing: AI Vulnerability Detection Has Crossed a Threshold
OpenAI Daybreak Aims For The Agentic AppSec Workflow
Palo Alto Networks Bets Frontier AI Can Make Pentesting Continuous
Why AI Learned to Attack Before It Learned to Defend
Author Information
Vikram Rathnam is Research Director, Software Lifecycle Engineering at The Futurum Group. His research examines how software is built, tested, secured, and operated as AI agents move from assisting developers to doing the work. His coverage includes observability, platform engineering, and the control planes that govern agents. Before joining Futurum, he spent 24 years on the vendor side: ten in engineering at Silicon Labs and fourteen in product and partner strategy, with roles at Cisco, Secureworks, Dell, and CMIT Solutions. Across those roles, he managed B2B cybersecurity, IoT, and AI products, with partner and channel ecosystems as the common thread. At CMIT Solutions, he led product strategy across a franchise network of 180 locations. He reads a vendor’s roadmap as someone who has had to ship one and take it to market.
Mitch Ashley is VP and Practice Lead for the CIO & Technology Buyers and Software Lifecycle Engineering practices at The Futurum Group. A multi-time CIO and CTO with 30+ years leading technical organizations, Mitch built and operated production systems spanning cybersecurity for the U.S. Department of Defense, PKI services for the broadband and 5G industries, SaaS platforms, large-scale telecom and banking systems, and a national broadband network. His work with AI began early, developing expert systems that diagnosed and repaired complex mainframe environments. That operator foundation grounds his analysis in operational consequence, covering the technology buyer's world of software engineering, cybersecurity, DevOps, cloud, and AI.
