--- title: "Beyond the Model: Benchmarking Codex Security, Claude Security, and Nebu" description: "We benchmark our pipeline against Codex Security and Claude Security. Our pipeline gives you a better trade-off: match competing scanners in a fraction of the time and cost, or scale up to find more bugs than either of them." url: https://nebusec.ai/research/nebu-benchmark/ published: 2026-09-01T00:00:00.000Z --- # Beyond the Model: Benchmarking Codex Security, Claude Security, and Nebu Nebula Security · September 1, 2026 · 5 min read * [benchmark](https://nebusec.ai/tags/benchmark) * [security](https://nebusec.ai/tags/security) Table of contents * [Introduction](#introduction) * [Experiment Setup](#experiment-setup) * [Results](#results) * [Codex Security and Claude Security](#codex-security-and-claude-security) * [Nebu changes the tradeoff](#nebu-changes-the-tradeoff) * [The effect of the pipeline](#the-effect-of-the-pipeline) * [Takeaway](#takeaway) ## Introduction Codex Security and Claude Security are built for the same task, but they make different tradeoffs. We run both on the same open source projects to compare bug coverage, cost, and time. Codex Security can move from a fast low-cost scan to much higher coverage with deep scan. Claude Security follows a different curve, and its result changes sharply when its report cutoff is removed. The more surprising result comes from Nebu. With GPT-5.4, Nebu reaches 80% coverage. This matches the best Codex Security result in our benchmark with GPT-5.6-Sol. It also takes far less time and has a lower cost. Our results suggest that pipeline design can be worth several generations of model progress: a weaker model, given the right orchestration, can reach the performance of much newer models. ## Experiment Setup We benchmark our pipeline against **Codex Security** and **Claude Security** on open source projects, using 76 known bugs as ground truth. We use the frontier models from OpenAI and Anthropic: `GPT-5.4`, `GPT-5.6-Sol`, and `Opus 5`. We do not have access to Mythos, but according to [Anthropic](https://www.anthropic.com/news/claude-opus-5), Opus 5 has the same vulnerability identification capability as Fable 5, which uses the same underlying model as Mythos. We divide the benchmark into four groups by model and product: 1. Codex Security (GPT-5.4) 2. Codex Security (GPT-5.6-Sol) 3. Claude Security (Opus 5) 4. Nebu (GPT-5.4 & GPT-5.6-Sol) For each group, we test different security scan modes and model reasoning efforts. Codex Security has a [deep scan mode](https://learn.chatgpt.com/docs/security/plugin/deep-scans) toggle, while Claude Security has optional scan levels (low/medium/high/max). Those options change the orchestration of the security scan and are orthogonal to reasoning efforts. We test the following candidate configurations: | Group | Base | Enhanced | Extended | | ---------------------------- | ---------------------------- | --------------------------------- | ---------------------------------------- | | Codex Security (GPT-5.4) | High reasoning Standard scan | **Xhigh reasoning** Standard scan | Xhigh reasoning **Deep scan** | | Codex Security (GPT-5.6-Sol) | High reasoning Standard scan | **Xhigh reasoning** Standard scan | Xhigh reasoning **Deep scan** | | Claude Security (Opus 5) | High reasoning High scan | **Xhigh reasoning** **Max scan** | Xhigh reasoning Max scan (**no cutoff**) | | Nebu | High reasoning GPT-5.4 | **Xhigh reasoning** GPT-5.4 | Xhigh reasoning **GPT-5.6-Sol** | We cap Codex Security at xhigh reasoning because it is the recommended setting for best security scan performance in the official documentation. For Claude Security, the default scan returns only the top 45 reports and drops the rest. We therefore include a no-cutoff variant that keeps all reports, so the comparison is not affected by this internal limit. We measure performance along three axes: **bug coverage**, **cost**, and **time**. The benchmark shows clear differences between Codex Security and Claude Security. It also shows how much these tradeoffs can change with a different security pipeline. ## Results Bug coverage vs. estimated cost Data labels Each line traces a benchmark group from its base configuration to stronger settings. At comparable coverage, Nebu has a lower estimated cost. ### Codex Security and Claude Security Codex Security has two very different performance profiles. With GPT-5.6-Sol in Standard scan, xhigh reasoning reaches 41% coverage in 29 minutes at a cost of about $15. Deep scan raises coverage to 80%, but runtime increases to 5h 50m and cost rises to about $1.5k. Claude Security follows a different curve. Opus 5 with xhigh reasoning and Max scan reaches 42% coverage in 2h 48m at about $270. That is similar coverage to Codex Security in Standard scan, but it takes longer and costs more. Claude Security also keeps only the top reports by default. Removing this cutoff raises measured coverage to 74%. Runtime increases to 4h 39m, with a cost of about $445. At the strongest settings we tested, Codex Security reaches higher coverage than Claude Security, at 80% versus 74%. Claude Security runs faster and at a lower cost, but reaches lower coverage. ### Nebu changes the tradeoff Bug coverage vs. time Data labels At a lighter setting, Nebu reaches 62% coverage in 50 minutes with GPT-5.4 as the audit model. This is already well above Codex Security with GPT-5.4 Deep scan at 43%, and Claude Security with xhigh reasoning and Max scan at 42%. A stronger Nebu configuration reaches 80% coverage in 1h 26m. It matches the best Codex Security result we measured and exceeds the 74% reached by Claude Security without the report cutoff, while finishing sooner and at a lower cost than both. Nebu can scale further to 88% coverage, the highest result in the benchmark. This run finishes in 3h 25m. It is still faster than the highest coverage Codex Security and Claude Security configurations we tested, although it uses more compute. ## The effect of the pipeline Same model improvement (GPT-5.4 xhigh) Codex coverage · Nebu coverage · Cost per bug With GPT-5.4 and xhigh reasoning fixed for the audit, Codex Security goes from 24% coverage in Standard scan to 43% with Deep scan. Nebu reaches 80%. ## Takeaway Standard Codex is fast and cheap, but its coverage is limited. Claude Security reaches similar coverage with more time and cost. Removing Claude’s report cutoff improves coverage substantially. Codex Deep reaches much higher coverage, but requires a large increase in both runtime and cost. Nebu changes that tradeoff. It can reach the best Codex Security coverage we measured much faster and at a lower cost. It can also use more compute to reach 88% coverage, the highest result in the benchmark. A better pipeline can be worth several generations of model progress. > **Notes** > > * We have never used any benchmark test data, in any form, to improve our pipeline. > * We have never intentionally removed any test cases that were unfavorable to our pipeline. > * All benchmark runs were conducted with web search and network access disabled.