FixBench Security Patch Remediation & Invariant Preservation
Evaluating whether automated security agent patches invalidate real exploit payloads while preserving complete functional test suites and introducing zero secondary vulnerabilities.
Python, Go, Java, Node.js, C/C++
Verified CVE Remediation Topologies
Dual-Gate (PoC Invalidation + Regressions)
Exploit Invalidation / Zero Regressions
Core Research Problem
While general software engineering benchmarks evaluate generic bug fixing, vulnerability remediation is fundamentally adversarial. A security patch must satisfy three simultaneous criteria: it must neutralize the active exploit payload, cause zero regression on legitimate application unit tests, and avoid introducing subtle secondary security defects. FixBench formalizes these three criteria into an automated evaluation protocol.
Exploit Vector Invalidation
The candidate patch must fundamentally disrupt the causal path of the reproducible PoC test case.
Zero Functional Regression
The complete upstream unit test suite must pass without any behavioral drift or breaking API changes.
Secondary Flaw Prevention
Automated AST and sanitizer audits ensure the patch does not introduce new type errors, denial-of-service, or injection bypasses.
Minimal Patch Diff Footprint
Penalizes over-broad refactoring in favor of focused, maintainable, minimal invariant-preserving edits.
Deterministic Execution Pipeline
Every evaluation task follows a structured, auditable lifecycle from ingestion to oracle confirmation.
Isolated Container Build
Vulnerable repository is compiled and booted inside an isolated container with baseline regression tests verified.
Candidate Patch Synthesis
Autonomous agent inspects vulnerability report and code context, synthesizing a candidate patch diff.
Dual-Oracle Automated Trial
Container applies patch: PoC exploit must fail, full regression test suite must pass, and static sanitizers confirm safety.
Falsifiable Scoring Standards
SecDome enforces strict ground-truth verification standards. No speculative scoring or synthetic shortcuts.
Deterministic Exploit Disruption
Patch must cause the existing automated PoC script to fail cleanly without throwing unexpected crashes.
Regression Test Invariance
100% of the repository existing unit, integration, and end-to-end tests must execute successfully.
Structural Safety Audit
Post-patch AST analysis verifies no dangerous sinks, permissive typecasts, or swallowed exceptions were introduced.
Evaluation harness and dual-oracle specification finalized. Corpus containerization in progress; baseline comparative results will be released openly upon completion of verification campaigns.
Reproducible & Auditable.
All SecDome benchmark specifications, task schemas, and evaluation protocols are maintained openly under Apache 2.0 / MIT. We invite external research groups to audit our task definitions and contribute reproducible evaluation environments.