How to Evaluate AI Agent Cybersecurity Capabilities

Wiz Research outlines the principles behind the relaunch of its Cyber Model Arena, a benchmark of 26 models, four general-purpose harnesses, 59 agents and 327 challenges across code vulnerabilities, API, web and cloud security, arguing that benchmarks should fix their objective first, use real-world tasks, grade deterministically with code rather than LLM judges, isolate agents to prevent cheating, and weigh success against cost. The post reports that agents perform very differently by category (average pass@1 of 78.0% on API security versus 38.2% on code vulnerabilities) and that pairing harness with model matters, with a mean absolute gap of 5.1 pass@1 points across 22 models run on both Claude Code and ADK React. It also lists limitations including simple harnesses, string-equality grading on code tasks, public web tasks that may be contaminated, a 1,000-second time cap and unpublished confidence intervals.