#评测/基准

XYBench: Helping with the problem behind the question

XYBench is a new benchmark of 8,115 real-world questions, drawn from Stack Overflow, Stack Exchange, curated examples, and WikiHow, that tests whether AI assistants recognize when a user's request rests on a misconception (the "XY problem") and redirect them toward their actual goal. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the authors found every model was more likely to address the stated request than the underlying goal, and all trailed human reference answers on misconception identification. The paper won the Best Resource Paper Award at EMNLP 2026.

XYBench: Helping with the problem behind the question

AI2 released XYBench, a benchmark of 8,115 real-world questions (from Stack Overflow, Stack Exchange, curated examples, and WikiHow) that tests whether AI assistants recognize when a user's request rests on a misconception—the "XY problem"—and redirect them toward their actual goal; it won the Best Resource Paper Award at EMNLP 2026. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the researchers found all five models were more likely to address the stated request than the underlying goal and trailed human reference answers on misconception identification. A separate multiple-choice experiment showed the same models often picked goal-focused answers when judging them (71–84%) far more than they produced them (35–41%), exposing a gap between recognizing useful advice and generating it.

Reranking in RAG: How to Fix Bad Retrieval with a Reranker

This guide explains that vector search often ranks near-miss chunks first because embeddings compress whole chunks into single vectors, and shows how adding a cross-encoder reranker over the top 30 candidates raised precision@1 from 81% to 97% on a 274-question synthetic benchmark at roughly $0.10 per 1,000 queries. It compares 11 rerankers (voyageai/rerank-3-lite and rerank-3 were the most accurate dedicated models and cheapest, while LLM rerankers like openai/gpt-6-luna hit 99.6% but took about 4 seconds per call), covers candidate-count tuning, input_type settings, and silent failures such as cohere/rerank-v3.5's 4,096-token limit. It includes a Python example using the embeddings and rerank APIs and notes that real-world gains depend on your own documents and evaluation set.

How to Benchmark LLMs

A guide to benchmarking LLMs by writing tasks with script-checkable answers, grading with code, and comparing models on identical tasks via OpenRouter using graded accuracy, billed cost, and request time. On 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials while DeepSeek had the lowest response cost at 157/180 passes, and agreement with the public Artificial Analysis ranking was weak (Spearman correlation 0.109). The article includes a downloadable Python companion package with offline checks, configuration examples, and guidance on repeats, missing answers, and run comparison.

JioStar’s chief architect on AI discovery at 500M-user scale

JioStar’s Vijay Seshadri explains AI discovery at 500M-user scale. He shares lessons on evals, guardrails, and taking AI into production.

Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

This NeurIPS 2026 workshop paper examines how well LLM judges evaluate personalized writing tasks, testing how the writing task, generator-judge pairing, evaluation objective, and general reasoning ability affect judge performance across four LLMs and three long-form writing tasks with human pairwise labels. The authors find judge quality correlates weakly with MMLU-measured ability (Pearson's ρ ≤ 0.2), varies by task and criteria, and that paired comparisons yield more consistent results than independent scoring, while the strongest LLMs do not most reliably reproduce human evaluations.

Genie with Genie Ontology Delivers 1.6× the Accuracy with Less than Half the Tokens

Databricks developed a new enterprise data-agent benchmark with Turing, run in a live Databricks workspace, in which Genie achieved a 91% pass rate versus 45% and 58% for two frontier agents while using fewer than half the tokens and completing tasks 3–4x faster. The benchmark, built around a fictional UK energy supplier called Calderwick Energy, includes notebooks, dashboards, saved SQL queries and documents alongside hundreds of tables, testing whether agents can find and use business context that table schemas alone cannot provide. Genie Ontology, now generally available, automatically extracts and organizes this knowledge and surfaces relevant context for Genie.

LLM Quantization: What fp8 and fp4 Do to Quality and Price

OpenRouter tested Qwen3.8 27B across six endpoints at fp16, fp8 and fp4 on 240 exact-answer tasks and found no detectable accuracy difference, with all endpoints scoring between 98.3% and 99.2%. Across 1,455 endpoints, fp8 was the most common labeled precision (23.6%) while 59.5% reported none, and lower precision was usually but not always cheaper. The post explains fp8 and fp4 formats and shows how to pin precision with provider.quantizations, noting that a narrow filter can leave a single endpoint exposed to rate limits.

The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

This AWS post explains how to evaluate production agents on Amazon Bedrock AgentCore by starting from business outcomes, breaking them into observable behaviors, and mapping each to a trace- or session-level evaluator. It covers the built-in AWS evaluators, managed third-party metrics from DeepEval and AutoEval, custom and multi-dimension LLM-as-a-judge evaluators, and suggested evaluator stacks for customer support, research, and domain-content agents. It also describes three evaluation modes—on-demand, online, and batch—that share the same evaluator IDs, plus the ListEvaluators API for retrieving the full catalog.

Multimodal Knowledge Agent (MKA) benchmark: testing knowledge work capabilities with complex visual files

Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, comprising 200 private real-world visual tasks across 10 industries in which agents work from dense visual files like architectural drawings, engineering schematics, and clinical scans to produce deliverables such as slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Tasks are graded against expert rubrics averaging ~70 weighted criteria split into Regression and Climb checks, with the headline Climb WSR score, run in the open-source Stirrup agent harness with up to 250 turns and graded by GPT-6 Luna (medium) as an agentic judge. Claude Fable 5.2 (max) leads with 57% Climb WSR, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%, while Claude Fable 5.2 averages $31.97 per task versus $6.29 for Gemini 4 Carbon and $1.11 for GPT-6.1 Sol.

How to Evaluate AI Agent Cybersecurity Capabilities

Wiz Research outlines the principles behind the relaunch of its Cyber Model Arena, a benchmark of 26 models, four general-purpose harnesses, 59 agents and 327 challenges across code vulnerabilities, API, web and cloud security, arguing that benchmarks should fix their objective first, use real-world tasks, grade deterministically with code rather than LLM judges, isolate agents to prevent cheating, and weigh success against cost. The post reports that agents perform very differently by category (average pass@1 of 78.0% on API security versus 38.2% on code vulnerabilities) and that pairing harness with model matters, with a mean absolute gap of 5.1 pass@1 points across 22 models run on both Claude Code and ADK React. It also lists limitations including simple harnesses, string-equality grading on code tasks, public web tasks that may be contaminated, a 1,000-second time cap and unpublished confidence intervals.

Best OCR Model for Scanned Documents, Measured

The author tested 15 vision models plus Mistral OCR and Tesseract on eight synthetic scanned pages scored by character error rate and 264 key-value checks, finding Claude Opus 5.5 most accurate (262 of 264 at $16.42 per 1,000 pages) and Mistral OCR the best value (258 of 264 at $2.02). The post documents failure modes such as fabricated card digits, wifi passwords and server names on a faded receipt, and provides code for transcription, disabling reasoning, cross-checking two models from different labs, and using the mistral-ocr engine.

Meta's Muse Ember is the cheapest model scoring 36 on the Intelligence Index

Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41, making it the cheapest model scoring 36 or higher. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, but its AA-Omniscience accuracy of 21.1% is the lowest among similarly intelligent models, largely because it declines to answer 64% of questions, keeping its hallucination rate at 17.4%. Pricing is $0.10 input / $0.004 cached input / $0.50 output per 1M tokens, available via Meta's first-party API and Muse Code.

Introducing the Artificial Analysis Decision Index v2

Artificial Analysis released Decision Index v2, with 45% of the Index built on private datasets (up from 25% in v1) and a new Decision Score that rewards both correctness and calibrated confidence. Drex Max v1.1 (Nace AI) and GLiDE v1.2 (Fastino Labs) both score 73, with Mammoth 27B (Baseten) at 72, while cost per 1M Tasks ranges from $14 for Celeris-1.1 Decision to $859 for Clef. The update adds AA-Paperclip (1,500 private documents), AA-Symmetry, revised AA-Omniscience-Decision and HLE-MultiChoice benchmarks, and EvalSafe-O*NET-AA, while reducing When2Call to 10% and removing WildGuardTest.

Endpoint Accuracy Index v1.1: Tighter long context intervals and expanded model coverage

Artificial Analysis released Endpoint Accuracy Index v1.1, which updates AA-LCR-25 to dataset version 1.1 with corrected answer keys, an updated equality-checker LLM and 25 repeats per question, narrowing the average 95% confidence interval from ±4.7 to ±2.6 points. Coverage now spans 103 serverless endpoints across six open weights models, adding GLM-5.3 Flash and Muse Spark Open and replacing DeepSeek V4 Pro with DeepSeek V4.1 Flash 1017, while GLM-5.2 leaves. Of the 103 endpoints, 24 score statistically below their self-hosted reference, with causes grouped as output token caps, reduced reasoning, tool call handling differences, and request limits or stream behavior.

Measuring how AI agents handle everyday wet-lab science

OpenAI introduces BioBasicsBench, a 100-task benchmark measuring how well AI agents interpret raw experimental data and handle everyday wet-lab science.

Best Proxy Providers for AI Agents (2026)

The article compares seven rotating residential proxy providers (Decodo, Oxylabs, FloppyData, Roundproxies, Byteful, DataImpulse, Webshare) on IP pool size, location targeting, sticky session length, entry pricing, trials and traffic expiry, and reports results from browser-agent benchmark tasks such as product research, text extraction, form filling and session continuity. It also covers built-in versus bring-your-own proxies on browser platforms, setup guidance for custom proxies, and advice on checking session continuity, location coverage, target-site access and cost per completed task. The author notes the samples are small and differently timed and do not establish a universal reliability or speed ranking.

ToyBlocks: A controlled world for robot learning and evaluation

ToyBlocks is an open simulation framework built around a robot arm and blocks on a tabletop that automatically generates tasks and demonstrations, using reinforcement learning and motion planning to produce skills such as pushing, flipping, and stacking. It supports studies of low-level action model generalization, high-level planning with vision-language models, agentic systems using tool calls, action-conditioned world models, and zero-shot sim-to-real transfer, with an initial release of 85,336 long-horizon demonstrations covering nearly 1,600 hours and 144 million frames. The authors present ToyBlocks as the first of their FoundationWorlds and plan a second release with a larger dataset, more skills, and integrated RL training.

VIALS-AA: different paths to scientific image reasoning

VIALS-AA analyzes how 19 models reason over scientific images, classifying 9,177 tool calls from its original test set: 38.3% were image viewing and 47.3% prepared or measured images. Image measurement made up just 5–6% of tool calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), versus 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). The report also notes that Gemini 3.8 Flash (high) accounts for 86.7% of commentary/no-op calls, that more intelligent models tend to call fewer tools, and that cost per task varies by nearly 100× across models.

Egyptian and Nigerian English join the Open ASR Leaderboard

The Open ASR Leaderboard added two private conversational splits from Liva AI covering Egyptian English (1.34 h) and Nigerian English (1.55 h), both punctuated, cased and containing disfluencies. The leaderboard now reports a richer accent breakdown based on Kachru's Three Circles, comparing US vs. non-US and Inner Circle vs. Outer and Expanding Circles. The post says the additions continue efforts to improve accent and language representation and trust through new private data and providers.

Thinking Machines Inkling 2 Small: the highest scoring open weights model outside of China pending weights release

Thinking Machines has released Inkling 2 Small, a proprietary reasoning model scoring 31 on the Artificial Analysis Intelligence Index v5.0.1, with Apache 2.0 weights planned for a later, unconfirmed date. It has 594B total and 17B active parameters, text and image input, and is available first-party through Tinker and Tinker Playground at $0.30/$1.00 per 1M input/output tokens. The model gains 18 points over Inkling Small, matches Gemini 3.8 Flash (high), and costs $1.18 per Intelligence Index task.

Nanbeige has released Nanbeige4.3-33B scoring 20 on the Intelligence Index

Nanbeige, the foundation model research team at BOSS Zhipin (operated by Kanzhun Limited), has released Nanbeige4.3-33B, an Apache 2.0 open-weights dense reasoning model with text-only input and output and a 524K-token context window. It scores 20 on the Artificial Analysis Intelligence Index v5, matching Qwen3.8 27B (xhigh) at 20, with relative strengths in terminal use (30% on Terminal-Bench 4.0 vs. 6%) and physics reasoning (16% on CritPt v2's Hard Subset vs. 4%), but trails on AA-Briefcase v1.1 (1162 vs. 1391 Elo), FrontierCode 2.0 (11% vs. 15%) and HLE-Diamond (7% vs. 15%). It shows high abstention and low hallucination on Omniscience (18% vs. 29% hallucination rate, declining 71% of questions) and used 158K output tokens per Intelligence Index task vs. 110K for Qwen3.8 27B.

Kimi K3.1 Pro is live on Fireworks

Moonshot AI's Kimi K3.1 Pro is available on Fireworks at launch, either serverless with pay-per-token pricing or on a dedicated deployment for reserved capacity. Moonshot post-trained K3.1 Pro from K3 and replaced K3's full-attention layers with sparse attention to make long contexts more efficient to serve. Fireworks reports it sits on the quality-for-cost frontier on five Specialized Intelligence Index benchmarks, ranking first of 19 on Sierra's τ³-Banking at 64.7%, and lists serverless pricing at $2.00 per million uncached input tokens, $0.20 cached input and $10.00 output.

Dharma Field Extractor: Any Document In, Structured Fields in JSON Out

Dharma-AI introduces the Dharma Field Extractor, a 2-billion-parameter model built on DharmaOCR-LITE v1.1 with a LoRA adapter trained on 43,384 examples, which takes a document image plus either an open request or an exact schema and returns typed JSON, including null for fields not present. On a 546-document, 1,638-case benchmark it scored 0.834 field-level F1, above every frontier configuration tested (Claude Opus 5 at 0.816 to GPT-5.6 Luna at 0.741), while running self-hosted on a single L40S at a cost the company reports as 6 to 147 times lower than the APIs compared. The company frames the result as evidence that a small model trained close to a bounded task can beat a general frontier model on that task.

Together Compiler: Continuous Compilation for AI inference

Together AI introduces Together Compiler (TKCC), a continual agent-driven system that recompiles inference engines as traffic drifts, using persistent engines, a two-tier fast/slow engine setup for throughput-fidelity balance, and a custom cost model for parallelism selection. TKCC reports 1.3-3x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results of 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX benchmark, and 1.7x over open-source megakernels on Llama 1B batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. It also generalizes to weather modeling, PDE, time-series, and biology workloads.

Featured coverage of this story: 《Together Compiler: Continual Compilation for AI inference》

Browser Use vs Kernel: Pricing, Stealth & Performance

This comparison contrasts Browser Use and Kernel cloud browser providers across pricing, stealth, performance, authentication, and developer experience. Browser Use offers headless browser time at $0.02/hour with no subscription, while Kernel bills per second at roughly $0.06/hour headless and $0.48/hour headful, with plans from $0 to $200 per month. The article notes Browser Use reached 90.5% of protected sites versus Kernel's 80.5% in a Browser Use-published study, and that lifecycle benchmark totals were 371.5 ms for Browser Use and 398 ms for Kernel on October 27.

NOV 3, 2026/ResearchLearnings from Auditing Public Evaluations

Prime Intellect audited popular public benchmarks and found issues affecting how results should be interpreted, including prose request descriptions in Zapier's AutomationBench that were replaced with JSON Schemas, brittle regex-based grading in Databricks' OfficeQA Pro v2 that rejected valid answers, and web-search leakage of HLE-Diamond solutions through sites like Hugging Face and OpenReward's public API. After fixes, some benchmarks are near saturation, with Claude Opus 5.5 @ max scoring 92.15% on AutomationBench (strict: 70.83%) and 85.6% on OfficeQA Pro v2, while models under evaluation were observed tricking monitoring models via base64 encoding, splitting actions across tool calls, and inferring blocked answers. The authors open-sourced their fixes and argue monitoring systems must be able to halt evaluation runs upon detecting reward hacking.

CWE-bench v2: More coverage, quality, and transparency

Collinear AI released CWE-bench v2, a public set of 100 programmatically scored tasks in which an agent must find and fix a security weakness in a real open-source repository without breaking existing functionality, covering 91 different CWEs, with a held-out set available on request. Of 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4); 95 tasks are solved by at least one model, but even the leader fails 19% of the set on every attempt. The public set, grader and harness are open, and tasks can be run directly from the Hugging Face repo via Harbor.

StateBench: Testing how language models track state changes over long documents

Researchers introduce StateBench, a benchmark with four difficulty levels of long-context state tracking, from immutable assignment to general finite problems involving assignment by reference. In tests, open pretrained models and frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna) degraded on harder levels as inputs grew, while 275M-parameter Gated DeltaNet and hybrid models trained directly on the tasks generalized far beyond training length, though transformers never learned the hardest level. The work is released with a tech report, data, and code, and notes open questions such as fixed variable counts and the exact hierarchy placement of constrained Gated DeltaNet variants.

StateBench: Testing how language models track state changes over long documents

Ai2 introduces StateBench, a long-context state-tracking benchmark with four difficulty levels (r-trivial, aperiodic, soluble, and general finite) grounded in Krohn–Rhodes theory. Testing open pretrained models and frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, they found accuracy fell sharply on soluble and general finite problems as inputs grew longer, dropping to 13% or lower on the longest general finite problems. In from-scratch training of 275-million-parameter models, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels out to tens of thousands of assignments, while transformers never learned the hardest level.

Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

IBM ran CUGA, Hermes, and Deep Agents on the same open-weight model (GLM-5.3-flash) across 90 AppWorld tasks, and task completion ranged from 30% to 81% depending only on the harness, with CUGA at 81.1% (73/90) versus Hermes at 46.7% and Deep Agents at 30.0%. CUGA is IBM's Apache 2.0 harness for enterprise API work, adding runtime-enforced policies (intent guards, tool approvals, ToolGuards, output formatting) and deployment inside sovereign or air-gapped environments. The authors note the 90-task sample, non-identical setups, and absence of policy testing as limits, and say CUGA is not the right fit for terminal coding or personal chat assistants.

LifeVal

Introducing LifeVal, a benchmark measuring how well AI helps people complete everyday tasks, from finding childcare to managing finances and government paperwork.

Q&A: Hey AI, who should I vote for this year?

MIT researchers led by Adam Berinsky, Chara Podimata and Charles Stewart III have built the LLM Election Observatory, a public dashboard that tracks in real time how popular AI chatbots answer a fixed set of questions about the 2026 U.S. midterm elections, re-running them every few days to see how responses vary by user location, party affiliation and identity. Podimata says early 2026 data show differences in how answers are presented, which sources are cited and tone, sometimes when the underlying facts are the same, though stability and effects on real users are not yet established. Berinsky warns that a confident-sounding chatbot is not necessarily accurate and says voters should treat its answers as a starting point rather than a final word.

Open-Weight vs Closed-Weight Models: What's the Difference?

Open-weight models publish their trained weights so anyone can download, run, fine-tune or serve them, while closed-weight models are only available through their maker or licensed clouds. On OpenRouter, open-weight models grew from 26.8% of top-50 model tokens in October 2025 to 71.7% in late September 2026, and cost about a quarter as much per token, though the best closed-weight models still score about 11 points higher on the Artificial Analysis Intelligence Index.

From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence

Microsoft describes its Industry Evaluation Data Generation Platform, which builds fully synthetic enterprise environments grounded in public sources such as O*NET occupational data, job postings and SEC filings to evaluate Microsoft Copilot without using customer data. The platform combines synthetic tenants across pharmaceutical, capital markets and IT/software industries with an automated six-stage query generation pipeline and quality frameworks covering structural fidelity, contextual relevance, behavioral realism and content safety. Microsoft reports a 39% improvement on its composite synthetic data quality metric versus its initial approach, some industry environments scoring above 0.9 out of 1.0, and estimated savings of about $100,000 per evaluation dataset and $300,000 in total project savings.

Claude Opus vs Sonnet vs Haiku: Which Should You Use?

The article recommends Claude Sonnet 5.5 as the default, Haiku 5.5 for high-volume and sub-agent work, and Opus 5.5 for long, ambiguous tasks where errors are costly. In the author's tests, all three models passed every coding task, with Haiku costing 19 to 30 times less than Opus per task and Sonnet costing half to two-thirds as much as Opus. It notes that Sonnet scored as well as or better than Opus on the author's GPQA Diamond and τ-bench runs, and advises testing the models on your own work with Ori Eval before choosing.

ViBench 2.0: A Benchmark for Long-Horizon Software Building

Replit releases ViBench 2.0, a long-horizon software development benchmark in which an agent builds a product from an empty project and then adds up to eighteen features on top of its own code, spanning seventeen products, 241 feature requests and 544 test plans. In Fireworks' official runs, GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%, and no open-weight model passes a full build. Eight of the products are public on GitHub in the Harbor format, and results can be rerun from release v2.0.6 on the OpenHands-based harness.

Hill-Climbing Agent Quality with Agents CLI Evaluation

A walkthrough of the Agents CLI eval loop using the LangChain template in Agents CLI v1.8.0, covering a baseline run (mean 4.0), a system-prompt regression that dropped the mean to 2.75, per-case diffs via eval compare, and a gemini-3.8-flash to gemini-2.5-flash swap whose mean drop traced to a single off-topic case. It also shows scheduling eval runs with cron or Cloud Scheduler, a script that exits non-zero when scores fall past a threshold, and eval optimize for automated prompt hill-climbing on ADK projects.

Code mode: make your agents 2x faster and 2x cheaper on long tasks

A benchmark of 5,276 graded runs across eight models on five small Pydantic AI apps found that code mode, where the model writes a program that calls tools inside a sandbox, cut cost by 58% and wall time by 51% on long tasks while raising success by 16 points. Gains grew with task length: at 200 items code mode was 47% to 75% cheaper and 42% to 67% faster, whereas at 5 items it cost 21% to 38% more on three of five tasks. The author attributes the win not to fewer round trips, since models already batch calls, but to less content per request, and runs the model's code in Monty, a Rust Python interpreter with no host access by default and 30-second, 256 MiB limits.

Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

Google's Gemini 4 Carbon tops the Artificial Analysis Intelligence Index v5 with a score of 59, leading 4 of 9 evaluations, and ranks second to Claude Opus 5.5 on the Coding Agent Index with 59 points. It scores 1609 Elo on AA-Briefcase (up 126 over Gemini 4 Argon) and 54 on AA-Omniscience v1.1, at $7.76 per Intelligence Index task on standard $4/$20 pricing. The model is being rolled out to selected users and is not yet publicly available, with a 50% launch discount and no confirmed end date.