#评测/基准

Wojak-Bench 3D: Which AI is best at 3D printing?

Wojak-Bench 3D is a community preference poll that hides model names until you vote, with one vote per browser per pair. The leaderboard currently shows Claude Fable 5.2 at 64% share (7/0/4, 11 votes) and GPT-6 Astra at 36% (4/0/7, 11 votes), where share = (wins + 0.5 × ties) / votes on same-prompt pairs. The site notes it is community preference rather than a controlled evaluation, that small samples are unreliable, and that votes compare photos of physical prints where available, otherwise renders of the print files.

Gemini 4 Carbon takes the MathArena crown but costs 15 times more than GPT-6.1 Sol

Google's Gemini 4 Carbon became the first non-OpenAI model in a while to top the MathArena math benchmark, displacing GPT-6.1 Sol, with Claude Fable 5.2 taking second place. Its most common failure mode is exceeding the time limit, and in one case it ran a script for more than eleven hours without checking its progress. With the reasoning settings used in testing, Carbon and Fable 5.2 cost roughly 15 and 40 times as much as GPT-6.1 Sol, respectively.

New 'Access to Justice' Benchmark: Leading AI Models Miss Critical Legal Details

Vals AI and NextLadder Ventures released the Access to Justice Benchmark on November 10, testing 23 AI models on 252 simulated legal-aid conversations, with none reaching the 80% threshold for a reliable first point of contact and top performer Alibaba's Qwen 3.8 Max scoring 76.4%. Models met 78–87% of criteria for general legal explanations but only 30–50% for identifying specific local resources, with 21 of 23 falling back on national directories in at least 70% of final answers. Free-tier models averaged 69.7% versus 70.4% for premium ones, and housing questions scored highest at 75.4% while public benefits ranked lowest at 57.3%.

Show HN: Portal2Bench – Can AI agents solve Portal 2?

Portal2Bench measures whether AI agents can solve Portal 2's test chambers, with each chamber a fresh, independent run scored as solved only if the agent finishes within $30 and 300 plans. In the reported results, GPT-6.1 Sol (Codex) and GPT-6 Astra (Codex) each solve 20 of 26 chambers, GPT-6.1 Terra 19, Opus 5.5 11, Sonnet 5.5 6, GPT-6 Luna 0, and Fable 5.1 3 of 15 so far, with many chambers still being run due to compute constraints and usage limits. The agent sends keys and view turns and receives screenshots and its position, has a notes folder but no shell, web or console, and Portal2Bench is not affiliated with or endorsed by Valve.

AI models read the law to the right of their politics

Across 200 randomly selected split Supreme Court decisions, 13 AI models were asked whose legal reading was correct and, separately, whether the outcome was good for the country; when the two answers disagreed, it was almost always "the conservative reading of the law is right, but the liberal result is better," by 420 to 41. On the law alone, every model landed left of the Court's center, with the Claude models siding with the majority 85–87% of the time, like Roberts and Barrett. The author notes caveats including one answer per model per case and no direct comparison to how justices judge outcomes.

Frontier AI takes only months to get 10 times cheaper

Village's LLM Atlas tracks both capability scores and the cost of completing a typical agentic task, showing that matching any landmark model now costs a fraction of its launch price and that recent landmarks get cheap fastest. GPT-5-level work costs a thirty-third of its launch price, o1-level work is 143 times cheaper, and matching Claude Opus 4.6 is 51 times cheaper today. Levels set since o1 are falling 11 to 25 times a year, versus 3 to 5 times a year for GPT-4 and GPT-4o levels.

Best AI Video Generation Models in 2026: Elo, Price per Second, Open Weights

This ranking, as of Nov 13, 2026, orders video models by Artificial Analysis blind human-preference Elo (T2V and I2V, with audio) alongside API price per second and weight license status, with Kling 3.0 1080p Pro fixed as the 1000-Elo anchor. Alibaba's Wan 3.0 leads text-to-video at 1175 Elo ($0.20/sec, closed), ByteDance's Seedance 2.5 leads image-to-video at 1161 Elo ($0.569/sec), and MiniMax H3 is the top open-weights model at #3 (1161 Elo, $0.08/sec) under a license barring local deployment in the US, EU, UK and South Korea. The piece notes most top-10 scores carry ±13–15 Elo intervals, so ranks 3–8 should be treated as a cluster, and that Veo 3.1 (964) now scores below the Kling 3.0 anchor.

XYBench: Helping with the problem behind the question

XYBench is a new benchmark of 8,115 real-world questions, drawn from Stack Overflow, Stack Exchange, curated examples, and WikiHow, that tests whether AI assistants recognize when a user's request rests on a misconception (the "XY problem") and redirect them toward their actual goal. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the authors found every model was more likely to address the stated request than the underlying goal, and all trailed human reference answers on misconception identification. The paper won the Best Resource Paper Award at EMNLP 2026.

XYBench: Helping with the problem behind the question

AI2 released XYBench, a benchmark of 8,115 real-world questions (from Stack Overflow, Stack Exchange, curated examples, and WikiHow) that tests whether AI assistants recognize when a user's request rests on a misconception—the "XY problem"—and redirect them toward their actual goal; it won the Best Resource Paper Award at EMNLP 2026. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the researchers found all five models were more likely to address the stated request than the underlying goal and trailed human reference answers on misconception identification. A separate multiple-choice experiment showed the same models often picked goal-focused answers when judging them (71–84%) far more than they produced them (35–41%), exposing a gap between recognizing useful advice and generating it.

1 in 4 3rd Party AI Endpoints Underperform Their Advertised Models

Artificial Analysis released version 1.1 of its Endpoint Accuracy Index, which compares more than 100 serverless endpoints hosting open-weight models against reference deployments of the official weights, finding roughly one in four score below their reference. The index reruns each endpoint on BFCL v4-500, HLE-250 and AA-LCR-25, and version 1.1 raises long-context repeats from 10 to 25, cutting the average 95% confidence interval from plus or minus 4.7 to plus or minus 2.6 points, while coverage expands to six models including DeepSeek V4.1 Flash 1017, GLM-5.3, GLM-5.3 Flash, Muse Spark Open, Kimi K3 and gpt-oss-120b. Reported gaps trace to output caps that truncate answers, shorter reasoning, and a schema bug encoding string tool arguments as numbers, with GLM-5.3 endpoints scoring from 67.2% to 99.3% and all 22 DeepSeek endpoints statistically consistent with the reference.

AI Mixing and Mastering Services Compared

This guide compares seven AI mixing and mastering services—Moozix, Bandmixr, LANDR, eMastered, RoEx Automix, Cryo Mix and BandLab Mastering—distinguishing multitrack stem mixing from stereo mastering and noting which platforms do which. It reviews each service's features, track limits, export formats, privacy policies and pricing as of November 12, 2026, and offers selection guidance based on whether you have separate stems, a finished stereo mix, or long rehearsal recordings.

Stanford CS 329Z: Engineering AI Agents

Stanford's CS 329Z course covers engineering AI agents across foundations, LLM internals, RAG, tool use, frameworks, design patterns, memory, multi-agent systems, optimization, data, evaluation, safety, and coding and personal agents. It includes guest lectures on Terminal-Bench, SWE-bench, and a fireside chat with Andrew Ng, plus a final project demo day.

Reranking in RAG: How to Fix Bad Retrieval with a Reranker

This guide explains that vector search often ranks near-miss chunks first because embeddings compress whole chunks into single vectors, and shows how adding a cross-encoder reranker over the top 30 candidates raised precision@1 from 81% to 97% on a 274-question synthetic benchmark at roughly $0.10 per 1,000 queries. It compares 11 rerankers (voyageai/rerank-3-lite and rerank-3 were the most accurate dedicated models and cheapest, while LLM rerankers like openai/gpt-6-luna hit 99.6% but took about 4 seconds per call), covers candidate-count tuning, input_type settings, and silent failures such as cohere/rerank-v3.5's 4,096-token limit. It includes a Python example using the embeddings and rerank APIs and notes that real-world gains depend on your own documents and evaluation set.

AI-written peer reviews outscored top human experts

A PsyPost article reports on a Harvard working paper finding that AI-written peer reviews outscored top human experts in blind comparisons. The item links to the PsyPost article and a Hacker News discussion thread.

Can AI chair a multilateral negotiation?

Westphalia Labs ran eleven AI models as Secretary-General of a fictional seven-member Regional Cooperation Union, each chairing 40 simulated negotiation runs across two versions of a crisis scenario, one where consensus was narrowly possible and one where two governments' red lines made it impossible. Only GPT-6.1 Sol passed every run, while Claude Opus 5.5, Grok 4.7 and the low-cost DeepSeek V4.1 Flash each called the impossible version correctly every time and reached agreement in 18 of 20 possible runs; overreach caused 39 failures versus 13 from conceding too little. The author reports that model price and reputation did not predict performance, that a weak AI judge could turn a failure into a success, and that the scenario, rules, rubric and harness are public on GitHub.

Google's Gemini 4 Hydrogen outscores the bigger Argon just four weeks after its launch

Google released Gemini 4 Hydrogen, a fast general-purpose model for developers and everyday users that scores 51 on the Artificial Analysis Intelligence Index at its default "high" thinking level (54 at the new "xhigh" tier), beating the larger Gemini 4 Argon's 50. It has a one-million-token input and output window and costs $1.50 per million input tokens and $7.50 per million output tokens, with Google reporting a jump from 18.2% to 57% on Terminal-Bench 4.1 versus Gemini 3.8 Flash. Artificial Analysis found Hydrogen hallucinates only 8% of the time but lags on agentic terminal work, scoring 54% on Terminal-Bench 4.0 and 50% on Terminal-Bench-Science 0.1.

Featured coverage of this story: 《Introducing Gemini 4 Hydrogen》

AI Models Know What Women, Peace and Security Is. They Just Don't Bring It Up

Our Secure Future describes building the WPS AI Leaderboard, a 30-scenario benchmark testing whether LLMs apply Women, Peace and Security principles when users don't explicitly ask, finding that frontier models' average score fell from 0.82 on cued prompts to 0.16 on sparse ones. Models also varied unpredictably in resisting adversarial requests to sideline WPS requirements, which the authors attribute to post-training rather than general capability. A multi-agent WPS workflow scored 0.84 overall with flat gender-sensitivity across tiers, though the authors stress the benchmark is not a deployment certification and cannot replace human WPS advisors.

AI chatbots are recommending fake retail websites, study finds

A study by IP protection firm Podqi, shared with The Verge, tested ChatGPT, Gemini, and Perplexity with shopping-related prompts about 151 brands and found all three cited risky sites, including fake retailer impersonators, re-registered old domains, and dead sites. Across 6,795 answers, Perplexity showed risky sites in 15 percent of answers, ChatGPT in 8 percent, and Gemini in 4 percent, with smaller brands appearing especially at risk.

Open benchmark of AI decision APIs vs. XGBoost for fraud triage

Smart Mobile House released fraud-decision-lab, a vendor-neutral benchmark that sends the same transactions to TypeSafe Jev, OpenAI Decisions, Cloudflare Clef/Clef-flash and a structured-output LLM, then measures how much labelled fraud each ranks into the top 1% and 5% of an analyst review queue against an XGBoost baseline, with cost and latency. It runs offline by default on bundled synthetic data, saves predictions and experiment metadata for local verification, and requires API keys, --allow-external-requests and a --max-cost-usd estimate for live provider runs. The project notes that XGBoost learns from labelled history while the AI models score transactions zero-shot, that the bundled data is synthetic, and that the current source distribution contains no benchmark result files.

Did it use REA? Checking viral AI reverse-engineering demos

A verification page checks 39 cases described as "built with REA" against each project's README, agent files, author posts or official showcase, finding 26 confirmed, 3 partial, 4 unverified and 6 not REA (mostly Claude Code or Codex with Ghidra or IDA, or manual work). The README column was re-checked on 2026-11-10 with no changes, and X posts could not be opened from the authors' environment. It notes that two small controlled comparisons, rea-lab (482 preregistered runs) and leanrea (36 runs), found no significant gain in solve rate from REA at higher time and cost.

ElevenLabs Scribe v3 Live uses conversational context to stop voice agents from cutting off callers mid-sentence

ElevenLabs released Scribe v3 for pre-recorded audio and Scribe v3 Live for low-latency real-time use, both available through the ElevenAPI and ElevenAgents platform. The company says Scribe v3 cuts word errors by about 25 percent versus its predecessor across 102 languages, while Artificial Analysis benchmarks confirm top-tier accuracy but note slower speed and mid-pack pricing. Scribe v3 Live analyzes conversational context alongside the audio signal to detect when a speaker has actually finished, and both models support preloaded specialized terms, natural-language formatting rules, and mid-sentence language switching.

Show HN: A free course from engineer to AI engineer, one Git tag per episode

A free YouTube-based course, "From engineer to AI engineer," teaches experienced software engineers with no generative-AI background to build a retrieval-augmented generation (RAG) pipeline in eight episodes, with a second seven-episode series, "Prove it works," building an evaluation harness on the same repository. Each episode is one commit and one Git tag (s1e1–s1e8 and s2e1–s2e7), so checking out a tag reproduces the repository exactly as that episode leaves it. The series uses TypeScript, SQLite, curl and an HTTP server, answering with claude-sonnet-5-5 over a fictional 26,000-word corpus (Lantern Supply Co.) that includes deliberate traps such as near-identical product names and a policy changed in later release notes.

Remote Labor Index: Measuring AI Automation of Remote Work

The authors introduce the Remote Labor Index (RLI), a multi-sector benchmark of real-world, economically valuable projects designed to evaluate end-to-end AI agent performance in practical settings. AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%. The authors state these results help ground discussions of AI automation in empirical evidence and provide a common basis for tracking AI impacts.

Show HN: MetaArena – Compare & Rank 5,353 Meta Global AI Hackathon Projects

The author built MetaArena, a site that presents two projects from Meta's Global Developer Hackathon head-to-head and lets users pick the stronger one, ranking them on a leaderboard using the Bradley-Terry algorithm used by LLMArena and DesignArena. The hackathon had a $1M prize pool, 34K applicants, 20K accepted developers, and over 5K submitted projects, which made human review infeasible and AI rubric scoring disappointing.

[AINews] D-Day (Decision Models Day)

Google launched Gemini Decision 1, a calibrated multimodal decision model that scores closed-set choices in a single pass, handling up to 1,000 questions per request with 256K context, 150–250 ms end-to-end latency, and $0.10 per 1M input tokens with no output charge. Google's own tests show it beating TypeSafe's Jev 1.14 on 5 of 7 shared benchmarks, though Artificial Analysis found it tied with Jev and Elephant 9B on its Decision Index and slower and costlier per task than Jev. The roundup also covers Fable 5.2 topping Agent Arena, AlphaGenome 2, Runway's Praxis-1, Gemini Robotics 2.5, and reports that Nvidia's board is preparing to remove Jensen Huang.

How to Benchmark LLMs

A guide to benchmarking LLMs by writing tasks with script-checkable answers, grading with code, and comparing models on identical tasks via OpenRouter using graded accuracy, billed cost, and request time. On 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials while DeepSeek had the lowest response cost at 157/180 passes, and agreement with the public Artificial Analysis ranking was weak (Spearman correlation 0.109). The article includes a downloadable Python companion package with offline checks, configuration examples, and guidance on repeats, missing answers, and run comparison.

Gemini Decisions API

Google has joined Jev and OpenAI in the decision model category with "Gemini Decision 1", an API that supports the same three question types as Jev (yes/no, choice, and rubric-scoring) and reuses Jev's term "Noul" for yes/no questions. It is priced like GPT-6 Luna at 10 cents per million input tokens with free output, while Jev remains cheapest at 4.2 cents per million; unlike Jev (text and JSON only) and OpenAI (text and images), Gemini Decision 1 covers text, images, audio, video, and PDFs. Testing it on a video returned probability values rather than the 1.0 or 0.0 yes/no answers Jev and OpenAI give.

Pundit Claims – Open-Source Register of AI Claims by Gary Marcus and Ed Zitron

Pundit Claims is an open-source register that tracks statements about AI made by Gary Marcus and Ed Zitron and records what happened, finding Marcus mostly right (88% of 780 claims with a clear outcome supported, 94% of 82 predictions) and Zitron right about as often as wrong (49% of 230 claims, 50% of 30 predictions). Each square represents one claim, with lighter squares marking probable verdicts where both AIs agree and lightest squares where only one AI makes the call, as of 2026-11-11. The site also offers breakdowns by topic, year, company and prediction status, plus a methodology page.

JioStar’s chief architect on AI discovery at 500M-user scale

JioStar’s Vijay Seshadri explains AI discovery at 500M-user scale. He shares lessons on evals, guardrails, and taking AI into production.

Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

This NeurIPS 2026 workshop paper examines how well LLM judges evaluate personalized writing tasks, testing how the writing task, generator-judge pairing, evaluation objective, and general reasoning ability affect judge performance across four LLMs and three long-form writing tasks with human pairwise labels. The authors find judge quality correlates weakly with MMLU-measured ability (Pearson's ρ ≤ 0.2), varies by task and criteria, and that paired comparisons yield more consistent results than independent scoring, while the strongest LLMs do not most reliably reproduce human evaluations.

Genie with Genie Ontology Delivers 1.6× the Accuracy with Less than Half the Tokens

Databricks developed a new enterprise data-agent benchmark with Turing, run in a live Databricks workspace, in which Genie achieved a 91% pass rate versus 45% and 58% for two frontier agents while using fewer than half the tokens and completing tasks 3–4x faster. The benchmark, built around a fictional UK energy supplier called Calderwick Energy, includes notebooks, dashboards, saved SQL queries and documents alongside hundreds of tables, testing whether agents can find and use business context that table schemas alone cannot provide. Genie Ontology, now generally available, automatically extracts and organizes this knowledge and surfaces relevant context for Genie.

Modernizing NetHack's C engine with .NET MAUI, Avalonia and a 31-tool AI advisor

Sound Mind Games (Hyvän mielen pelit ry), a Finnish nonprofit, describes how GnollHack wraps NetHack's roughly 500,000-line C99 engine as a native library behind a shared C# client, migrated from Xamarin.Forms and SkiaSharp to .NET MAUI by April 2025 and extended to Linux, the Steam Deck and macOS in 2026 via Avalonia.Controls.Maui. The game's opt-in Gnoll Overseer advisor embeds a web app that reads live game snapshots from the C engine and uses 31 tools (20 server-side, 11 on-device) to look up facts, and GnollBench grades models with the production prompt and tools using a blind two-provider judge panel, a geometric-mean score, a 25-point critical-error cap and a 50-point honesty floor. The post also covers the October 2026 nine-model benchmark results and the team's agentic workflow, in which coding agents write most C# and TypeScript code while the C engine, art, music and voiceovers remain human-made.

We need more science in the judgment of AI

The author argues that LLM capabilities are too often judged from subjectively impressive one-off results rather than pre-registered hypotheses and evaluation criteria, a practice akin to HARKing and p-hacking that is fine for hypothesis generation but not for scientific testing. He notes that impressive feats like one-shot coding often reflect abundant training data for a domain, while areas with sparse data such as Zig, Nextflow and much of biology perform poorly, and that a researcher involved in rewriting bioinformatics software in Rust now uses LLMs to build a deterministic translation tool rather than translating code directly, because too many corner cases are missed.

The Four Tiers of AI Intelligence

The article groups LLM intelligence into four tiers by difficulty of attainment: Tier 1 (Absolute Frontier) is limited to Anthropic, OpenAI and Google, with Anthropic described as the most consistent lately; Tier 2 sits behind the frontier but above the best open source, where Moonshot's Kimi series is said to define the tier; Tier 3 is close to the best existing open source and the most crowded; Tier 4 is "the rest," which may still include strong models valued for size or cost. The author notes tiers reflect a model's position at release and that lab-specific analysis is as of November 10th, 2026.

LLM Quantization: What fp8 and fp4 Do to Quality and Price

OpenRouter tested Qwen3.8 27B across six endpoints at fp16, fp8 and fp4 on 240 exact-answer tasks and found no detectable accuracy difference, with all endpoints scoring between 98.3% and 99.2%. Across 1,455 endpoints, fp8 was the most common labeled precision (23.6%) while 59.5% reported none, and lower precision was usually but not always cheaper. The post explains fp8 and fp4 formats and shows how to pin precision with provider.quantizations, noting that a narrow filter can leave a single endpoint exposed to rate limits.

The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

This AWS post explains how to evaluate production agents on Amazon Bedrock AgentCore by starting from business outcomes, breaking them into observable behaviors, and mapping each to a trace- or session-level evaluator. It covers the built-in AWS evaluators, managed third-party metrics from DeepEval and AutoEval, custom and multi-dimension LLM-as-a-judge evaluators, and suggested evaluator stacks for customer support, research, and domain-content agents. It also describes three evaluation modes—on-demand, online, and batch—that share the same evaluator IDs, plus the ListEvaluators API for retrieving the full catalog.

In open-weight AI, America is the one playing catch-up

According to Village's LLM Atlas, which scores 857 models on a single capability scale, China's best open-weight models have led America's every day since September 2024, when Meta's Llama 3.1 405B last matched Alibaba's Qwen2.5. The 12 record-setting US open models since 2025 arrived a median of 3.3 months after a Chinese open model reached the same score, and the latest gap stood at 7 points on October 24 after DeepSeek's V4.1 Flash and Moonshot's Kimi K3.1 Pro widened a lead Meta's Muse Spark Open had cut to 2.9 points. Both countries' open models still trail the best closed models, by about five months for China and nine for America.

Multimodal Knowledge Agent (MKA) benchmark: testing knowledge work capabilities with complex visual files

Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, comprising 200 private real-world visual tasks across 10 industries in which agents work from dense visual files like architectural drawings, engineering schematics, and clinical scans to produce deliverables such as slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Tasks are graded against expert rubrics averaging ~70 weighted criteria split into Regression and Climb checks, with the headline Climb WSR score, run in the open-source Stirrup agent harness with up to 250 turns and graded by GPT-6 Luna (medium) as an agentic judge. Claude Fable 5.2 (max) leads with 57% Climb WSR, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%, while Claude Fable 5.2 averages $31.97 per task versus $6.29 for Gemini 4 Carbon and $1.11 for GPT-6.1 Sol.

Doc2Harness: Benchmarking how well AI builds agent harnesses

The authors propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering: a build model constructs a candidate harness from an interface-only contract skeleton and a development document consolidating behavior across twelve frontier harnesses, and the frozen harness is then run per test case by a scripted mock model and scored via four observable channels. Across nineteen frontier models, each with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05% while the weakest model reaches 27.44%, and failures cluster around how cross-cutting functionality is split across modules. The authors describe it as the first deterministic benchmark for measuring long-horizon harness engineering.

NaiveAI Promises 2,100 Tokens per Second. How Well Does It Code?

Latent Space benchmarks NaiveAI's open-weights Naive-N0.5-Flash, a MiMo-V2.5 derivative adapted to DeepSeek Sparse Attention with about 3.25T extra training tokens, against DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash on speed, verbosity and three app-building prompts. It reports peak throughput near 2,100 tokens/s and a median response time of 3.1 seconds versus 5.3 and 10.1 seconds, but far higher thinking-token use on hard questions and mean times similar to DeepSeek. In the apps, Naive-N0.5-Flash produced the best solar system, tied DeepSeek on the tree model and trailed it on ray-tracer efficiency, while its no-thinking variant failed to build a usable tree app.