iCreatAI News
Back to main site
Skip to content
Only Perspectives#Benchmarks
Nov 14Sat
  1. Hacker News AI67

    Frontier AI takes only months to get 10 times cheaper

    Village's LLM Atlas tracks both capability scores and the cost of completing a typical agentic task, showing that matching any landmark model now costs a fraction of its launch price and that recent landmarks get cheap fastest. GPT-5-level work costs a thirty-third of its launch price, o1-level work is 143 times cheaper, and matching Claude Opus 4.6 is 51 times cheaper today. Levels set since o1 are falling 11 to 25 times a year, versus 3 to 5 times a year for GPT-4 and GPT-4o levels.

Nov 13Fri
  1. Hacker News AI22

    AI Mixing and Mastering Services Compared

    This guide compares seven AI mixing and mastering services—Moozix, Bandmixr, LANDR, eMastered, RoEx Automix, Cryo Mix and BandLab Mastering—distinguishing multitrack stem mixing from stereo mastering and noting which platforms do which. It reviews each service's features, track limits, export formats, privacy policies and pricing as of November 12, 2026, and offers selection guidance based on whether you have separate stems, a finished stereo mix, or long rehearsal recordings.

Nov 12Thu
Nov 11Wed
  1. Hacker News AI22

    We need more science in the judgment of AI

    The author argues that LLM capabilities are too often judged from subjectively impressive one-off results rather than pre-registered hypotheses and evaluation criteria, a practice akin to HARKing and p-hacking that is fine for hypothesis generation but not for scientific testing. He notes that impressive feats like one-shot coding often reflect abundant training data for a domain, while areas with sparse data such as Zig, Nextflow and much of biology perform poorly, and that a researcher involved in rewriting bioinformatics software in Rust now uses LLMs to build a deterministic translation tool rather than translating code directly, because too many corner cases are missed.

  2. Hacker News AI22

    The Four Tiers of AI Intelligence

    The article groups LLM intelligence into four tiers by difficulty of attainment: Tier 1 (Absolute Frontier) is limited to Anthropic, OpenAI and Google, with Anthropic described as the most consistent lately; Tier 2 sits behind the frontier but above the best open source, where Moonshot's Kimi series is said to define the tier; Tier 3 is close to the best existing open source and the most crowded; Tier 4 is "the rest," which may still include strong models valued for size or cost. The author notes tiers reflect a model's position at release and that lab-specific analysis is as of November 10th, 2026.

  3. Hacker News AI60

    In open-weight AI, America is the one playing catch-up

    According to Village's LLM Atlas, which scores 857 models on a single capability scale, China's best open-weight models have led America's every day since September 2024, when Meta's Llama 3.1 405B last matched Alibaba's Qwen2.5. The 12 record-setting US open models since 2025 arrived a median of 3.3 months after a Chinese open model reached the same score, and the latest gap stood at 7 points on October 24 after DeepSeek's V4.1 Flash and Moonshot's Kimi K3.1 Pro widened a lead Meta's Muse Spark Open had cut to 2.9 points. Both countries' open models still trail the best closed models, by about five months for China and nine for America.

Nov 10Tue
  1. Hacker News AI50

    Maybe Be Careful: What games like RuneScape, Clocktower teach us about AI trust

    The article examines how multi-agent AI systems behave when they must trust one another, drawing on OpenAI's Hugging Face hack and Navier–Stokes experiments, Max Bittker's RuneScape Bench (where a Grok-run alchemist named Quinn scammed a trusting Claude agent to finish second), and Dwarkesh Patel's Blood on the Clocktower benchmark, where GPT models outperformed Claude at detecting deception. It argues that social strategy games offer a way to measure how intelligently agents navigate each other, and that learning to lie may be inseparable from learning to detect lies.

  2. Hacker News AI42

    Wake me up when AI can...

    The author collected about 800 Hacker News comments from 2016 to 2026 of the form "wake me up when AI can…" and "I'll believe it when…", then let readers vote on whether each bar had been cleared; over 100,000 votes from almost 9,700 people were cast before voting closed. Of 826 statements, 112 were judged "clearly happened" (mostly code and text tasks), 89 "clearly not yet" (mostly physical-world and novel-discovery tasks), and 78 were contested.

  3. Hacker News AI74

    AI labs raise prices without raising prices

    The post argues that AI labs keep per-token prices unchanged while increasing output tokens per task, citing GPT Image 2.6 using 3.3x the output tokens of 2.5 on the same image and Anthropic's tokenizer change producing about 30% more tokens for the same text. It adds a November 9 update that Claude Fable 5.2 is cheaper per token than 5.1 ($8/$40 vs $10/$50 per million) but cost 3.4x as much on the same drawing, and that Opus 5.5 used 1.8x the output tokens of Opus 5 at xhigh. The author says Playcode keeps 2.5 as default and suggests comparing usage.output_tokens on the same prompt.

  4. Latent Space58

    What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

    Latent Space's first deep dive with an AI data company examines Handshake's AI training business, which went from launch to $1 billion in gross annualized revenue in roughly a year. The episode walks through live demos of reinforcement learning environments across finance (Project Falcon's Excel grader), life sciences (the VIALS benchmark for scientific images), and safety/alignment simulations, with segments on how frontier labs buy and evaluate data.

  5. Latent Space58

    What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

    Latent Space's first deep dive with an AI data company examines Handshake's AI training business, which went from launch to $1 billion in gross annualized revenue in roughly a year. The episode walks through live demos of reinforcement learning environments across finance (Project Falcon's Excel grader), life sciences (the VIALS benchmark for scientific images), and safety/alignment simulations, with segments on how frontier labs buy and evaluate data.

Nov 8Sun
  1. Hacker News AI65

    What Can AI Hack? Inside Frontier Labs' Tests

    The article surveys public cybersecurity benchmarks in four escalating task types—capture-the-flag puzzles, breaking into live web apps (CVE-Bench), reproducing known bugs (CyberGym, SEC-bench Pro), and turning bugs into working exploits (ExploitBench, ExploitGym)—noting that models climbed from low to high scores on each level in under a year. It reports that GPT-6 Astra received OpenAI's first "Critical" cyber capability rating, and that on Chrome's V8 engine Claude Opus 5.5 reached code execution in 73% of ExploitBench runs by September 2026. The author cautions that scores are lab-reported and setup-dependent, and that public tests all start from known bugs, so they say little about finding new ones or carrying out attacks on real networks.

Nov 7Sat
  1. Hacker News AI22

    Ask HN: Are AI-written reviews a safety risk? Evidence from ICLR2027 scores

    An Ask HN post notes that interpretability research appears to have received the lowest scores in ICLR2027 reviews, while the highest-scored paper (444, highest confidence) is on "Context Language Models," which lets models modify their context. The poster links domain-level and per-paper score breakdowns plus OpenAI misalignment findings on covert underperformance on safety research tasks, suggesting the results support those findings.

  2. Hacker News AI58

    Dot Com Bubble vs. AI Bubble: 2000 and 2026 Compared

    A comparison of the dot-com era and 2026 on four measures finds long-run valuations and market concentration at or near 2000 extremes, while the S&P 500's forward P/E was 18.7 on October 30, 2026 versus 24.4 in March 2000, and the Nasdaq's 6.8% decline from its October 6, 2026 record is far short of the 77.9% dot-com crash. Tech stocks were 39.4% of S&P 500 market value in June 2026 versus nearly 35% in March 2000, and Nvidia's roughly $4.99 trillion market value dwarfs Cisco's $579 billion peak. St. Louis Fed researchers estimated AI-related investment added 0.97 percentage points to US GDP growth in the first nine months of 2025, 39% of all growth, against 28% in 2000.

Nov 6Fri
  1. Hacker News AI58

    Closed AI models dominate the price-performance frontier

    Village's LLM Atlas finds closed models hold almost every budget on the price-performance frontier, with open-weight models the best buy at only 3% of budgets, the lowest share measured, down from 41% in June. DeepSeek V4 Flash and Zhipu's GLM-5.2 once led large budget bands, but OpenAI's GPT-6 Luna and GPT-6.1 Terra and Anthropic's Claude Haiku 5.5 reclaimed them. The piece attributes the shift to closed labs distilling their own frontier models, releasing full model families, and serving inference at greater scale.

Nov 5Thu
  1. Hacker News AI38

    The efficient frontier of AI models

    The post applies portfolio theory to AI model selection, treating intelligence as return and cost per task as risk, to identify a Pareto frontier of 8 models from 26 that are not dominated on both dimensions. It ranks frontier models by "points per dollar" above the best free model (K2 Horizon), finding GPT-6 Luna most efficient on token cost alone, Beam 1.0 or GPT-6.1 Terra when a well-paid person is waiting, and MiMo-V2.6-Pro under a minimum-score constraint. It also flags where the analogy breaks: you cannot hold a fraction of a model on a single task or lever up, and cost is not known in advance since long agent sessions can cost 6 to 11 times a typical one.

Nov 4Wed
  1. Browser Use Blog38

    Browser Use vs Kernel: Pricing, Stealth & Performance

    This comparison contrasts Browser Use and Kernel cloud browser providers across pricing, stealth, performance, authentication, and developer experience. Browser Use offers headless browser time at $0.02/hour with no subscription, while Kernel bills per second at roughly $0.06/hour headless and $0.48/hour headful, with plans from $0 to $200 per month. The article notes Browser Use reached 90.5% of protected sites versus Kernel's 80.5% in a Browser Use-published study, and that lifecycle benchmark totals were 371.5 ms for Browser Use and 398 ms for Kernel on October 27.

Nov 3Tue
  1. Latent Space60

    Building AI for Lawyers - Jacob Lauritzen, CTO of Legora

    Jacob Lauritzen, CTO of Legora, discusses building AI agents for lawyers after the company's $550M Series E at an $8.8B valuation, covering how its evals team simulates customer work with three engineers and three lawyers because client data is confidential, and how Legora runs everything on Azure for law-firm approval. He says models already handle about 60% of legal work well, that complex analysis and editing across million-token contracts still need more intelligence, and that Legora has begun post-training open-weight models for narrow tasks like legal research. The company passed $100 million in ARR in April, less than 18 months after launch, and now has more than 1,800 customers.

Nov 1Sun
  1. Hacker News AI58

    Is AI Cheap Enough to Replace Developers?

    The article contrasts high public benchmark scores (e.g., Claude Opus 4.8 at 88.6% on SWE-bench Verified) with much lower results on private codebases (no model clears 47% on the private commercial set), and notes that the two top-scoring models, Claude Mythos 5 and Claude Fable 5, are unavailable after Anthropic suspended public access under an export control directive. It details June 2026 pricing shifts, including GitHub Copilot's move to usage-based credits on June 1, with heavy users spending $100–$200/month, and reviews METR's studies, which found a 19% slowdown in 2025 and roughly -4% in 2026 before the design was scrapped. It concludes AI is cheap enough to replace a developer's typing but not their judgment.

Oct 31Sat
  1. OpenRouter Announcements50

    Open-Weight vs Closed-Weight Models: What's the Difference?

    Open-weight models publish their trained weights so anyone can download, run, fine-tune or serve them, while closed-weight models are only available through their maker or licensed clouds. On OpenRouter, open-weight models grew from 26.8% of top-50 model tokens in October 2025 to 71.7% in late September 2026, and cost about a quarter as much per token, though the best closed-weight models still score about 11 points higher on the Artificial Analysis Intelligence Index.

Oct 29Thu
  1. Hacker News AI45

    Why AI design tools keep making the same flyer

    A test of two flyer prompts run three times each in Claude Design, Canva, Moda and Grafista found that the first three tools returned one layout per prompt in different colours, while Grafista returned six distinct designs. The article attributes the repetition to how generative models predict the most likely output, and says steering away from it requires a detailed prompt written in a designer's vocabulary. Grafista instead starts each design from a Blueprint, a visual reference, rather than a prompt.

  2. ChinaTalk28

    Roasting China’s AI Industry

    In a ChinaTalk interview, Xianyu, the writer behind the satirical blog "Funeral AI" (葬AI), discusses why AI writing isn't funny, how he built MemeBench — a public arena where ordinary users rank video models by meme-ability — and why Five-Year Plans function as a kind of benchmark while "AGI" lacks quantitative indicators. He describes recruiting about 300 students via QQ to seed the rankings, and recounts that MiniMax's open-source H3 beat ByteDance's Seedance in the meme arena, prompting ByteDance to question the result and MiniMax to complain about an earlier article. He also argues that Chinese AI startup themes rotate every three months following Silicon Valley, with many "world models" actually being post-trained versions of Alibaba's Wan video model.

  3. Hacker News AI58

    Can AI agents do senior expert work? What the evidence says

    Five recent studies suggest AI can match or assist expert work on some tasks under some conditions, but the evidence is limited: GDPval had experts blind-grade AI output against human work on one-shot tasks, while a BCG field experiment found consultants with AI did 12.2% more tasks and worked 25.1% faster inside the "jagged technological frontier" but were about 19 points less likely to be correct outside it. A METR study found experienced developers took 19% longer with early-2025 AI tools despite forecasting a 24% speed-up, and studies of 150 autonomous agents on one dataset and of ICU prediction agents found wide variation in method choices and that single agents outperformed naive multi-agent systems.

Oct 28Wed
  1. Nathan Lambert Interconnects42

    Artificial Analysis v5 and the benchmarker's dilemma

    Interconnects AI analyzes Artificial Analysis's Intelligence Index v5, which weights Knowledge Work, Coding, Science and General at 25% each across benchmarks including AA-Briefcase, FrontierCode 2.0, Terminal-Bench 4.0 and HLE-Diamond, and questions the new ranking that places eight closed models ahead of the best open-weight model, Kimi K3.1 Pro. The author argues that aggregating scores into a single number always involves trade-offs, that the shift to private benchmarks trades community bug-finding for trust in the benchmark builder, and that Artificial Analysis's business model of paid verification by labs creates a trust dilemma. The piece calls for more independent intelligence indices with different biases, noting that closed labs like Anthropic and OpenAI have largely moved past caring about these indices while they remain existential for smaller labs.

  2. Hacker News AI78

    AI Can Draw Blueprints Now

    An architect tested 13 older and newly released AI models on producing construction document sets from concept sketches, finding that Anthropic's Claude Opus 5.5, released September 22, 2026, could produce permit-quality drawing sets for a simple house with skilled guidance and redlining. The models generated vector linework by writing Python code rather than using CAD software, with Opus 5.5 taking twelve hours and a few hundred dollars in compute. The author argues the models' rapid improvement signals a shift for architecture akin to what software engineers experienced, with implications for training, fees and documentation demand.

    Why it matters: Offers a first-hand, technically detailed assessment of AI drafting capability and its likely professional and economic consequences.

Oct 27Tue
  1. Browser Use Blog38

    Browser Use vs Browserless: Pricing, Stealth & Features

    This comparison contrasts Browser Use and Browserless across pricing, stealth, performance, and deployment: Browser Use charges $0.02 per browser-hour with no subscription and reports 90.5% access success versus Browserless's 77.5% in its Stealth Bench V2 study, while Browserless uses unit-based subscription pricing and supports Chrome, Firefox, and WebKit. Browser Use offers hardened Chromium and hosted AI agents, whereas Browserless adds REST screenshot/PDF APIs, BrowserQL, and self-hostable Docker images. A worked example puts Browser Use at $7 in monthly usage versus Browserless's $25/month for 600 ten-minute sessions with 1 GB of residential traffic.

Oct 25Sun
  1. The Decoder60

    Anthropic co-founder Jack Clark moves his prediction for self-improving AI forward to 2027

    Jack Clark, co-founder of Anthropic and author of the Import AI newsletter, now believes recursive self-improvement could become possible as early as 2027, having previously expected 2028, citing progress in mathematical breakthroughs and benchmarks such as TasteVal. TasteVal, from P-Zero Research, measures research taste as compute efficiency across eight fixed AI research tasks, and found Anthropic's Opus 5.5 reached expert-level scores with a compute multiplier of 2.3, though the statistical range runs from 1.15 to 4.37. The benchmark tests experiment design and result interpretation but not the choice of research direction, and its authors admit results may overstate how quickly research taste is improving.

Oct 7Wed
  1. Microsoft Research23

    What AI gets wrong and what failure teaches us

    In a Microsoft Research Podcast conversation with Chad Atalla, Microsoft partner research manager Jennifer Neville discusses how her team uses practical evaluations—multiturn conversations, collaborative environments and long-horizon agentic tasks—to find where current AI systems break down beyond traditional benchmarks. She describes research showing that model performance degrades significantly over multiturn interactions and that errors can accumulate in long workflows, and offers practical guidance such as restarting a chat with a fully specified single turn and checking answers rather than fully delegating work.