iCreatAI News
Back to main site
Skip to content

Technology areas

Deployment engineering Latest updates

Engineering practices for getting models running: inference optimization, VRAM and cost, serving architecture, and infrastructure selection.

74 条精选Past 30 days: 71 updatesTotal: 1,011 updates

Updated

精选归档 · 第 3 页

Nov 5Thu 41–60 updates
  1. Baseten Blog62

    Introducing Baseten Grid

    Baseten Grid is a single layer for enterprises to route requests across approved open and closed models, monitor usage, and govern access, with a gateway supporting providers including OpenAI, Anthropic, Gemini Enterprise Agent Platform, Amazon Bedrock, and Baseten. In early benchmarks, its Baseten Router cut cost per coding task by up to 70% with minimal quality loss, and Grid also adds centrally configurable guardrails. Grid is in public preview starting today.

    Why it matters: Useful for teams weighing AI coding costs, as it details a routing-and-governance layer with specific benchmark claims and supported providers.

  2. vLLM Blog78

    vLLM Delivers Day-0 Support for Kimi K3.1 Pro

    vLLM announces day-0 support for Kimi K3.1 Pro, Moonshot AI's 2.8T-parameter (104B active) multimodal MoE with a 1M-token context that adds a DSA sparse-attention indexer to all 24 full-attention layers of its KDA + MLA hybrid. The post details a packed KV cache for three kinds of state, token-sharded prefill attention (up to 1.69× faster TTFT at 340K tokens on 8× GB300), decode context parallelism with an exact top-k merge, a fused low-latency decode path, a DSpark speculator preview, AgentX benchmark results on GB300 and Rubin, zero-mismatch RL, and support on NVIDIA, AMD, TPU and Alibaba T-HEAD Zhenwu PPUs.

    Why it matters: It is a detailed engineering account of how sparse attention, hybrid KV cache layout and parallelism choices interact, with measured numbers for anyone serving long-context MoE models.

Nov 4Wed
  1. Together AI Blog78

    Together Compiler: Continual Compilation for AI inference

    Together AI introduces Together Compiler (TKCC), a continual optimization loop that uses high-throughput agents to generate inference-engine optimizations, built on "persistent engines" that hot-reload candidate forward passes without restarts and a two-tier engine design balancing measurement throughput against fidelity. TKCC reports up to 2.6x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results on SemiAnalysis' InferenceX benchmark (1111 TPS on MiniMax M3.1-Flash, 1205 TPS on DeepSeek v4.1-Flash), and 1.7x over open-source megakernels on Llama 1B at batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. The post also says TKCC generalizes to weather modeling, PDE, time-series, and biology workloads, and is serving trillions of tokens today.

    Why it matters: It offers a detailed look at how agent-driven, continual compilation addresses drifting inference traffic, with concrete throughput and benchmark numbers.

  2. Hacker News AI78

    Revolut swaps Claude for GLM open-weight model, cutting AI coding bill by 90%

    Revolut has moved its roughly 2,000 software engineers off Anthropic's Claude Code and OpenAI's Codex as default coding tools onto GLM, an open-weight model from Chinese lab Z.ai that Revolut hosts itself, bringing its bill to about a tenth of what it paid Anthropic and OpenAI. David Abitbol, who runs AI operations and infrastructure at Revolut, said the company serves GLM from about 56 Nvidia B200 GPUs at a blended cost of around $0.30 per million tokens, and reported roughly 60% shorter coding time and 70% shorter lead time in a sample of about 2,000 engineers, though the comparison does not isolate the model change from other causes. Abitbol said the 90% saving excludes cloud, staff and monitoring costs and depended on heavy tuning, with out-of-the-box inference "terrible" until time to first token fell from about 14 seconds to 1.7 seconds on the same GPUs.

    Why it matters: It shows a major European fintech running a self-hosted Chinese open-weight model as its engineering default, with concrete cost and latency figures and caveats worth noting.

Nov 3Tue
  1. Claude Blog60

    Make long-running agents faster with message threads

    Anthropic announced that message threads is in public beta on the Claude Platform, letting agents send only new messages plus the previous response ID instead of resending full conversation history each turn. With message threads on by default for most Claude Code users, requests of 500k to 1M tokens saw median time to first token drop by about 35%, though token usage stays the same since Claude still reads the full conversation each turn. Threads are held encrypted in memory, expire 5 minutes after the last request, and are available on the native Claude Platform, Claude Platform on AWS, Claude in Microsoft Foundry, and Claude Models on Google Cloud (Anthropic managed).

    Why it matters: It gives developers concrete latency figures and setup details for a new API feature aimed at long-running agent sessions.

Nov 2Mon
  1. Hacker News AI82

    China's First Frontier AI Export Controls: Mofcom Announcement No. 48

    On October 16, 2026, MOFCOM and MOST issued Announcement No. 48, proposing to add 40 technologies to China's export catalogue—23 of them in AI, covering frontier model weights, agent architectures, and offensive cyber capability—while an interim pre-export registration requirement is already in effect. The catalogue additions are a draft open for comment until November 15, 2026, whereas registration applies now, including to exports already started but not yet complete, and is triggered by licensing for overseas use or changes of ownership or control. MOFCOM says API, app, web, SaaS, and lightweight on-device use do not count as exporting a large model, but local deployment of a restricted model outside China does, and open-sourcing a new restricted model requires reporting under automatic licensing.

    Why it matters: It maps a new Chinese outbound control regime onto concrete enterprise exposure points—self-hosted deployments, intra-group code sharing, and M&A—making it useful for compliance and security teams tracking frontier AI supply-chain risk.

Nov 1Sun
  1. OpenRouter Announcements60

    Build an AI Code Reviewer with OpenRouter

    This guide presents a 233-line Python script (review.py) that uses the OpenRouter Responses API with the Files API, Shell server tool, and Apply Patch server tool to run changed files in a hosted Linux sandbox, confirm bugs by executing checks, and return fixes as V4A patches. It covers setup, a five-bug sample repository, a Git pre-push hook, and a GitHub Actions workflow, with benchmark results across four models (e.g., Claude Sonnet 5.5 fixing all five bugs at $0.0447 and 60 seconds). The script writes a report to .git/ai-review.json and exits 1 when a finding meets the --block-at severity threshold.

    Why it matters: It offers a concrete, reproducible implementation with cost and latency benchmarks, plus practical notes on API pitfalls, useful for developers building automated code review.

  2. Browser Use Blog60

    BU Ultrafast is live

    Browser Use has made BU Ultrafast, its fastest browser agent, available on every Browser Use Cloud plan in the US, including the free plan, after a waitlist period that ended Oct 30. The company reports a median task time of 22 seconds and $0.0064 per task for Ultrafast, versus 79 seconds and $0.0098 for its default agent on GPT-6 Luna, based on medians of completed runs from Oct 16-29, 2026. Users enable it by setting the model to bu-ultrafast, and bu-fast offers the same agent with more reasoning per step.

    Why it matters: It gives concrete speed and cost benchmarks for a newly released browser agent and explains how to switch to it via the API.

Oct 31Sat
  1. Hacker News AI76

    Open Science: The open-source AI workbench for scientific research

    OpenScience is an open-source research agent, available as a desktop app, browser workspace or CLI, that plans tasks, gathers evidence, runs code and experiments, and returns checkable results with every step visible. It offers shell, Python and R kernels, file access with explicit grants, remote compute, hundreds of bundled scientific skills, and connectors to databases such as ChEMBL, UniProt, PubMed and arXiv, with support for your own API keys, local models or the managed Ace option. It is Apache-2.0 licensed and reports 53/70 (75.7) on Terminal-Bench Science and 82.2 on BiomniBench-DA.

    Why it matters: Useful for researchers weighing an open-source, traceable agent workbench, since it lists concrete capabilities, model-access options and published benchmark scores.

  2. Cloudflare Blog78

    4 real? Tracking exploitation and patching of eMudhra’s weak keys

    Cloudflare reports that eMudhra's eight weak intermediate keys (derived from numbers like 4, 56, 66, 115, 155, 156, 230 and 237) could be recovered by anyone from public certificates, and that it patched all high-priority systems after learning of the issue on October 19, 2026. Logging on Pingora recorded about 22,000 attempted handshakes with the weak keys over a week, two-thirds of them fetches of test sites, with roughly 160 presenting forged certificates naming unrelated domains, one dated October 20 at 18:19 UTC, less than an hour after Mozilla's bug report became public. Probe data from 50 million challenge-page connections shows modern browsers mostly patched while bots and embedded browsers such as Android WebView lag, with more than half of probes on large networks still accepting the compromised chain.

    Why it matters: Documents real-world exploitation evidence and patching progress for a CA key compromise, with concrete data on forged certificates and client-side trust.

Oct 30Fri
  1. Cursor Blog82

    Security notice — malicious version of Anthropic.claude-code (2.1.313) identified in the Cursor Extension Marketplace

    Cursor reported that a maliciously modified Anthropic.claude-code version 2.1.313 was available in its VS Code extension Marketplace between about 2:27am and 1:35pm PT, affecting at most 5,000 users; the extension attempted to install a persistent SSH backdoor, though the attacker server has since been taken offline. Cursor removed the version, shut down and rebuilt the marketplace, rotated keys, and published indicators of compromise plus macOS and Windows check and remediation scripts. Users who installed the version in that window should check for the svc-telemetry SSH key, remove persistence and tunnel processes, and rotate credentials from a trusted system.

    Why it matters: The notice gives concrete indicators of compromise and remediation steps that Claude Code users can act on directly.

  2. OpenRouter Announcements60

    How Much Does Claude Code Cost? Real Session Costs by Model

    Claude Code is free to install, but users pay for the model it calls, either through a Claude subscription or per token on an API key. Median API-key session costs from OpenRouter data over the 30 days to October 18, 2026 were $0.31 for a 2–9-turn session and $1.39 for a 10–49-turn session on Claude Opus 5.5, versus $0.16 and $0.60 on Claude Sonnet 5.5. Most of the cost comes from Claude Code's own context written to the prompt cache on the first request and read back on later turns, so the article recommends Sonnet as default, Haiku for background jobs and sub-agents, shorter sessions, and a credit limit on the key.

    Why it matters: It gives concrete per-session cost medians by model and length, plus practical levers for controlling Claude Code spend.

  3. Factory Blog78

    Droid 1.0: Building a Frontier Harness

    Factory released Droid 1.0, an agent harness it says outperforms native harnesses built for their models by up to 20% more efficiently on the same tasks, with programmatic tool calling (Script mode) now the default for models including GPT-6.1 Sol, Claude Opus 5.5 and Kimi K3. The post reports that Script mode cuts Droid's cost by 15% on Claude Opus 5.5, that the base prompt shrank 37% between versions 0.170.0 and 0.250.0, and that median time to first paint fell from 1,839 to 488 ms. It also describes the internal Agent Factory loop, in which agents propose harness changes that engineers review before shipping.

    Why it matters: A detailed first-party account of harness design choices, benchmark methodology and latency work that is useful for teams building or evaluating coding agents.

Oct 29Thu
  1. vLLM Blog62

    Day 0 support for Mistral Large 4 on vLLM

    vLLM announces day-0 support for Mistral Large 4, a 1T-parameter MoE multimodal model with 49B active parameters per token, running on the V2 model runner with breakable CUDA graphs and a new vllm/models/mistral/ package. The work adds cross-layer KV sharing, DCP/PCP context parallelism, DSpark speculative decoding, NIXL-based disaggregated serving and HiSparse KV offload, plus day-0 ROCm support on AMD Instinct MI350 Series GPUs with MXFP4 experts. Support ships in the vllm/vllm-openai:mistral-large-4 Docker image, with FP8 and NVFP4 checkpoints and serving commands documented.

    Why it matters: A detailed engineering account of what it takes to land a frontier model on an inference engine, useful for teams planning long-context or disaggregated deployments.

  2. OpenRouter Announcements60

    How we rebuilt provider routing for better, faster inference

    OpenRouter announced a new default provider routing algorithm that scores endpoints on realized price, time to first token, throughput, error and rate-limit rates, and output quality rather than input token price alone. In a week-long switchback experiment over 24 million requests, the new router cut TTFT p50 by 16.9%, raised throughput p50 by 16.3%, and reduced requests hitting at least one 429 by 67%, with cost per K requests up 6.0%. Explicit sorts, provider.order, and cache-aware conversation pinning still work as before, and price-only routing remains available via provider.sort "price" or the :floor suffix.

    Why it matters: Useful for teams on OpenRouter deciding whether to keep default routing or opt back into price-only sorting, since it quantifies the speed, reliability, and cost tradeoffs.

  3. Pydantic AI Articles74

    Code mode: make your agents 2x faster and 2x cheaper on long tasks

    A benchmark of 5,276 graded runs across eight models on five small Pydantic AI apps found that code mode, where the model writes a program that calls tools inside a sandbox, cut cost by 58% and wall time by 51% on long tasks while raising success by 16 points. Gains grew with task length: at 200 items code mode was 47% to 75% cheaper and 42% to 67% faster, whereas at 5 items it cost 21% to 38% more on three of five tasks. The author attributes the win not to fewer round trips, since models already batch calls, but to less content per request, and runs the model's code in Monty, a Rust Python interpreter with no host access by default and 30-second, 256 MiB limits.

    Why it matters: Offers a detailed, methodology-heavy benchmark with per-model cost, latency and correctness numbers, plus an honest list of where code mode costs more.

  4. Artificial Analysis Articles72

    GitHub Copilot Fusion orchestrates Anthropic and OpenAI models to balance performance and cost

    GitHub has released Copilot Fusion in GitHub Copilot, a runtime orchestration system that selects and coordinates models from Anthropic and OpenAI, including Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6 Sol, to optimize performance, cost and latency per task. In Artificial Analysis' Coding Agent Index v2.0, its max setting in GitHub Copilot CLI scored 2 points below Opus 5.5 in Claude Code at half the cost per task ($5.59 versus $11.42), leading Terminal-Bench 4.0 at 71 while ranking third on FrontierCode at 51 and scoring 57 on SWE-Atlas-QnA-AA. It used 7.2M tokens per task, roughly half of Claude Opus 5.5 in Claude Code and 30% less than Devin Fusion with Opus 5.5 (max) + GPT-6 Luna (high).

    Why it matters: Offers concrete benchmark and cost comparisons for a multi-model orchestration approach, useful for readers weighing coding-agent performance against token spend.

Oct 28Wed
  1. Google Developers Blog67

    Fast, reliable A2UI, with swappable inference formats

    A2UI now supports swappable inference formats, with A2UI Express as the new default that requires up to 3x fewer tokens than Direct JSON and can cut latency by up to 3x, while decision models like Jev can reduce latency by up to 80x for layouts defined in advance. Express is available in the Python agent SDK (a2ui-agent-sdk 0.9.0), the TypeScript agent SDK (@a2ui/agent) and the Dart SDK (a2ui_agent 0.0.1-wip004), and the SDK compiles the model's reply into validated A2UI messages. The post also describes the Apparite (jev2ui) example, where Jev answers layout questions and code builds the component tree, with screens complete in 0.8 to 1.7 s across 11 prompts in a small first-look sample.

    Why it matters: It gives concrete latency and token figures plus working SDK versions for teams weighing Express against decision-model-driven layouts.

  2. Replit Blog72

    Anyone can now bring their ideas into the company codebase

    Replit launched Cloud Agents for existing repositories, letting anyone on a team describe a change in plain language and have an agent find the code, test it in cloud sandboxes, and open a tested pull request in GitHub or GitLab with screenshots and video. The feature includes shared environments set up once by an admin, review feedback that agents turn into proposed team skills for engineer approval, and a Slack integration for starting changes from threads; it is available to all Pro and Enterprise customers.

    Why it matters: Announces a notable shift in how non-engineers can contribute tested code changes to production repositories, with concrete details on testing, review, and setup.

Oct 27Tue
  1. NVIDIA Newsroom75

    NVIDIA and Meta to Expand Collaboration on Muse and Muse Family of Models Amid Robust Growth

    NVIDIA and Meta announced they will expand their partnership on Muse and the Muse family of models to improve inference and training performance, citing popular demand for AI agents. Meta intends to expand deployment of NVIDIA Vera Rubin and Vera CPUs, plus Spectrum-X Ethernet networking and BlueField-4, and the companies will consider collaboration on NVIDIA OpenShell Reference Architecture and NVIDIA Sentry for agent security and monitoring. NVIDIA will invest $3 billion in Meta's equity offering.

    Why it matters: The announcement details a major AI infrastructure partnership and a $3 billion equity investment, useful for readers tracking NVIDIA-Meta ties and agent infrastructure.