Python vs Rust for AI Agents: When the Runtime Matters

You’re standing up an agent service. Same shape everyone’s building right now: a supervisor that fans out to a research worker and an analysis worker, calls a model, hits a tool or two over MCP, aggregates, returns. The question in front of you isn’t “which language do I like.” It’s “will the runtime I pick show up in my latency, my bill, or my pager.”

I built the exact same agent twice to find out. Once in Python LangGraph and FastAPI, once in Rust Axum, graph-flow, and rig. Then I put both under identical load.

Here’s the part that surprised me. At 10 requests per second, they tie. Python’s median response was 656 ms; Rust’s was 629 ms. That gap is noise. If your service lives at that load, you can pick Rust or Python by flipping a coin and your users will never know.

So the interesting question isn’t “is Rust faster.” Of course Rust is faster somewhere. The question is where? At what point does the runtime stop being free, and will your workload ever get there? That’s what this post is about.

The setup: one agent, built twice

There’s one DAG. It runs the same way in both languages:

pre → supervisor → worker_a (research / MCP) → worker_b (analysis / LLM) → aggregate → post

Everything around the runtime is shared and identical. Same Postgres, same MinIO, same model (a deterministic mock in the isolated benchmarks, a real one in end-to-end mode), the same DeepWiki MCP server. The only thing that differs between the two stacks is the language the DAG executes in. That’s the whole point: if latency or memory moves, the runtime moved it, because nothing else can.

The pre- and post-processing steps are written in plain language code i.e. no numpy on the Python side. In-node concurrency is idiomatic for each: Rust leans on rayon and threads, Python stays single-threaded behind the GIL. Those are deliberate choices, and they cut in Rust’s favor. I’ll come back to them in the caveats, because they matter and I’m not going to bury them.

I didn’t benchmark two different programs

This is the part most language shootouts get wrong, and it’s the first thing a skeptical reader should check. If you write two programs that merely claim to do the same work, any performance gap might just be a gap in what they actually compute. One version skips a sort. The other allocates differently. You end up comparing implementations, not runtimes.

So I made the equivalence a test, not a promise.

In isolated mode: deterministic mock model, deterministic research stub, both runtimes must produce byte-for-byte-equivalent output. Same ranked_findings, same chunk-id and source ordering, scores within a floating-point tolerance, identical summaries. A target called make equiv diffs the two /invoke responses field by field and fails the build if anything drifts. The published numbers only mean something because that gate is green:

make equiv
PASSED — 1 finding matches (chunk_ids + sources identical, scores within 1e-09, summaries equal).

If the two agents ever computed different answers, the benchmark would refuse to run. That’s the guardrail that lets the rest of these numbers stand up.

What I measured, and how

The priorities, in order: latency, then throughput, then memory, then CPU, then cold start. That ordering drove the design and when two goals fought, the one on the left won.

Two modes. Isolated swaps in the mock model and research stub so there’s no network or model jitter in the numbers; this is where the headline figures come from, because it’s the only way to attribute a delta to the runtime instead of to a slow API call. Realistic wires up a real model end-to-end to sanity-check that the whole thing holds together.

CPU and memory are self-reported. Each agent reads its own /proc/self and exports process_cpu_seconds_total and process_resident_memory_bytes. No cAdvisor, no cgroup accounting, both runtimes measure themselves the same way, which keeps the comparison fair across container engines. One honest consequence of that choice: process_cpu_seconds_total counts every thread. Rust spreads CPU work across cores with rayon, so under heavy load it can post more total CPU-seconds than single-threaded Python while still winning on wall-clock latency. More cores burning for less time. Keep that in mind when you get to the CPU number.

The results

Rust follows the load you give it. Python follows it up to a point, then stops following and starts drowning.

Throughput and the cliff

Python’s achieved throughput climbs with offered load until about 20–25 rps, then it falls as you push harder, the classic single-worker queueing collapse, not a graceful slope. Its sustainable ceiling, the highest offered load it held with under 1% of requests dropped, is 10 rps, at a p95 of 676 ms. Push to 25 rps and 16% of requests get dropped while p95 blows out to 20.5 seconds. Past 100 rps it stops producing measurable responses at all. Rust, on the same ladder, tracks offered load one-to-one all the way to 150 rps (p95 925 ms) and doesn’t start shedding until 250.

Memory

Peak resident set: Rust 307 MiB, Python 1194 MiB. Nearly 4x. If you’re packing agents onto a box or paying for memory by the gigabyte-hour, that’s a line item, not a rounding error.

Cold Start

First healthy response from a fresh container: Rust 1319 ms, Python 3316 ms. Two seconds sounds trivial until you’re scaling to zero and eating that gap on every cold request.

CPU (read this one carefully)

Over the run, Python burned 721 CPU-seconds and Rust burned 588. Closer than you’d expect, and this is exactly the all-threads effect from earlier: Rust parallelizes, so its total looks bigger than its latency win would suggest. The number that actually separates them is per-request cost. In the sweep, Rust spends roughly 3–10 CPU-ms per request; Python spends 38 to 179, climbing as it saturates. Same work, an order of magnitude apart in efficiency. Python’s single core just can’t get out of its own way once the queue builds.

For what it’s worth, the per-step processing latency in isolation is basically identical (235.6 ms Python, 235.0 ms Rust). At the level of one request doing one unit of work, the languages are even. The gap is entirely about what happens when requests pile up.

What the numbers are actually telling you

Here’s how I’d use this if I were making the call.

The runtime is invisible below Python’s concurrency cliff, somewhere around 10 to 20 rps per instance. Below that line, your latency isn’t set by the language. It’s set by the fixed costs every request pays no matter what: the model call, JSON serialization, the network hop. Those dwarf the DAG execution engine, so both runtimes land in the same place. That’s why they tied at 10 rps and it wasn’t a fluke, it’s structural.

Above that line, everything changes. Python can’t add workers to a single process to climb out; the GIL sees to that. You scale by running more processes, which means more memory (and you saw the memory number) and more orchestration. Rust just keeps taking the load on one process.

So the decision comes down to two numbers you already have, or can get in an afternoon:

Your offered load per instance: Not aggregate. Per instance, since that’s what hits the cliff.

Your p95 latency SLO: The number your on-call actually gets paged on.

If your per-instance load sits comfortably under ~10–15 rps and your SLO has slack, Python is fine. You’ll trade some memory and some cold-start time for a faster iteration loop and the entire Python ML ecosystem. If you’re pushing past that, or your p95 budget is tight, or you’re paying for memory and scale-to-zero latency, Rust stops being a preference and starts being an answer.

The biases

No benchmark is neutral, and pretending otherwise is how you lose a technical audience. Here’s where this one leans, and which way.

Pure-language processing favors Rust. The similarity-matrix work in pre/post-processing is hand-written in both languages, with no numpy on the Python side. That’s the naive-developer default, and it’s a real scenario, but a numpy version would push that math down into optimized C and narrow the CPU gap. If your Python is numpy-heavy where it counts, discount the CPU efficiency delta accordingly.

Concurrency is idiomatic, not matched. Rust uses rayon and real threads; Python stays single-threaded behind the GIL. I could have reached for multiprocessing or an async worker pool on the Python side. I didn’t, because the point was to compare what each runtime gives you by default, the way most teams actually write the first version. That’s a defensible choice and also a thumb on the scale. Name your own poison.

The load ladder is asymmetric on purpose. Python’s sweep stops at 150 rps because it’s thoroughly dead by then and burning more wall-clock time on a corpse proves nothing. Rust’s ladder runs further specifically to find its ceiling. If a chart shows Rust flat to the top of its range, that means the ladder didn’t reach the ceiling, not that there isn’t one.

Sample size. Each sweep step is three repeated measurements. The lowest-rps points can still carry warm-up jitter, so don’t over-read a 5 ms wobble at 2 rps.

When each one wins

Run it yourself

Checkout: https://github.com/sanketdaru/python-vs-rust-ai-agents-runtime-matters

Everything here comes out of one command:

make benchmark

It brings up the shared infra, seeds a deterministic corpus, builds both agents, runs cold-start, latency, throughput, and the load sweep, and writes docs/RESULTS.md. The equivalence gate runs as part of it, so if the two agents ever disagree you’ll know before you trust a single number.

These numbers came off a native Ubuntu box, and that detail matters more than it looks. I moved the whole setup off WSL2 onto native Ubuntu with docker-ce specifically because the WSL2 results weren’t reproducible enough to trust, which is its own small lesson about benchmarking on virtualized storage and cgroup layers.

The setup for this benchmark:

OS Ubuntu 24.04.4 LTS, kernel 6.8.0-134-generic (native, not WSL2)

CPU Intel Core i7-1355U, 12 logical cores

RAM 16 GiB

Engine Docker 27.3.1, overlay2 driver, cgroup v2

It’s a laptop-class part with a mix of performance and efficiency cores and a thermal ceiling. Good enough to compare two runtimes against each other on identical hardware; not a stand-in for what either would do on a tuned server. Take the ratios seriously and the absolute numbers with a grain of salt.