sandbox-benchmarking: Measuring the 6-Second Ghost in Vercel Sandbox
How a minimal benchmarking suite exposed the hidden first-command penalty haunting ephemeral compute.
`Sandbox.create()` resolves before the VM is fully ready. The first `runCommand` absorbs the remaining boot time. Every subsequent command is fast.
- Vercel Sandbox reports a successful creation before the underlying Firecracker microVM has finished booting.
- The first command sent to a new sandbox absorbs the hidden boot time and suffers a latency penalty of up to 33 seconds.
- Restoring from a snapshot often takes longer than a fresh boot because the infrastructure must pull heavy state data across the network.
- This hidden startup tax creates a significant performance bottleneck for interactive AI agents that rely on ephemeral compute.
The Invisible Wall
Cloud providers market their ephemeral compute environments with promises of sub-second startup times. For Vercel Sandbox, the marketing highlights millisecond readiness. But developers building interactive AI agents began noticing a phantom delay. The infrastructure felt fast, yet the first response always dragged.
The repository vercel-labs/sandbox-benchmarking is a technical detective story. It operates as a bug report disguised as a benchmarking suite. Developer John Lindquist built it to isolate a specific bottleneck in Vercel's Firecracker-based microVMs. The finding is stark. The system suffers from a massive first-command penalty.
The Leaky Abstraction
The core issue lies in how the Vercel Sandbox SDK handles state. When a developer calls Sandbox.create(), the API returns a success response almost immediately. The hypervisor has allocated the resources. To the control plane, the sandbox is running.
But running does not mean ready. The underlying operating system is still booting. When the developer sends the very first sandbox.runCommand(), that command hits a queue. It sits there, waiting for the VM to actually accept instructions. The SDK makes the command execution look incredibly slow, but the command is simply absorbing the hidden boot time.
Benchmarking the Benchmarks
To prove this wasn't just local network latency, the repository includes two identical test environments. One runs locally via bench.mjs. The other runs inside Vercel's own infrastructure as a serverless function via api/bench.mjs. Both scripts execute a trivial command twice in a row.
Minimal reproduction: first runCommand after Sandbox.create() has 7-112x latency penalty vs subsequent commands
| Environment | create() Latency | First Command (Penalty) | Second Command (Ready) |
|---|---|---|---|
| Local CLI | ~1.5s | ~2.0s - 4.0s | < 100ms |
| Vercel Function (IAD1) | ~800ms | ~6.5s - 33.0s | < 100ms |
The data reveals a counter-intuitive truth. Running the benchmark from within Vercel's own network often results in a worse first-command penalty than running it over the public internet from a local machine.
The Snapshot Irony
The most surprising revelation involves snapshots. Snapshots are designed to be an optimization. You freeze a VM state and restore it later to skip the boot sequence. But the benchmark shows that restoring from a snapshot can actually be slower than starting fresh.
The production penalty is **~6 seconds on every restore**, consistently.
If the underlying physical host has not seen that specific snapshot recently, the infrastructure must pull the heavy state data across the network. This cold restore process completely negates the benefit of snapshotting in the first place.
The Agent Tax
This repository highlights a critical challenge for the next generation of software. AI frameworks like OpenClaw rely on persistent, conversational contexts. If every user message requires waking up a sandbox from a snapshot, a six-second penalty on every interaction breaks the user experience.
The sandbox-benchmarking project proves that infrastructure providers need to rethink their readiness metrics. Until the API accurately reflects when a machine can actually execute code, developers will continue paying this hidden tax.