Evaluating an AI agent means handing it a real task, letting it work, and checking the result. Along the way the agent installs packages, edits files, and runs commands — untrusted, AI-written code that a benchmark may execute thousands of times. Agents can also game their checks through reward hacking, tampering with tests or overwriting the file that records their score. Docker's new integration addresses both safety and fairness by running evals in isolated environments.
According to Docker, Harbor can now run every eval task in Docker Sandboxes. The concrete changes:
- Docker Sandboxes are MicroVM environments that run agents such as Claude Code, Codex, and Cursor more safely.
- With Sandboxes, teams can define custom network policies, which Docker says enables fair, reproducible evals.
- Every Harbor eval task runs in its own clean, isolated Sandbox so results stay consistent across runs.
- Evals can run locally or in the cloud: Docker offers free Sandboxes on local surfaces, scaling to Cloud Sandboxes for larger workloads.
- The setup pairs an eval framework — which runs each task, lets the agent work, checks the result, and collects scores and logs — with a clean environment per task.
For developers building or benchmarking agents, the implication is practical. Reward hacking is a growing concern across the eval community, and running untrusted, AI-generated code thousands of times without isolation risks contaminating a machine or skewing scores. MicroVM isolation contains the agent's shell commands and file edits, while custom network policies limit what an eval can reach, so a passing score better reflects real task completion rather than a gamed check. Starting on local Sandboxes for free and scaling to Cloud Sandboxes means smaller teams can adopt the same reproducible harness that larger benchmarking runs need, without rebuilding their evaluation infrastructure.
Source: https://www.docker.com/blog/harbor-docker-sandboxes-run-safe-reproducible-agent-evals/