Google DeepMind open-sources AlphaCode 3 agent evaluation harness

Read the original article →

What happened

Google DeepMind open-sourced the AlphaCode 3 agent evaluation harness on July 19 under Apache 2.0. It provides reproducible benchmarks for multi-step coding agents with sandboxed execution, hidden tests and cost tracking.

The release includes reference scores for Gemini 3.5 Pro, Claude Sonnet 5 and Codestral 2. DeepMind said it built the harness to standardize comparisons as labs gate frontier models behind preview lists. Contributors can submit new task suites via GitHub pull request.

Why it matters

When frontier models are hard to access, trustworthy benchmarks matter more than launch blog scores. A harness that tracks cost alongside pass rate matches how teams actually choose models in 2026.

Open sourcing it lets vendors and enterprises run the same tests instead of relying on vendor-selected highlights.

MintedBrain take

Use AlphaCode 3 tasks as one input, not the sole decision criterion. Hidden tests help, but your codebase, dependencies and CI setup will still differ; run a short internal pilot on top of the harness results.

References

Discussion

  • Loading…

← Back to News