Supabase’s open benchmark pits AI coders against real tasks
Supabase has just open-sourced Evals, a framework that measures how well AI coding agents—including Claude Code, Codex and OpenCode—handle real Supabase projects. Instead of synthetic puzzles, Evals runs agents against everyday tasks such as building a database schema, debugging a failing Edge Function or repairing a broken Row-Level Security policy. The project ships with a public leaderboard at supabase.com/evals and an internal regression suite that runs daily, giving teams a way to gate SDK releases or compare agent harnesses head to head.
A sandbox that feels like production
Every evaluation spins up a real Supabase stack inside Docker containers, complete with the CLI, an MCP server, and isolated ports. Agents interact with the actual Management API surface through a lightweight runtime called “platform-lite,” so the benchmark isn’t just emulating work—it’s doing it. Scenarios are organized across three dimensions: products (database, auth, storage, Edge Functions, etc.), topics (RLS, security, migrations, SQL, SDK, observability), and stages (build, deploy, investigate, resolve). Each scenario is grounded in real support tickets, bug reports or GitHub issues, ensuring the tests reflect what engineers actually face.
Transparent scoring—with room to improve
Evals combines deterministic checks (did the schema deploy correctly?) with LLM-as-a-judge grading. Agents get one retry before the final score, and the framework keeps two suites: a public benchmark for breadth and a daily regression set that tracks known failure modes without moving published scores. Local runs require Docker, free ports 54321–54329, and provider API keys, but the GitHub repo supabase/evals is Apache-2.0 and installs via pnpm.
Why it matters
For teams shipping regulated backends in fintech or healthcare, a single misstep in RLS policy can turn into a security incident. By open-sourcing its evaluation harness, Supabase gives the industry a shared yardstick to compare agents, catch regressions early, and harden tools before they reach production. The public leaderboard also pressures vendors to improve on real-world Supabase tasks, turning opaque benchmarks into transparent, reproducible evidence.
Source: MarkTechPost. AI-assisted editorial synthesis — TechnoExpress.