Introducing Scope: Open-source evaluation for AI coding agents

AI coding agents are becoming users of your software. They discover APIs, choose software development kits (SDKs), follow documentation, and invoke tools on a developer’s behalf. For product teams, that creates a new challenge: an experience that works well for a human developer may still leave an agent choosing the wrong API, missing an important instruction, or getting stuck before the task is complete.

Teams have long invested in developer experience (DevEx) through user testing, interviews, and product feedback. Those practices remain essential, but they don’t tell the whole story when an agent is using the product. Developer Agent Experience (DevAx) extends that work: understanding how agents interact with your software and using evidence to improve the outcomes they achieve.

The opportunity is to make that learning continuous. Give an agent a representative task, inspect its interactions, improve the product or guidance, and evaluate again. But a successful demo or a single continuous integration (CI) test isn’t enough. Models, agent runtimes, skills, plugins, Model Context Protocol (MCP) servers, and custom instructions all influence behavior. Results vary between attempts, and the configurations themselves keep changing.

Today, we’re announcing Scope, an open-source platform built on Kubernetes for turning those interactions into evidence product teams can act on. Microsoft Developer Relations began building Scope more than a year ago to automate these evaluations and help teams decide what to improve next. Available under the MIT License, Scope supports a continuous DevAx learning loop grounded in repeatable experiments.

From evaluation to product improvements with Azure Cosmos DB

The Azure Cosmos DB team’s experience shows what that loop can deliver. Using Scope, the team compared bare models with its Agent Kit across end-to-end scenarios, then used the findings to address issues in both the Agent Kit and the documentation.

Using SCOPE, we compared bare models with our Agent Kit across eight end-to-end Azure Cosmos DB scenarios in five languages. The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT usage fell from 47% to 0%.

—Jay Gordon, Senior Program Manager, Azure Cosmos DB

These results connect evaluation to decisions a product team can make: which documentation needs clarification, where tooling needs a fix, and whether an update changes agent behavior. The gains describe the Cosmos DB scenarios evaluated, rather than a general ranking of agents. They demonstrate how targeted changes, informed by observed behavior, can improve the way agents use a product.

For more on the shift from building for human developers to supporting agents, watch Your Product Was Built for Humans. Is It Ready for Agents? or join us at AI DevCon NYC conference in November.

Turn agent behavior into better product experiences

Whether you’re maintaining an SDK, building a command-line interface (CLI), or publishing an MCP server or skills, the question is broader than which agent scores highest. You need to understand whether agents can discover and use what you ship, and what your team can change when they cannot.

Scope helps teams investigate improvements across three areas:

• Documentation that guides agents to successful outcomes. Look for recurring missed instructions or incorrect assumptions, update the guidance, and rerun the scenarios to see whether those failures become less frequent.
• SDK discovery and correct adoption. Check whether agents select the intended SDK, initialize it correctly, and use its APIs as expected. Compare a baseline with revised examples or an agent skill to understand what helps.
• Tooling that supports task completion. Examine where agents struggle with command-line tools or MCP servers. Use the evidence to prioritize clearer tool descriptions, more actionable errors, or fixes to the integration itself.

For example, an SDK team can ask agents to add an integration to a sample application, then evaluate whether it builds, uses the intended API, and handles a documented error response. If missing-item handling repeatedly fails, the team has a specific behavior to investigate. After updating the guidance or tooling, repeated evaluations can show whether that change improves the task’s success rate.

Why AI agent evaluation needs an open-source foundation

The challenges behind agent experience aren’t unique to one product or organization. SDK maintainers, tool authors, researchers, and agent developers are all asking how agents behave and what helps them succeed. No single team can represent every language, workflow, or integration. An open project can bring more scenarios, more failure cases, and more viewpoints into the evaluation process.

Evaluation also needs scrutiny. When a result informs a documentation update or an SDK change, teams should be able to inspect the task, evidence, and judgment behind it. Opening Scope’s execution and evaluation code gives contributors a way to question the methods, investigate unexpected results, and improve the evaluation itself.

A shared foundation also reduces duplicated infrastructure. Agent integrations, reusable criteria, and trace-analysis improvements can help other teams study their own products without rebuilding the same machinery. The goal isn’t one universal benchmark or definition of success. It’s a common starting point that teams can adapt to their needs. The MIT License lets teams use and extend Scope, while self-managed deployments let them evaluate private workloads without publishing their code or run data.

Build an evaluation loop around your product

Scope drives existing coding agents, including GitHub Copilot, Claude Code, and Visual Studio Code with Copilot. You define representative tasks, provide a starting codebase when needed, and choose the agent setup. Configurable personas let you vary the simulated user’s perspective and communication style.

Reusable, versioned profiles capture the configuration behind each run, including the worker, model, agent version, skills, MCP servers, and supported extensions. Keep the task and criteria consistent while changing the part you want to evaluate. Repeated runs help distinguish recurring problems from outliers, while preserved configurations make it easier to understand what changed between experiments.

Success is defined by your product’s requirements. Scope represents evaluation criteria as a directed acyclic graph (DAG), so checks can stand alone or depend on prerequisites. A runtime check, for example, can depend on a successful build. The Judge service uses large language model (LLM) based evaluation to assess the output against those criteria, helping teams see which requirements were met and where further investigation is needed.

Criteria-level results are a starting point for investigation. Scope streams activity as the agent works and retains interactions and workspace snapshots for later review. Your team can follow tool calls, inspect generated files, and connect a failed requirement to the steps that preceded it. Those traces provide observable evidence of what the agent did, helping turn a failure into a concrete documentation, SDK, or tooling improvement.

Clearly, repeatable configurations don’t guarantee identical agent behavior, and automated judges can make mistakes. So, even when using Scope, it is necessary to use representative tasks, repeated runs, and human review of the evidence when making decisions.

Integrate AI agent evaluation into your development workflow

As your product and its agent integrations evolve, the same scenarios can help your team track improvements and investigate regressions. Use the Portal to configure and inspect experiments, or the CLI and REST API to submit benchmarks and retrieve results from scripts and pipelines. Evaluation becomes a recurring source of product feedback rather than a one-time check.

The repository includes the application services, agent workers, evaluation service, and documentation. You can start a local stack with Docker Compose using the quick-start guide.

Scope is software you deploy, not a hosted service included with the release. You choose the connected services and manage access to experiment data and credentials. Running agents and model-based evaluation requires access to the corresponding providers.

Get started with Scope

We’re making Scope open source so developers and researchers can build on a shared foundation for improving DevAx. We welcome new agent integrations, reusable criteria, improvements to analysis, and lessons from teams evaluating their own products. The core contributors to Scope—Waldek Mastykarz, Cedric Vidal, Garry Trinder, Josh Duffney, Wassim Chegham, Christopher Maneu, Patrick Chanezon—and I would love to discuss the ideas behind Scope on GitHub and work together with the community to grow the DevAx discipline.

Next step: Start with one task that matters to your product. Define a successful outcome, run it repeatedly, and investigate where agents struggle. Use what you learn to improve the experience, then evaluate the change.

Explore the official Scope repository on GitHub, run your first evaluation, and contribute to the project.