Measuring Agent Experience: open sourcing the platform we use, Project Scope, and an AX practitioner handbook

19 October 2026

I lead Microsoft Developer Relations and one of my primary responsibility is to define our team’s mission (why), vision (where), strategy (what) and culture (how), in collaboration with many stakeholders across the company, and with the influence of many outside. Our mission has been quite stable over the past 5 years, and is very aligned with Microsoft’s mission: Help technology communities achieve more with AI on Microsoft Cloud.​ Our culture is also quite stable, based on growth mindset, respect, integrity, accountability. Strategy evolves quickly with quarterly revisions, to align with product teams and a fast evolving competitive landscape. 18 months ago, with the drastic changes AI Agents introduced in how we work and how our products evolved, I shared with my team an updated vision and a few months later shared some aspects of it in a public blog post about Reinventing Developer Relations in the age of AI Agents, with 3 dimensions: helping developers become managers of agents, transforming our own Developer Relations workflows with AI, and considering agents as users, helping them discover and use your services to build applications, treating AI Agents as a new type of customer of our developer services, and developing a new discipline of Agent Experience (AX).

In the post I gave details and examples about the first 2 aspects, but for AX I just set the problem and suggested that “in order to improve something you need to measure it: in the same way that Developer Relations uses metrics to measures developer experience, we need to define metrics for AX and build benchmarking tools to understand what AI Coding Agent Experience looks like with your company’s services.”. My team and many of our colleagues across many Microsoft product teams have been busy in the past year working on this agenda. We have co-evolved the methodology to measure AX in a way that is helpful to improve our products, and the tooling, building and rebuilding our tool as we learned what worked and what we needed. From the beginning we realized we were probably ahead of the curve in this area, and that everyone building software for developers would need this, so we built it from day one with open source technologies, on top of Kubernetes and KEDA, with the goals of open sourcing it at some point.

Today we’re open sourcing the The AX Practitioner Playbook, describing our methodology to measure AX, and Scope, an open-source evaluation system for AI coding agents. Scope lets you evaluate how AI Coding Agents are implementing specific scenarios, across a matrix of (Harness x Models x skills/Plugins/MCP x OS), with multi-turn interactions and an LLM judge to evluate the results. Scope is easy to deploy on AKS, we have been running it internally at Microsoft for months. In terms of governance, if the project is successful, we’re planning to donate it to AIIF or CNCF so it can be grown as a shared common system for the industry.

Learn more about Scope, contribute or log issues in the repository, and join the talk Jay Gordon and I give at AI DevCon NYC on November 4 AI Agents Are Your New Users.