{"id":98436,"date":"2026-04-09T08:00:00","date_gmt":"2026-04-09T15:00:00","guid":{"rendered":""},"modified":"2026-04-09T06:54:54","modified_gmt":"2026-04-09T13:54:54","slug":"how-drasi-used-github-copilot-to-find-documentation-bugs","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2026\/04\/09\/how-drasi-used-github-copilot-to-find-documentation-bugs\/","title":{"rendered":"How Drasi used GitHub Copilot to find documentation bugs"},"content":{"rendered":"\n

For early-stage open-source projects, the \u201cGetting started\u201d guide is often the first real interaction a developer has with the project. If a command fails, an output doesn\u2019t match, or a step is unclear, most users won\u2019t file a bug report, they will just move on.<\/p>\n\n\n\n

Drasi<\/a>, a CNCF sandbox project that detects changes in your data and triggers immediate reactions, is supported by our small team of four engineers in Microsoft Azure’s Office of the Chief Technology Officer. We have comprehensive tutorials, but we are shipping code faster than we can manually test them.<\/p>\n\n\n\n

\n
Detect and react to your first database change using Drasi<\/a><\/div>\n<\/div>\n\n\n\n

The team <\/a>didn’t realize how big this gap was until late 2025, when GitHub updated its Dev Container infrastructure, bumping the minimum Docker version. The update broke the Docker daemon connection, and every single tutorial stopped working. Because we relied on manual testing, we didn’t immediately know the extent of the damage. Any developer trying Drasi during that window would have hit a wall.<\/a><\/a><\/a><\/p>\n\n\n\n

This incident forced a realization: with advanced AI coding assistants, documentation testing can be converted to a monitoring problem<\/strong>.<\/p>\n\n\n\n

<\/a><\/a>The problem: Why does documentation break?<\/h2>\n\n\n\n

\n Documentation usually breaks for two reasons:\n<\/p>\n\n\n\n

1. The curse of knowledge<\/h3>\n\n\n\n

Experienced developers write documentation with implicit context. When we write “wait for the query to bootstrap,” we know to run `drasi list query` and watch for the `Running` status, or even better to run the `drasi wait` command. A new user has no such context. Neither does an AI agent. They read the instructions literally and don’t know what to do. They get stuck on the “how,” while we only document the “what.” <\/p>\n\n\n\n

<\/a>2. Silent drift<\/h3>\n\n\n\n

\n Documentation doesn’t fail loudly like code does. When you rename a configuration file in your codebase, the build fails immediately. But when your documentation still references the old filename, nothing happens. The drift accumulates silently until a user reports confusion.\n<\/p>\n\n\n\n

\n This is compounded for tutorials like ours, which spin up sandbox environments with Docker, k3d, and sample databases. When any upstream dependency changes\u2014a deprecated flag, a bumped version, or a new default\u2014our tutorials can break silently.\n<\/p>\n\n\n\n

<\/a>The solution: Agents as synthetic users<\/h2>\n\n\n\n

To solve this, we treated tutorial testing as a simulation problem. We built an AI agent that acts as a \u201csynthetic new user.\u201d<\/p>\n\n\n\n

\n This agent has three critical characteristics:\n<\/p>\n\n\n\n

    \n
  1. It is na\u00efve<\/strong>: It has no prior knowledge of Drasi\u2014it knows only what is explicitly written in the tutorial.<\/li>\n\n\n\n
  2. It is literal<\/strong>: It executes every command exactly as written. If a step is missing, it fails.<\/li>\n\n\n\n
  3. It is unforgiving<\/strong>: It verifies every expected output. If the doc says, \u201cYou should see \u2018Success\u2019\u201d, and the command line interface (CLI) just returns silently, the agent flags it and fails fast.<\/li>\n<\/ol>\n\n\n\n

    <\/a>The stack: GitHub Copilot CLI and Dev Containers<\/h3>\n\n\n\n

    We built a solution using GitHub Actions, Dev Containers, Playwright, and the GitHub Copilot CLI. <\/p>\n\n\n\n

    \n Our tutorials require heavy infrastructure:\n<\/p>\n\n\n\n