Skip to content

How the assistant is tested

← Architecture

How the requests to GitHub are tested

Most tests replace GitHub with the fake in tests/support.py, which answers the calls of the script from the issues, comments and pull requests that a test sets up, as the end of this part shows. The fake knows a GraphQL query by its words, so it can't tell a misspelt field from a right one, or a field that GitHub renamed last week. The script, for its part, falls back where one of GitHub's newer fields fails: labels go without their rationale, and an issue closes through REST, without the link to its original. Such a mistake would show only as a notice in the log of a repository that uses the action, so three checks look for it before a release:

Three checks of the requests to GitHub: The GraphQL documents of issue_assistant.py meet a copy of GitHub's schema in the required check of every pull request. The commands sweep, sync-labels and context run against GitHub itself with a token that can only read, on every pull request, on main and every week. Every week the documents also meet the schema that GitHub publishes that day, and a notice says when the copy is behind it.Three checks of the requests to GitHub: The GraphQL documents of issue_assistant.py meet a copy of GitHub's schema in the required check of every pull request. The commands sweep, sync-labels and context run against GitHub itself with a token that can only read, on every pull request, on main and every week. Every week the documents also meet the schema that GitHub publishes that day, and a notice says when the copy is behind it.

  1. Every GraphQL document meets GitHub's schema. tests/test_graphql_contract.py runs each method of the class GitHub that sends GraphQL, catches what it hands to gh and checks it against GitHub's public schema, whose copy is tests/github/schema.docs.graphql: every field and argument must exist and none may be deprecated, and every variable must fit its type, such as the reason for closing an issue. The answer is worked out from the schema as well, so it holds only the fields that the document asks for, and the method can read no other. A document that the test doesn't know fails it, so a new one comes with its case.
  2. The commands that read run against GitHub. The job egress of the Action test runs the action of the pull request as dry runs of sweep and sync-labels on this repository, and context for a closed issue of HomeDarts for Autodarts, a public repository that uses the action, with its comments, the other issues and the discussions. Its token can only read, so a write that a dry run let through would fail too.
  3. Every week, the schema of the day. GitHub changes its API without a release of the action. The job schema of the Action test checks the documents every week against the schema that GitHub publishes that day. GitHub deprecates a field, an argument or a value of an enum months before it removes it, and says when, so the run turns red as soon as the script uses something deprecated, while it still works. When the copy in tests/github/ is behind, a notice says so, and a pull request refreshes it with the command in tests/test_graphql_contract.py.

The fake itself is as strict as GitHub, so that what it lets through works on GitHub too:

How the tests stand in for GitHub: The tests of the events, the sweeps and the commands call the fake GitHub in tests/support.py, which gives long lists in pages and checks the body of every write. The tests of the class GitHub run Python's real subprocess.run, which starts tests/fake_gh.py in place of gh; it writes down each command and what it read, and answers as the test asks. What the assistant writes, its comments, the context of each task, the plan of each event and the job summaries, is compared word for word with the golden files in tests/golden/.How the tests stand in for GitHub: The tests of the events, the sweeps and the commands call the fake GitHub in tests/support.py, which gives long lists in pages and checks the body of every write. The tests of the class GitHub run Python's real subprocess.run, which starts tests/fake_gh.py in place of gh; it writes down each command and what it read, and answers as the test asks. What the assistant writes, its comments, the context of each task, the plan of each event and the job summaries, is compared word for word with the golden files in tests/golden/.

  • Long lists come in pages. The fake gives every list of the REST API in pages, of 30 items unless the request asks for up to 100. A reader that pages gets every item, and one that forgets to gets the first page alone, as from GitHub. Tests with 150 comments, events, reactions, issues, labels, milestones and alerts show that every reader reaches the last page, also in a sweep and in an event of an issue whose reporter answered in its 140th comment.
  • Every write has the body that GitHub reads. The fake refuses a write whose body lacks a key that GitHub needs or has one that GitHub ignores, and a reaction, a colour or a reason for closing that GitHub doesn't know.
  • gh runs for real, with a stand-in. The tests of the class GitHub, which builds the commands of gh, run Python's real subprocess.run with the options of the script. In place of gh, it starts tests/fake_gh.py, which writes down its arguments and what it read, and answers as the test asks, also with an error. So the tests see whole commands and the JSON that a write sends, and that a failing gh ends in the error that the script handles, never in an exception that it doesn't expect.
  • What the assistant writes is pinned word for word. Golden files in tests/golden/ hold every comment in English and in German, the files of the context of each task, the plan and the requests of each kind of event, and the job summaries. A change of a text, of its Markdown or of the schema of an answer fails tests/test_golden.py until its golden file shows it, so the pull request shows what readers will see differently. tests/README.md says how to write them anew.

How the names of the engines are checked

What keeps each engine to reading rests on names that its vendor owns: the inputs of the Codex action that choose its sandbox, the keys of the settings that leave the Gemini CLI only its tools that read, the flags that do so for the Copilot CLI, and the outputs that hold the answer. tests/test_templates.py and tests/test_engines.py check our side of these names. The other side changes with a release of the engine, and nothing would fail: GitHub only warns about an input that an action doesn't have and runs the step without it, the Gemini CLI ignores a key of its settings that it doesn't know, and so does Codex with a key of its configuration. No check of a pull request runs an engine with its secret, and the Canary runs Claude alone, so the job engine contracts of the Action test runs the pinned versions without one:

How the names of the engines are checked: A pull request that changes the step of an engine or a lock file of engines/, such as an update of Renovate, or one that changes what the check reads, and a run every week start the job engine contracts of the Action test. It holds no secret, and its block lets through GitHub, npm's registry and Sigstore's root of trust alone. It reads the action.yml of each engine's action at the commit that the workflows pin, and every input that a step passes and every output that a job reads must be there. It installs each engine's command line at its pinned version, as analyze does: the Gemini CLI and the Copilot CLI from their lock files, whose every package must have a valid signature of npm's registry, and a copy of the lock file with another hash of the command line must fail. Every flag of its step must be in its help, with its value, and every key of the Gemini CLI's settings in the schema of that version. Then each runs without a key, in a folder like that of analyze, and must stop at the missing key alone, with its settings taken. A renamed input, flag or key, a signature that doesn't verify or a hash that npm takes fails the check, which names it, before a release brings the new version to every repository.How the names of the engines are checked: A pull request that changes the step of an engine or a lock file of engines/, such as an update of Renovate, or one that changes what the check reads, and a run every week start the job engine contracts of the Action test. It holds no secret, and its block lets through GitHub, npm's registry and Sigstore's root of trust alone. It reads the action.yml of each engine's action at the commit that the workflows pin, and every input that a step passes and every output that a job reads must be there. It installs each engine's command line at its pinned version, as analyze does: the Gemini CLI and the Copilot CLI from their lock files, whose every package must have a valid signature of npm's registry, and a copy of the lock file with another hash of the command line must fail. Every flag of its step must be in its help, with its value, and every key of the Gemini CLI's settings in the schema of that version. Then each runs without a key, in a folder like that of analyze, and must stop at the missing key alone, with its settings taken. A renamed input, flag or key, a signature that doesn't verify or a hash that npm takes fails the check, which names it, before a release brings the new version to every repository.

  1. The actions. scripts/engine_contract.py actions finds every step whose id is the key of an engine in the workflows of .github/workflows/, and reads the action.yml of its action at the commit that the step pins through GitHub's API. Every input that the step passes must be there and not deprecated, every input that the action requires must be passed, and every output of the step that its job reads, such as structured_output or final-message, must be there. The Codex action must still say that it hands permission-profile to Codex as default_permissions of its configuration, which the next step checks.
  2. The command lines. clis installs Codex at the version of codex-version, without the scripts of its packages, as its action does, and the Gemini CLI and the Copilot CLI from their lock files in engines/ with npm ci, as analyze does. For those two, npm audit signatures must verify the signature of npm's registry of every package that npm installed from the lock file on GitHub's Ubuntu x64 runner, which is every package that analyze installs, and the attestations of the provenance of those that have them, and npm ci must refuse a copy of the lock file with another hash of the command line, with EINTEGRITY: a lock file that npm installed anyway would no longer bind analyze to the packages that this check verified. Every flag of the command line that analyze or the action builds must be in the --help of that version, with each value that the step gives it among those that the help lists, such as each level of effort. Every key of the Gemini CLI's settings must be in the JSON schema of the settings at the tag of that version, in the repository that its package names; without a schema, the check fails.
  3. A run without a key. The Gemini CLI and the Copilot CLI run as analyze runs them, with a home of their own in a folder of the runner's temporary folder and their settings where analyze writes them, and must stop asking for their secret, never at an argument, and without naming the file of their settings: the Gemini CLI names it only to refuse it or to say that it skips it, as it does for system settings that root doesn't own. The first run of this check found that it did so on GitHub's runners, so analyze now writes them with sudo into a folder of root's below /etc/issue-assistant. Codex checks its key only at OpenAI, whose host the block stops, so it runs with --strict-config, which fails on a key of its configuration that it doesn't know, and passes when it starts its turn.

The job runs on a pull request that changes a workflow with the step of an engine, a lock file of engines/, issue_assistant.py, action.yml, the script, the Action test or .github/renovate.json5, so on every update of an engine that Renovate opens, every Thursday, by the schedule of the Action test, and by hand. Its block lets through api.github.com, github.com, registry.npmjs.org and tuf-repo-cdn.sigstore.dev, Sigstore's root of trust, which npm reads for the attestations, alone, and no command line sees the token of the job. A part that fails names the workflow, the step and the input, or the version and the flag or the key, and the PR coach explains it from .github/checks.project.toml. It is a required check, and it reports on every pull request: on one that changes no engine and nothing that it reads, it passes without running. So Renovate merges a patch version of an engine on its own once it passed, in a pull request of its own, while a minor or major version waits for the maintainer, as decision 0020 says. What the check can't see is the behaviour behind a name that stays: the pull request of Renovate still asks to read the release notes before the next release, as the security design says.

How the mutation tests test the tests

A test that runs a line doesn't yet check it. Every Saturday the Mutation workflow changes issue_assistant.py in thousands of small ways, each a mutant, such as < for <= or another string, and runs the tests that call the changed function against each: a mutant that no test notices shows a line that the tests run but don't check. mutmut does this in a copy of the repository, mutants/, in which each function of the script has every one of its mutants beside it.

How the weekly mutation run tests the tests: In every pull request, tests/test_mutation_safety.py fails for a test that reads issue_assistant.py as text without the marker mutation_unsafe, and names the fix. Every Saturday the Mutation workflow copies the repository into mutants/, where each function of the script has its mutants beside it, and runs the tests there without the marked ones, first without any mutant; a test that fails then stops the whole run. Then each mutant runs against the tests that call its function, and the report lists the mutants that no test noticed.How the weekly mutation run tests the tests: In every pull request, tests/test_mutation_safety.py fails for a test that reads issue_assistant.py as text without the marker mutation_unsafe, and names the fix. Every Saturday the Mutation workflow copies the repository into mutants/, where each function of the script has its mutants beside it, and runs the tests there without the marked ones, first without any mutant; a test that fails then stops the whole run. Then each mutant runs against the tests that call its function, and the report lists the mutants that no test noticed.

  • The tests pass in the copy first. Before any mutant, the run takes every test in mutants/ once, with -m "not mutation_unsafe"; a test that fails there stops the whole run, which then measures nothing. A test that reads the script as text, such as the one that collects every GraphQL document of the script, sees each mutant in that copy, whichever one runs, so it can't notice one and fails without any. It carries @pytest.mark.mutation_unsafe, and a check of behaviour that it makes as well is a test of its own, which the mutants run against.
  • A guard finds an unmarked one in every pull request. tests/test_mutation_safety.py reads every test with Python's ast and fails for one that reads issue_assistant.py with read_text(), open() or inspect.getsource(), itself or through a helper, a fixture or a constant of the same module, without the marker, and names the fix. It doesn't follow the shared helpers of conftest.py and support.py, so these never read the script as text. So the weekly run doesn't learn of it days later.
  • The mutants change logic, not help. mutmut leaves the constants of a module alone and copies a whole function for each of its mutants. The commands of the script and the help of each command and option are the tables SCRIPT_COMMANDS and SCRIPT_OPTIONS, from which command_line() builds the parser, and main() hands each command to a function of its own, such as apply_command(). A mutant of main() is a copy of 66 lines, not of 333, whose 949 mutants made almost a third of the 43 MB of the copy, and none changes a text of the help, which no test needs to check.

When a check of this project fails

The checks of this project alone, the three jobs of the Action test, the check of the Workflow test and the fuzzing of fuzz/, explain themselves as the checks of the blueprint do. Each job of the Action test is made of parts, such as the workflows of the templates, their security and the set-ups of the sample repository and of this repository, and each part runs also when one before it failed, so that the check reports every failure at once. Its last step, Explain the failure, writes the explanation of each part that failed, from .github/checks.project.toml, into the summary of the check and as its annotation: what is wrong, why it matters, how to fix it, and the commands that run the part on your machine. After the checks, the PR coach of the blueprint keeps one comment on the pull request with these explanations, also on a pull request from a fork, whose own runs can't write to it.

How a failed job of the Action test explains itself: Each part of a job of the Action test runs also when one before it failed. When a part failed, the last step of the job asks blueprint.py for the explanation of each part that failed, from .github/checks.project.toml, and writes it into the summary of the check and as its annotation. After the checks, the PR coach reads the annotations from the API and keeps one comment on the pull request up to date, also on a pull request from a fork, with what failed, why it matters and the commands that run the part on your machine.How a failed job of the Action test explains itself: Each part of a job of the Action test runs also when one before it failed. When a part failed, the last step of the job asks blueprint.py for the explanation of each part that failed, from .github/checks.project.toml, and writes it into the summary of the check and as its annotation. After the checks, the PR coach reads the annotations from the API and keeps one comment on the pull request up to date, also on a pull request from a fork, with what failed, why it matters and the commands that run the part on your machine.

No command can fix what these checks find on its own: a template, a request to GitHub or an input that fuzzing found needs a person, so each explanation says why in its autofix, and the fix bot leaves the check alone. tests/test_coach.py keeps the explanations true to the workflow: every job ends with that step, every step after the checkout belongs to one part, every job and every part has its explanation, and the commands of a part are those that its steps run.