Zum Inhalt

The canary and the Workflow test

← Architecture

The canary

The tests replace the engine, and the Action test runs everything but it, so the path that matters most, from an issue through the engine to the checked answer, ran only when someone opened an issue: here that is rare, and every repository that uses the action runs it with its own secret. A renamed output of the engine's action, a host that the firewall blocks, an expired secret, or a secret or variable that no longer reaches the called workflow would show as issues without an answer, in every repository at once. The Canary, .github/workflows/canary.yml, runs the whole path every Monday, after every change on main of what it runs, and by hand:

How the Canary runs the path of an issue through the AI: The Canary starts every Monday, after a change on main of the script, the prompts, the action, the reusable workflow, the firewall policy, the settings or the canary's issue, and by hand. It calls the reusable workflow as every repository does, with the permissions of the template, the engine's secrets passed by name and the input canary. Its job event takes the canary's issue and sets no labels; outside this repository, or when the called workflow doesn't see the variable ISSUE_ASSISTANT_AI set to on, the canary fails. Its job analyze runs on the firewall runner with the engine's secret of the environment issue-assistant, context reads the issue of tests/fixtures/canary-issue.json instead of one on GitHub, and the engine answers with JSON. Without the secret or without an answer, the canary fails. Its job apply checks the answer as for a real issue, as a dry run: an answer that breaks a check fails the canary, and one that passes shows the comment it would post in the summary. Beside the call, its job the AI's wait for its runner reads the jobs of the run until analyze starts: it warns when analyze waited for more than 10 minutes, and when it waits for more than 30, the canary fails and opens the issue of the bot-health bot at once.How the Canary runs the path of an issue through the AI: The Canary starts every Monday, after a change on main of the script, the prompts, the action, the reusable workflow, the firewall policy, the settings or the canary's issue, and by hand. It calls the reusable workflow as every repository does, with the permissions of the template, the engine's secrets passed by name and the input canary. Its job event takes the canary's issue and sets no labels; outside this repository, or when the called workflow doesn't see the variable ISSUE_ASSISTANT_AI set to on, the canary fails. Its job analyze runs on the firewall runner with the engine's secret of the environment issue-assistant, context reads the issue of tests/fixtures/canary-issue.json instead of one on GitHub, and the engine answers with JSON. Without the secret or without an answer, the canary fails. Its job apply checks the answer as for a real issue, as a dry run: an answer that breaks a check fails the canary, and one that passes shows the comment it would post in the summary. Beside the call, its job the AI's wait for its runner reads the jobs of the run until analyze starts: it warns when analyze waited for more than 10 minutes, and when it waits for more than 30, the canary fails and opens the issue of the bot-health bot at once.

  • It calls the reusable workflow as every repository does. Its job is the call of the template templates/.github/workflows/issue-assistant.yml, with the same name, the same permissions and the engine's secrets passed by name, which tests/test_templates.py keeps equal to the template's. Only the workflow differs, ./.github/workflows/reusable-issue-assistant.yml of the commit, and the input canary. So the run takes the path of production: the job that calls can't name an environment, and the job analyze gets the value of the secret from the environment issue-assistant; the called jobs see the variable ISSUE_ASSISTANT_AI of the repository; and the engine runs on the firewall runner, with the steps and pins of every repository. Only with canary do the engines also accept a run that a bot started, as the canary's own runs after a push are.
  • The issue exists nowhere. tests/fixtures/canary-issue.json holds an issue of the bug report form of this repository and a comment, as GitHub's REST API gives them. With canary, the environment variable ISSUE_ASSISTANT_FIXTURE gives them to context and apply in place of an issue on GitHub; the other issues, the discussions, the labels and the latest release come from GitHub, as for a real issue. The job event takes the canary's issue in place of that of an event and sets no labels, and with a fixture apply is a dry run whatever dry-run says, so nothing reaches an issue, although the token of apply may write, as in every repository. A test keeps the issue fitting the form and the labels of this repository.
  • Where a repository's run skips the AI, the canary fails. The job event fails when the called workflow doesn't see ISSUE_ASSISTANT_AI set to on, which the settings bot sets here from .github/repository.toml, and the job analyze fails without the engine's secret or without an answer. A repository's run would skip analyze or apply there, and the canary would stay green.
  • Only this repository passes canary. The job event fails a call with canary from another repository, which has no canary's issue. Every step and value of the canary in the reusable workflow names inputs.canary, which every other caller leaves false, and a test keeps the job analyze of the discussions equal to that of the issues without them.
  • The repository of the caller is that of the workflow. The Canary calls the workflow of its own repository, so it shows that the secret and the variables reach a called job, not whose they are when the two repositories differ: that a called job sees the caller's rests on GitHub's documentation, as decision 0015 says.
  • The firewall policy is this repository's. The runner enforces .github/egress-firewall.yaml, the blueprint's copy of the template of the release that it pins, so a change of templates/.github/egress-firewall.yaml reaches the canary with the blueprint's next update.
  • Only main runs it. Only the default branch may use the environment issue-assistant, so no pull request runs the canary. Every update of the engine's action merges on its own once the required checks of its pull request passed, engine contracts among them, as .github/renovate.json5 says, and the push of the merge starts the canary with the new version; the release waits for it, as the next section shows. The engine's action runs for a schedule or a run by hand, not for a push, so after a push the job start, whose token may only start workflows, starts the canary as a run by hand and waits for its result. GitHub may answer 404 for the run that it just named in the first moments after the dispatch, so the job asks for it every 5 seconds before it watches it; when GitHub still doesn't show it after a minute, the job ends with a warning that links to the run, rather than with a red run of main while the canary passes.
  • It measures the wait for the AI's runner. Its job the AI's wait for its runner runs scripts/canary_queue.py beside the call and reads the jobs of its own run until the job of the AI starts: analyze on the firewall runner, or, when the variable ISSUE_ASSISTANT_AI_RUNNER names another runner, analyze-in-reserve in its place, while the run skips the other one. The wait counts from the end of the job event, which both need, so neither the event nor an earlier canary counts; the summary shows it with the name of the job, with a warning above 10 minutes and a failure above 30, also while the job still waits, and with the advice that fits its runner, which the PR coach explains from .github/checks.project.toml. On main the job then opens the issue of the bot-health bot for the Canary at once, since the run, and with it GitHub's mail and bot-health's look, ends only when the job of the AI does.
  • A red canary is seen. GitHub mails the failure of a scheduled run to whoever last changed its schedule, here the maintainer. A run that the job start started has nobody to mail, so that job fails with it, and GitHub mails the failure of the push's run to the person who merged; a canary that a later push's canary replaced fails nothing. Every red run also shows under Actions, and keeps the release back.

The canary before the release

Nobody looks at an update of an engine before it merges: Renovate opens a pull request for each engine a week after its release, a minor or major version too, and it merges on its own once its required checks passed (decision 0030). So the release waits for the canary instead of a person who watches it:

How an update of an engine reaches the repositories: Renovate opens a pull request for each engine, a week after the release of its new version, a minor or major one too. Its required checks, engine contracts among them, run the engine without a key; when one fails, the update doesn't merge, and no other engine waits for it. When they pass, the update merges on its own. The push to main starts the Canary, which runs the AI's whole path with the engine's action and its secret, and the release bot updates its pull request. That pull request's required check, the Canary of main, waits while a Canary of main runs. When the latest one passed, the release merges and reaches every repository that uses the action. When it failed, the release waits, the bot-health bot opens an issue, and a revert of the update or another run of the Canary lets the release go.How an update of an engine reaches the repositories: Renovate opens a pull request for each engine, a week after the release of its new version, a minor or major one too. Its required checks, engine contracts among them, run the engine without a key; when one fails, the update doesn't merge, and no other engine waits for it. When they pass, the update merges on its own. The push to main starts the Canary, which runs the AI's whole path with the engine's action and its secret, and the release bot updates its pull request. That pull request's required check, the Canary of main, waits while a Canary of main runs. When the latest one passed, the release merges and reaches every repository that uses the action. When it failed, the release waits, the bot-health bot opens an issue, and a revert of the update or another run of the Canary lets the release go.

  • The release's pull request has a check of its own. The Canary of main, the job of .github/workflows/release-gate.yml, is a required check of .github/repository.project.toml. It runs on the pull request of the release bot alone, from its branch release-please--… of this repository; on every other pull request GitHub skips it, which counts as passed, without a runner.
  • It waits while a canary runs. scripts/release_gate.py reads the runs of the Canary on main every minute while one of them is unfinished, such as the one that the push of the latest change started, which the release bot's update of its pull request follows. Then the latest run decides: the check passes only when it passed. A run that a later one replaced never is the latest once that one ended, so a latest run that was cancelled is one that a person stopped, and the check fails for it, as for a push whose job start couldn't start the canary. After 100 minutes, the check fails and can run again.
  • A red canary keeps every release back. Not only one with the update of an engine: the release ships what is on main, and a red canary means that the path of an issue through the AI is broken there, or not shown to work. The summary of the check names the run and its commit, and the PR coach explains it from .github/checks.project.toml. A revert of the update, or a later one that fixes it, runs the canary again and updates the release's pull request, whose check then passes; a canary that failed for a moment needs another run of it, on Monday or by hand with gh workflow run canary.yml. The release bot's auto-merge then merges a release of dependency updates.
  • Every canary lets the release decide again. The check decides when the release's pull request gets a new head. The last job of every canary of main that runs the AI, the release decides again, runs scripts/release_gate.py again: when the check of the release's pull request ended with another verdict than this canary, a failure after a canary that passed or a pass after one that failed, it runs the check again, which waits until this canary has ended and decides on it. Nobody has to run the check by hand, and a canary that fails after the check passed holds the release back, unless it merged before.
  • The other engines aren't run with a key. The canary runs Claude, the engine of this repository. Codex, Gemini and Copilot rest on the week that every update waits and on engine contracts; decision 0030 names what stays a risk.

The Workflow test

The tests and actionlint only read the reusable workflows, the Action test runs the action by itself, and the Canary calls only the workflow of the issues, only on main and with the engine's secret. So nothing ran the jobs that every repository calls on a pull request: a wrong needs, if, output or permission would show first in those repositories, after a release, often without a red run, since a job that its condition skips passes. The Workflow test, .github/workflows/workflow-test.yml, calls each of them on every pull request that changes what they run, and on every push to main; "Re-run all jobs" starts it again:

How the Workflow test runs the reusable workflows: Every pull request and every push to main start the Workflow test. Its first job asks GitHub for the files of a pull request: one that changes nothing that the dry runs run or read skips the calls. Otherwise one job calls each reusable workflow of the commit as a repository does, with the permissions of its template and as a dry run, also from a fork with a token that can only read. Before its checkouts, every job of a reusable workflow stops when GitHub names no commit of the workflow. The labels, the findings and the lifecycle run in full; the issues run with the input smoke, whose job event takes the canary's issue, whose analyze and analyze-in-reserve are skipped and whose apply checks the answer of tests/fixtures/smoke-answer.json; the discussions run with the input smoke too, whose job event takes the discussion of tests/fixtures/smoke-discussion.json, whose analyze and analyze-in-reserve are skipped and whose apply checks the answer of tests/fixtures/smoke-discussion-answer.json. The last job, the required check reusable workflows, reads the jobs of the run: each one passed with the step that writes the plan of its dry run, or was skipped where its call has to skip it, and it fails on every other.How the Workflow test runs the reusable workflows: Every pull request and every push to main start the Workflow test. Its first job asks GitHub for the files of a pull request: one that changes nothing that the dry runs run or read skips the calls. Otherwise one job calls each reusable workflow of the commit as a repository does, with the permissions of its template and as a dry run, also from a fork with a token that can only read. Before its checkouts, every job of a reusable workflow stops when GitHub names no commit of the workflow. The labels, the findings and the lifecycle run in full; the issues run with the input smoke, whose job event takes the canary's issue, whose analyze and analyze-in-reserve are skipped and whose apply checks the answer of tests/fixtures/smoke-answer.json; the discussions run with the input smoke too, whose job event takes the discussion of tests/fixtures/smoke-discussion.json, whose analyze and analyze-in-reserve are skipped and whose apply checks the answer of tests/fixtures/smoke-discussion-answer.json. The last job, the required check reusable workflows, reads the jobs of the run: each one passed with the step that writes the plan of its dry run, or was skipped where its call has to skip it, and it fails on every other.

  • Each call is that of its template. One job calls each reusable workflow of the commit, ./.github/workflows/reusable-*.yml, with the permissions of its template in templates/ and dry_run, which tests/test_workflow_test.py keeps equal. On a pull request from a fork GitHub gives every job a token that can only read, which a dry run needs; on one of this repository a token that may write, so a test makes sure that every command of the action that writes gets the dry_run of its call.
  • The labels, the findings and the lifecycle run in full. sync-labels, findings and sweep read this repository as every repository's morning run does, and write nothing. A dry run also waits for no run of its group and replaces none, so the pull requests don't queue behind each other or behind the real sweep. An alert of code scanning on main that the Findings workflow there hasn't reported yet fails the dry run of the findings, as it fails that workflow, whose record in the cache the dry run reads and never writes; since a pull request can't close it before its merge, the check passes then, with a warning, as long as open alerts are all that its job names.
  • The issues run without the AI. The input smoke, which only this repository may pass, is the counterpart of the Canary's: the job event takes the issue of tests/fixtures/canary-issue.json and sets no labels, analyze and analyze-in-reserve are skipped, and apply checks the answer of tests/fixtures/smoke-answer.json in place of the engine's, as a dry run whose summary shows the comment. A test runs apply with this answer for that issue, so a change of the checks that refuses it fails before the workflow does. The engine, its secret and its runner stay the Canary's.
  • The discussions run without the AI too. A pull request is no event of theirs, so the call passes smoke to them as well: the job event takes the discussion of tests/fixtures/smoke-discussion.json, analyze and analyze-in-reserve are skipped, and apply checks the answer of tests/fixtures/smoke-discussion-answer.json as a dry run. As in a run by hand, apply answers it whatever this repository's categories; a test runs it too.
  • Every job checks the commit of its workflow first. A reusable workflow can't pin the commit that it is part of, so each job checks out the issue assistant of the commit that GitHub names in its context, job.workflow_repository and job.workflow_sha (decision 0015). Without them, actions/checkout would take the caller's own repository at its default branch, and the job would run the caller's action.yml, or fail with a puzzling error. The step before the checkouts stops the job unless GitHub names a repository and a full commit SHA, and says why; a test keeps it in every job of every reusable workflow.
  • The last job is the check. A job that its condition skips passes as well as one that runs, so the job reusable workflows, the check that main requires, reads the jobs of its own run with scripts/workflow_test.py: each job passed, with the step that writes the plan of its dry run, or was skipped where its call has to skip it. GitHub lists the steps of a job some time after the job ends, once for minutes (#288), so while it lists a job that has to run as not yet completed, or without the steps that it ran, the check reads the jobs again every 20 seconds, for up to five minutes, before it judges, and so it does after a read that may succeed when it comes again: one without an answer, or with an error of GitHub's servers, a timeout or a rate limit. A read that GitHub refuses with any other error of the client, such as 403 or 404, fails the check at once and names the status, GitHub's message and where to look (#327): every read again would repeat it. A job that ended earlier with at least one step listed, such as one that failed before its plan, fails the check at once. A job that the step Make sure that Harden-Runner blocks stopped shows as stopped by Harden-Runner; when only such stops failed, the check says so in the words of that step, and the flaky-test bot counts no flaky test (#392). Its summary shows a table of every job, and the PR coach explains a failure from .github/checks.project.toml. A pull request that changes nothing that the dry runs run or read, such as one of the documentation alone, skips the calls, and the check passes in seconds.

The runners that the action is tested on

The workflows of the templates run on ubuntu-24.04, yet a repository's own job may call the action on ubuntu-latest or on the free ARM runners. So the jobs action and egress of the Action test each run on ubuntu-24.04, ubuntu-26.04 and ubuntu-24.04-arm, side by side. The leg on ubuntu-24.04 keeps the name of its job and alone lints the templates. Another leg names its runner in its check, such as action (ubuntu-26.04), and its part other-runner says what to compare. check prefers PyYAML, so each leg of action also runs it once with PyYAML hidden, and the real yq of that image reads the set-up. A leg on another runner isn't a required check; one that fails on main gets an issue from the bot health.

The first step of action.yml compares python3 with the version of decision 0003, since the script may not get far enough to do so. The job old-python runs the action on Python 3.11 and passes only when it failed. It runs that step once more, to read that its output names the version and holds no traceback.