Skip to content

The canary and the Workflow test

← Architecture

The canary

The tests replace the engine, and the Action test runs everything but it, so the path that matters most, from an issue through the engine to the checked answer, ran only when someone opened an issue: here that is rare, and every repository that uses the action runs it with its own secret. A renamed output of the engine's action, a host that the firewall blocks, an expired secret, or a secret or variable that no longer reaches the called workflow would show as issues without an answer, in every repository at once. The Canary, .github/workflows/canary.yml, runs the whole path every Monday, after every change on main of what it runs, and by hand:

How the Canary runs the path of an issue through the AI: The Canary starts every Monday, after a change on main of the script, the prompts, the action, the reusable workflow, the firewall policy, the settings or the canary's issue, and by hand. It calls the reusable workflow as every repository does, with the permissions of the template, the engine's secrets passed by name and the input canary. Its job event takes the canary's issue and sets no labels; outside this repository, or when the called workflow doesn't see the variable ISSUE_ASSISTANT_AI set to on, the canary fails. Its job analyze runs on the firewall runner with the engine's secret of the environment issue-assistant, context reads the issue of tests/fixtures/canary-issue.json instead of one on GitHub, and the engine answers with JSON. Without the secret or without an answer, the canary fails. Its job apply checks the answer as for a real issue, as a dry run: an answer that breaks a check fails the canary, and one that passes shows the comment it would post in the summary. Beside the call, its job the AI's wait for its runner reads the jobs of the run until analyze starts: it warns when analyze waited for more than 10 minutes, and when it waits for more than 30, the canary fails and opens the issue of the bot-health bot at once.How the Canary runs the path of an issue through the AI: The Canary starts every Monday, after a change on main of the script, the prompts, the action, the reusable workflow, the firewall policy, the settings or the canary's issue, and by hand. It calls the reusable workflow as every repository does, with the permissions of the template, the engine's secrets passed by name and the input canary. Its job event takes the canary's issue and sets no labels; outside this repository, or when the called workflow doesn't see the variable ISSUE_ASSISTANT_AI set to on, the canary fails. Its job analyze runs on the firewall runner with the engine's secret of the environment issue-assistant, context reads the issue of tests/fixtures/canary-issue.json instead of one on GitHub, and the engine answers with JSON. Without the secret or without an answer, the canary fails. Its job apply checks the answer as for a real issue, as a dry run: an answer that breaks a check fails the canary, and one that passes shows the comment it would post in the summary. Beside the call, its job the AI's wait for its runner reads the jobs of the run until analyze starts: it warns when analyze waited for more than 10 minutes, and when it waits for more than 30, the canary fails and opens the issue of the bot-health bot at once.

  • It calls the reusable workflow as every repository does. Its job is the call of the template templates/.github/workflows/issue-assistant.yml, with the same name, the same permissions and the engine's secrets passed by name, which tests/test_templates.py keeps equal to the template's. Only the workflow differs, ./.github/workflows/reusable-issue-assistant.yml of the commit, and the input canary. So the run takes the path of production: the job that calls can't name an environment, and the job analyze gets the value of the secret from the environment issue-assistant; the called jobs see the variable ISSUE_ASSISTANT_AI of the repository; and the engine runs on the firewall runner, with the steps and pins of every repository. Only with canary do the engines also accept a run that a bot started, as the canary's own runs after a push are.
  • The issue exists nowhere. tests/fixtures/canary-issue.json holds an issue of the bug report form of this repository and a comment, as GitHub's REST API gives them. With canary, the environment variable ISSUE_ASSISTANT_FIXTURE gives them to context and apply in place of an issue on GitHub; the other issues, the discussions, the labels and the latest release come from GitHub, as for a real issue. The job event takes the canary's issue in place of that of an event and sets no labels, and with a fixture apply is a dry run whatever dry-run says, so nothing reaches an issue, although the token of apply may write, as in every repository. A test keeps the issue fitting the form and the labels of this repository.
  • Where a repository's run skips the AI, the canary fails. The job event fails when the called workflow doesn't see ISSUE_ASSISTANT_AI set to on, which the settings bot sets here from .github/repository.toml, and the job analyze fails without the engine's secret or without an answer. A repository's run would skip analyze or apply there, and the canary would stay green.
  • Only this repository passes canary. The job event fails a call with canary from another repository, which has no canary's issue. Every step and value of the canary in the reusable workflow names inputs.canary, which every other caller leaves false, and a test keeps the job analyze of the discussions equal to that of the issues without them.
  • The repository of the caller is that of the workflow. The Canary calls the workflow of its own repository, so it shows that the secret and the variables reach a called job, not whose they are when the two repositories differ: that a called job sees the caller's rests on GitHub's documentation, as decision 0015 says.
  • The firewall policy is this repository's. The runner enforces .github/egress-firewall.yaml, the blueprint's copy of the template of the release that it pins, so a change of templates/.github/egress-firewall.yaml reaches the canary with the blueprint's next update.
  • Only main runs it. Only the default branch may use the environment issue-assistant, so no pull request runs the canary. A minor or major version of the engine's action waits for the maintainer in a pull request of its own, and a patch version merges on its own once the check engine contracts passed, as .github/renovate.json5 says. The push of either merge starts the canary with the new version; the maintainer merges the next release only once the canary passed, and a failed update is reverted. A release of dependency updates alone merges itself, which decision 0020 names as a residual risk. The engine's action runs for a schedule or a run by hand, not for a push, so after a push the job start, whose token may only start workflows, starts the canary as a run by hand and waits for its result.
  • It measures the wait for the firewall runner. Its job the AI's wait for its runner runs scripts/canary_queue.py beside the call and reads the jobs of its own run until analyze starts. The wait counts from the end of the job event, which analyze needs, so neither the event nor an earlier canary counts; the summary shows it, with a warning above 10 minutes and a failure above 30, also while analyze still waits, which the PR coach explains from .github/checks.project.toml. On main the job then opens the issue of the bot-health bot for the Canary at once, since the run, and with it GitHub's mail and bot-health's look, ends only when analyze does.
  • A red canary is seen. GitHub mails the failure of a scheduled run to whoever last changed its schedule, here the maintainer. A run that the job start started has nobody to mail, so that job fails with it, and GitHub mails the failure of the push's run to the person who merged; a canary that a later push's canary replaced fails nothing. Every red run also shows under Actions.

The Workflow test

The tests and actionlint only read the reusable workflows, the Action test runs the action by itself, and the Canary calls only the workflow of the issues, only on main and with the engine's secret. So nothing ran the jobs that every repository calls on a pull request: a wrong needs, if, output or permission would show first in those repositories, after a release, often without a red run, since a job that its condition skips passes. The Workflow test, .github/workflows/workflow-test.yml, calls each of them on every pull request that changes what they run, and on every push to main; "Re-run all jobs" starts it again:

How the Workflow test runs the reusable workflows: Every pull request and every push to main start the Workflow test. Its first job asks GitHub for the files of a pull request: one that changes nothing that the dry runs run or read skips the calls. Otherwise one job calls each reusable workflow of the commit as a repository does, with the permissions of its template and as a dry run, also from a fork with a token that can only read. Before its checkouts, every job of a reusable workflow stops when GitHub names no commit of the workflow. The labels, the findings and the lifecycle run in full; the issues run with the input smoke, whose job event takes the canary's issue, whose analyze is skipped and whose apply checks the answer of tests/fixtures/smoke-answer.json; the discussions get no event of theirs and skip every job. The last job, the required check reusable workflows, reads the jobs of the run: each one passed with the step that writes the plan of its dry run, or was skipped where its call has to skip it, and it fails on every other.How the Workflow test runs the reusable workflows: Every pull request and every push to main start the Workflow test. Its first job asks GitHub for the files of a pull request: one that changes nothing that the dry runs run or read skips the calls. Otherwise one job calls each reusable workflow of the commit as a repository does, with the permissions of its template and as a dry run, also from a fork with a token that can only read. Before its checkouts, every job of a reusable workflow stops when GitHub names no commit of the workflow. The labels, the findings and the lifecycle run in full; the issues run with the input smoke, whose job event takes the canary's issue, whose analyze is skipped and whose apply checks the answer of tests/fixtures/smoke-answer.json; the discussions get no event of theirs and skip every job. The last job, the required check reusable workflows, reads the jobs of the run: each one passed with the step that writes the plan of its dry run, or was skipped where its call has to skip it, and it fails on every other.

  • Each call is that of its template. One job calls each reusable workflow of the commit, ./.github/workflows/reusable-*.yml, with the permissions of its template in templates/ and dry_run, which tests/test_workflow_test.py keeps equal. On a pull request from a fork GitHub gives every job a token that can only read, which a dry run needs; on one of this repository a token that may write, so a test makes sure that every command of the action that writes gets the dry_run of its call.
  • The labels, the findings and the lifecycle run in full. sync-labels, findings and sweep read this repository as every repository's morning run does, and write nothing. A dry run also waits for no run of its group and replaces none, so the pull requests don't queue behind each other or behind the real sweep. An open alert of code scanning on main fails the dry run of the findings, as it fails the Findings workflow there, which reports it; since a pull request can't close it before its merge, the check passes then, with a warning, as long as open alerts are all that its job names.
  • The issues run without the AI. The input smoke, which only this repository may pass, is the counterpart of the Canary's: the job event takes the issue of tests/fixtures/canary-issue.json and sets no labels, analyze is skipped, and apply checks the answer of tests/fixtures/smoke-answer.json in place of the engine's, as a dry run whose summary shows the comment. A test runs apply with this answer for that issue, so a change of the checks that refuses it fails before the workflow does. The engine, its secret and its runner stay the Canary's.
  • The discussions skip. A pull request is no event of theirs, and their job event runs only for a new discussion or a run by hand, so every job of the call is skipped.
  • Every job checks the commit of its workflow first. A reusable workflow can't pin the commit that it is part of, so each job checks out the issue assistant of the commit that GitHub names in its context, job.workflow_repository and job.workflow_sha (decision 0015). Without them, actions/checkout would take the caller's own repository at its default branch, and the job would run the caller's action.yml, or fail with a puzzling error. The step before the checkouts stops the job unless GitHub names a repository and a full commit SHA, and says why; a test keeps it in every job of every reusable workflow.
  • The last job is the check. A job that its condition skips passes as well as one that runs, so the job reusable workflows, the check that main requires, reads the jobs of its own run with scripts/workflow_test.py: each job passed, with the step that writes the plan of its dry run, or was skipped where its call has to skip it. Its summary shows a table of every job, and on a failure the PR coach explains it from .github/checks.project.toml. A pull request that changes nothing that the dry runs run or read, such as one of the documentation alone, skips the calls, and the check passes in seconds.