The flaky-test bot reports only a failure of the code¶
Context¶
On 2026-11-22 the flaky-test bot opened #1090 for a run in which no test had failed: in the first attempt of run 45004845410 of CI (python), scripts/check.sh had passed, and the step Keep the coverage report failed with Failed to CreateArtifact: Unable to make request: ETIMEDOUT. The client of GitHub's artifact service in actions/upload-artifact v7.0.2 tries a 5xx or a 429 five times, but throws a network error such as ETIMEDOUT or ECONNRESET at once, so a step that only keeps a report failed a required check, and the issue asked the maintainer to fix a test that didn't fail (#1095).
The bot counted every failure as a test unless it knew an exception: a first attempt that a newer run cancelled, a job cancelled without its time limit (0014), a job that ran out of time after every step had passed, one that Harden-Runner stopped (0134), and one that says CHECK_UNAVAILABLE's words or only sums up other jobs. A failed checkout, a failed setup-uv and a runner that hung after its work all got rows. Of the 294 rows in 26 issues of six repositories, from 2026-10-08 to 2026-11-22, 80 were such failures, 77 of them the hangs of October that 0088 fixed. And the exceptions for CHECK_UNAVAILABLE and for a job that sums up worked on no real job: the PR coach's explanation, which every failed job of a check has, counted as an error of the job's own.
The maintainer decided on 2026-11-22: when a job that failed at first and passed when it ran again has no failed test, a failure that is foreign to the code opens no issue for him; asked about a foreign failure that keeps coming back, he chose that it is reported when it came back for the third time in 7 days.
Options¶
continue-on-erroron the uploads: a report that a second try would have kept is lost, and every other foreign failure still opens the issue.- A list of the messages of the artifact client, or of the names of actions and steps that are foreign: a list that lags behind every new action and every new wording, and a test that prints
ETIMEDOUTwould count as foreign. - Close the issue after a time without a flake: a test that flakes once a month disappears with its history.
- A second try of each upload in a required check, and a bot that tells the code from what is outside it by where the job broke, as the runner's own lines of the log show it, with a threshold for a foreign failure that keeps coming back.
Decision¶
Option 4:
- The uploads: every upload of an artifact in a required check tries once more after half a minute, with the same action at the same commit, the same inputs and
overwrite: true, since a first try that lost its answer may have left the artifact behind.actions/upload-pages-artifactoverwrites nothing, so its second try takes the namegithub-pages-again, which the job docs gives as its outputpagesand the job deploy takes for its download and foractions/deploy-pages. A second failure of the coverage report, the report of the taint analysis, the website of a pull request and the fixes of docs on main warns and passes, since only a report or a bot reads them and the next run writes them again; the outputtaintof check (php) counts either try. Every other upload, such as the fixes of a job that has failed already or the website of GitHub Pages, fails the job as before. - Today's exceptions come first, unchanged, but the PR coach's explanation of the check is no error of the job's own.
- A sign of the code keeps the row: a test, in an annotation of
blueprint/annotate.pyor in the log; a step ofscripts/check.shthat stopped, which the PR coach names in a notice; a command that exited; a finding, a failure at a file of the repository that doesn't carry CHECK_UNAVAILABLE's words. - Otherwise the section of the log where the job broke decides, as the runner opens it:
##[group]Runfor a step, withshell:in its group for a command, andPost job cleanup.for a post step. The sections of the PR coach and of each step whose##[error]said why it failed don't count, with their exits. The job broke in the section with##[error]The operation was canceled., else in that of its first##[error]. A command,./and an action of the repository itself, and any other header are the code; an action of another repository, a post step and the set-up of the job are outside it, and so is a log without an error left that ends in the runner's Cleaning up orphan processes. A log without a##[group]Runcounts as none; without one, a job that ran out of time without a step that broke and with no annotation but its time limit broke in its runner when GitHub answers 404, and any other job is outside the code only when each step that broke is GitHub's name of an action of another repository,Run owner/repo@ref. - A foreign failure runs again as before, but gets no row and no comment; a notice and the section Outside the code of the summary name it. When a job of the same name without the values of its matrix broke in the same steps in two runs of the same workflow of the 7 days before that passed at a later attempt, with today's exceptions applied, the third time gets a row with
outside the code. The look-back reads 10 pages of 100 runs at most, so that the report ends within its 5 minutes, and says how far it reached when it stopped there. - The issue says what it holds: its first sentence names the foreign failures that came back, and a hidden mark in its body tells an issue of this bot from one of an older bot. A run without a row judges each row of the comments of an unmarked issue once, 20 at most, by its job; when none is the code's, the bot closes the issue with a comment, else it writes the mark and the new sentence and leaves the counts as they are.
- What says why:
.github/download.shuses CHECK_UNAVAILABLE's words when GitHub kept failing a download, and each step Explain the failure passes withoutblueprint.py, as after a checkout that failed, so that its exit doesn't look like a command of the code without a log.
Consequences¶
- The issue of flaky tests holds what the maintainer can fix: of the 39 rows since 2026-11-08, #1090 and one runner that hung after its work drop out; of the 100 rows of jobs that ran out of time, 23 stay, each with a log that shows a command of the tests.
- A foreign failure that stays, such as a mirror that fails or a pin that expired, reaches the maintainer on its third run in 7 days; the first two of each kind stay quiet. The look-back compares names and steps only, so a failure of the code with the same key counts as well, which can make it fire sooner, never later.
- A step can print
::group::Run a/b@cand so open a section that looks like an action of another repository. Only the code of the repository or of a pull request can do that; it hides a row of the issue and changes no check, so the bot doesn't defend against it. - Tests that run inside an action of another repository, such as
docker/build-push-action, would count as foreign; none of the watched workflows of the fleet has one, and the threshold still brings such a failure back. - A failure outside the code inside a command, such as a download of fonts, Playwright or Packagist that times out, stays a row unless the command says CHECK_UNAVAILABLE's words: a command is the project's code.
- A job of a project that only sums up others writes
this job only sums up the jobs that it needsin its error, or its exit keeps a row next to the job that failed; the changelog says so, and how a project's own upload gets the same pair of tries.