Find failing, slow and flaky CI

Anvil records every CI run and job from GitHub Actions and from repositories hosted on Anvil, across all your repositories. This guide shows where to look when CI fails, slows down or flips between passing and failing.

To run jobs on Anvil's machines, see Quickstart: CI runners.

Run history #

CI → Run history lists every job, newest first, with its status, repository, branch, commit, who triggered it, the event (push, pull request, schedule, manual), and how long it took compared with the slowest job in view.

Filter by status, event, user, repository, job and branch in the left column. Each filter shows counts, and the long lists are searchable. The time range at the top (1 hour to 30 days) applies to every page and is kept as you move around. Above the list, the run distribution chart shows how many runs succeeded and failed over time, with the average duration.

The overview page adds the headline numbers for the range: jobs, pass rate, median job duration and queue time, flaky jobs and runner minutes.

Failure diagnosis on the PR #

When a job fails on a commit that's the head of an open pull request, Anvil explains it in the PR. It waits 90 seconds so the other jobs of the run can finish, then for up to six failed jobs it reads:

• the end of the job's log (the last 250 lines, without colour codes and timestamps);
• how often that job failed in the last 30 runs in the repository;
• whether the same job passed on the same commit in another attempt;
• the code, in a fresh checkout of the PR.

It posts one comment per pull request, headed "CI failed on <commit>", and edits it on later failures. For each failed job you get the kind of failure, the file and line when it can tell, the cause, the fix, the relevant log lines, and a note when the job was seen passing on the same commit. If a re-run has passed by the time it looks, it says nothing.

Turn it off for a repository with ci.analyze_failures: false in .anvil.yaml. Each diagnosis is capped at $1.

Anvil also sees CI results when it reviews: the job results recorded so far for the head commit are part of its input.

Flaky jobs #

A job is flaky when it both failed and passed on the same commit, usually after a re-run. CI → Flaky jobs lists them with the repository, how many commits were affected, the failure rate and when it last happened. When a flaky job fails on a pull request, the failure diagnosis says so instead of sending you to look for a bug.

Detection works on whole jobs here. For individual tests, see Tests and flaky tests.

Job pages and logs #

Click any job in Run history to open its page. It shows the job's steps on a timeline, how long its last 30 runs took, the other jobs of the same run, failed tests when the run uploaded a test report, and the log.

Anvil fetches a job's log when the job finishes and keeps it for 30 days. Very long logs are cut in the middle to 5 MB (the start and the end, where failures are, stay). In the log:

• each step is a fold; failed steps open by default, and ##[group] blocks fold inside them;
• colours from the tools are kept;
• click a line number to link to it (#L120), shift-click a second one for a range;
• Search this log finds every line containing the text, opens the steps it's in, and moves between hits with Enter and Shift+Enter;
• the download button saves the log as a text file.

Older jobs without a stored log are fetched when you open them, as long as the host still has the log.

Log search #

CI → Log search searches the stored logs of every job you can see, in the time range at the top. Every word must appear; put phrases in quotes. Narrow it with filters:

For example, status:failure ECONNRESET or job:e2e "Timed out". Each result shows the first matching line; clicking it opens the job at that line with the search filled in. A search covers the newest 40 matching jobs.

Tests and flaky tests #

Anvil reads JUnit XML test reports from a run's artifacts when the run finishes. Most test runners can write JUnit: Vitest (--reporter=junit), Jest (jest-junit), pytest (--junitxml), Go (gotestsum --junitfile), Maven and Gradle. Upload the file as an artifact whose name starts with junit or test-results; name it junit-<job name> when a run has several test jobs, so results land on the right job:

- run: npx vitest run --reporter=default --reporter=junit --outputFile=junit.xml
- uses: actions/upload-artifact@v4
  if: always()   # upload the report when tests fail too
  with:
    name: junit-${{ github.job }}
    path: junit.xml

CI → Tests lists:

• Flaky tests: a test that passed and failed on the same commit, or that keeps flipping between passing and failing (three or more changes across its recent runs while mostly passing). Each shows why, its last 20 results and its failure rate.
• Failing tests that aren't flaky: broken until someone fixes them.
• Slowest tests, by average time.

Click a test for its history: every result with the commit, branch, time and failure message. Test results are kept for 90 days. Failed tests also appear on the job's page.

Machine metrics #

For jobs on Anvil runners, the runner manager records the container's CPU and memory every 10 seconds. The job page charts both against the size's limits and gives a sizing hint from the peaks with 20% headroom, for example "Peaked at 1.4 CPU and 2.1 GB; anvil-2cpu would do.", or suggests a bigger size when the job ran out of CPU or came close to its memory limit. Jobs on the runners for hosted repositories don't have metrics yet. While a job runs, its chart updates every 10 seconds.

CI → Compute shows CPU and memory across all jobs: running jobs live, the runner pool's capacity, and compute history by repository and runner size. See Watch CPU and memory.

Monitors #

CI → Monitors sends you a notice when something goes wrong in a repository's CI. Set them up per repository:

• CI fails on the branch: any job on the watched branch (main by default) fails. One notice per commit.
• A job gets slower: a successful job on the branch took more than the threshold (50% by default) longer than its median over its last 20 successful runs. It needs five earlier runs, and ignores jobs under a minute and slowdowns under 30 seconds.
• A test turns flaky: from test reports, as above. One notice per test.

Notices appear at the top of the Monitors page. With Email too, each one is also emailed to the addresses you list, or to you if the list is empty. The quiet period (30 minutes by default) stops a burst of failures from sending one email each: notices of the same kind within it are only shown in Anvil.

Runner sizing #

CI → Rightsize runners groups jobs by their runs-on label and shows, for each: the number of jobs, the duration at the median and the 95th percentile, the queue time at the same percentiles, and the runner minutes used. Each row gets a plain recommendation, for example "Jobs wait 3m 10s at p95. Add capacity to this pool." or "Sized well."

Below that, Runners shows each runner's job count and average duration and queue time, and Heaviest jobs lists the jobs that use the most runner minutes, with their label: the first candidates for a bigger or smaller runner.

Runners for hosted repositories #

A workflow that runs on GitHub's ubuntu-latest usually runs here unchanged:

# .gitea/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm test

CI for invited users' repositories is switched on per repository after the admin has looked at it, because jobs currently share the host's Docker. See Security and data.