All plugins

testsift

Exact, local analysis of JUnit XML test reports (pytest --junitxml, Maven Surefire, Gradle, Jest/Vitest junit reporters, go-junit-report/gotestsum, PHPUnit, .NET, CI artifacts), from one file or a directory of them. test_report_summary gives exact pass/fail/error/skip counts, failures grouped by cause (messages that differ only in numbers, ids or quoted values share a group), the slowest tests and flaky suspects; test_report_failures lists failing tests with type, message, file/line and the start and end of the trace; test_report_compare diffs two runs into newly failing, fixed, newly skipped, still failing, added, removed and much slower tests. Streamed, standard library only: no network, no credentials, no third-party dependencies, no self-update; DOCTYPE and entity declarations are refused. Disclosure: reads the local report paths you pass it (read-only), only regular files, and honours Hermes' core read denylist.

rodricksz4h5Toolsv1.0.0Updated 7 days ago

About

Exact, local analysis of JUnit XML test reports (pytest --junitxml, Maven Surefire, Gradle, Jest/Vitest junit reporters, go-junit-report/gotestsum, PHPUnit, .NET, CI artifacts), from one file or a directory of them. test_report_summary gives exact pass/fail/error/skip counts, failures grouped by cause (messages that differ only in numbers, ids or quoted values share a group), the slowest tests and flaky suspects; test_report_failures lists failing tests with type, message, file/line and the start and end of the trace; test_report_compare diffs two runs into newly failing, fixed, newly skipped, still failing, added, removed and much slower tests. Streamed, standard library only: no network, no credentials, no third-party dependencies, no self-update; DOCTYPE and entity declarations are refused. Disclosure: reads the local report paths you pass it (read-only), only regular files, and honours Hermes' core read denylist.

testsift

Exact, local analysis of JUnit XML test reports, as three Hermes tools.

Every major test runner and CI system can write JUnit XML: pytest
(--junitxml), Maven Surefire/Failsafe, Gradle, Jest (jest-junit), Vitest
(--reporter=junit), Go (go-junit-report, gotestsum --junitfile), PHPUnit,
.NET (JunitXml.TestLogger), Playwright, Cypress. A real suite's report is
thousands of test cases with full stack traces, far too much to put in a model's
context, and "which tests broke and why" is exactly the question a coding agent
gets asked after a red CI run. testsift reads the report on disk and returns
exact answers.

Why a tool and not a prompt or skill

Three things a model can't do reliably over a file it can't read whole:

  • Exact counts across one report or a whole directory of them
    (target/surefire-reports/, a downloaded CI artifact).

  • Failures grouped by cause. 300 failing tests are often one or two root
    causes. Messages that differ only in numbers, ids, hashes or quoted values
    share a group (real output on a pytest report):

    failed: AssertionError: expected even, got <NUM>              x2
        test_sample::test_parity[3], test_sample::test_parity[17]
    error: setup: RuntimeError: database not reachable at <IP>    x1
        test_sample::test_uses_broken
    

    The cause is the failure message, or, when an emitter writes the same
    placeholder for every failure (Go's go-junit-report and gotestsum write
    Failed), the first line of the trace that isn't Go runner output
    (=== RUN, --- FAIL). pytest's failed on setup with "..." wrapper is
    unwrapped to the error inside it. Grouping is by text, so it is a strong hint
    about shared causes, not a proof.

  • Run-to-run diffs. Given a baseline run and a new run, which tests
    newly fail, which were fixed, which were already failing, what was added or
    removed and what got much slower. This separates "my change broke it" from
    "it was already red".

Tools

All three always return a JSON string; a bad path or an unreadable report
returns { "ok": false, "error": "..." } rather than raising. path (and
base/head) is a report file, or a directory searched recursively for
*.xml reports. XML that isn't a test report (pom.xml, coverage.xml) is
skipped and listed in files_skipped.

test_report_summary

argtyperequireddefaultmeaning
pathstringyes-report file or directory
max_groupsintegerno10failure groups to return (cap 100)
slowestintegerno10slowest tests to return (cap 100)

Returns tests, passed, failed, errors, skipped, duration_seconds,
failure_groups:[{signature,count,example_message,tests}],
suites_with_most_failures, slowest:[{test,seconds}] and flaky_suspects
(tests with Surefire flakyFailure/flakyError entries, i.e. failed and then
passed on a rerun, or the same test id both passing and failing in the input).

test_report_failures

argtyperequireddefaultmeaning
pathstringyes-report file or directory
containsstringno-only failures whose id, type or message contains this (case-insensitive)
include_outputbooleannofalseadd the tail of each failing test's system-out/system-err
offsetintegerno0failures to skip
limitintegerno20failures to return (cap 200)

Each failure has test (classname::name), status (failed or error),
suite, file/line when the report has them, type, message, detail
(the trace, as its first and last 2,000 characters when longer) and report.

test_report_compare

argtyperequireddefaultmeaning
basestringyes-baseline report file or directory
headstringyes-new report file or directory
limitintegerno50entries per list (cap 500)

Returns totals for both sides and newly_failing, added_failing, fixed,
newly_skipped (passed before, skipped now, which can hide a regression),
still_failing, added, removed and slower (at least twice as slow and at
least one second slower), each with a *_total count.

How results are counted

  • A test is identified by classname::name (or name alone when there is no
    classname). When the same id appears more than once, the worst status wins
    (error > failed > skipped > passed) and its times are added up.
  • A test case with both <failure> and <error> counts as an error.
  • Surefire reruns don't change the final status. flakyFailure/flakyError
    (failed, then passed) mark the test as a flaky suspect; rerunFailure/
    rerunError (every rerun failed too) are a consistent failure, reported as
    failed_reruns on that failure.
  • time attributes that aren't a non-negative number are ignored. A thousands
    separator is accepted only together with a decimal point (1,234.5), since
    1,5 is ambiguous.
  • A report cut off mid-write (a killed CI job) keeps the test cases before the
    break and is listed under files_with_parse_errors.

Safety and disclosure

  • Reads only the paths you pass, read-only. Never writes, deletes or runs
    anything.
  • Honours Hermes' read denylist. Inside the agent every file, including
    each file found in a directory, goes through the same check Core's
    read_file uses, so credential stores, internal caches and .env files are
    refused, also through symlinks. Like Core, the check runs on the path as
    given as well as on its absolute form, so Windows NT namespace paths
    (\??\, \\?\UNC\, \\.\, GLOBALROOT) are refused before anything
    touches them.
  • Regular files only. Paths under /proc and /dev (including symlinks
    that resolve there) are refused, and so is anything that isn't a regular
    file (FIFOs, sockets, devices), before it is opened. Symlinked directories
    are not followed during a directory walk.
  • Safe XML. Reports are parsed by a streaming expat parser that refuses any
    DOCTYPE or entity declaration, so entity expansion ("billion laughs") and
    external entities can't be used. JUnit reports never need either.
  • Bounded. Reports are streamed in 64 KB pieces. Each failure keeps at most
    4,000 characters of trace and 1,000 of output tail, however large the report.
    A call reads at most 5,000 files and 2,000,000 test cases; hitting either is
    flagged in the result.
  • No network, no credentials, no telemetry, no background processes, no
    self-update.
    Standard library only.

Install

hermes plugins install rodricksz4h5/testsift

License

MIT, see LICENSE.

README from the reviewed commit ebf734e; it updates when the author re-pins.

More by rodricksz4h5

8 plugins by rodricksz4h5 in the catalog.

rodricksz4h5covsiftExact, local analysis of code coverage reports: LCOV (c8, nyc, Jest, Vitest, cargo-llvm-cov, lcov), Cobertura XML (coverage.py, Jest/Vitest, coverlet, gcovr, PHPUnit), JaCoCo XML, Clover XML, SonarQube generic XML, coverage.py, gcovr and Coveralls JSON, and Go cover profiles, from one file or a directory of them. coverage_diff takes a unified diff (git diff) and returns the added lines no test ran, per file, with diff coverage and an optional fail_under gate; coverage_summary gives exact line, branch and Go statement coverage, the least covered files with their uncovered lines and a per-directory rollup; coverage_file lists one file's uncovered lines and partial branches; coverage_compare shows which files lost coverage between two runs. Standard library only: no network, no credentials, no subprocess (never runs git or tests; the diff is passed in), no third-party dependencies, no self-update; XML entity declarations are refused. Disclosure: reads the local report and diff paths you pass it (read-only), only regular files, and honours Hermes' core read denylist.Communitytoolsrodricksz4h5dataprobeExact, local structure and quality checks over large CSV/TSV and JSONL data files that don't fit in a model's context. data_profile reports per-column inferred type, null rate, distinct count, numeric statistics and top values (for JSONL, the inferred union schema with per-field presence); data_validate reports only the actionable problems — mixed-type columns with the offending line numbers, ragged rows, duplicate rows, non-unique key candidates, constant or all-empty columns and malformed JSONL lines; data_head returns a clean parsed preview that handles quoted CSV and JSON correctly. Streamed, standard library only: no network, no credentials, no third-party dependencies, no self-update. Disclosure: reads the local data path you pass it (read-only) and honours Hermes' core read denylist.Communitytoolsrodricksz4h5layersiftExact, local reading of saved container images: docker save and podman save tars (classic and Docker 25+ layouts) and OCI image archives, including multi-platform indexes and gzip, bzip2 or xz compression. image_summary says where the size goes: each layer with the Dockerfile instruction that made it, the final filesystem size, bytes still shipped after a later layer deletes or overwrites them (which layer added them, which removed them), caches and archives left behind (apt lists, apk/yum/dnf, pip/uv/npm/yarn, Go and Cargo caches, temp directories, .git, compilers) with the fix, and the biggest directories and files; image_layer opens one layer; image_compare compares two images before and after a Dockerfile change. Standard library only: no network, never runs docker or contacts a registry, no credentials, no subprocess, never extracts to disk, no third-party dependencies, no self-update. Disclosure: reads the local image archives you pass it (read-only), only regular files, and honours Hermes' core read denylist.Communitytoolsrodricksz4h5locksiftExact, local review of dependency lockfiles: npm package-lock.json, pnpm-lock.yaml, yarn.lock (v1 and Berry), uv.lock, poetry.lock, Pipfile.lock, Cargo.lock, composer.lock, Gemfile.lock and go.sum. lock_diff compares two versions of a lockfile (a branch against main) and returns the dependency changes instead of thousands of lines of churn, with supply-chain signals per installed copy: a source that moved (registry to git or URL, another registry host, a different tarball or commit at the same version, an alias or patch that changed, a tarball that belongs to another package), checksums changed, removed or weakened, new install scripts, new registries. lock_summary gives counts, packages at several versions, non-registry sources and install scripts; lock_why shows the dependency chains that pull a package in. Standard library only: no network, no credentials, no subprocess (never runs git or a package manager), no third-party dependencies, no self-update. Disclosure: reads the local lockfile paths you pass it (read-only), only regular files, and honours Hermes' core read denylist.Communitytools

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section