All plugins

hermes-regress

Regression tests built from your own sessions. Capture is passive: every finished turn is written as a plain JSON case holding the prompt, the model, each tool call with its exact arguments and the output it returned. Replay a case through the real agent loop with every tool call vetoed at pre_tool_call and answered from the recording, so a model, provider or config change can be measured without side effects — /regress run --model X reports pass, drift, regressed and error per case, with markdown, JUnit XML and a hash-chained export bundle for CI. Fixtures match only on identical tool and arguments; a case whose tool plan cannot be reproduced is reported as a gap instead of being answered from the wrong recording, and a replay window that never reaches the hook is marked error and stops the run rather than executing live tools. Disclosure — writes captured cases under HERMES_HOME/plugin-data/hermes-regress/ (mode 0600) with the prompt, answer, tool arguments and tool results all passed through redaction first (provider keys, GitHub/Slack/AWS tokens, JWTs and inline password/token values become [REDACTED], counts recorded per case); keeps replay state in the same directory (armed.json plus an audit line per armed tool call in hook.log), scopes the replay veto to the single session that claims the armed case (only a first turn whose message exactly matches the recorded prompt can claim it, recorded from pre_llm_call), keeps that scope through a mid-turn context compression by matching the claimed task id as well as the session id, and serialises claims with a file lock, so an unclaimed window serves and blocks nobody, a late claim cannot revive a disarmed window, and other chats and cron runs are never affected, and runs the local hermes CLI as a subprocess when you ask for a replay; no network calls of its own, no telemetry.

harishkotraToolsv0.1.0Updated 2 days ago

About

Regression tests built from your own sessions. Capture is passive: every finished turn is written as a plain JSON case holding the prompt, the model, each tool call with its exact arguments and the output it returned. Replay a case through the real agent loop with every tool call vetoed at pre_tool_call and answered from the recording, so a model, provider or config change can be measured without side effects — /regress run --model X reports pass, drift, regressed and error per case, with markdown, JUnit XML and a hash-chained export bundle for CI. Fixtures match only on identical tool and arguments; a case whose tool plan cannot be reproduced is reported as a gap instead of being answered from the wrong recording, and a replay window that never reaches the hook is marked error and stops the run rather than executing live tools. Disclosure — writes captured cases under HERMES_HOME/plugin-data/hermes-regress/ (mode 0600) with the prompt, answer, tool arguments and tool results all passed through redaction first (provider keys, GitHub/Slack/AWS tokens, JWTs and inline password/token values become [REDACTED], counts recorded per case); keeps replay state in the same directory (armed.json plus an audit line per armed tool call in hook.log), scopes the replay veto to the single session that claims the armed case (only a first turn whose message exactly matches the recorded prompt can claim it, recorded from pre_llm_call), keeps that scope through a mid-turn context compression by matching the claimed task id as well as the session id, and serialises claims with a file lock, so an unclaimed window serves and blocks nobody, a late claim cannot revive a disarmed window, and other chats and cron runs are never affected, and runs the local hermes CLI as a subprocess when you ask for a replay; no network calls of its own, no telemetry.

hermes-regress

Regression tests for your agent, built from the sessions you already had.

Every Hermes plugin, prompt tweak, model swap and config change alters how your agent behaves. Nothing tells you whether it still does the things you actually rely on. hermes-regress watches your real turns, turns each finished turn into a test case, and replays those cases against a different model or setup with every tool side effect stubbed out.

/regress run --model anthropic/claude-sonnet-4
run 20261111T081500  model anthropic/claude-sonnet-4  cases 12
  pass 9   drift 2   regressed 1   error 0
  regressed  a1b2c3d4  "deploy the staging build and confirm the health check"

Why this exists

A Hermes install accumulates trust in behaviour, not code. You know your agent checks the log file before restarting a service, writes the changelog in your format, and asks before touching production. Change one thing and that trust is silently invalid: no test fails, no error appears, the agent just answers differently.

Benchmarks do not cover this, because the tasks you care about are the ones you invented: your repo layout, your deploy script, your house style. hermes-regress builds the test suite from those.

Install

git clone https://github.com/harishkotra/hermes-regress ~/hermes-regress
ln -s ~/hermes-regress ~/.hermes/plugins/hermes-regress
hermes plugins enable regress

Restart Hermes. Plugins are loaded when the runtime starts, so a freshly installed or freshly edited plugin is invisible until then. /regress selftest proves the runtime is intercepting before you trust a replay.

Use

Capture is passive. There is nothing to switch on: install, restart, and work as usual.

/regress status                    capture on/off, case count, last run
/regress list 10                   newest cases
/regress show a1b2c3d4             turn summary; --full adds arguments and outputs
/regress expect a1b2c3d4 "tests pass"    add a must-appear assertion
/regress run --model gpt-5         replay every case against another model
/regress run --case a1b2c3d4 --limit 5
/regress report                    markdown report for the last run
/regress export                    zip + hash-chained manifest
/regress prune --keep 200

Headless, for CI:

python3 rg_cli.py run --model anthropic/claude-sonnet-4 --fail-on-regressed
python3 rg_cli.py report

Exit codes: 0 clean, 1 error, 2 a case regressed.

What a case is

A plain JSON file under $HERMES_HOME/plugin-data/hermes-regress/cases/ (mode 0600). Readable, diffable, safe to commit once you have read it.

{
  "case_id": "a1b2c3d4e5f60718",
  "created_at": "2026-11-11T08:05:16+0530",
  "model": "anthropic/claude-sonnet-4",
  "prompt": "deploy the staging build and confirm the health check",
  "answer": "Staging deployed. Health check returns 200 after 3 retries.",
  "tool_calls": [
    {"tool": "terminal", "args": {"command": "make deploy ENV=staging"},
     "result": "deployed build 481 to staging", "source": "post_tool_call"},
    {"tool": "terminal", "args": {"command": "curl -sf localhost:8080/healthz"},
     "result": "200", "source": "post_tool_call"}
  ],
  "expect": [],
  "redactions": 0
}

tool_calls is the fixture set for replay. Each entry is the tool name, its exact arguments, and what it returned during capture.

Safety model

Replay runs the real agent loop but answers every tool call from the capture. Three rules enforce that:

  1. Every tool call is vetoed at pre_tool_call. Nothing reaches your shell, filesystem, network or APIs while a case is armed.
  2. The veto only applies to the replay's own turn. The replay runs inside your gateway, so an unscoped veto would freeze every other chat, messaging session and cron job for minutes. Scoping cannot be done with an environment variable: hermes chat --oneshot runs the turn in the gateway process, and the hook runs there too, so nothing the runner sets in its child is visible to it.
    Instead the armed case has no owner until pre_llm_call reports is_first_turn=True for a turn whose message is exactly the recorded prompt (whitespace collapsed). Only the top of a fresh session is the replay the runner started, and full equality stops a cron, kanban or skill turn that merely shares the opening words from claiming the case. That turn's session_id and task_id are written to armed.json, and every later hook call is compared against both: context compression mid-turn hands the agent a new session_id while the task_id stays put, and the task keeps the veto on the replay's own calls across that boundary. Claims are serialised with flock on replay/armed.lock, each writer uses its own temp file, and the write is skipped if armed.json has already been disarmed, so a late claim cannot resurrect a finished window. An armed window that has not been claimed serves nobody and blocks nobody, and every other session gets {} throughout.
  3. The hook fails closed. If fixture lookup raises while a case is armed, serve() still returns a block, so a bug in this plugin cannot turn a replay into a live run.
  4. A replay never becomes a case. Capture is skipped whenever a case is armed, so /regress run cannot feed its own output back into the suite.
  5. Only an arg-identical recorded call is served. A different command with the same tool name is not answered from a fixture, it is blocked and recorded as a gap: replaying the output of a different command would answer the wrong question.
  6. Armed replay state expires. plugin-data/hermes-regress/replay/armed.json carries expires_at (15 min default), and the hook re-reads it on every call. A crashed or abandoned run disarms itself.
  7. A run that executed real tools says so, after the fact. Every claim and every served call appends a line to replay/hook.log. A case whose tool calls were not fully answered from the recording is marked error with the count: no claim at all, no hook activity, or only n of m calls served — in each case the rest ran for real. This is detection, not prevention: the plugin cannot un-execute a tool, it can only refuse to report the run as a clean replay. The suite stops there instead of piling more live side effects on top.

Before any run, preflight() replays a probe case. If the runtime does not intercept it, the run aborts with an explanation instead of executing real tools. /regress selftest runs the same probe on demand.

Everything written to disk passes through a redaction layer first: the prompt, the answer, every tool result and every set of tool arguments (a curl -H "Authorization: Bearer ..." never lands raw). Provider keys (sk-, pk-, rk-), GitHub tokens, Slack tokens, AWS access keys, JWTs, inline password= / token= / api_key= values and long hex strings become [REDACTED]. Cases, run results, reports, exported bundles, armed.json, the lock file and hook.log are all written 0600, and the redaction count lands in the case as redactions. The store is always get_hermes_home()/plugin-data/hermes-regress, so a per-profile gateway keeps each profile's cases apart; there is no environment override to point it somewhere else.

Verdicts

VerdictMeaning
passAnswer matches the recording and the tool plan matched.
driftAnswer or plan moved but stayed above the similarity threshold. Worth reading.
regressedAn expectation missing, a tool step dropped, or similarity under threshold.
errorThe replay itself failed: timeout, provider error, runtime not intercepting.

Thresholds live in plugin-data/hermes-regress/config.json (threshold, plan_threshold). Expectations always win over similarity: add one with /regress expect for the part that must never change.

Evidence bundle

/regress export writes a zip with results.json, report.md, junit.xml and a manifest.json whose hash chain covers every case. Change one result by hand and the chain no longer verifies. JUnit output drops straight into any CI renderer.

Verified

  • python3 -m unittest discover -s tests runs 32 tests covering redaction, the case store, hash chain, arming and expiry, fixture matching, unrecorded-tool blocking, verdicts, expectations, markdown, JUnit, the export bundle, session/task scoping, claim tightening, the late-claim race, live-execution reporting and file modes.
  • Capture is exercised against real turns: 28 cases recorded from live sessions while this plugin was developed, including turns that ran shell commands.
  • Hook serving is exercised through the real dispatch path (get_pre_tool_call_block_message returns a block with the recorded output and writes the trace line).

Limits, stated plainly

  • Replay sends the recorded prompt, not the full original conversation, so a case measures decision-making on that task, not session-history effects.
  • Answers are compared by normalised similarity plus expectations. There is no semantic judge, so unusually phrased but correct answers can read as drift.
  • Fixtures match on exact arguments. A case whose tool calls embed timestamps or run ids will show as a plan gap, which is usually the honest signal that the task is non-deterministic.
  • Replay proves the agent still chooses the same steps with the same data. It does not prove the live tool still works.

Layout

plugin.yaml      manifest: hooks, command, metadata
__init__.py      register(ctx): hooks + /regress
rg_capture.py    post_tool_call / post_llm_call / on_session_end capture and redaction
rg_replay.py     signature capture, fixture serving, preflight, scoring, verdicts
rg_store.py      case and run storage, config, hash chain
rg_report.py     markdown, JUnit XML, export bundle
rg_cli.py        /regress command surface and headless entry point
tests/           unittest suite

License

MIT

README from the reviewed commit fbd570b; it updates when the author re-pins.

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section