keppygonogo — a Hermes plugin
47 out of 50 is not 94%. It lands somewhere between 84% and 98%, so if you
were aiming at 95%, you cannot yet say you got there.
This plugin gives a Hermes agent a decision instead of a number. Ask it whether
something is good enough to ship and it answers with one of five verdicts, the
interval behind it, the confidence threshold that routes uncertain cases to a
human, and — when the sample is simply too small to say — that it is too small
to say.
hermes plugins install keppy/hermes-plugin-gonogo
hermes plugins enable gonogo
Requires gonogo-eval>=0.4,<0.5, which Hermes installs for you from the
plugin's pyproject.toml. Expected actions (a case whose right move is to
wait, ask or not reply) and the borderline compare verdict arrived in 0.4.
The ecosystem this belongs to
This plugin is the in-session half of a small ecosystem that takes a task
from which model? to ship it or not:
your task ──► evalroute ─ choose a (model, effort) arm;
│ measured where available, priors marked
▼
your cases ──► thomas ── train on labels (encoder) or score_text (RL);
│ evaluate on untouched cases with gonogo
▼
gonogo ── ship it, test a threshold on fresh cases, or walk away
- gonogo — the library behind this
plugin, and the root of the map. Any agent, your real cases, a target;
the verdict comes with the interval behind it. - thomas — the training harness. RL
sharesscore_textacross training and held-out evaluation; encoder SFT
instead learns labels and needs a separate held-out evaluation. - evalroute — the
routing layer: classify the task, hand back the arm with measured
costs where available and priors otherwise; collect ratings to prioritize
the next controlled batch. - The Hermes plugins — the same three, inside your agent's session:
this one for decisions, thomas
with a human approval hook on its training tool, and evalroute's/route
before the first turn.
What it adds
| Tool | The question it answers |
| --- | --- |
| gonogo_decide | "Is this good enough to ship?" — from a pass count or a results file |
| gonogo_report | "Give me the report artifact" — for a doc, a PR, a pilot review |
| gonogo_compare | "Is B actually better than A?" — McNemar on paired cases |
Plus a bundled skill, gonogo-verdict, that loads the reporting discipline
with it: quote the interval and not the point estimate, read the verdict off
the lower bound, keep the calibration caveat, keep the self-implicating
detail in.
Why a plugin and not a library
The library already exists — pip install gonogo-eval. What a plugin changes is
who asks. Any Hermes session can now end a measurement with a defensible call
instead of a plausible-looking percentage, in the same conversation where the
measurement happened, without anyone remembering to go run a script.
A worked exchange
You: The invoice router got 47 of 50 right. Ship it?
Agent: (calls
gonogo_decidewith passed=47, total=50, target=0.95)
**ASSIST ONLY** — pass rate 94.0% [83.8%, 97.9%] is below the 95% target
and no confident subset reaches it; useful as a draft-generator, not as an
unattended step
Pass rate 94.0% [83.8%, 97.9%] over 50 cases (47 passed), target 95%
Note: Every case reported the same confidence, so no abstention threshold can be
derived. Emit a real per-case confidence to unlock selective automation.
Two things that matter there. The interval reaches down to 83.8%, so 94% is not
the number the decision gets made on. And the tool noticed the results carried no
per-case confidence — which is what a review threshold would have been built
from, so it says so rather than silently reporting an overall rate.
Give it a results file with confidence on it instead and the same call finds the
operating point:
**AUTOMATE WITH REVIEW** — overall pass rate 85.0% [73.9%, 91.9%] misses the 90%
target; a candidate threshold at confidence 0.86 reaches 100.0% precision on 85% of cases on this selection set, but needs a fresh holdout before deployment
Candidate operating point (fresh holdout required): abstain below 0.86 → 85% cases selected at 100.0% precision 100.0% [93.0%, 100.0%], 9 cases to review
Calibration error 0.10
Note: The 0.86 threshold was chosen by searching this same case set, so its
precision is optimistically biased. Re-measure it on fresh cases before relying on it.
That last note is the plugin's own caveat, not a user's: a threshold found by
searching the case set it is scored on is biased upward. can_automate remains
false for this candidate; holdout signoff is a separate step.
The results file
One JSON object per line. passed alone is enough for a one-run verdict.
For gonogo_compare, every row in both runs needs the same stable, explicit
id (and matching group membership if grouped); generated row positions are
not paired evidence. The file is plain enough to build in a spreadsheet.
{"id": "inv-001", "passed": true, "confidence": 0.97, "expected": "Acme Supply", "output": "Acme Supply"}
{"id": "inv-006", "passed": false, "confidence": 0.42, "expected": "Acme Supply", "output": "Cinder Works", "detail": "normalized the trading name to its parent entity"}
{"id": "inv-007", "passed": false, "error": "TimeoutError: 30s"}
{"id": "inv-008", "passed": false, "abstained": true}
{"id": "msg-014", "passed": true, "confidence": 0.91, "expected_action": "no_reply", "action": "no_reply"}
{"id": "msg-015", "passed": false, "confidence": 0.88, "action": "wait"}
confidence(0–1) is optional for an entire file, but mixed missing and present
values are rejected; it is what unlocks a candidate review threshold.erroris a crashed case — a failure, not an aborted run.abstainedlands at
zero confidence, so a threshold defers it first.expected_actionsays the right move for a case is not to answer (wait,
ask,no_reply, any name); absent means answer.actionis what the agent
did. A hold-back case passes exactly when they match, and any action other
thanabstainkeeps its confidence: a confident wrongwaitis a silent miss
and counts against precision. Reports then carry the actions table (acted
when it should have held back / held back when it should have acted). Name
your say-nothing actionno_replyorwait, notabstain.groupon every case folds correlated checks into one trial per group.examples/invoice_routing.jsonlis a 60-case synthetic set to try it on.
gonogo_report with save_report writes the report JSON, and gonogo_compare
pairs two of those by case id (or by group id with matching case membership) —
so a run measured today can be checked against one measured last week.
What it can touch
Three tools, and nothing else: no hooks, no middleware, no environment
variables, no credentials, no network. Count-only calls need no file; results
calls read the path you give them and optionally write report JSON or HTML when
requested. The whole blast radius is the one dependency and those opt-in files.
Development
uv venv .venv && uv pip install --python .venv/Scripts/python.exe gonogo-eval pytest pyyaml
.venv/Scripts/python.exe -m pytest tests -q
hermes plugins validate . # the catalog admission gate, including the security scan
hermes plugins doctor . --ci # discovery, manifest, import, registration
The tests assert the verdicts themselves — that 47/50 at a 95% target is
ASSIST ONLY, that the same 47/50 at 90% is INSUFFICIENT EVIDENCE needing
about 204 cases, and that a 3-point gap on 60 cases is not an improvement.
MIT. The statistics are gonogo's; this plugin
is the surface that lets an agent ask for them.
README from the reviewed commit faf3e6a; it updates when the author re-pins.