All plugins

evalroute

Route a task to the right (model, reasoning effort) arm before you start. Classifies the task into a lane, prints a provenance-carrying route card, and installs per-model reasoning-effort defaults. Strong rules run locally; weak-signal classification calls the host LLM and consumes tokens. The separate harness and shim can make paid provider calls when explicitly run; no plugin-owned credentials. Disclosure — weak-signal task classification sends the task text to your active Hermes model via the host LLM facade (token cost). The model-callable tool, /route, /rate and observed /model and /reasoning commands append task text and command args to a local ledger at <HERMES_HOME>/evalroute/labels.jsonl. "hermes evalroute install-routes" rewrites agent.reasoning_overrides in config.yaml when you run it. The bundled harness and Hermes shim are never run by the plugin. When run by hand they make paid provider calls, execute model-generated code on the host via shell checkers, and run hermes -z with approvals bypassed. Disclosure — "hermes evalroute sync" (run by hand) downloads the route table from the Hugging Face dataset keppy/evalroute-flywheel into HERMES_HOME and route cards then show that dataset's text; "hermes evalroute dispatch" (run by hand) spawns a single-query hermes chat on the routed model (Nous provider by default) — or, with --runner claude-code, a `claude -p` (Claude Code) process under the user's own Claude account, whose per-run cost is recorded in the local sidecar — with the brief as its query, inheriting the current environment except HERMES_YOLO_MODE (the worker never inherits --yolo; its approvals follow approvals.single_query_mode, default deny), and writes its report and a .dispatch.json sidecar beside the brief. Disclosure — "hermes evalroute contribute" (run by hand only) refuses to run unless evalroute.contribute: true is set in config.yaml, and then uploads redacted outcome rows only — lane, routed arm, confirmed arm, verdict, method, facet counts, ISO week and an HMAC-SHA256 task hash; never task text, notes, paths or the salt — to the user's own path (contributed/<hf-username>/) in the Hugging Face dataset keppy/evalroute-flywheel using the user's existing huggingface_hub login; "--dry-run" prints the exact rows and sends nothing; the per-install salt stays local in <HERMES_HOME>/evalroute/contribute.salt. Since evalroute 0.9.1 (schema 2) the same upload also carries lane-correction rows as lane pairs (from_lane, to_lane, method, week) with no task text, plus the max_turns value and harness name the arm ran under and, since evalroute 0.10.0, whether the arm was an explore draw and which default arm it replaced (arm ids only). Disclosure — since evalroute 0.10.0, setting evalroute.explore.fraction in config.yaml (or EVALROUTE_EXPLORE) makes that fraction of routes recommend the lane's runner-up arm from the bundled explore table instead of the default arm (and "hermes evalroute dispatch" then runs that runner-up model), marked on the route card; off by default. Disclosure — "hermes evalroute install-encoder <dir|owner/name>" (run by hand only) downloads a small text-classification model (the default id keppy/evalroute-lane-encoder is a 146 MB ModernBERT fine-tune on the Hugging Face Hub) into <HERMES_HOME>/evalroute/encoder/ and from then on "hermes evalroute route" runs it locally, offline, to classify the task when the keyword rules are weak, instead of the LLM call; it needs torch and transformers in the plugin environment (pip extra evalroute[encoder]), which the plugin does not install; "install-encoder --remove" deletes it. "hermes evalroute export-cases" (run by hand only) writes your --lane-pinned route rows, task text included, to a local file you name, for training your own encoder; stdout carries counts only and nothing is uploaded. As of 0.6.2 the plugin is a thin adapter: all behaviour lives in the PyPI package evalroute (>=0.10.0,<0.11), declared as a Python dependency that Hermes installs on your confirmation.

keppy1Toolsv0.6.5Updated 2 days ago

About

Route a task to the right (model, reasoning effort) arm before you start. Classifies the task into a lane, prints a provenance-carrying route card, and installs per-model reasoning-effort defaults. Strong rules run locally; weak-signal classification calls the host LLM and consumes tokens. The separate harness and shim can make paid provider calls when explicitly run; no plugin-owned credentials. Disclosure — weak-signal task classification sends the task text to your active Hermes model via the host LLM facade (token cost). The model-callable tool, /route, /rate and observed /model and /reasoning commands append task text and command args to a local ledger at <HERMES_HOME>/evalroute/labels.jsonl. "hermes evalroute install-routes" rewrites agent.reasoning_overrides in config.yaml when you run it. The bundled harness and Hermes shim are never run by the plugin. When run by hand they make paid provider calls, execute model-generated code on the host via shell checkers, and run hermes -z with approvals bypassed. Disclosure — "hermes evalroute sync" (run by hand) downloads the route table from the Hugging Face dataset keppy/evalroute-flywheel into HERMES_HOME and route cards then show that dataset's text; "hermes evalroute dispatch" (run by hand) spawns a single-query hermes chat on the routed model (Nous provider by default) — or, with --runner claude-code, a `claude -p` (Claude Code) process under the user's own Claude account, whose per-run cost is recorded in the local sidecar — with the brief as its query, inheriting the current environment except HERMES_YOLO_MODE (the worker never inherits --yolo; its approvals follow approvals.single_query_mode, default deny), and writes its report and a .dispatch.json sidecar beside the brief. Disclosure — "hermes evalroute contribute" (run by hand only) refuses to run unless evalroute.contribute: true is set in config.yaml, and then uploads redacted outcome rows only — lane, routed arm, confirmed arm, verdict, method, facet counts, ISO week and an HMAC-SHA256 task hash; never task text, notes, paths or the salt — to the user's own path (contributed/<hf-username>/) in the Hugging Face dataset keppy/evalroute-flywheel using the user's existing huggingface_hub login; "--dry-run" prints the exact rows and sends nothing; the per-install salt stays local in <HERMES_HOME>/evalroute/contribute.salt. Since evalroute 0.9.1 (schema 2) the same upload also carries lane-correction rows as lane pairs (from_lane, to_lane, method, week) with no task text, plus the max_turns value and harness name the arm ran under and, since evalroute 0.10.0, whether the arm was an explore draw and which default arm it replaced (arm ids only). Disclosure — since evalroute 0.10.0, setting evalroute.explore.fraction in config.yaml (or EVALROUTE_EXPLORE) makes that fraction of routes recommend the lane's runner-up arm from the bundled explore table instead of the default arm (and "hermes evalroute dispatch" then runs that runner-up model), marked on the route card; off by default. Disclosure — "hermes evalroute install-encoder <dir|owner/name>" (run by hand only) downloads a small text-classification model (the default id keppy/evalroute-lane-encoder is a 146 MB ModernBERT fine-tune on the Hugging Face Hub) into <HERMES_HOME>/evalroute/encoder/ and from then on "hermes evalroute route" runs it locally, offline, to classify the task when the keyword rules are weak, instead of the LLM call; it needs torch and transformers in the plugin environment (pip extra evalroute[encoder]), which the plugin does not install; "install-encoder --remove" deletes it. "hermes evalroute export-cases" (run by hand only) writes your --lane-pinned route rows, task text included, to a local file you name, for training your own encoder; stdout carries counts only and nothing is uploaded. As of 0.6.2 the plugin is a thin adapter: all behaviour lives in the PyPI package evalroute (>=0.10.0,<0.11), declared as a Python dependency that Hermes installs on your confirmation.

hermes-plugin-evalroute

Route a task to the right (model, reasoning effort) arm before you start.

This plugin is a thin adapter: all behaviour lives in the
evalroute library (PyPI) — the
classifier, the flywheel ledger, dispatch, the dataset sync, and the route
table. The plugin changes only when the Hermes manifest/hooks wiring or the
lib↔plugin contract changes; classifier, ledger, dispatch, dataset and
route-table work ship as library releases with no catalog PR.

What it registers

| registration | name | provider |
| --- | --- | --- |
| tool | evalroute_route | evalroute.routing.evalroute_route |
| slash command | /route | evalroute.routing.handle_route_command |
| slash command | /rate | evalroute.flywheel.handle_rate |
| CLI command | hermes evalroute … | evalroute.cli.setup_cli / evalroute_cli |
| hook | pre_llm_call | first-turn sniff (plugin-local sniff.py) |
| hooks | pre_command, post_llm_call | evalroute.flywheel.on_pre_command / on_post_llm_call |
| skill | evalroute-routing | bundled in this repo |

A contract guard (evalroute.contract.CONTRACT_VERSION == 1) refuses to
register when the installed library speaks a different contract than this
adapter was written for — update this plugin instead of half-working.

Commands

  • /route <task> (or hermes evalroute route "<task>") — classify a task
    into a lane, print a route card: model, effort, provenance, route id.
    --lane <id> pins; --json emits the machine envelope.
  • /rate pass|fail|skip [--route-id <id>] [--lane <id>] [--model <id> --effort <level>] [--note <text>] (or hermes evalroute rate …) — label
    the outcome; this is the flywheel's only data source.
  • hermes evalroute … — also dispatch <brief.md> (route a brief, spawn a
    worker on that arm, print the rate line), install-routes (effort
    defaults into agent.reasoning_overrides), sync (pin the published
    route table), report (a static HTML page over the ledger, sessions,
    trains, drift), contribute (opt-in: upload whitelist-redacted outcome
    rows to your own path in the shared dataset — off until you set
    evalroute.contribute: true with hermes config set; --dry-run shows
    the exact rows first), and more; hermes evalroute with no verb prints the
    workflow epilog, --version the resolved library.

Install

hermes plugins install keppy/hermes-plugin-evalroute
hermes plugins enable evalroute
hermes evalroute install-routes --dry-run   # review, then drop --dry-run

Installs from the plugin catalog; pyproject.toml pulls
evalroute>=0.8,<0.9 from PyPI into Hermes's managed environment. No
credentials, no requires_env.

Why the split

Before the split, the routing core shipped inside the plugin: every
classifier tweak, ledger field, or route-table regen was a plugin release,
and a plugin release means a catalog pin, an install-probe-tag cycle, and a
review of unrelated routing changes under Hermes' manifest rules. After the
split the plugin is manifest + hooks + a contract guard — the pieces that
are actually about Hermes — and the library carries everything else. A
library release is uv publish; a plugin release is now rare by design.

Details

Classifier design (rules vs LLM fallback), facets, the ledger row kinds,
the route-table provenance states, the harness, the shim, and the dataset
sync are documented in the library README.

README from the reviewed commit da75935; it updates when the author re-pins.

More by keppy

3 plugins by keppy in the catalog.

keppygonogoTurn a pilot-scale eval score into a deployment decision — Wilson intervals, a confidence threshold that routes uncertain cases to a human, and an honest INSUFFICIENT EVIDENCE verdict when the sample cannot support the claim. Disclosure — gonogo_report writes report JSON/HTML to the path the model supplies.CommunitytoolskeppythomasFine-tune a small calibrated encoder classifier on Modal and rate it with gonogo. Every GPU launch goes through the human approval gate with the exact config shown; data checks and evaluation run locally for free. Disclosure — thomas_encoder_train spends the user's Modal GPU credits and uploads the chosen training JSONL rows to Modal after the human approval gate; the base model is downloaded from Hugging Face on the Modal side. Training and inference run in a separate Python interpreter the user configures via THOMAS_PYTHON (with the author's thomas package, torch and modal installed); the Hermes venv only gets gonogo-eval. Runs and artifacts are written under HERMES_HOME/thomas/runs; the eval tool writes report files to the path the model supplies.Communitytools