skill-retrieval

skill-retrieval

BM25-based skill retrieval plugin for Hermes Agent. Replaces the full skill list in the system prompt with a names-only compact view (~2K tokens) and injects top-K relevant skill descriptions per turn (~300 tokens), saving ~9K tokens/turn while keeping skills discoverable by name. Those figures were measured on a Hermes install with ~300 skills; the saving scales with your own skill count.

This is a Hermes Agent plugin. It is not a Claude Code plugin and will not load in Claude Code — that runtime has no pre_llm_call event, no Python register() entry point, and reads .claude-plugin/plugin.json rather than plugin.yaml. Requires Hermes Agent >=0.21.1.

See SKILL.md for full architecture, token measurements, how it works, performance, limitations, and how to verify a healthy install.

Contents

• Installation
• Configuration
• Read-only core references
• Uninstall
• Data handling (Jev rerank)
• License

Installation

# From the agent_skills repo root
ln -s "$(pwd)/plugins/skill-retrieval" ~/.hermes/plugins/skill-retrieval

# Named profile: the plugin reads its own directory relative to the plugin
# folder, and keys/paths resolve under $HERMES_HOME (set per profile) — no
# cross-profile fallback. Copy the plugin into the profile's plugins/ dir
# and set any needed keys in that profile's environment (Hermes loads the
# profile's .env into os.environ at startup).

# Dependencies (Hermes Python env)
# pyyaml is the only dependency — usually already present in a Hermes env.
# On CPython 3.14 the plugin uses the host's hermes_yaml facade instead.
pip install pyyaml

User plugins are opt-in — Hermes discovers the directory but does not load it until you enable it:

hermes plugins enable skill-retrieval

Then restart the Hermes session so the plugin's register() runs.

Configuration

| Setting | Default | Override |
|---------|---------|----------|
| Top-K results | 6 | SKILL_RETRIEVAL_TOP_K env var |
| Injected results | TOP_K | SKILL_RETRIEVAL_INJECT_K env var; a pool wider than the injected set (e.g. TOP_K=12, INJECT_K=6) lets the Jev rerank filter, not just reorder |
| System prompt compaction | enabled | SKILL_RETRIEVAL_COMPACT=0 disables compaction but keeps retrieval injection |
| BM25 k1 | 1.5 | edit scripts/bm25_retriever.py |
| BM25 b | 0.75 | edit scripts/bm25_retriever.py |
| Jev rerank | off | SKILL_RETRIEVAL_RERANK=jev enables; key TYPESAFE_API_KEY (legacy TYPESAFE_KEY accepted) from os.environ |
| Session rerank cache | off | SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1 reuses a named session's confirmed candidate workset and avoids redundant Jev calls while BM25 stays inside it |
| Rerank provider | typesafe | SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter calls Jev through OpenRouter with OPENROUTER_API_KEY |
| Rerank model | jev-latest (typesafe), ~typesafe/jev-latest (openrouter) | SKILL_RETRIEVAL_RERANK_MODEL, e.g. typesafe/jev-1.14 to pin a version |
| Rerank A/B log | $HERMES_HOME/data/jev-trial/rerank_ab_log.jsonl (rotates to .1 past 5 MB) | SKILL_RETRIEVAL_RERANK_LOG env var |

export SKILL_RETRIEVAL_TOP_K=8
export SKILL_RETRIEVAL_INJECT_K=6
export SKILL_RETRIEVAL_COMPACT=0  # retrieval-only mode; the skills prompt is left unchanged (the builder is still wrapped to record tool capabilities)
export SKILL_RETRIEVAL_RERANK=jev  # optional: rerank the top-K with the Jev system-one model
export SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1  # optional: reuse a named session's confirmed workset
export SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter  # optional: call Jev through OpenRouter with OPENROUTER_API_KEY instead of a TypeSafe key

Jev rerank

Optional, off by default. When enabled, the BM25 top-N shortlist is reranked by the Jev system-one judgment model (api.typesafe.ai/v1/systemone, model jev-latest). It is fail-soft: a 3.0 s timeout with no retries, and any error falls back to the BM25 order. The key is read from os.environ only (Hermes loads the active profile's .env into environ at startup — no cross-profile file reads). Every query is logged to the A/B log path; scripts/rerank_ab_report.py summarizes the log locally. See Data handling below for exactly what leaves the machine.

Evaluate over a period of time before deciding to keep it. A single session cannot show whether the rerank helps, so the practical benefit is hard to judge turn by turn. Enable the rerank, use it for real work for at least 1-2 weeks, then review the A/B log with scripts/rerank_ab_report.py — look at how often the Jev order differs from the BM25 order (rank1_flip), what the per-turn probability spread looks like, and whether the flips are right. If the spread is consistently tiny or the flips look arbitrary, turn it off.

Session rerank cache

Optional and off by default. This mode is intended for conversations whose
topic stays coherent for several turns. The first successful Jev rerank freezes
that named session's BM25 candidate pool as a workset. On later turns, local
BM25 still runs and determines the current order. If every candidate that would
be injected is already in the workset (or was selected by a later successful
rerank), the external Jev call is skipped. A new would-be-injected candidate
triggers Jev again; only a successful Jev result can confirm it for reuse.

The cache never changes the system prompt, never reuses anonymous requests,
and never crosses session ids. It is bounded to 256 sessions and is cleared on
skill installation, removal, or other skill-catalog mutations. On a cache hit,
the current BM25 order is injected, so enable this only when avoiding repeated
network reranks is worth that semantic tradeoff.

Read-only core references

The plugin imports ten Hermes core helpers read-only (never assigned, never wrapped). From agent.prompt_builder:

• _current_session_platform_hint — session platform for cache keying and visibility gating
• _parse_skill_file, _build_snapshot_entry — SKILL.md frontmatter/description parsing for the corpus
• _skill_should_show — per-skill visibility gating (tools, toolsets, platform)
• extract_skill_conditions — condition extraction feeding _skill_should_show

From agent.skill_utils:

• get_all_skills_dirs, get_project_skills_dirs — skill root resolution (profile, external dirs)
• get_disabled_skill_names — disabled-set gating
• iter_project_skill_files, iter_skill_index_files — SKILL.md file enumeration
• (skill_matches_platform is consulted via getattr when present)

These are read-only imports disclosed per the catalog's rule-9 note. If a Hermes upgrade changes their signatures, the corpus loader is guarded: any mid-scan exception falls back to the standalone loader, so the index is never silently empty while compaction strips descriptions. If a future change needs an assignment or wrap there, that counts against rule 9's strict reading and warrants a core seam request instead.

In codex_app_server mode the prompt goes to Codex as developerInstructions and bypasses llm_request, so that mode gets no compaction and no snapshot, while turn-1 fail-open still retrieves.

Uninstall

1. Disable the plugin (hermes plugins disable skill-retrieval), then delete the plugin folder from ~/.hermes/plugins/ with your file manager or shell.
2. Restart the Hermes session
3. The system prompt reverts on restart

The middleware and hook are session-scoped, so removal is otherwise instant.

Data handling (Jev rerank)

The rerank is off by default. With no SKILL_RETRIEVAL_RERANK=jev set,
the plugin makes no network calls and writes nothing.

When you opt in (SKILL_RETRIEVAL_RERANK=jev + a key present):

• Sent to api.typesafe.ai (normally one POST per retrieval turn; with
  SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1, confirmed named-session turns can
  skip the POST; model
  jev-latest): the user message flattened, stripped of leading imperative
  shapes over the full shape set (shape plus its following sentence), capped
  at 2000 characters, plus one
  question per top-12 shortlisted skill naming that skill's id and its
  description — the description is whitespace-normalized and unambiguous
  imperative shapes are removed wherever they appear (the shape itself plus
  its trailing whitespace and terminator; sentence text around it stays;
  ordinary wordings like 'new task:', 'delete all' are kept), capped at 400
  characters. BM25 scores never leave the machine. The response
  carries one relevance probability per candidate.
• Key: TYPESAFE_API_KEY (legacy TYPESAFE_KEY accepted), read from
  os.environ only — Hermes loads the active profile's .env into
  environ at startup, so no other profile's file is ever read. The key is
  never logged, never written, never included in any record.
• Through OpenRouter (SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter): the
  same payload goes to https://openrouter.ai/api/v1/systemone, which serves
  Jev on the same System One contract and forwards it to TypeSafe. The key is
  OPENROUTER_API_KEY, also from os.environ only. Each provider reads only
  its own key, so a TypeSafe key is never sent to OpenRouter and vice versa.
• Local log: each reranked turn appends one JSONL line to
  $HERMES_HOME/data/jev-trial/rerank_ab_log.jsonl (override:
  SKILL_RETRIEVAL_RERANK_LOG) containing the timestamp, session id, a
  200-character query excerpt, the BM25 order, the Jev order, probabilities,
  token count, and latency. The log rotates to <name>.1 past 5 MB, so it
  never grows without bound. scripts/rerank_ab_report.py summarizes it
  locally; no log data is uploaded anywhere.

License

MIT. See LICENSE for the upstream SR-Agents notice and this project's notice.

README from the reviewed commit 86a8ace; it updates when the author re-pins.