About
Your skills, judged blind: finds out whether each skill your Hermes agent loads makes its answers better. /skilluminati <skill> takes the latest 12 times you used the skill, each with the chat as it stood at that moment, and your model answers each one twice, once with the skill and once without. The Owl, a blind judge, sees both answers and the skill's one-line description of its job, never which answer had the skill, and sees every pair twice with the answers swapped; if it picks differently the second time, the pair counts as Same. Verdicts: Enlightened (answers are clearly better with it), Dead weight (the model does this anyway, it only costs tokens), Hindrance (answers are worse with it) or Unproven (too few uses to tell yet), each marked Probable or Plausible. Built-in question sets for frontend-design, systematic-debugging and hermes-agent let a fresh install run a test with no history, and /skilluminati selftest plants a helpful, a harmful and an empty skill to prove the Owl can tell them apart. Surfaces: the /skilluminati command, two read-only agent tools, a Desktop page with a sidebar entry (plugin SDK only) and a read-only dashboard backend. Disclosure — opens HERMES_HOME/state.db read-only and reads installed skill files, skills/.usage.json and skills.disabled/model.default from config.yaml; for each test it sends up to 12 past tasks (the stored session system prompt, the chat up to the moment the skill loaded and any tool output in it, after best-effort regex blanking of emails, phone numbers, key-shaped strings, token-carrying URLs, money amounts and names in plugin-data/skilluminati/names.txt) plus the skill text to the active provider and model, or a per-role provider/model you configure, through ctx.llm, at up to 64 model calls per skill on your own allowance; runs start only from a local CLI/TUI/Desktop/ACP session (never cron or a messaging gateway) on a background thread; writes only under HERMES_HOME/plugin-data/skilluminati/; imports one private core helper (agent.skill_utils.get_all_skills_dirs) to list skill folders; no other network calls, no shell commands, no stored credentials.
Skilluminati
A Hermes Agent plugin that checks whether your skills still earn their place. It replays a skill's latest uses twice, once with the skill and once without, and the Owl, judging blind, picks the better answer each time. The verdict is Enlightened, Dead weight, Hindrance or Unproven.
Install
hermes plugins install yxssxn/skilluminati/plugin
hermes plugins enable skilluminati
In Hermes Desktop, also turn on Capabilities -> Plugins -> Skilluminati to add the Skilluminati page.
Try it
/skilluminati systematic-debugging
A new install has no history to test, so this uses the skill's 8 built-in questions and takes about 8 minutes. When it is done a popup says so: press Show in chat to put the verdict in the chat. On the Skilluminati page you can press Ask the Owl on any skill instead.
What it touches
- Model calls: a test calls your configured model through Hermes (about 48 to 64 calls, one at a time). Runs start only when you type the command or press Ask the Owl; never from cron or a messaging platform.
- Your history: it reads your past chats from Hermes's
state.db read-only. Keys, emails and other private details are removed (names only if you list them) before any text is sent to a model.
- Files it writes: results go only to
<HERMES_HOME>/plugin-data/skilluminati/ (owner-only).
- Background work: each test runs in one background thread inside Hermes.
/skilluminati stop ends it.
- Chat: when a test ends you get a popup; the verdict goes into the chat only when you click "Show in chat".
- No other network calls, no shell commands, no stored credentials.
The full guide, including how the judging works and what it cannot tell you, is in the main README.
MIT licence. See LICENSE.
README from the reviewed commit 0affc46; it updates when the author re-pins.
yxssxn
