salch-credshadow
Find out which model is better at your work, from your real prompts.
Benchmarks rank models on other people's tasks. You want to know: would Hermes 4 have answered my questions better than the model I use now? Shadow answers that, blind.
/shadow vote
Your prompt: "How should I handle flaky uploads?"
— Answer 1 —
Add retries with exponential backoff, and make the upload idempotent so a retry can't …
— Answer 2 —
Wrap the call in a retry loop …
Which is better? /shadow 1, /shadow 2, /shadow tie, or /shadow skip.
/shadow 1
You picked nousresearch/hermes-4-70b. Answer 1 was nousresearch/hermes-4-70b, Answer 2 was glm-5.3.
/shadow
Shadow: which model is better at your work (blind votes on your own prompts)
glm-5.3 vs nousresearch/hermes-4-70b: 4-11, 2 tie(s). glm-5.3 wins 27% (95% range 11-52%):
no clear winner yet.
How it works
- After a plain chat turn, meaning one where the agent called no tools, Shadow sends the same conversation in the background to the second model you chose. It goes through Hermes's own plugin model access (
ctx.llm), so your session never waits. Turns with tool calls are skipped, because a second model can't redo the tools fairly. /shadow voteshows both answers blind, as Answer 1 and Answer 2 in random order. You see which model wrote which only after you vote.- Your leaderboard counts wins, losses and ties for each pair of models, with a 95% range (a Wilson interval). It names a winner only when the range is clearly above or below 50%, and says "too early to call" under 10 votes.
- Each chat sees only its own pairs. On a gateway (Telegram, Discord…),
/shadow voteshows only pairs captured in that same conversation, so one user never sees another user's prompt or answers. At your own terminal, all local pairs are yours. - Answers are deleted once you vote. Only the result is kept: which model won, and when.
Turn it on
Shadow is off until you name a model, and Hermes must allow the plugin to choose one:
plugins:
entries:
shadow:
settings:
compare_model: nousresearch/hermes-4-70b # the model to compare with
compare_provider: "" # if it's on another provider
sample: 1.0 # share of eligible turns to compare
max_per_day: 40 # cost cap
llm:
allow_model_override: true # required by Hermes for any plugin choosing a model
# allow_provider_override: true # only if you set compare_provider
Use it
/shadow your leaderboard, and how many answers wait for a vote
/shadow vote the next pair, blind
/shadow 1 | 2 | tie your vote
/shadow skip skip this pair
hermes shadow the leaderboard from your shell
What it touches
- Model calls: one extra call to your chosen model per compared turn, up to
max_per_day(40 by default). They cost what that provider charges. - What the second model receives: the plain text of the conversation, up to the last 12 user and assistant messages. Tool calls and results are not sent, and neither is the main model's answer.
- Stores locally: the prompt, both answers (up to 4,000 characters each) and the model names for pairs waiting for your vote, plus vote results, in
<hermes home>/plugin-data/shadow/shadow.json. Answers are deleted when you vote or skip, and at most 100 pairs wait at once. - No network of its own, no telemetry, no dependencies.
Limits
- Conversational turns only. Tool-using turns can't be replayed fairly.
- One second model at a time. Change
compare_modelto start a new pairing; old results stay in the leaderboard under their own names. - Your votes are your taste. That's the point, but it isn't a general benchmark.
Development
pytest tests
19 tests cover per-chat privacy on gateways, the per-profile data path, the off state, what is sent (and what is not), skipping tool turns, the blind order and reveal, deleting answers after a vote, the leaderboard and its interval, the trust-error hint, the daily cap and sampling, the provider setting and the CLI.
License
MIT
README from the reviewed commit 0a55597; it updates when the author re-pins.
