crucible

Crucible

Agents say things confidently. Crucible makes them earn it.

Crucible is a Hermes Agent plugin that turns a code audit or a research
question into an adversarial game. Independent proposer agents commit findings with quoted evidence. Attacker agents
try to break them. A blind referee checks every quote against its source and re-runs every proof of concept in a
sealed container. Belief is computed from checked evidence, never asserted. What you get back is a map of what
survived: each surviving finding with the evidence that carried it, everything held back with the reason, and an
honest status for every question.

A real run: a standard-depth audit of PraisonAI v4.6.77 (an open-source AI-agent framework, the release before a
published sandbox-escape fix), with the agents' terminal in Docker. The report, verbatim but abridged (two of five
survivors and three of four held-back findings shown):

### Survived (5)
- **In _resolve_vars, `${{ env.VAR }}` substitutes os.environ.get(var_name) for any variable name with no
  allowlist, leaking arbitrary secrets into later shell/python step commands.** (97%, entailed) [C4]
  - “def env_replacer(m): var_name = m.group(1).strip() return os.environ.get(var_name, f"${{{{ env.{var_name} }}}}")”
    — file:praisonai/cli/features/job_workflow.py
  - code re-run in the sandbox: re-run reproduced 'VULNERABLE: leaked-secret'
- **_exec_file_action loads and exec_modules any .py found under actions/ next to the workflow, so a
  workflow-controlled action name runs arbitrary Python with full process privileges.** (95%, entailed) [C8]
  - code re-run in the sandbox: re-run reproduced 'VULNERABLE_Q4_ACTION'

### Not established (4)
- In _find_action_file, action_name is concatenated into a filesystem path with only '-'→'_' normalization and no
  containment check, so names containing traversal sequences load and exec arbitrary .py files via importlib. —
  unconfirmed: its proof of concept failed (reproduced, but does not demonstrate the claim: It prints the marker
  unconditionally after resolving a path still inside the actions directory …)
- In praisonai/cli/features/job_workflow.py, _exec_inline_python allows ast.Attribute/ast.Call and does not block
  dunder attributes, so workflow-controlled script code can walk object.__class__.__bases__[0].__subclasses__() to
  subprocess.Popen and execute arbitrary commands (RCE). — unconfirmed: its proof of concept failed (re-run printed
  "RESULT: {'ok': False, 'error': 'Evaluation error: _Final() takes no arguments'}", not 'RCE_MARKER')

The first held-back finding's proof printed VULNERABLE without the flaw happening, and the referee caught it. The
second is a real flaw (its proof indexed the wrong class; with the right one the sandbox runs a shell command), held
back because no proof reproduced it: Crucible misses before it overclaims. Run 20261108-215109-093e2d.

Reports now open with a bottom line. For this run it reads: 5 findings confirmed by a reproduced proof of concept,
but not every part of the question is answered; 2 findings are read in the code but have no reproducing proof of
concept yet.

Why this is more than a prompt

A prompt can ask a model to be careful. It can't make careless answers fail. Crucible changes what reaches you:

• A referee that never sees who argued. It fetches the cited page and checks that the quoted span is really
  there. That's string matching, no model: a typo or a ligature is forgiven, a changed number, a swapped word or a
  dropped "not" is not. It then answers five separate yes/no questions about the passage: same
  population, same measure, same direction, the right evidence state (a projection isn't a count, a correlation isn't
  a cause), and within scope. The engine, not the referee, turns those flags into a grade. The referee never sees the
  author, the prose or the confidence. It also rules whether each claim answers the question it was filed under: a
  true fact that doesn't is shown as a fact on the way and cannot mark the question answered.
• Belief is computed from evidence. A claim's log-odds move only with live, graded evidence. Repeat evidence from
  one source is discounted, so one paper quoted three times counts less than three papers. One quote can never carry
  a claim alone: a second from the same site counts half, from an independent one in full, and testimony or a quote
  that doesn't clearly state the claim can't stand in for that second source. A re-run calculation shows arithmetic,
  not facts, so in research it carries only a claim stated as computed from figures it names. A claim may rest on
  another claim only once the referee finds it follows from it: true but unrelated premises carry nothing. Agents' stated confidences are bets that get scored, not verdicts.
• Attackers see everything. A run is settled only when a full round upheld no attack and added no new claim
  or evidence. If the last round did add support, attackers get a closing turn at it; anything they still never saw
  is named in the report's notes.
• Attacks bite. An upheld attack from another group of agents caps a claim at contested until it's answered,
  and the cap passes to every claim that rests on it. Causal claims need RCT, systematic-review or natural-experiment
  evidence. "No effect" claims need a systematic review or registry.
• Nothing is rewritten. The board is an append-only log. A claim's falsifier and confidence are frozen before
  its evidence. A repair is a new, narrower claim, and the old one stays on the map with its reason. Restoring a run
  replays every move through the same rules, so a tampered log won't load.
• Honest endings. Every question ends answered, contested, ambiguous (with the reading assumed) or
  blocked (no checkable evidence was reachable). Blocked is reported plainly, never filled in from memory.

It uses Hermes surfaces a skill can't reach: parallel subagents (ctx.subagent_lifecycle), out-of-band model calls on
its own pinnable model slot (ctx.llm), a message of its own in the conversation (ctx.post_message), lifecycle hooks
(listed under Disclosures), a Desktop pane with live events, a transcript directive, and backend routes.

Ways to use it

Crucible also keeps a ledger across runs. Claims that survived become known findings. Claims that were killed
become known traps: guard mode flags an answer that repeats one, at no extra cost, with the evidence that killed it.
When runs disagree about a claim, the latest one decides, and entries expire after 180 days.

Live card. Crucible posts its own card the moment a run starts and edits it as the run plays: a progress bar
through framing, commits and attack rounds, the spend against the cap, and every claim moving between ⏳ being checked,
✅ survived, ⚔️ contested, 🔍 not established and 💀 killed, with the source behind each survivor and the attack behind
each kill. When the run ends the card leads with the answer, a bottom line on what is still missing, and how the
question was split into sub-questions. In a messaging app (Telegram, Discord, Slack…) the card is a message in the chat, so when the run ends it
is the map and the agent adds only a takeaway. In the CLI and Desktop, Hermes shows plugin posts as notifications
beside the chat, so the agent presents the full report as well. The card uses Hermes's post_message.

In Hermes Desktop, the Crucible pane shows each run live, claims are colour-coded by verdict, and you can
attack any claim yourself, or a piece of evidence, or the way a question was split. Your attack goes to the same referee as the agents' attacks. A run that fails shows as failed, and one whose Hermes process was killed shows as abandoned (the run holds a lock its process drops on exit, so a reused process id cannot revive it), so the pane stops offering attacks it could never play.

Code mode: findings that need no sources

Bug hunting has no web pages to quote. Pass workspace and the same game runs on a repository:

• Proposers (a security reviewer, an exploit developer, a parser specialist) report one vulnerability per claim,
  quoting the exact lines (file:path/to/mod.py) and, where they can, attaching a proof of concept.
• Attackers (a maintainer, a triager, an auditor) look for the input check that makes a bug unreachable, re-run
  the proofs of concept, and attack overstated impact.
• The referee works on a copy of the repository frozen when the audit began, since the agents' terminal can
  write to the live one. It checks each quote is really in the file (nothing outside the repository can be read), answers the
  same yes/no flags reframed for code (right code path, shows the named flaw, no guard in sight, impact not
  overstated), and re-runs every proof of concept in a throwaway Docker container: no network, the frozen copy
  mounted read-only, 2 GB, 1 CPU, 60 seconds. Reproducing the output is not enough: it then asks whether the code calls the
  repository's own code and prints its marker only because the flaw happened, so print("VULNERABLE") proves nothing.
• The rule: quotes from one repository are one source, and a finding backed only by quoted code is reported as
  unconfirmed. Only a reproduced, genuine proof of concept confirms one.

Code mode never fetches the web, keeps its findings out of the cross-run ledger, and refuses a workspace that is /,
your home folder, a hidden folder in it (~/.ssh, ~/.aws), or Hermes's own data. If the agents see the repository at a different path (a container mount),
map it with the workspace_map setting.

Install

hermes plugins install samshapley/hermes-crucible
hermes plugins enable crucible

After updating Crucible, restart Hermes (hermes gateway restart) so a running gateway loads the new version.

Then ask: "Put this through the Crucible: …". Optional: pin a different model family for the referee under
Settings → Models → Auxiliary → Crucible referee, for a more independent check.

Settings live under plugins.entries.crucible.settings (hermes config set plugins.entries.crucible.settings.<key> <value>):

How a run works

R0 FRAME     framer: sub-questions, each with a contract (population, exposure, outcome, comparison, timeframe)
R1 COMMIT    3 proposers commit atomic claims blind to each other, each with quoted evidence
R2 CHECK     referee: span present? 5 flags → grade; population/measure mismatches file a visible attack
R3 ATTACK    3 attackers (skeptic, methodologist, source auditor) search independently, aimed by leverage
R4 CHECK     referee rules on each attack on the narrowest question that settles it
R5 RESPOND   proposers answer, concede or narrow; refused moves come back with the rule and the fix
             repeat R3–R5 until a cycle produces no upheld attack, or the budget is spent
R6 RESOLVE   verdicts, Shapley credit for load-bearing evidence and decisive attacks, calibration, receipt

Design lineage

• Debate with verified quotes: Khan et al. 2024 (arXiv:2402.06782) found that debate makes judges more
  accurate when quotes are machine-checked against the source. Irving et al. 2018 (arXiv:1805.00899).
• Why not debate to consensus: multi-agent consensus loses accuracy to conformity
  (arXiv:2509.05396, 2506.01332). Crucible never treats agreement as evidence.
• Claim-level verification: FActScore, SAFE, VeriScore; "quote exists" ≠ "quote supports" (ALCE, AttributionBench).
• NVIDIA Auto Ontology's validation discipline: resolve the question contract first, separate evidence states,
  separate verdict flags instead of free-form judgements, fail closed, keep receipts.

Evaluation

Pre-registered before any arm ran: see evals/PREREGISTRATION.md and its four dated
amendments.

Stage 1 (baseline pilot, 24 questions, plain Hermes with web search): 53% of the control's 208 factual claims
were supported by the source it cited. 24% carried no citation, and of the cited claims the checker could read, 27%
were not supported by what they cited. 15 of 24 questions fell below the 80% bar and form the comparison set.

Stage 2 (control vs guard vs run, 15 questions, two models): a full Crucible run raised the share of claims
supported by their cited source by about 15 points over plain Hermes on both models (61% vs 45% on gpt-6.1-sol,
61% vs 43% on gpt-6.1-terra; pooled +15.1 points, 95% interval +0.5 to +31.6), and halved fabricated or unreadable
citations on terra. Guard mode's gain did not replicate and is reported as a null. Pilot numbers, one answer per
question: details, intervals and every deviation in evals/RESULTS.md.

Bug hunt (code mode): 17 real Python advisories published after the models' release, each checked out at the
commit before its fix, audited offline by plain Hermes and by Crucible on the same free model. The pre-registered
hypothesis holds by its letter and not in substance. Precision is 13% for Crucible and 12% for the control
(3/23 vs 5/42), a one-point difference that is noise. Detection is 3/17 vs 5/17, within the allowed margin, and
equal at 5/17 when tentative findings are counted. Crucible confirmed about half as many findings that were not
the advisory's bug (1.2 per case vs 2.2). In a hand-check of four per arm, three of the control's were false as
written and none of Crucible's were; Crucible's were true behaviour, but not shown to be security flaws. Crucible
took about five times as long. Details, every amendment and the per-case table are in
evals/bughunt/RESULTS.md.

Honest limits

• LLM agents don't optimise points the way rational players do. Scores act as selection pressure (what survives,
  where the next round's attacks go, what feedback agents get), not motivation.
• The referee is a model answering narrow questions. It's tested on fixtures, every grade links to the exact page
  snapshot it was checked against, and it can still be wrong. In code mode it errs toward holding a real finding back
  when no proof reproduces it, as in the example above.
• No board can see evidence an agent chose not to write down. Independent attackers who search for themselves are the
  answer to cherry-picking, not a stricter rule.
• No claim survives on one quote, so a claim about what one source says ("Britannica states X") cannot survive: no
  second source can confirm it. Proposers are told to narrow the fact instead.
• Runs cost real tokens. Guard mode is cheap; full runs are for questions that deserve them.

Disclosures

• Network: fetches cited pages through Hermes web_extract and directly over HTTPS; uses NCBI E-utilities and
  OpenAlex for walled scholarly abstracts. Agents choose the URLs, so Crucible fetches only public addresses: never
  this machine, its local network or a cloud metadata service, before or after a redirect. Proposer and attacker
  subagents use the web toolset. Code mode makes no network requests of its own; its subagents get the file and
  terminal toolsets, and their commands run wherever Hermes's terminal backend runs: on this machine unless
  you set terminal.backend to docker. They read the code under audit, which can try to instruct them, so
  audit untrusted repositories with a Docker terminal backend (the bug-hunt eval does).
• Code execution: in research mode nothing is executed. Research agents read the web, and a page can tell an
  agent what to write, so their calc evidence is evaluated by a small arithmetic interpreter (numbers, names,
  operators, print, a few pure builtins and math) that has no route to files, imports or the network, and that
  refuses numbers, text and nested lists past fixed sizes, so a calc cannot exhaust memory or time. In code mode, proofs of concept run in a docker run --rm --network none --read-only
  container with the frozen copy of the repository mounted read-only and every Linux capability dropped; without Docker they are not run at all, and the report says so.
• Model calls: subagents and the referee spend your configured model's tokens. Each run has a USD cap, priced
  with Hermes's own tables from every model call, the subagents' included (their context compression too); agents
  still working when it is reached are stopped, and agents that had already finished keep their work. A model Hermes has no price for makes the shown cost a lower bound, and the report says so. Pricing uses one private Hermes function (agent.usage_pricing), since
  plugins have no public pricing API; if a release changes it, costs show as a lower bound instead of failing.
• Storage: runs, page snapshots and the ledger live under $HERMES_HOME/plugin-data/crucible/. A code-mode run
  also copies the repository there while it plays, and deletes the copy when it ends. No credentials
  are read or stored, and there is no telemetry.
• Hooks: post_tool_call records pages fetched during a turn or run; transform_llm_output rewrites the final
  answer only when guard mode is on; subagent_start, post_api_request, post_auxiliary_call and
  subagent_stop price Crucible's own subagents' model calls for the run's cap; other subagents' calls are dropped
  when those subagents stop.

Development

python -m pytest tests           # board rules, credence, credit, referee, code mode, full scripted games, guard, API, the plugin surface
ruff check .                     # lint (ruff.toml)
hermes plugins validate .        # catalog checks

The board (engine/) is pure Python with no Hermes imports. The game (core/) takes Hermes surfaces as injected
callables, so all of it runs under test with fakes.

MIT licensed.

README from the reviewed commit c8d73eb; it updates when the author re-pins.