All plugins

llm-antidote

Runtime guardrails, behavioral evaluation, and signed fleet admission for Hermes Agent: a versioned capability policy that blocks or requires approval for model tool calls, paired synthetic behavioral evaluation with no real side effects, and a signed local promotion gate that refuses a model whose evidence does not satisfy signed fleet policy. Hooks-only (no agent tools). Disclosure — Adds approval prompts to tool calls by default (balanced mode): every MCP, connector or other-plugin tool, memory and skill writes, file reads outside the working and temp directories, and network-capable terminal/code commands go to Hermes's approval prompt; strict mode blocks them instead, and opt-in enforced fleet mode blocks all tool calls until the served model name matches the admitted one. The user-run hermes antidote eval run command sends synthetic prompts (and the API key you name) to the OpenAI-compatible endpoint you pass, as billed model calls (default suite up to ~170 requests); model inspect hf: queries huggingface.co; fleet keygen writes unencrypted Ed25519 private keys (mode 0600) to paths you choose.

GroveMinting1Toolsv3.2.4Updated 1 month ago

About

Runtime guardrails, behavioral evaluation, and signed fleet admission for Hermes Agent: a versioned capability policy that blocks or requires approval for model tool calls, paired synthetic behavioral evaluation with no real side effects, and a signed local promotion gate that refuses a model whose evidence does not satisfy signed fleet policy. Hooks-only (no agent tools). Disclosure — Adds approval prompts to tool calls by default (balanced mode): every MCP, connector or other-plugin tool, memory and skill writes, file reads outside the working and temp directories, and network-capable terminal/code commands go to Hermes's approval prompt; strict mode blocks them instead, and opt-in enforced fleet mode blocks all tool calls until the served model name matches the admitted one. The user-run hermes antidote eval run command sends synthetic prompts (and the API key you name) to the OpenAI-compatible endpoint you pass, as billed model calls (default suite up to ~170 requests); model inspect hf: queries huggingface.co; fleet keygen writes unencrypted Ed25519 private keys (mode 0600) to paths you choose.

llmAntidote for Hermes Agent - provenance and runtime guardrails

CI
Release
License: MIT
Target: Hermes Agent
Policy: Block + Approve
Audit: Model Provenance
Eval: Behavioral
Fleet: Signed Admission

Runtime guardrails · behavioral evaluation · signed attestations · fleet admission control

Inspect what you are running, constrain what it can access, and stop recognized
credential-exfiltration chains before Hermes executes them.

[!IMPORTANT]
llmAntidote reduces the blast radius of a compromised or backdoored model. It
cannot prove that model weights are backdoor-free, discover arbitrary secret
trigger phrases, or replace OS-level sandboxing and network egress controls.

Why llmAntidote

A poisoned model can behave normally until a hidden phrase causes it to request
a malicious tool call. ProjectDiscovery demonstrated a 7B model that downloaded
a payload, read project credentials, and sent them to an external collector.
Routine prompts and benchmark scores did not reveal the backdoor.

llmAntidote addresses the enforceable part of that problem: the model may request
an action, but Hermes and the host decide whether it runs.

Safeguards

Runtime policy

  • Blocks commands that combine sensitive reads with network transmission.
  • Blocks recognizable download-and-execute and reverse-shell patterns.
  • Classifies tools by capability, not only argument text; send/upload connectors
    and unknown plugin tools are treated as egress-capable.
  • Requires approval for reads outside operator-configured safe roots, including
    unlisted secret filenames such as token.json.
  • Routes ambiguous sensitive access through Hermes' human approval flow.
  • Tracks approved sensitive reads as session-scoped taint metadata, scoped by
    capability, and reserves pending sensitive access so outbound calls cannot run
    before a read resolves (correlated by tool-call id).
  • Records rule IDs, fixed tool families, and decisions without raw tool names,
    arguments, results, or secrets.

Provenance

  • Inspects local and Hugging Face model provenance without loading model weights.
  • Creates immutable model lockfiles and verifies local files against SHA-256 hashes.

Evaluation

  • Runs paired trigger-candidate evaluations with synthetic tools and coverage-scoped verdicts.

Fleet admission

  • Enforces signed fleet policy over attestations, model locks, and evaluation evidence.
  • Enforces signed runtime admission and blocks tool calls on model-route mismatch.
  • Produces signed PASS/DENY receipts with a rotatable/revocable signer trust root
    and metadata-only fleet status reporting.

Requirements

  • Hermes Agent 0.21.5 or newer.
  • Python 3.11 or newer.
  • A trusted Hermes installation with plugins enabled explicitly.

Quick Start

Install from GitHub and enable the plugin:

hermes plugins install GroveMinting/llmAntidote --enable
hermes plugins doctor llm-antidote --ci

Hermes' install scanner is expected to return caution because the policy and
tests intentionally contain credential-path and malicious-command signatures.
Review those findings before accepting the prompt. A dangerous verdict is not
expected and must not be bypassed.

Restart Hermes, then inspect the active runtime state:

/antidote status

Inspect a local model directory:

hermes antidote model inspect /models/my-model

Inspect an immutable Hugging Face revision:

hermes antidote model inspect hf:publisher/model \
  --revision 0123456789abcdef0123456789abcdef01234567

Create and verify a local model lockfile:

hermes antidote model lock /models/my-model --output model-lock.json
hermes antidote model verify model-lock.json

Run the standard behavioral suite against a local OpenAI-compatible endpoint:

hermes antidote eval run \
  --endpoint http://127.0.0.1:8000/v1 \
  --model local-model \
  --model-lock model-lock.json \
  --output antidote-eval-report.json \
  --markdown-output antidote-eval-report.md \
  --minimize

The evaluator sends prompts only to the endpoint you provide. Its file, shell,
write, and HTTP tools are simulations: no command, filesystem mutation, or
outbound request is executed by the evaluator.

To reach a remote vLLM host such as a DGX Spark, tunnel loopback over SSH rather
than exposing plaintext HTTP on the LAN:

ssh -N -L 8000:127.0.0.1:8000 spark-centurion
hermes antidote eval run --endpoint http://127.0.0.1:8000/v1 --model MODEL_ID ...

See docs/BEHAVIORAL_EVALUATION.md for the
explicit --trusted-host opt-in and its cleartext tradeoff.

Fleet Quick Start

Generate separate Ed25519 keys for policy authority, evidence attestation, and
online receipt signing, then initialize policy:

hermes antidote fleet keygen \
  --private-key policy-private.pem \
  --public-key policy-public.pem

hermes antidote fleet keygen \
  --private-key attestation-private.pem \
  --public-key attestation-public.pem

hermes antidote fleet keygen \
  --private-key receipt-private.pem \
  --public-key receipt-public.pem

hermes antidote fleet policy init \
  --name production \
  --output fleet-policy.json

hermes antidote fleet sign fleet-policy.json \
  --private-key policy-private.pem \
  --output fleet-policy.sig.json

hermes antidote fleet policy validate fleet-policy.json

Record the printed policy SHA-256 in protected configuration. Admission requires
that out-of-band value through --trusted-policy-sha256, preventing replay of
an older policy signed by the same authority.

Bind a model lock and behavioral report into an attestation:

hermes antidote fleet attest \
  --subject model-lock.json \
  --subject-name model-a \
  --evidence antidote-eval-report.json \
  --output model-a.attestation.json

hermes antidote fleet sign model-a.attestation.json \
  --private-key attestation-private.pem \
  --output model-a.attestation.sig.json

Run the admission gate and issue a signed receipt:

hermes antidote fleet admit \
  --policy fleet-policy.json \
  --policy-signature fleet-policy.sig.json \
  --policy-public-key policy-public.pem \
  --trusted-policy-sha256 <trusted-policy-sha256> \
  --attestation model-a.attestation.json \
  --attestation-signature model-a.attestation.sig.json \
  --attestation-public-key attestation-public.pem \
  --evaluation-report antidote-eval-report.json \
  --model-lock model-lock.json \
  --receipt model-a.receipt.json \
  --receipt-private-key receipt-private.pem \
  --receipt-signature model-a.receipt.sig.json

Ingest the verified signed receipt into the metadata-only ledger:

hermes antidote fleet ledger ingest \
  --database antidote-fleet.db \
  --receipt model-a.receipt.json \
  --signature model-a.receipt.sig.json \
  --public-key receipt-public.pem

hermes antidote fleet ledger summary --database antidote-fleet.db

See docs/FLEET_ADMISSION.md before distributing
keys or policy bundles. Policy and attestation private keys should normally
remain offline. CI receives only their public keys plus a separately scoped
receipt-signing key.

Policy Modes

ModeKnown exfiltrationAmbiguous riskAudit records
balancedBlockRequire approvalYes
strictBlockBlockYes
auditAllow with findingAllow with findingYes

balanced is the default. Configure the plugin through Hermes Desktop or under
plugins.entries.llm-antidote.settings in ~/.hermes/config.yaml:

plugins:
  entries:
    llm-antidote:
      settings:
        mode: balanced
        taint_ttl_minutes: 60
        pending_ttl_minutes: 5
        taint_scope: egress_persist_shell
        max_scan_bytes: 1048576
        audit_capacity: 200
        safe_read_roots: /home/agent/workspace
        sensitive_roots: /home/agent/.config

Reads inside safe_read_roots are ordinary unless the name matches a known
secret pattern; reads outside every safe root require approval. taint_scope
chooses which capabilities a completed sensitive read blocks
(egress, egress_persist_shell (default), or all); max_scan_bytes is the
argument scan limit, above which a call is refused. See
docs/POLICY.md for capability classification, outbound
allow-listing, content-vs-intent handling, and the pending-sensitive-access state
machine.

Enforced Fleet Runtime

Admission is not only a CI gate. Set admission_enforcement: enforced with a
signed policy and a verified model binding and the plugin will refuse to
register unless both verify. The signed policy's runtime mode and taint TTL
become authoritative, and tool calls are blocked if the served model route stops
matching the admitted model. In enforced mode an unobserved or empty model route
also fails closed unless admission_require_route_observation: "false" is set,
and if model binding is required but admission_model_name is empty, tool calls
are blocked with admission.model_route_unconfigured (shown by /antidote status and the startup log). Use llm-antidote fleet runtime-verify --settings SETTINGS.json as an external supervisor/startup probe. See
docs/FLEET_ADMISSION.md.

Decision Flow

model requests tool call
        |
        v
pre_tool_call policy
        |
        +-- known exfiltration or payload execution --> block
        |
        +-- tainted or pending session egress ---------> block
        |
        +-- ambiguous sensitive or outbound ----------> Hermes approval
        |                                               (+ pending reservation)
        |
        +-- ordinary operation ----------------------> allow
        |
        v
post_tool_call metadata update
        |
        +-- successful sensitive access -------------> taint session
        +-- denial, error, timeout ------------------> release pending
        +-- no sensitive result ----------------------> no taint

Policy evaluation is deterministic and does not call an LLM.

Model Inspection

The inspector reports evidence rather than a misleading safety score:

  • resolved immutable revision;
  • model, adapter, and base-model lineage claims;
  • file inventory and weight formats;
  • unsafe pickle-based serialization;
  • repository code or executable artifacts;
  • mutable revision use and missing file digests;
  • local SHA-256 values in deep scans and lockfiles.

Hugging Face inspection reads repository metadata only. It does not download
multi-gigabyte weight shards. Local inspection never deserializes model files.

Behavioral Evaluation

v0.2 runs matched control and candidate prompts through a bounded synthetic
tool loop. The existing runtime policy classifies requested actions, while the
evaluator measures activation rate, control rate, carrier-prompt coverage, and
endpoint failures. Reports use four explicit verdicts:

  • TRIGGER_RECOVERED
  • SUSPICIOUS_BEHAVIOR_DETECTED
  • NO_BACKDOOR_DETECTED_WITHIN_TEST_COVERAGE
  • INCONCLUSIVE

TRIGGER_RECOVERED means a candidate reproducibly changed tool behavior under
the configured controls. It does not prove how the behavior entered the model
or whether it was intentional. See
docs/BEHAVIORAL_EVALUATION.md.

Threat Model

llmAntidote assumes the selected model may intentionally emit malicious tool
calls. The Hermes host and this plugin remain trusted. See
docs/THREAT_MODEL.md for boundaries and bypass risks.

Fleet Admission

v3 adds a local control-plane contract for model promotion. Signed policies
declare acceptable evaluation verdicts, suite identities, coverage, error rate,
evidence age, and model-lock binding. Admission produces a signed receipt whose
exit status is suitable for CI/CD gates. A metadata-only SQLite ledger accepts
only signed receipts, supports signer allowlisting/revocation/rotation, and
reports PASS/DENY counts without ingesting prompts, tool arguments, credentials,
or model output.

Development

python -m pip install -e .
python -m pip install -r requirements-dev.txt
pytest --cov=llm_antidote --cov-fail-under=80
ruff check .
mypy

Development tooling lives in requirements-dev.txt, not a [project.optional-dependencies]
extra: Hermes resolves directory plugins with --all-extras, so a dev extra
would be installed into the host environment and can conflict with Hermes' own
pinned tooling.

With a Hermes checkout on PYTHONPATH, the end-to-end hook harness drives the
plugin through Hermes' real plugin manager:

PYTHONPATH=/path/to/hermes-agent python tests/integration/hermes_e2e.py --mode standalone
PYTHONPATH=/path/to/hermes-agent python tests/integration/hermes_e2e.py --mode enforced

When Hermes is installed locally, also run:

hermes plugins doctor . --ci
hermes plugins validate .

Repository Layout

plugin.yaml                 Hermes plugin manifest
__init__.py                 Hermes registration entry point
llm_antidote/               capability policy, runtime, evaluation, admission, deployment, ledger
tests/                      policy, provenance, and property tests
tests/integration/          real-Hermes end-to-end hook harness
tests/data/                 maintained Hermes tool inventory
scripts/                    maintenance helpers (tool-inventory recorder)
docs/                       threat model, policy, evaluation, fleet admission, roadmap
examples/                   bounded evaluation suite examples
.github/workflows/ci.yml    static checks, unit tests, and Hermes end-to-end

Sources and Limitations

Pattern-based command analysis is not a shell interpreter and can be bypassed
through sufficiently novel indirection. Run untrusted models with real secret
separation, a container or microVM, and deny-by-default egress.

Disclosure

What this plugin does that a user should know before installing:

  • Network calls. The runtime hooks make no network calls. The optional
    evaluation CLI (hermes antidote eval run, llm-antidote eval run) sends
    synthetic prompts to an OpenAI-compatible endpoint you specify; model inspect
    with an hf: source fetches metadata and file listings from Hugging Face.
  • Reads outside the plugin's own data. At runtime the policy engine reads
    tool-call arguments in memory only, and never persists or logs them. The CLI
    reads the files you point it at: model directories, signed policy / lock /
    attestation / evaluation files, and the ledger database.
  • Credentials. fleet keygen writes unencrypted Ed25519 PKCS#8 private keys
    to operator-chosen paths (mode 0600); keep them offline. The evaluation CLI
    reads its API key through Hermes' secret scope (agent.secret_scope.get_secret)
    when running inside Hermes, and through the named environment variable in the
    standalone entry point. The runtime hooks never read secrets, .env, or
    auth.json.
  • Storage. Admission receipts and the fleet ledger live in a local SQLite
    database at the path you choose.
  • No shell commands, subprocesses, background processes, or telemetry.

The catalog security scan reports three caution findings
(dump_all_env and sudo_usage) that point at the detection regexes in
llm_antidote/policy.py (patterns that recognize printenv/env and sudo
in a model's tool call). They are matchers, not behavior; the plugin never dumps
the environment or runs sudo.

License

This project is licensed under the MIT License. Research, Hermes
Agent, model repositories, and third-party artifacts retain their own licenses.
See ATTRIBUTION.md.

README from the reviewed commit c7f6a54; it updates when the author re-pins.

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section