All plugins

lemma

An AI-scientist plugin that audits what your agent reads. Paste or mention a paper link and the ambient pre-LLM hook offers an audit unprompted; six tools (quick_audit, audit, doctor, status, judge, improve) extract testable claims, generate and run honest audit scripts behind a mandatory positive control, and render verdicts — supported, falsified, inconclusive — over an append-only evidence trail. The Desktop Audit Room pane shows live runs (stage, attempts, trace tail) and self-distrust flags (verify_gain) when a post-feedback flip to supported needs eyeballing. Everything stays local under plugin_data_dir. Disclosure — paper text and audit prompts go to your configured LLM provider (Anthropic, or any OpenAI-compatible endpoint you set via OPENAI_BASE_URL); papers are fetched from arxiv.org, export.arxiv.org and openreview.net; LLM-written audit scripts run inside a sandbox (bubblewrap on Linux, sandbox-exec on macOS: no network, workdir-only writes, minimal env) and are refused where no sandbox exists unless you turn on the plugin's `host_exec` setting, which runs them on the host without isolation; `lemma_audit` and `lemma_improve` run as detached background processes; a pre-LLM hook adds an "offer an audit" hint when a paper link appears in the conversation; all artifacts stay under the plugin data directory.

udirobertToolsv0.3.0Updated 5 hours ago

About

An AI-scientist plugin that audits what your agent reads. Paste or mention a paper link and the ambient pre-LLM hook offers an audit unprompted; six tools (quick_audit, audit, doctor, status, judge, improve) extract testable claims, generate and run honest audit scripts behind a mandatory positive control, and render verdicts — supported, falsified, inconclusive — over an append-only evidence trail. The Desktop Audit Room pane shows live runs (stage, attempts, trace tail) and self-distrust flags (verify_gain) when a post-feedback flip to supported needs eyeballing. Everything stays local under plugin_data_dir. Disclosure — paper text and audit prompts go to your configured LLM provider (Anthropic, or any OpenAI-compatible endpoint you set via OPENAI_BASE_URL); papers are fetched from arxiv.org, export.arxiv.org and openreview.net; LLM-written audit scripts run inside a sandbox (bubblewrap on Linux, sandbox-exec on macOS: no network, workdir-only writes, minimal env) and are refused where no sandbox exists unless you turn on the plugin's `host_exec` setting, which runs them on the host without isolation; `lemma_audit` and `lemma_improve` run as detached background processes; a pre-LLM hook adds an "offer an audit" hint when a paper link appears in the conversation; all artifacts stay under the plugin data directory.

lemma — the agent that audits what agents read

Lemma audits scientific claims end to end: paste a paper (arXiv id,
OpenReview id, or a local PDF) and it extracts testable claims, writes
and runs numerical audit scripts with a mandatory positive control, and
judges the resulting evidence trail — supported / falsified /
inconclusive, never a bare paragraph. Every artifact is preserved under
plugin_data_dir("lemma")/papers/<slug>/ (claims.json,
audit_report.json, judge_report.json, trace.jsonl).

Setup

hermes plugins install lemma        # catalog install, pinned commit

At install you are prompted for ANTHROPIC_API_KEY (declared
requires_env, stored through Hermes's credential route — the plugin
reads it via agent.secret_scope.get_secret, never from .env or
files). Prefer a different provider? Set OPENAI_API_KEY instead —
plus OPENAI_BASE_URL/OPENAI_MODEL for OpenAI-compatible endpoints
(Venice, OpenRouter, a local server) — and skip the Anthropic prompt;
agent/llm.py's provider order applies (named endpoints → anthropic →
openai). Python deps install into the Hermes environment from
python_dependencies in plugin.yaml.

The audit pipeline ships inside the plugin: catalog installs use
subdir: hermes-plugin, so only this directory is checked out. The
pipeline modules the audit/judge/improve/evidence stages need live in
lemma_pipeline/ — a vendored copy of agent/ generated by
scripts/sync_hermes_plugin.py (the package is deliberately not named
agent, which would collide with hermes-agent's own agent package on
sys.path). agent/ remains canonical; the sync script is enforced by
a pre-commit check. Repo-only surfaces (lemma design, lemma eval,
lemma room, extract-server) are not vendored — the plugin's tools
never invoke them. The code that runs is still the code that was
reviewed: the vendored tree is part of the pinned commit.

Tools / hooks / commands

| Surface | What it does |
|---|---|
| lemma_doctor | preflight: pipeline launch, python deps, trackio, sandbox, credential route — JSON checks + issues |
| lemma_quick_audit | extract → audit first N claims (≤5) → judge; bounded ~10 min |
| lemma_audit | full pipeline in a detached background run; poll lemma_status |
| lemma_status | per-workdir verdicts, judge result, trace/evidence paths |
| lemma_judge | re-run the evidence-trustworthiness judge on a workdir |
| lemma_improve | feedback loop → re-audit → before/after record (advisory) |
| /audit <paper> | slash command → lemma_quick_audit |
| pre_llm_call hook | notices paper links in user turns and the agent's own citations; injects an offer, never runs unasked |
| Audit Room pane | Desktop view backed by dashboard/plugin_api.py at /api/plugins/lemma/* |

Network endpoints

  • api.anthropic.com — auditor/extractor/improve LLM calls (your key);
    or your OPENAI_BASE_URL endpoint when using the OpenAI-compatible route
  • export.arxiv.org, arxiv.org, openreview.net — paper fetches
  • api.firecrawl.dev — only when FIRECRAWL_API_KEY is set in env
    (PDF/page parsing; lemma falls back to PyMuPDF without it)
  • paperclip.gxl.ai — only for lemma search (not wired to a tool)
  • *.wandb.ai — only when the plugin's Weave setting is enabled
    (off by default; WANDB_API_KEY alone never enables tracing)
  • pypi — plugin dependency install, at enable time (includes trackio
    for the evidence/logbook stage)

Generated code — how it executes

The audit stage runs scripts written by an LLM from untrusted paper
text — a prompt-injected paper can smuggle code. Policy
(agent/script_sandbox.py):

  1. macOS: sandbox-exec deny-by-default profile — no network, no exec
    outside the interpreter, writes only under the paper workdir, home
    directories unreadable, rlimits + minimal env.
  2. Linux: bwrap — private namespaces (no network), read-only system
    binds, workdir is the only writable mount, same env/rlimits.
  3. Elsewhere: generated code does not run — attempts record
    sandbox_unavailable and the claim is left inconclusive.
    The plugin's Host exec setting is an explicit operator opt-in for
    unsandboxed platforms (still a minimal env — no inherited secrets —
    but no isolation; off by default).

Pipeline children themselves spawn with a minimal env (PATH, HOME,
the LLM credential vars above resolved through agent.secret_scope,
lemma markers, and the dependency env's site-packages as
PYTHONPATH) — never the whole Hermes process environment.

Settings

Two behaviour toggles live in plugin config (config_schema in
plugin.yaml, editable under Desktop → Settings → Plugins or
hermes config set plugins.entries.lemma.settings.<key>), not .env:

| Key | Default | Effect |
|---|---|---|
| host_exec | false | run generated audit scripts without the sandbox (unsafe opt-in; equivalent to LEMMA_AUDIT_HOST_EXEC=1 for direct runs) |
| weave | false | send pipeline traces to Weave (equivalent to LEMMA_WEAVE=1) |

The evidence stage needs the trackio CLI, installed automatically via
python_dependencies; it lands next to the plugin interpreter where
lemma_pipeline.evidence looks for it first.

Detached background runs

lemma_audit / lemma_improve return immediately and run detached
(start_new_session) — they keep working after the turn ends, logging
to lemma_audit.log / lemma_improve.log inside the workdir. Poll
with lemma_status or the Audit Room pane. lemma_quick_audit and
lemma_judge run in-process with timeouts.

File access

Reads/writes only under plugin_data_dir("lemma") plus the pinned
pipeline source. Local paper sources are restricted to .pdf files
with a %PDF header inside the plugin papers dir (or a
LEMMA_LOCAL_PDF_DIRS-listed dir). Workdir paper parameters accept
names only — paths and .. are rejected.

README from the reviewed commit 98b061d; it updates when the author re-pins.

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section