All plugins

sluice-search

Tiered free-first web search and extraction router and backend manager for Hermes Disclosure — routes web search and page extraction through third-party engines the user enables (keyless public endpoints such as DuckDuckGo via ddgs and the keyless modes of Exa, Parallel, Firecrawl, Tavily and Keenable; paid APIs with the user's own keys; an optional local SearXNG), so queries and fetched URLs go to those services; fetches pages itself with httpx and, when installed, a headless Playwright Chromium and a local Camofox server, refusing redirects and later navigations to private or metadata addresses on the httpx and Chromium paths (Camofox's own redirects are not checked); hermes sluice-search install runs npm/podman/docker/playwright installs (Camofox pinned by digest/version) with a minimal environment.

Abhigyan ChatterjeeWebv1.0.0Updated 7 days ago

About

Tiered free-first web search and extraction router and backend manager for Hermes Disclosure — routes web search and page extraction through third-party engines the user enables (keyless public endpoints such as DuckDuckGo via ddgs and the keyless modes of Exa, Parallel, Firecrawl, Tavily and Keenable; paid APIs with the user's own keys; an optional local SearXNG), so queries and fetched URLs go to those services; fetches pages itself with httpx and, when installed, a headless Playwright Chromium and a local Camofox server, refusing redirects and later navigations to private or metadata addresses on the httpx and Chromium paths (Camofox's own redirects are not checked); hermes sluice-search install runs npm/podman/docker/playwright installs (Camofox pinned by digest/version) with a minimal environment.

Sluice

Sluice is the web layer for Hermes. One plugin serves both native web tools,
web_search and web_extract, through a single routing provider over twelve backends, and
manages every engine it can use: a local SearXNG, keyless public endpoints, free-plan keys
and paid keys. Configure it once and every Hermes surface, including cron sessions, gateway
sessions, subagents and the Desktop app, routes the same way without learning a new tool
name.

  • Free first. Routing order is local, keyless, free-plan keys, then paid. Paid is last
    by default and never exclusive.
  • Visible state. Per-backend health, latency, quota and circuit state survive restarts.
    Every result row names the backend and tier that answered it.
  • Local extraction. httpx plus trafilatura first, real Chromium through Playwright
    second, the Camofox REST server third, with a deterministic block classifier deciding
    when to climb and when to stop.
web_search ──┐
             ├─► SluiceWebSearchProvider ─► Router ─► 12 backends ─► SQLite state
web_extract ─┘                              │
                                            └─► Fetch Ladder ─► http │ chromium │ camofox

Requirements: Hermes 0.22.5 or newer (plugin.yaml), Python 3.11 or newer.

Why Sluice

Hermes ships a provider registry and a config key you edit by hand, but no surface to
install, test, monitor or switch backends. Sluice provides that surface, and it serves the
native tools rather than adding a second search API to learn. It keeps per-backend health,
latency, quota and circuit state across restarts, caches searches and extracts, and names
the backend that answered on every row.

Install

hermes plugins install sluice-search
hermes plugins enable sluice-search
hermes plugins doctor sluice-search --ci
hermes sluice-search doctor

The plugin is named sluice-search everywhere (catalog key, manifest name,
install directory, CLI command and provider id) because an unrelated plugin
already owns the sluice name in the upstream catalog, and Hermes names the
install folder and the settings key from the manifest name.

hermes plugins install asks before installing the plugin's four Python dependencies
(httpx, trafilatura, playwright, ddgs). doctor --ci exits non-zero on a manifest or
registration error. hermes sluice-search doctor prints the twelve-backend inventory: tier, plan,
key state and what is missing.

To route Hermes through Sluice:

hermes sluice-search setup --apply

That runs three hermes config set commands and nothing else:

hermes config set web.search_backend sluice-search
hermes config set web.extract_backend sluice-search
hermes config set web.cache_enabled false

Observed:

Sluice config route (Sluice never writes config.yaml itself):
hermes config set web.search_backend sluice   [ok]
hermes config set web.extract_backend sluice   [ok]
hermes config set web.cache_enabled false   [ok]

done: Hermes now routes web_search and web_extract through Sluice, outer cache off

The outer Hermes cache is turned off so there is exactly one cache with one TTL. Without
--apply, setup only prints the commands. Sluice never writes config.yaml itself and
never patches Hermes core.

Each route is also available on its own as hermes sluice-search use search [--apply] and
hermes sluice-search use extract [--apply]. The same operations work from a checkout through the
standalone console script:

.venv/bin/sluice doctor
.venv/bin/sluice test --backend exa --limit 2 "sqlite wal checkpoint"

Minimal configuration

None. The keyless tier is on by default, so Exa's MCP endpoint, Parallel's MCP endpoint,
Firecrawl, Tavily and Keenable answer with no key at all, and ddgs runs from a local child
process with no key either. A backend with a key runs keyed; a backend without one runs its
keyless path when keyless tiers are enabled; otherwise it is skipped and the reason is
printed.

API keys are set in Settings ▸ Plugins, not through the CLI and not through the pane:
Hermes stores them in the profile's .env and Sluice reads presence through
get_provider_env(), never writes a value. There is no key input anywhere in the plugin.

The full settings reference, including every default and effect, is in
docs/configuration.md.

Quick start

Save this as example.py at the root of a checkout and run it with the project's
interpreter. It does one keyless search and one extraction against the deterministic fixture
server in demo/, so it needs no keys and no configuration. The search needs network; the
extract does not.

"""One keyless search and one extract. No API keys, no config."""

from demo.fixture_server import running_server
from sluice.provider import SluiceWebSearchProvider

if __name__ == "__main__":
    provider = SluiceWebSearchProvider()

    result = provider.search("sqlite wal checkpoint", limit=2)
    print("search success:", result["success"])
    for row in result["data"]["web"]:
        print(f'  {row["position"]}. {row["title"]}  [{row["backend"]}/{row["tier"]}]')
        print(f'     {row["url"]}')

    with running_server() as base:
        for url in (f"{base}/ok", f"{base}/js", f"{base}/private"):
            item = provider.extract([url])[0]
            print(f'{url.rsplit("/", 1)[-1]:>8}: {item["block_class"]} via {item["stage"]}')
            print("          ", (item.get("content") or "").splitlines()[0])
uv venv
uv pip install -e .
.venv/bin/sluice install playwright
.venv/bin/python example.py

sluice install playwright downloads Playwright's Chromium build. If it is skipped, the
Chromium rung uses a google-chrome, google-chrome-stable, chromium,
chromium-browser or chrome binary already on PATH instead. With neither present, the
/js line prints js: js_required via http and the other two lines are unchanged. The
output below was reproduced with the normal locale, LANG=C.UTF-8 and env -i.

Output from a run on 5 November 2026:

search success: True
  1. Write-Ahead Logging  [exa/keyless]
     https://sqlite.org/wal.html
  2. Checkpoint a database  [exa/keyless]
     https://sqlite.org/c3ref/wal_checkpoint_v2.html
      ok: ok via http
           [sluice: stage=http status=ok]
      js: ok via chromium
           [sluice: stage=chromium status=ok]
 private: policy_blocked via policy
           [sluice: stage=policy status=policy_blocked]

The three extract lines show the ladder at work on the fixture pages: /ok is read at
the HTTP rung, the JavaScript shell /js is read at the Chromium rung, and /private never
gets a request because robots.txt disallows it. Which keyless backend answers a search
varies with load and latency; every row names it. ddgs runs its query in a separate
interpreter that reads one JSON request on stdin and prints one JSON object, so the
if __name__ == "__main__": guard in the sample is conventional rather than required.

The same thing through Hermes, once the plugin is installed:

hermes sluice-search test --backend exa --limit 2 "sqlite wal checkpoint"
exa (keyless, keyless) answered in 886 ms with 2 result(s)

 1. Write-Ahead Logging
    https://sqlite.org/wal.html
 2. Checkpoint a database
    https://sqlite.org/c3ref/wal_checkpoint_v2.html

Commands

CommandWhat it does
hermes sluice-search doctortwelve-backend inventory, free/paid, key state, browser engines, the exact next command per gap
hermes sluice-search statuscircuit, quota, p50/EMA latency, calls, spend, cache size, per-domain block memory
hermes sluice-search setup [--apply]print (or run) the three config routes
hermes sluice-search use search|extract [--apply]switch one Hermes backend to Sluice
hermes sluice-search test --backend <id> "query"one backend, one query, latency and rows
hermes sluice-search cache clearclear the Sluice SQLite cache and Hermes' cache/web
hermes sluice-search install playwright|camofox|browserbaserun an install job; never automatic

Every command is unattended-safe: it never prompts, returns non-zero on failure, and prints
the command that comes next.

Backends

One entry per vendor. A vendor with a key runs keyed; without one it runs its keyless path
when keyless tiers are enabled; otherwise it is skipped.

BackendTierSearchExtractKey
SearXNGlocalyesnonone (self-hosted URL)
ddgskeylessyesnonone
Exakeyless / free / paidyesyesEXA_API_KEY
Parallelkeyless / paidyesyesPARALLEL_API_KEY
Firecrawlkeyless / free / paidyesyesFIRECRAWL_API_KEY
Tavilykeyless / free / paidyesyesTAVILY_API_KEY
Keenablekeyless / paidyesyesKEENABLE_API_KEY
Bravefree / paidyesnoBRAVE_API_KEY
PerplexitypaidyesyesPERPLEXITY_API_KEY
DeepSeekpaidyesyesDEEPSEEK_API_KEY
TinyFishpaidyesnoTINYFISH_API_KEY
xAIpaidyesnoXAI_API_KEY

Search-only backends raise NotImplementedError from extract, and the router never calls
them for extraction: web_extract is served by the local Fetch Ladder. Per-vendor plans
(plan_exa, plan_brave, ...) only change routing order inside the keyed tiers. A free key
is never labelled paid, and a paid backend never jumps a free one.

Each search attempt gets a 2500 ms window by default. A backend may declare a wider window
of its own, and ddgs does: it is a local multi-engine scrape rather than an API call, and
a live call regularly exceeds the 2500 ms default, so ddgs declares 8000 ms. The wider
window only ever widens that attempt, and the 8000 ms total budget is shared by the whole
ladder: while a candidate is still behind the current one, the router keeps a default 2500 ms
window in reserve for it, so with the defaults ddgs gets 5500 ms and the next backend is
still attemptable.

Network and process disclosures

Sluice makes network calls only when you or a Hermes tool ask it to. The destinations:

  • Search and extract APIs: the twelve vendors at their documented hosts:
    api.exa.ai and mcp.exa.ai, api.parallel.ai and search.parallel.ai,
    api.firecrawl.dev, api.tavily.com, api.keenable.ai, api.search.brave.com,
    api.perplexity.ai, api.deepseek.com, api.search.tinyfish.ai, api.x.ai, plus your
    own SearXNG at the searxng_url setting. The keyless paths (Exa MCP, Parallel MCP,
    Firecrawl, Tavily, Keenable public) send no credential.
  • Target pages: extraction fetches the URL you asked for, its robots.txt, and, only
    when the ladder climbs, the same URL through Chromium or Camofox. The redirect-hop
    SSRF guard covers the httpx and Chromium rungs; the Camofox rung hands the URL to
    your Camofox server, whose own redirects the guard cannot see. Responses are not
    sent anywhere else.
  • Local servers: Camofox at CAMOFOX_URL (default http://localhost:9377) and SearXNG
    at searxng_url or SEARXNG_URL, if you run them. Health is checked with GET /health,
    and SearXNG with GET /search.

Local processes and binary use:

  • Playwright Chromium is launched headless by the ladder, one browser per fetch, with a
    15 s cap. hermes sluice-search install playwright runs python -m playwright install chromium,
    which downloads the browser build. Nothing is installed otherwise.
  • Camofox is a Node server you run yourself. hermes sluice-search install camofox either
    starts the container sluice-camofox from ghcr.io/jo-inc/camofox-browser@sha256:49f8… (podman
    first, then docker) or, with no container runtime, installs @askjo/camofox-browser@1.19.0 into
    the plugin data directory and prints the npx command that starts it. The first start
    downloads Camoufox (the vendor calls it "hundreds of MB"). The pane's Stop button runs
    podman|docker stop sluice-camofox.
  • Packaged Chromium (google-chrome, chromium, ...) is detected by the doctor and
    never launched by Sluice.

Shell commands Sluice may run, always from an explicit user action:

python -m playwright install chromium
podman|docker run -d --name sluice-camofox -p 9377:9377 -e CAMOFOX_PORT=9377 ghcr.io/jo-inc/camofox-browser@sha256:49f8e02f9fb0e191b406845482745efc72f835db06789e3a34e77a6ff1497046
podman|docker stop sluice-camofox
npm install --prefix <plugin data dir>/camofox @askjo/camofox-browser@1.19.0
hermes config set web.search_backend sluice-search      # only with --apply
hermes config set web.extract_backend sluice-search     # only with --apply
hermes config set web.cache_enabled false        # only with --apply

Data and credentials

The SQLite store lives in the plugin data directory, ctx.plugin_data_dir(), normally
~/.hermes/plugin-data/sluice/sluice.db, or ~/.hermes/plugins/sluice-search/data outside a
Hermes context. It holds:

  • the search and extract cache, with TTLs (search 900 s general and 300 s news, extract
    3600 s, failed URL 60 s) and LRU eviction down to a low-water mark when the file passes
    200 MB;
  • backend state: circuit breakers (3 consecutive failures, 15 min; one quota error, 60 min),
    quota windows, latency EMA and p50 samples;
  • usage rows (backend, kind, ok, latency, cost, time) and per-domain block memory for 24 h;
  • install job records.

No credential is ever written to that store. Keys live in the profile .env through Hermes
and are read through get_provider_env(), or os.environ outside Hermes; the pane and the
API expose presence and the env var name only.

The dashboard API is mounted at /api/plugins/sluice-search-search/ and speaks JSON only. The Desktop pane
polls /status every 5 s while visible and holds no secret.

Documentation

  • docs/configuration.md: every setting, environment variable,
    default and effect.
  • docs/architecture.md: module map, the search and extract data
    paths, the Hermes seams, and the extension points.
  • docs/development.md: dev environment, tests, the fixture server,
    local SearXNG, validation, release.
  • DESIGN.md and REQUIREMENTS.md: the design and the requirements behind the
    implementation.
  • references/endpoints.md: request and response shapes for all twelve vendors.

Scope for 1.0

Not in this release:

  • Parallel or racing fan-out and result fusion. Routing is sequential.
  • Per-domain routing rules set by the user. Domain memory is automatic.
  • TLS impersonation profiles.
  • Answer synthesis and deep research mode.
  • A web dashboard that is not the Desktop pane.
  • The openai-native backend.
  • A custom CDP browser provider.
  • Automatic SearXNG deployment. The compose file is a reference; nothing starts it for you.
  • CAPTCHA solving, paywall or login circumvention.
  • Per-call backend pinning in web_search(query, limit) or web_extract(urls, char_limit).

Development

uv venv
uv pip install -e ".[dev]"
.venv/bin/pytest -q

Live vendor calls are opt-in with SLUICE_LIVE_TESTS=1. The fixture server for the demo is
python demo/fixture_server.py --port 8765. The full loop is in
docs/development.md.

License

MIT. See LICENSE.

README from the reviewed commit 2737453; it updates when the author re-pins.

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section