All plugins

live-video

Give Hermes eyes in a live voice conversation. Point your phone at the room, talk normally, and the agent answers in duplex voice whilst the camera streams: ask what it can see and it answers from the current frame; ask to be told when something comes into view and the relay checks the frames for 90 seconds, verifies a sighting against the held image before claiming it, and says it once. The six-action tool starts the relay, hands you the URL to open, reads a session's transcript back to the agent, and replays what it saw — measured first-token latency was 0.57 s and 1.03 s across two turns on a modest VPS. Disclosure — audio and camera frames stream from your browser to a relay this plugin starts on port 8098 (loopback by default; a LAN bind is opt-in and protected by a token minted per start), and from there to generativelanguage.googleapis.com (Gemini Live, which also answers watch checks) using GEMINI_API_KEY read from Hermes' secret scope; start spawns node server.js from the plugin's own directory as a background process, which spawns a Python frame-change tracker (numpy, Pillow, torch); the tool reads the relay's transcript.log and server.log back to the agent; watch verification spends the key's quota. No telemetry, and the key is never written to disk.

dacheahVoicev1.3.8Updated 3 days ago

About

Give Hermes eyes in a live voice conversation. Point your phone at the room, talk normally, and the agent answers in duplex voice whilst the camera streams: ask what it can see and it answers from the current frame; ask to be told when something comes into view and the relay checks the frames for 90 seconds, verifies a sighting against the held image before claiming it, and says it once. The six-action tool starts the relay, hands you the URL to open, reads a session's transcript back to the agent, and replays what it saw — measured first-token latency was 0.57 s and 1.03 s across two turns on a modest VPS. Disclosure — audio and camera frames stream from your browser to a relay this plugin starts on port 8098 (loopback by default; a LAN bind is opt-in and protected by a token minted per start), and from there to generativelanguage.googleapis.com (Gemini Live, which also answers watch checks) using GEMINI_API_KEY read from Hermes' secret scope; start spawns node server.js from the plugin's own directory as a background process, which spawns a Python frame-change tracker (numpy, Pillow, torch); the tool reads the relay's transcript.log and server.log back to the agent; watch verification spends the key's quota. No telemetry, and the key is never written to disk.

Hermes Live Video

Point your phone at the room, talk to your agent, and let it see.

Live Video: the page answering from the camera

Live Video gives Hermes a duplex voice conversation with a camera attached. Open a
page on your phone, speak normally, and the agent answers in under a second while
watching the feed. When you ask it to tell you that something has come into view, it
checks the frames and speaks the moment it appears — then goes quiet again.

The plugin is the control surface. The relay beside it is the capability: a Node
service that carries the audio, streams frames to Gemini Live, runs a local
frame-change tracker, and verifies sightings with a second model.

  phone browser ── audio + frames ──► relay (Node, :8098) ──► Gemini Live (duplex voice)
                                          │
                                          ├── vision_tracker.py   "has the scene changed?"
                                          └── watch verifier      "is the thing there yet?"
                                                    │
  you ── "let me know when …" ─────────────────────┘
                                                    │
                            agent speaks: "The blue screen is showing now, sir."

What it actually does

  • Duplex voice, sub-second. Model audio streams back while you speak; measured
    first-token latency on a modest VPS was 0.57 s and 1.03 s across two turns.
  • Sees, and says what it sees. Ask "describe what you're seeing" and the agent
    answers from the current frame — a white desk, a blue globe, a toy car — and it is
    instructed to say plainly when it cannot see, rather than inventing detail.
  • Watch mode. "Let me know when the kettle boils" arms a 90-second window. The
    relay looks at the view every few seconds and, on a sighting, asks the model to say
    so. If nothing appears, it asks whether to keep watching. One scene is described
    once, not once per glance.
  • A transcript you can review. Every session writes transcript.log; the plugin
    turns it into counts, turns, watch receipts and verifier latencies, so you can ask
    your agent afterwards what happened.

Why this is a plugin and not a skill

A skill could describe this workflow. It could not carry 16 kHz audio in both
directions, run a change detector on 224×224 frames at four frames a second, hold a
90-second window open across conversational turns, or verify a sighting against the
image before claiming it. That is real-time code on both sides of the conversation,
which is the point of the plugin system.

Requirements

PieceNeeds
RelayNode.js 18+ (ws)
Frame trackerPython 3 with numpy, Pillow, torch (CPU is fine; no model download)
ModelA Gemini API key with access to gemini-3.8-live; watch mode verifies with gemini-3-flash-preview

Linux, macOS and Windows are supported. On Windows the relay is spawned into its own
process group and stopped with taskkill /T, so the frame tracker is not orphaned; if
node or Python is not on PATH, set node_bin and PYTHON_BIN explicitly.

On the free tier, gemini-3-flash allows 20 generate-content requests per day per
project
, which one 90-second watch can exhaust. Watch mode realistically wants a
billing-enabled key; without it, the relay spends its budget, says so in the log, and
stops asking.

Install

The plugin directory is the relay: plugin.yaml, server.js, public/ and the
vision tracker ship in one tree, so what Hermes loads is the code the catalogue
pinned. Nothing is copied out of the repository.

# from the catalogue, or straight from the pinned commit
hermes plugins install live-video
hermes plugins install dacheah/hermes-live-video
hermes secrets set GEMINI_API_KEY                  # the plugin passes it to the relay

# the relay's own dependencies, in the plugin directory
cd ~/.hermes/plugins/live-video && npm ci
python3 -m pip install numpy pillow torch

On Windows, in PowerShell, the same three steps are:

hermes plugins install live-video
hermes secrets set GEMINI_API_KEY
cd "$env:USERPROFILE\.hermes\plugins\live-video"
npm ci
py -m pip install numpy pillow torch

npm ci and pip are only needed once per install; the relay reads ws from its own
node_modules, and the tracker picks the interpreter that can import numpy, PIL
and torch.

Then, in a Hermes session:

You: start live video

Hermes: (live_video → start) Relay is up on http://127.0.0.1:8098/.

Reaching it from a phone

The relay binds to 127.0.0.1 by default: nothing on the network can open it, and
nothing needs a token. That is the right default, and it is the only mode a
catalogue install uses unless the user changes it.

A phone is not on the loopback interface, so reaching the relay from one means
either a tunnel in front of it (keep the loopback bind — recommended, and camera
access requires a secure origin anyway) or an explicit LAN bind:

plugins:
  entries:
    live-video:
      bind: 0.0.0.0

On a non-loopback bind the plugin mints a random token per start, writes it to
<HERMES_HOME>/plugin-data/live-video/relay.token with mode 0600, passes it to
the relay as LIVE_TOKEN, and returns URLs that already carry it —
http://192.168.1.24:8098/?token=…. The relay refuses /, the static files and
the /ws upgrade without that token, and refuses any request whose Origin is
not the host it was addressed as. /healthz stays open so a supervisor can probe
it; it reports no key material. Stopping the relay deletes the token.

The tool

One tool, six actions:

ActionWhat it does
startLaunch the relay if it is not already up; returns local and LAN URLs, pid and health
stopStop a relay this plugin started (a systemd-run relay is left to systemd, and stop says so)
urlThe address to open on the phone, plus health
statusRelay health, service state, tracker processes, watch state
sessionRecent transcript: counts by tag, window, user turns and replies
checksWatch verifier behaviour, grouped per armed window: triggers, verdicts, latencies

start reads GEMINI_API_KEY through Hermes' secret scope and hands it to the child
in its environment. It never writes the key to disk. A relay you run yourself as a
service keeps running its own way — the plugin does not restart, reconfigure or
attach to it.

Configuration

Everything is optional. The defaults are this plugin's own directory — the relay
ships in the same reviewed tree as the manifest — port 8098, and a loopback bind.

plugins:
  entries:
    live-video:
      relay_dir: ""                    # empty = this plugin's own directory
      port: 8098
      bind: 127.0.0.1                  # loopback; see "Reaching it from a phone"
      node_bin: ""                     # default: node on PATH, else Hermes' bundled node
      unit: hermes-live-video          # systemd user unit, if you run it as one
      window_s: 90                     # watch window length used when reading logs
      page_title: ""                   # name on the phone page; empty = "Live Video"

Relay-side tuning lives in the environment, not the source:

VariableDefaultMeaning
LIVE_USER_NAMEthe userName used in relay-injected context
LIVE_USER_ADDRESSsirHow the agent addresses you
LIVE_PAGE_TITLELive VideoName in the phone page's heading and tab (the plugin sets it from page_title)
LIVE_SYSTEM_PROMPT_FILE—Path to a text file with the full system instruction
WATCH_WINDOW_SECONDS90Watch window
WATCH_HEARTBEAT_MS5000Re-check interval; 0 disables heartbeat checks
WATCH_MAX_CHECKS18Verifier calls per window (the spend cap)
WATCH_VERIFIER_MODELSgemini-3-flash-previewComma-separated REST fallback models
WATCH_VERIFIER_MODEautolive (short Live session), generate (one REST call), auto (live, REST on failure)
WATCH_VERIFIER_TIMEOUT_MS8000Ceiling for one verdict
LIVE_MODELmodels/gemini-3.8-liveThe Live model, used for the conversation and for verification

What goes over the network

Audio and video go from your phone to the relay, and from the relay to Google's
Gemini Live endpoint. Watch-mode frames go to the verifier model, also Google. Nothing
else leaves the machine: no telemetry, no analytics, no updates fetched at runtime.
The relay reads its key from GEMINI_API_KEY or its own config.env — never from
another application's credential store.

If you expose the relay beyond your network, put authentication in front of it. An
open relay spends its owner's key: the page takes camera frames and starts model
sessions for anyone who finds it.

Honest limits

  • The frame tracker is a change detector, not an object detector. It answers
    "has the view changed enough to look?" with a seeded random-projection sketch, and
    leaves "is the thing there?" to the verifier model. With the heartbeat disabled it
    can only react to a change large enough to move the sketch — a small object
    arriving in a still scene is the case it misses.
  • The verifier is slow: 3–7 seconds per look in testing. A sighting is voiced a few
    seconds after it appears, not instantly.
  • One watch window at a time, 90 seconds by default, with a per-window check cap.
  • The tracker needs PyTorch. It downloads no weights, but the dependency is large.
  • Windows is exercised, but not through the Hermes loader. The relay path has been
    run on Windows 11 (Node 24.18.0, Python 3.14.6, torch 2.12.1+cpu): detached spawn,
    /healthz with a key and the tracker available, taskkill /T teardown, pid gone, port
    free and no tracker orphaned — reproduced twice, and reachable here without Hermes via
    py tools.py --selftest (see docs/windows-test.md). What has
    not been exercised on Windows is the Hermes plugin loader calling those same
    functions; if that misbehaves, the log tail from start says where. A real phone on
    the LAN against that relay has since held a full spoken exchange with the camera in
    view, which is the only configuration that exercises the access token.

Tests

python3 tests/test_handlers.py          # 42 tests, stdlib only
node tests/test_token_gate.js           # 10 checks: the access token, the page's assets, its title
py tools.py --selftest                  # host self-test: spawn, health, stop, orphans
node tests/test_verifier.js             # 11 checks against the live API (needs a key)
hermes plugins validate .               # manifest, capabilities, security scan

test_verifier.js is not a mock: it opens real Live sessions and asserts the verdicts —
blue frame for "the screen turns blue" must answer YES, a red frame must answer NO, and
a one-millisecond timeout must return null rather than hang the watch loop.

Fixtures are real relay log lines, not synthesised ones, so a parser that stops
matching fails a test instead of returning an empty answer.

Screenshots

assets/live-page-answer.png and assets/live-page-sighting.png show the page mid-session —
the camera in view, the agent answering from the frame, then the sighting spoken once.

The sighting, spoken once

assets/live-transcript.png is the relay's own transcript for that session: WATCH_ARMED,
the VISION_EVENT, the WATCH_CHECK verdict and WATCH_FIRED visible (check 1, 971 ms).

The relay's transcript for that session

All three are frames from the demo recording described in docs/demo.md. The page, the
model's answers and the transcript rows are real; the camera frames in that recording are
synthetic stand-ins for a desk and a screen, and the voice on the user's own lines is
synthesised.

Licence

MIT. See LICENSE.

README from the reviewed commit 2474b58; it updates when the author re-pins.

More by dacheah

2 plugins by dacheah in the catalog.

dacheahdelegation-receiptsVerify what a subagent says it did before the parent reports the work as done. A "receipts" delegation backend asks every child to end its reply with a receipt (status, the files it changed, and the one command that proves the work); when a child stops, the plugin resolves and hashes each claimed path, matches every claim against the write targets that child's own tool history records so it cannot take credit for a file someone else wrote, and checks the cited command against a closed set of fixed check forms (pytest, uv run --no-sync pytest, cargo/go/npm/pnpm/yarn test), refusing a path rather than a bare name, a launcher, or inline evaluation. Host re-run is opt-in and off by default, because a hook-driven re-run bypasses Hermes' dangerous-command approval and the child's terminal backend; when an operator enables it, the command is re-run argv-style with no shell, in its own process group, with output and time bounded. Either way a verdict reaches the parent: a rejection is appended to the delegate_task result, with the failing output attached when there is one; a background child's verdict arrives on the parent's next turn instead. Every check stays in a local JSONL ledger shown by hermes receipts. One command per receipt, one receipt per child. Disclosure — the receipt's verified: command is written by the subagent being checked; host re-run is off by default, and when you enable plugins.entries.delegation-receipts.settings.host_rerun that command (one of pytest, uv run --no-sync pytest, cargo/go/npm/pnpm/yarn test) runs on this host with your privileges in the child's inferred workspace, outside Hermes' command approval and the configured terminal backend, and executes the project's test code and scripts, including files the subagent wrote (no shell, argv-split, filtered environment, 120 s default timeout); receipts and claimed paths are checked locally, the ledger is local at <HERMES_HOME>/plugin-data/delegation-receipts/ledger.jsonl, and the plugin makes no network calls and holds no credentials.Communitytools

Developer Mode

Explore built-in Hermes toolsets and actions.

HermesAgent

Developer?

Create a Hermes extension for Desktop or agent workflows.

Developer Section