hermes-xai

The xAI API surfaces core does not cover, nothing it already ships (provider, x_search, web search, TTS, STT and Imagine stay in core). Voice module: full-duplex Grok Voice (grok-voice-latest) in the Desktop composer, built like core's GPT-Live mode; the voice model hands real work to Hermes as a normal turn on the open chat (surface voice-live) and reads the result aloud; the backend mints a 300 s xAI client secret so XAI_API_KEY never reaches the renderer. Collections module: one read-only tool, xai_collections_search, over documents the user already indexed in xAI Collections (GET /v1/collections, POST /v1/documents/search). Disclosure — the voice button captures microphone audio and streams it to api.x.ai over a WebSocket; the backend calls api.x.ai once per call with XAI_API_KEY (read through agent.secret_scope) to mint the secret; delegated requests run as normal Hermes turns with the profile's full toolset and approvals, and their final text (up to 3000 characters) plus the last spoken turns are sent back to xAI. Each xai_collections_search call sends the query to api.x.ai with XAI_API_KEY and returns passages into the conversation. Nothing is uploaded, stored or deleted and no telemetry is emitted; xAI bills on the user's key.

Julientalbot7Voicev0.1.0Updated 19 days ago

About

hermes-xai

The xAI API surfaces Hermes core does not cover, as one plugin. Core already handles xAI as a
model provider (Grok on the Responses API, reasoning, function calling, server-side web search),
x_search, the xai web-search backend, TTS, STT and Grok Imagine (image generation/editing, video
generation/editing/extension). This plugin does not duplicate any of them. It adds two modules:

| module | what you get | xAI endpoint |
|---|---|---|
| voice | full-duplex Grok Voice in the Hermes Desktop composer; real work is handed to Hermes | POST /v1/realtime/client_secrets + wss://api.x.ai/v1/realtime |
| collections | xai_collections_search, a read-only tool over documents you indexed in xAI Collections | GET /v1/collections, POST /v1/documents/search |

Voice

You talk to grok-voice-latest (today grok-voice-3.0). When the request needs real work (files,
web, memory, an action), the voice model hands it to Hermes as a normal turn on the open chat and
reads out the result. Same pattern as core's GPT-Live mode (voice.voice_chat_mode: gpt-live); the
transport differs: xAI Realtime is a WebSocket API authenticated by an ephemeral client secret.

POST /api/plugins/hermes-xai/session: the backend reads XAI_API_KEY through
agent.secret_scope.get_secret, calls POST https://api.x.ai/v1/realtime/client_secrets (300 s
TTL) and returns {url, protocol_prefix, client_secret, model, session}. The key never leaves the
backend.
The desktop half opens wss://api.x.ai/v1/realtime?model=… with the subprotocol
xai-client-secret.<value> and sends the returned session.update (persona, server VAD, 24 kHz
PCM, tools hermes_task and end_call), then streams mic audio.
hermes_task becomes prompt.submit on the open chat with surface: "voice-live". The session's
model, toolset, memory and approvals apply unchanged. The turn's message.complete text goes back
to the voice as a [Hermes result] message.

Collections search

xai_collections_search(query, collection_ids?, limit?, filter?, list_collections?) returns the
best-matching passages (title, score, page, text ≤ 2000 chars) from collections you built on
console.x.ai. With no ids it uses collections_default_ids, else every collection of the team.
filter takes xAI's AIP-160 metadata syntax. Read only: nothing is uploaded, created or deleted.

Install

hermes plugins install https://github.com/Julientalbot/hermes-xai

Needs XAI_API_KEY in the profile's .env (the installer prompts for it). A Grok OAuth (SuperGrok)
sign-in alone is not used, because plugins never hold OAuth tokens (catalog rule 11). Then click the
waveform button in the composer, or ask Hermes about your documents.

Settings (plugins.entries.hermes-xai.settings, or the plugin's Settings page):

| key | default | |
|---|---|---|
| model | grok-voice-latest | or a pinned grok-voice-* id |
| voice | eve | built-in voice or custom voice id |
| reasoning_effort | none | none or high; none answers fastest, Hermes does the thinking |
| instructions | "" | appended to the voice persona (language, tone) |
| collections_enabled | true | register xai_collections_search |
| collections_default_ids | [] | collections searched when the model passes none (empty = all) |

Measured (real key, 2026-10-15)

End-to-end voice call with no microphone: the plugin was installed with hermes plugins install into
a throwaway HERMES_HOME, served by hermes serve, and a spoken question (macOS say, 24 kHz mono
PCM16, 3.3 s) was streamed into the realtime socket in 100 ms frames, exactly as the desktop button
does. The delegated request ran as a real Hermes turn (prompt.submit, surface voice-live, agent
model grok-4.3) on the serve's /api/ws gateway. Question: "What time is it right now in Tokyo?
Please check for me."

| step | run 1 | run 2 |
|---|---|---|
| POST /api/plugins/hermes-xai/session (secret minted; key absent from the response) | 925 ms | 394 ms |
| xAI WebSocket open | 855 ms | 924 ms |
| session.created → model | grok-voice-3.0 | grok-voice-3.0 |
| end of speech → transcript | 0.39 s | 0.55 s |
| end of speech → spoken acknowledgement (first audio) | 1.05 s | 1.03 s |
| end of speech → hermes_task call | 1.20 s | 1.34 s |
| Hermes turn (prompt.submit → message.complete) | 18.1 s* | 6.8 s |
| [Hermes result] → first audio of the answer | 1.02 s | 1.31 s |
| end of speech → spoken answer | 20.3 s* | 9.4 s |

Spoken answers: "It's 4:41 PM on October 15 in Tokyo." / "It's 4:42 PM in Tokyo right now." (both
correct). *Run 1 was the first turn of a fresh profile, which includes Hermes' first-run onboarding.
The voice layer itself answers in about 1 s; the rest is the Hermes turn, which depends on your
model and the task.

Collections, same install, through the registered tool handler: list_collections 0.73 s (1
collection, 26 documents); a search returned 3 passages (scores 0.49–0.51) in 1.61 s. The key never
appeared in a tool result.

Limits

Desktop only for voice. The button lives in the Hermes Desktop composer; the CLI, TUI and
messaging gateways have no realtime voice from this plugin.
API key, not OAuth. Plugins cannot use a Grok OAuth (SuperGrok) sign-in, so XAI_API_KEY is
required.
Results wait for the voice to finish. A Hermes result that arrives while the voice model is
speaking is queued and read right after.
Collections needs xAI-side storage. Collections keeps your documents on xAI's servers. If your
xAI team switches to zero data retention, Collections (and therefore xai_collections_search)
stops working. The voice module was not tested under zero data retention.
Read only. The plugin never creates, uploads or deletes collections or documents (that needs
xAI's separate management key).
Not affiliated with xAI or Nous Research.

Disclosure

Voice: pressing the button captures microphone audio and streams it to api.x.ai over a
WebSocket opened by the desktop with a 300 s client secret. The backend calls api.x.ai once per
call to mint that secret. Delegated requests run as normal Hermes turns with the profile's full
toolset; their final text (up to 3000 characters) plus the last spoken turns are sent back to xAI.
xAI bills voice minutes on your key.
Collections: each tool call sends the query (and optional filter) to api.x.ai with
XAI_API_KEY; returned passages enter the conversation. xAI bills searches on your key.
Nothing is stored by the plugin and no telemetry is emitted.

Develop

python -m pytest            # 20 tests, no network
HERMES_HOME=$(mktemp -d) hermes plugins validate .

MIT licensed.

README from the reviewed commit 8fd9bf8; it updates when the author re-pins.

More by Julientalbot

3 plugins by Julientalbot in the catalog.

Julientalbotfleet-policyFor administrators who pin settings through the managed scope: records what each agent role really sends to the provider (model, reasoning effort, service tier, plus the tier the provider says it served) next to the managed-scope state and the effective config (`hermes fleet-policy status --json`); measures cron cost per job — runs, empty runs, $/day, gate present (`crons --cost`); and `watch`, a generic wake gate for cron scripts that replays one read tool on an MCP server and wakes the agent only when new item ids appear (stock `wakeAgent` gate, no LLM call otherwise). Ships two skills (managed-scope admin, recurring tasks). Disclosure — records only model, reasoning effort and service tier per role (no message content, headers or keys) via the pre_api_request and pre_auxiliary_call hooks and a pass-through llm_execution middleware, stored in $HERMES_HOME/fleet-policy/observed.json; crons --cost opens cron/executions.db and state.db read-only and checks each cron run's final reply locally for silence; watch is an operator CLI command that calls one MCP tool at the config's server_url (or the admin-pinned watch_server_url) with config headers in which ${VAR} env values are expanded, follows redirects, and writes seen ids to a state file at the config-chosen path; the recurring-tasks skill has the agent write watch configs under $HERMES_HOME/watches, a script under $HERMES_HOME/scripts and a cron job; no other network access, no telemetry.Communityautomation Julientalbottool-call-salvageRemoves tool-call markup that a model wrote into its final reply as text instead of calling the tool (Qwen <function=...>, <invoke>, DeepSeek DSML and tool tokens, Harmony, Kimi, Mistral and Llama formats). Only complete markup that ends the reply is touched; code blocks and markup followed by more text are left as they are. An optional salvage mode, off by default and limited to the models you list, turns the markup into a real call to a tool offered in that request. Disclosure: no network, files, shell, subprocesses, telemetry or stored state; logs name the format and tools, never arguments; in salvage mode text the model wrote can become a real tool call (tools offered in the request, or Hermes's tool_call bridge when deferred_tools is on), run through Hermes's normal approvals.Communitymodels