hermes-voice-gateway
Hermes Voice Gateway connects your business phone line to Hermes. It answers calls, offers a restricted inventory/enquiry workflow, and makes operator-approved callbacks.
Status
Early-stage (v0.2). Inbound calls, operator-approved callbacks and call-status lookup are implemented, with a local browser pipeline for development. Not production-hardened: it has not been load-tested or verified at scale, and the limits below apply.
How a call works
Twilio -> POST <VOICE_WEBHOOK_URL>/voice (signature checked) -> TwiML opens a media stream to /media -> ElevenLabs speech-to-text -> the transcript goes to Hermes as a normal message from the caller's number -> Hermes' reply is spoken with ElevenLabs.
Setup
- Create a dedicated caller profile with
hermes profile create voice(do not clone your personal profile). Install and run this plugin in that profile, not your personal/default profile. Setmemory.memory_enabled: false,memory.user_profile_enabled: false, and leavememory.providerunset or empty; see the runtime policy below. Add credentials through Hermes’ secret configuration for that profile:TWILIO_ACCOUNT_SID,TWILIO_AUTH_TOKEN,ELEVENLABS_API_KEYVOICE_WEBHOOK_URL: the public https base URL Twilio can reach, no trailing slash (a tunnel such as ngrok works for local testing)VOICE_ALLOWED_USERS: comma-separated caller numbers (E.164). With no list, every call is refused, unlessVOICE_ALLOW_ALL_USERS=true.
- In the Twilio console, set the number's "A call comes in" webhook to
<VOICE_WEBHOOK_URL>/voice(HTTP POST). The URL must match exactly, because it is part of the signature check. - The call server listens on
127.0.0.1:8081by default (VOICE_WEBHOOK_HOST,VOICE_WEBHOOK_PORT). Point your tunnel or reverse proxy at it. It must forward WebSocket upgrades on/media.
Optional: VOICE_MAX_CALLS (default 5), VOICE_MAX_CALL_SECONDS (default 1800; 0 = no limit), ELEVENLABS_VOICE_ID, ELEVENLABS_STT_MODEL, ELEVENLABS_TTS_MODEL. VOICE_INSECURE_NO_SIGNATURE=true turns off signature checks (local dev only).
Callbacks (outbound calls)
- The agent calls the
voice_propose_callbacktool with a number, a one-sentence purpose and who asked. The number must be onVOICE_ALLOWED_USERS. This only records the request; nothing is dialed. - A human reviews it in Hermes:
/voice-callback list, then/voice-callback approve <id>or/voice-callback deny <id>. Proposals expire after 24 hours, and at most 20 can wait at once. - On approval the gateway asks Twilio to call the number from
TWILIO_PHONE_NUMBER. When they answer, Twilio fetches<VOICE_WEBHOOK_URL>/voice/outbound?cb=<id>(signature checked, usable once, only for the number that was dialed) and the call runs like an inbound call, except the agent speaks first and is told the approved purpose. - The agent can check progress with
voice_check_call_status, bycallback_idor by thecall_sidof a call this gateway placed. It cannot look up other calls on your Twilio account.
Callback statuses: pending, denied, expired (24 h), dialing, unanswered (Twilio never reached the answer URL within 3 minutes: no answer, busy or rejected), answered, completed (2 h after answering), failed. At most 20 proposals can wait, and at most 2 per number.
There is deliberately no agent-callable approve tool, and spoken words cannot run /commands. The "requested by" name is whatever the agent supplied and is not verified. Hermes does not tell a plugin command who ran it, so anyone Hermes lets run slash commands can approve: restrict command access in Hermes to your operators. State is kept in the plugin's own SQLite file (<HERMES_HOME>/plugin-data/hermes-voice-gateway/data.db).
Needs TWILIO_PHONE_NUMBER in addition to the inbound settings.
Caller inventory and enquiries
Phone turns can use only voice_search_inventory and voice_save_enquiry. The adapter overrides the caller toolsets with voice_caller and no_mcp; shell, file, browser, delegation, execution wrappers, connectors, callback/status tools and local-pipeline startup are not caller permissions. Operator-side callback tools and /voice-callback approval remain separate.
- Inventory search reads a packaged sample inventory snapshot (see Demo data), returns source URLs and caveats, and caps each response at five products. Model/title, maximum INR budget and minimum RAM/SSD filters are optional. There is no caller-controlled path or live-stock API.
- Saving requires a confirmed name, requirements and explicit
consent=trueto storing the enquiry with the current caller number. Unknown model, budget, RAM or SSD fields should be omitted. The caller cannot supply a different phone number, Call SID, database path or enquiry owner. - Enquiries are plaintext SQLite records in
<HERMES_HOME>/plugin-data/hermes-voice-gateway/data.db, tablevoice_enquiries. One enquiry is saved per call: identical retries return the same ID; changed details are refused rather than silently replacing the record. The response confirms local storage only: no team is notified, no slot is reserved and no callback is approved or scheduled. Operator viewing/handling is a separate follow-up feature. - Both handlers require matching task-bound gateway session/caller/thread/message identity, a bounded database grant and a live call in the admitting gateway process. Permissions close on completion, rejection, timeout, cancellation or hangup. Environment-only identities, stale calls and worker/host processes without the live grant are denied. Treat caller ID as unverified, not proof of a person's identity. Do not collect payment credentials or other sensitive data in enquiries.
Runtime policy prerequisite: Hermes can automatically append other installed plugin toolsets, context-engine tools or tool-search wrappers even after a platform override. Before each caller turn, the plugin checks the resolved tool definitions and refuses inference unless they are exactly the two caller tools. The pre-tool hook also explicitly denies all other tool names and denies permission-check errors; handler checks remain mandatory even for direct dispatch.
Create and configure a dedicated profile before accepting calls:
hermes profile create voice
hermes --profile voice config set memory.memory_enabled false
hermes --profile voice config set memory.user_profile_enabled false
hermes --profile voice config set memory.provider ''
Run the gateway with hermes --profile voice gateway run after installing the plugin and configuring its credentials in that profile. Do not clone or reuse an owner profile. Before every caller turn, the plugin refuses inference unless both built-in memory flags are explicitly boolean false and memory.provider is unset or empty. An external provider is refused even if both built-in flags are off, preventing caller turns from being synced to that provider.
What callers can see: this is still a normal Hermes agent in the dedicated profile, not a separate prompt sandbox. It can see Hermes’ system/safety instructions, that profile’s persona (SOUL.md and configured system prompts), any project/context instructions Hermes loads, the voice instructions (voice/PROFILE.md or VOICE_PROFILE_PATH), the current call’s history, approved callback purpose when applicable, and results from the two caller tools. Keep all of that context caller-safe; never copy personal persona, secrets or private business notes into it. The disabled memory stores (MEMORY.md/USER.md) are not loaded as built-in memory and caller turns are not synced to an external memory provider. Hermes session/transcript storage and consented local enquiry storage still apply; disabling memory is not a no-retention mode.
In the dedicated voice profile, save a Voice tool selection containing only voice_caller, with MCP disabled. known_plugin_toolsets.voice must include the omitted installed plugin toolsets (including voice) so Hermes does not add them by default. For a profile with only this plugin, that list is [voice, voice_caller]. Additional plugin toolsets must also be marked known-and-omitted for Voice. Context-engine additions and tool-search deferral must not broaden/replace the two exposed tool names. Use Hermes' tools/configuration interfaces; this plugin does not rewrite profile settings. Misconfiguration produces a caller-policy error in gateway logs and fails closed. This policy/context integration is checked against the installed Hermes APIs and must be reverified on runtime upgrades.
Speech recognition
Phone calls use scribe_v2 through ElevenLabsSTTService, matching the local development pipeline. Silero VAD detects the end of each spoken segment, then the audio is uploaded for transcription. Pause after speaking to let Hermes respond; this is segmented HTTP transcription, not the realtime WebSocket API. Each call owns an HTTP session that closes when the call ends or setup fails.
Limits (from the Pipecat telephony docs)
- Twilio WebSocket calls have basic call control only: no transfers or reconnects. Those need Daily PSTN/SIP.
- Pipecat's user-turn strategies can interrupt queued speech. Cancellation of Hermes' ongoing reasoning/tools is not coordinated with playback interruption, so full phone-call barge-in remains unverified.
- A dropped connection ends the call. Twilio call-status callbacks are not wired in, so a callback's stored status stays
answeredafter the call ends (usevoice_check_call_statusfor the live state). - Calls to voicemail are not detected; the agent may speak to an answering machine.
Security notes
- The allowlist trusts the caller ID Twilio reports, and phone caller IDs can be spoofed. Treat it as a filter, not as strong authentication. Do not give allowlisted callers more access than you would give anyone who can phone in.
- Pipecat logs caller speech (the transcript) at DEBUG level. Run Hermes with
LOGURU_LEVEL=INFOor higher so transcripts stay out of logs. - Spoken words never run Hermes
/commands; calls are plain conversation only.
Call behaviour (local pipeline and Twilio adapter)
- Greeting: the agent speaks first when a call connects.
VOICE_GREETINGoverrides the text;offdisables it. - Voice profile:
<HERMES_HOME>/voice/PROFILE.md(orVOICE_PROFILE_PATH) supplies short-reply, speakable instructions. If missing or empty, a built-in profile is used. Locally it is the OpenAI system instruction; the adapter snapshots it per call and supplies it to Hermes throughchannel_prompt, without changing the caller's transcript or replacing Hermes' own safety instructions. - Caller-profile template:
voice/PROFILE.mdis a sample laptop-sales profile (requirements intake, shortlist, privacy rules). Copy it to the profile location above or setVOICE_PROFILE_PATHto its absolute path; the repo file is not loaded automatically. The caller tool allowlist is enforced independently of the profile; a prompt is not an access-control boundary. Profiles are truncated at 6,000 characters. SetVOICE_GREETINGseparately to an opening that discloses the caller is talking to an AI, because the gateway's fixed greeting is not generated from the profile. - Smart turn: Pipecat's local smart-turn model decides from the audio when the caller has finished, instead of replying after any pause.
- Guardrails: a turn with no recognised words makes the agent ask the caller to repeat (up to twice in a row); silence gets "Are you still there?" and then a hang-up; three pipeline errors in one call end it with an apology.
- Adapter timing: one completed smart turn becomes one Hermes message. Idle detection pauses while Hermes is processing and while speech is queued or playing. Outbound callbacks retain their approved-purpose, Hermes-first opening instead of speaking a second generic greeting.
- Reply routing: each phone call has its own Hermes thread, keyed by Call SID. Speech delivery requires that call's thread or a valid reply-to turn; stale or unscoped replies cannot reach a later call from the same number. Only final user-facing text marked by Hermes with
notify=trueis spoken; busy acknowledgments and delivery-recovery notices stay silent. Direct adapter integrations must preserve that routing metadata. - Serialized phone turns: voice inputs run FIFO per canonical Hermes session, waiting until the gateway processing task exits, including cleanup. Phone input does not steer or merge into an in-progress response; audio playback interruption is still separate. Delivery bookkeeping settles once on completion, rejection, failure or hangup. A five-minute admission/processing wait fails the delivery without cancelling running tools, and late replies to that delivery are refused. Hangup drops queued delivery, not tool side effects already in progress. This relies on guarded private Hermes session/task APIs; incompatible state fails closed instead of guessing that input was accepted.
Local dev (no phone)
Use Pipecat 1.7.0 exactly, not a floating 1.7.x or 1.15.x release. Both dependency declarations and install hints use the same pin. Verify the actual interpreter as well as the dependency files; the Hermes-managed runtime and a shell's Python can be different environments.
voice_start_local_pipeline and frontend/ give a browser-mic test loop that does not involve Twilio or Hermes' agent. Supply ELEVENLABS_API_KEY and OPENAI_API_KEY in the process environment, then run python scripts/start_local_pipeline.py and open http://127.0.0.1:8765/. The standalone runner never reads ~/.hermes/.env or any profile dotenv file; when invoked through Hermes, secrets resolve through get_scoped_secret instead. Dev only; it is not part of the shipped plugin.
Tests
pytest tests from the repo root.
Tests simulate the model and audio; they are not a live phone or provider test. Caller-permission coverage is in tests/test_caller_tools.py.
Demo data
The opt-in local demo (VOICE_REFURBO_DEMO=true) uses a sample laptop-store inventory snapshot (voice/refurbo-inventory.json) and simulated appointments stored locally. The snapshot is demo data only: not live stock, and its prices are approximate and not quotes. The demo does not notify anyone, reserve real calendar slots or place calls, and does not change human approval for real callbacks. See docs/refurbo-demo.md.
License
MIT. See LICENSE.
README from the reviewed commit 71f5801; it updates when the author re-pins.

SRai22