meeting-overlay

Meeting Overlay — docs

Native Hermes desktop plugin (right-docked pane + status-chip), a phone-mirror
PWA, and a Hammerspoon menu-bar controller, for the meeting-mode copilot: live
transcription of a call, an answer panel, capture-source + model switching, and
on-card web/THI research.

Architecture

call audio ──► Multi-Output device (Speakers + BlackHole 2ch) ──► your ears
                      └─► BlackHole ──► listen.py (faster-whisper, LOCAL)
   phone mic ──► /ws/audio ──► pwa_server.py (same whisper)           ─┐
                                                                       ├─► rolling.txt (tail)
                                                                       └─► full.txt  (append)
                                             │
   ┌── desktop pane (polls /transcript + /answers + /settings) ◄───────┤
   ├── Hammerspoon 👂 menu — toggles the launchd daemon ────────────────┤
   └── answer.py (LLM loop) ──► answer.txt + answers.json ─────────────┘
                                    └─► save / recap → <vault>/Records/…/*.md

desktop/plugin.js — the Hermes pane (native plugin SDK). Polls the
backend REST namespace /api/plugins/meeting-overlay/* every 2s.
dashboard/plugin_api.py — FastAPI router (/transcript, /answers,
/status, /mode, /model, /run, /assist, /request, /sessions).
/status liveness pgreps the listen daemon's absolute path, so the daemon
must be launched with that path.
scripts/pwa_server.py — the phone PWA server: serves scripts/static/
and ingests the phone mic over /ws/audio.
scripts/install.sh — installs the durable launchd agents + tailnet serve
(see "Install").
~/.hammerspoon/meeting_listener.lua — the menu-bar controller (👂/🤐).
Toggles the launchd agent, switches capture source + model, fires the engine.
scripts/answer.py — the LLM answer loop.
--action what|followup|research|recap|save, --model host|local|off
(glm/qwen accepted as legacy aliases). Streams answer.txt (live panel)
and appends the answers.json deck; save/recap export to Records.

Install

Two scripts take a fresh machine to a running, durable system:

Audio — scripts/setup.sh: installs BlackHole and creates the
Multi-Output device (one-time).
Services — scripts/install.sh: writes + bootstraps the two durable
launchd agents (com.oz.meeting-pwa on --port 9877, com.oz.meeting-watch
for record mode) and exposes the phone PWA on the tailnet
(tailscale serve --https=9876 9877). Idempotent — re-run after a
tailscale serve reset to heal the mapping.

The PWA daemon itself binds only to 127.0.0.1 by default. Tailscale Serve
is therefore the intended phone path: it retains Tailnet identity and proxies
to the local listener. A direct LAN listener is intentionally not enabled by
installation. If it is required, start it explicitly with --lan and a
high-entropy --token-file with mode 0600; every HTTP route and audio
WebSocket then require that token before they can read or change meeting state.

The desktop com.oz.meeting-listener agent is written at runtime when a desktop
capture source is toggled (its MEETING_INPUT is mode-specific), so it needs no
install step.

Answer engine

| Action | --action | What it produces |
|---|---|---|
| 🗣 What should I say? | what | short in-the-moment phrasing (last ~90s) |
| 🔁 Follow-up questions | followup | 3–4 next questions |
| 🔬 Research | research | background/context/framing from model knowledge + transcript |
| 📋 Recap | recap | LLM summary (Decisions / Action items / Open questions) and save |
| 📦 Save | save | transcript-only save to Records (no summary) |

The three ask modes (what / followup / research) are mutually-exclusive,
persisted listen modes: selecting one arms the record-mode watch loop to drop a
card of that type whenever the transcript grows. Live web / THI research is a
separate on-demand path — each answer card carries web and thi inline actions
that queue a web-research / thi-research request seeded with that answer's
text, handed to the Hermes agent to run.

Model states

| Model | Meaning |
|---|---|
| host | the Hermes system default (hosted, auto-resolved) |
| local | local ollama fallback (offline) |
| off | no model engaged — transcription keeps flowing, no answers or cards |

Both model chips are toggles: click the selected one to turn it off. glm /
qwen are accepted as legacy aliases for host / local.

Capture sources

Four capture sources, one at a time (no cross-source merge/echo):

| Source | Label | Captures |
|---|---|---|
| phone | 📱 Phone | this phone streams its own mic — see "Phone PWA" |
| mic | 🎤 Mic | this Mac's built-in microphone |
| blackhole | 💻 Audio | system/computer audio (the other side of a call on this Mac) |
| both | 🔊 Both | computer audio + mic together, tagged [audio] / [mic] |

The desktop daemon's underlying MEETING_INPUT is blackhole (default), mic,
both/merge, or a device name; the pane exposes the same set as these four
sources.

Status semantics

green dot / 👂 = daemon alive (listening)
amber / stale = daemon dead, transcript older than 30s
gray / idle = daemon dead, transcript fresh or never ran

Daemon lifecycle

The durable agents are owned by scripts/install.sh (PWA + record-mode watch,
both KeepAlive + RunAtLoad); the desktop listener is generated at runtime.
Launch the desktop listener with the absolute script path (the liveness
pgrep needs it):
~/.hermes/hermes-agent/venv/bin/python ~/.hermes/skills/meeting-mode/scripts/listen.py
Own anything long-running under launchd, not a chat session — a daemon
spawned from a Hermes terminal dies with that session (the root cause of a
5-day STALE state).

Phone PWA (remote mic)

An installable PWA mirrors this panel on a phone over the Tailscale tailnet and
streams the phone's own microphone back into the SAME transcript. The phone is a
room mic — not a call-audio tap (call audio is not exposed by any OS/PWA API).

| Part | Location | Role |
|---|---|---|
| server | scripts/pwa_server.py | FastAPI; serves the bundle + /ws/audio; runs the same whisper + VAD/energy/min-duration gates as listen.py |
| bundle | scripts/static/ | index.html + manifest.webmanifest + sw.js + icons |
| source | phone | the phone streams its own mic; no local daemon is started |

Mic path: getUserMedia → ScriptProcessor → Int16 PCM 16k mono → /ws/audio.
Install: scripts/install.sh writes + bootstraps the com.oz.meeting-pwa
agent (--port 9877, KeepAlive) and runs tailscale serve --bg --https=9876 9877. Then open the .ts.net URL in mobile Safari/Chrome →
Share → Add to Home Screen. getUserMedia requires the Tailscale HTTPS secure
context — a plain http://<LAN-IP> origin is blocked.
Tailnet-only trust posture: the PWA has no application-level auth/CSRF
boundary, and its HTML rendering still lets server-controlled transcript/record
values reach the DOM. Accepted only while the PWA stays tailnet-only; this ends
if it is exposed beyond the tailnet or used for more sensitive records.

Known constraints

macOS 15.4 aggregate devices are GUI-only. AudioHardwareCreateAggregateDevice
is declared but returns 'nope' at runtime. The Multi-Output device
(Speakers + BlackHole, BlackHole on Drift Correction) is created in
Audio MIDI Setup. Name it whatever you like — the daemon taps the system
default output, not a hard-coded device name.
Continuity Camera is REMOVED from this build — it is Bluetooth/Wi-Fi
proximity only, cannot run over Tailscale, does not work with Managed Apple
IDs (work phones), and its mic only surfaces while the camera is actively in
use.
Menu-bar label text is single-line (native macOS menu) — keep captions short;
the status line and tooltip carry the detail.

Future work (someday-maybe)

Phone call-audio capture — capturing the audio of a call made on the phone
is permanently out of scope: no OS/PWA API exposes cellular call audio. The
shipped phone surface is the room-mic PWA (above); this item stays scoped to
call-audio only, not the phone PWA.

README from the reviewed commit 2c7859f; it updates when the author re-pins.

About