hermes-otel-plugin — OpenTelemetry (OTLP) Observability for Hermes

A native OpenTelemetry observability plugin for the Hermes agent.

This is a real Hermes turn in a stock Jaeger — one invoke_agent span per turn, a chat span per LLM call (with its measured latency), and an execute_tool span per tool call:

Every span carries the GenAI-convention attributes — model, tokens, cost, latency — so any conventions-aware backend can aggregate them:

The same trace in Grafana (Explore → Jaeger datasource) — any OTLP backend works:

It exports Hermes's built-in observability events over OTLP in three complementary ways:

1. GenAI-convention spans (gen_ai.*) — the standard trace model, so Hermes telemetry flows to any OpenTelemetry backend (an OpenTelemetry Collector, Jaeger, Grafana Tempo, Honeycomb, …) with no vendor lock-in.
2. Dashboard-shaped OTLP log records — the user_prompt / session_start / api_response / tool_result event model that agent-coding dashboards (the kind Claude Code / Codex / Copilot populate) aggregate on, so a Hermes session shows up in the Activity / Sessions / Leaderboards views — including its prompts — like any other coding agent.
3. Local-model cost — when the backend returns no price (Ollama / HF / vLLM local models), cost is estimated from the model's parameter size, so spans and log records carry a real cost_usd instead of $0.

Standalone and opt-in — it only loads when you explicitly enable it, and it makes no changes to Hermes core. Without the OTel SDK the hooks no-op silently (the plugin fails open).

Published as a standalone plugin per Hermes's contribution policy that third-party observability backends live outside the core tree (see hermes-agent#48184).

Install

Choose one install path. The catalog key is traceverse-hermes-otel; the plugin's enable key is traceverse_hermes_otel.

A. Hermes plugin catalog (recommended once listed)

hermes plugins install traceverse-hermes-otel
hermes plugins enable traceverse_hermes_otel

B. Direct GitHub install (available before catalog approval)

hermes plugins install Mandark-droid/hermes-otel-plugin
hermes plugins enable traceverse_hermes_otel

C. pip (entry-point discovery)

pip install git+https://github.com/Mandark-droid/hermes-otel-plugin.git
# then add `traceverse_hermes_otel` to plugins.enabled in ~/.hermes/config.yaml, or:
hermes plugins enable traceverse_hermes_otel

D. Manual clone into ~/.hermes/plugins/

git clone https://github.com/Mandark-droid/hermes-otel-plugin.git ~/.hermes/plugins/traceverse_hermes_otel
hermes plugins enable traceverse_hermes_otel

Quickstart

# 1. Install the OTel SDK + OTLP HTTP exporter (the only hard dependency)
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# 2. Install + enable the plugin (direct GitHub install works before catalog approval)
hermes plugins install Mandark-droid/hermes-otel-plugin
hermes plugins enable traceverse_hermes_otel

# 3. Send a turn, then look for an "invoke_agent" trace in your OTel backend
hermes chat -q "hello"

It works out of the box against a local collector on http://localhost:4318.

Configure (optional)

All configuration is optional. Set these in ~/.hermes/.env:

HERMES_OTEL_SERVICE_NAME=agent.coding.hermes   # service.name (backends group on this)
HERMES_OTEL_ENDPOINT=http://localhost:4318     # OTLP HTTP endpoint
HERMES_OTEL_AGENT_NAME=hermes                  # gen_ai.agent.name
HERMES_OTEL_CAPTURE_CONTENT=false              # attach prompt/response text (privacy-gated)
HERMES_OTEL_LOGS_ENABLED=true                  # set false for spans-only OTLP receivers (no /v1/logs): skips the dashboard log pipeline entirely
HERMES_OTEL_USER_ID=                           # stamp user.id on the resource

Standard OTel env vars are honoured as fallbacks: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT. Prompt / response / tool I/O is attached only when HERMES_OTEL_CAPTURE_CONTENT=true (privacy-gated, off by default).

Verify / disable

hermes plugins list                 # traceverse_hermes_otel should show "enabled"
hermes plugins disable traceverse_hermes_otel

Architecture

invoke_agent        per Hermes turn   (gen_ai.conversation.id = session id)
  ├── chat          per LLM API call  (tokens, cost, finish reason, TTFT)
  └── execute_tool  per tool call     (gen_ai.tool.name / .type, errors)

gen_ai.system = hermes on every span; service.name defaults to agent.coding.hermes. Spans and dashboard log records are emitted in parallel (a logs failure never disables spans). Identity (service.name, user.id, and any team.id / organization.id from OTEL_RESOURCE_ATTRIBUTES) is placed on the OTel resource so per-user / per-org leaderboard aggregations work.

Real span durations. Hermes's per-call hooks fire after each LLM/tool call and carry the measured wall-clock time (api_duration / duration_ms). Since v1.1.0 the plugin backdates each chat / execute_tool span start by that measured duration, so waterfalls show the true call windows — not zero-length ticks at the completion instant.

The mapping logic is pure, Hermes-agnostic, and unit-tested with in-memory exporters (no Hermes, no network) under tests/test_otel_plugin.py. The full reference lives inline, next to the code — each module's docstring is the source of truth:

Module	What its docstring documents
emitter.py	GenAI span model (invoke_agent / chat / execute_tool + attributes)
log_emitter.py	dashboard log-record model (user_prompt / session_start / api_response / tool_result — the field contract dashboards aggregate on)
cost.py	local-model cost estimation (model-size → price-tier methodology)
provider.py	tracer / logger bootstrap (genai_otel preferred, plain OTel SDK fallback)
__init__.py	the only Hermes-coupled module: register(ctx) + lifecycle hooks
plugin.yaml	manifest

Advanced — GPU / CO2 / eval metrics

Installing genai-otel-instrument instead of the plain OTel SDK unlocks on-prem GPU / energy / CO2 metrics and an inline eval suite (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) — scored in-process, on-prem, with no data leaving the host, no SaaS backend, and no separate eval service. It is driven entirely by genai_otel's own GENAI_* environment variables; no plugin code change is required.

→ See ADVANCED.md for the full setup: why genai_otel vs a vanilla OTel SDK (with a capability comparison), the install extras, the environment recipe, and exactly what attributes/metrics you get.

For AI agents

A machine-readable summary of this plugin's contract (also see llms.txt). Everything below is stable API; the per-module docstrings are the authoritative deep reference.

name: traceverse-hermes-otel
purpose: Export Hermes agent telemetry over OTLP (OpenTelemetry GenAI conventions)
install:
  recommended: ["hermes plugins install traceverse-hermes-otel",
                "hermes plugins enable traceverse_hermes_otel"]
  direct_fallback: "hermes plugins install Mandark-droid/hermes-otel-plugin"
  runtime_dependency: "pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http"
verify: "hermes plugins list   # 'traceverse_hermes_otel' shows enabled; then send one turn and look for an invoke_agent trace"
config_env:                                # all optional; set in ~/.hermes/.env
  HERMES_OTEL_SERVICE_NAME: "service.name resource attr (default agent.coding.hermes)"
  HERMES_OTEL_ENDPOINT: "OTLP HTTP endpoint (default http://localhost:4318)"
  HERMES_OTEL_AGENT_NAME: "gen_ai.agent.name (default hermes)"
  HERMES_OTEL_CAPTURE_CONTENT: "true attaches prompt/response/tool I/O text (default false, privacy-gated)"
  HERMES_OTEL_LOGS_ENABLED: "false skips the dashboard log-record pipeline entirely — for spans-only OTLP receivers without /v1/logs (default true)"
  HERMES_OTEL_USER_ID: "stamps user.id on the OTel resource"
span_model:
  invoke_agent: "one per Hermes turn; parent; gen_ai.conversation.id + session.id = session id"
  chat: "one per LLM API call; child; real measured duration (backdated start)"
  execute_tool: "one per tool call; child; real measured duration (backdated start)"
key_attributes:
  - gen_ai.operation.name        # invoke_agent | chat | execute_tool
  - gen_ai.system                # always "hermes"
  - gen_ai.request.model / gen_ai.response.model
  - gen_ai.usage.input_tokens / gen_ai.usage.output_tokens
  - gen_ai.response.finish_reasons   # array; [0] is the primary reason
  - gen_ai.tool.name / gen_ai.tool.type
  - cost_usd                     # real or size-tier-estimated for local models
  - copilot_chat.time_to_first_token  # measured API latency, ms
log_events: [user_prompt, session_start, api_response, tool_result]  # OTLP log records, dashboard-shaped
guarantees:
  - "fail-open: without the OTel SDK every hook no-ops; observability never breaks the agent"
  - "no Hermes core changes; hooks only"
  - "content capture is opt-in (HERMES_OTEL_CAPTURE_CONTENT), off by default"

Development

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http pytest pyyaml ruff
pytest tests/ -v       # 44 tests, in-memory exporters, no network
ruff check .

License

MIT