Credential Pools

Credential pools let you register multiple API keys or OAuth tokens for the same provider. When one key hits a rate limit or billing quota, Hermes automatically rotates to the next healthy key — keeping your session alive without switching providers.

This is different from fallback providers, which switch to a different provider entirely. Credential pools are same-provider rotation; fallback providers are cross-provider failover. Pools are tried first — if all pool keys are exhausted, then the fallback provider activates.

How It Works

Your request
  → Pick key from pool (round_robin / least_used / fill_first / random)
  → Send to provider
  → 429 rate limit?
      → Retry same key once (transient blip)
      → Second 429 → rotate to next pool key
      → All keys exhausted → fallback_model (different provider)
  → 402 billing error?
      → Immediately rotate to next pool key (24h cooldown)
  → 401 auth expired?
      → Try refreshing the token (OAuth)
      → Refresh failed → rotate to next pool key
  → Success → continue normally

Quick Start

If you already have an API key set in .env, Hermes auto-discovers it as a 1-key pool. To benefit from pooling, add more keys:

# Add a second OpenRouter key
hermes auth add openrouter --api-key sk-or-v1-your-second-key

# Add a second Anthropic key
hermes auth add anthropic --type api-key --api-key sk-ant-api03-your-second-key

# Add an Anthropic OAuth credential (Claude Code subscription)
hermes auth add anthropic --type oauth
# Opens browser for OAuth login

Check your pools:

hermes auth list

Output:

openrouter (2 credentials):
  #1  OPENROUTER_API_KEY   api_key env:OPENROUTER_API_KEY ←
  #2  backup-key           api_key manual

anthropic (3 credentials):
  #1  hermes_pkce          oauth   hermes_pkce ←
  #2  claude_code          oauth   claude_code
  #3  ANTHROPIC_API_KEY    api_key env:ANTHROPIC_API_KEY

The ← marks the currently selected credential.

Interactive Management

Run hermes auth with no subcommand for an interactive wizard:

hermes auth

This shows your full pool status and offers a menu:

What would you like to do?
Add a credential
Remove a credential
Reset cooldowns for a provider
Set rotation strategy for a provider
Exit

For providers that support both API keys and OAuth (Anthropic, Nous, Codex), the add flow asks which type:

anthropic supports both API keys and OAuth login.
  1. API key (paste a key from the provider dashboard)
  2. OAuth login (authenticate via browser)
Type [1/2]:

CLI Commands

Command	Description
`hermes auth`	Interactive pool management wizard
`hermes auth list`	Show all pools and credentials
`hermes auth list <provider>`	Show a specific provider's pool
`hermes auth add <provider>`	Add a credential (prompts for type and key)
`hermes auth add <provider> --type api-key --api-key <key>`	Add an API key non-interactively
`hermes auth add <provider> --type oauth`	Add an OAuth credential via browser login
`hermes auth remove <provider> <index>`	Remove credential by 1-based index
`hermes auth reset <provider>`	Clear all cooldowns/exhaustion status

Rotation Strategies

Configure via hermes auth → "Set rotation strategy" or in config.yaml:

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

Strategy	Behavior
`fill_first` (default)	Use the first healthy key until it's exhausted, then move to the next
`round_robin`	Cycle through keys evenly, rotating after each selection
`least_used`	Always pick the key with the lowest request count
`random`	Random selection among healthy keys

Error Recovery

The pool handles different errors differently:

Error	Behavior	Cooldown
429 Rate Limit	Retry same key once (transient). Second consecutive 429 rotates to next key	1 hour
402 Billing/Quota	Immediately rotate to next key	24 hours
401 Auth Expired	Try refreshing the OAuth token first. Rotate only if refresh fails	—
All keys exhausted	Fall through to `fallback_model` if configured	—

The has_retried_429 flag resets on every successful API call, so a single transient 429 doesn't trigger rotation.

Custom Endpoint Pools

Custom OpenAI-compatible endpoints (Together.ai, RunPod, local servers) get their own pools, keyed by the endpoint name from custom_providers in config.yaml.

When you set up a custom endpoint via hermes model, it auto-generates a name like "Together.ai" or "Local (localhost:8080)". This name becomes the pool key.

# After setting up a custom endpoint via hermes model:
hermes auth list
# Shows:
#   Together.ai (1 credential):
#     #1  config key    api_key config:Together.ai ←

# Add a second key for the same endpoint:
hermes auth add Together.ai --api-key sk-together-second-key

Custom endpoint pools are stored in auth.json under credential_pool with a custom: prefix:

{
  "credential_pool": {
    "openrouter": [...],
    "custom:together.ai": [...]
  }
}

Auto-Discovery

Hermes automatically discovers credentials from multiple sources and seeds the pool on startup:

Source	Example	Auto-seeded?
Environment variables	`OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`	Yes
OAuth tokens (auth.json)	Codex device code, Nous device code	Yes
Claude Code credentials	`~/.claude/.credentials.json`	Yes (Anthropic)
Hermes PKCE OAuth	`~/.hermes/auth.json`	Yes (Anthropic)
Custom endpoint config	`model.api_key` in config.yaml	Yes (custom endpoints)
Manual entries	Added via `hermes auth add`	Persisted in auth.json

Auto-seeded entries are updated on each pool load — if you remove an env var, its pool entry is automatically pruned. Manual entries (added via hermes auth add) are never auto-pruned.

Thread Safety

The credential pool uses a threading lock for all state mutations (select(), mark_exhausted_and_rotate(), try_refresh_current(), mark_used()). This ensures safe concurrent access when the gateway handles multiple chat sessions simultaneously.

Architecture

For the full data flow diagram, see docs/credential-pool-flow.excalidraw in the repository.

The credential pool integrates at the provider resolution layer:

agent/credential_pool.py — Pool manager: storage, selection, rotation, cooldowns
hermes_cli/auth_commands.py — CLI commands and interactive wizard
hermes_cli/runtime_provider.py — Pool-aware credential resolution
run_agent.py — Error recovery: 429/402/401 → pool rotation → fallback

Storage

Pool state is stored in ~/.hermes/auth.json under the credential_pool key:

{
  "version": 1,
  "credential_pool": {
    "openrouter": [
      {
        "id": "abc123",
        "label": "OPENROUTER_API_KEY",
        "auth_type": "api_key",
        "priority": 0,
        "source": "env:OPENROUTER_API_KEY",
        "access_token": "sk-or-v1-...",
        "last_status": "ok",
        "request_count": 142
      }
    ]
  },
  "credential_pool_strategies": {
    "openrouter": "round_robin"
  }
}

Strategies are stored in config.yaml (not auth.json):

credential_pool_strategies:
  openrouter: round_robin
  anthropic: least_used

How It Works​

Quick Start​

Interactive Management​

CLI Commands​

Rotation Strategies​

Error Recovery​

Custom Endpoint Pools​

Auto-Discovery​

Thread Safety​

Architecture​

Storage​