specsentry

SpecSentry

Real traffic. Real evidence. Reproducible tests.

SpecSentry is an API Reality & Contract Intelligence product for Hermes Agent.
It answers a question no generic scanner answers: does the traffic you
actually captured agree with the contract you actually published?

OpenAPI contract + real HAR traffic
  -> deterministic contract matching
  -> evidence-based contract drift detection
  -> coverage intelligence
  -> reproducible regression tests
  -> evidence-backed reports

It is not a Postman clone, not a generic API scanner, and not a conventional
vulnerability scanner. It never replays traffic, never contacts a host, and
never guesses.

The six core features

Projects (save / open / list) store derived results and source digests so
a prior assessment can be reopened without the original files.

Verdicts at a glance

Every report leads with one verdict:

Contents

• The six core features
• Verdicts at a glance
• The differentiator: an evidence chain, not a score
• Verdicts and evidence
• What it detects
• Coverage intelligence
• Reproducible regression tests
• Projects: save, reopen, retrieve
• Privacy by construction
• Try it — commands, exit codes, and a real report sample
• Installation — requirements first, then first-run verification
• As a Hermes tool
• Architecture
• Safety and scope
• Known limitations
• Tests
• Decision records
• License

The differentiator: an evidence chain, not a score

Every claim SpecSentry makes is either proven from captured evidence or
explicitly unverified. There is no confidence percentage and no "risk
score" computed from counts.

Each finding carries a severity from a fixed per-rule table (high,
medium, low). Severity is never computed from counts, confidence, or any
other aggregate, and it never changes a verdict.

Verdicts and evidence

Every rule evaluation produces exactly one verdict, and each carries a
deterministic identity and the evidence that supports it.

A run-level verdict is also reported. It is derived from the same counts the
engine already produces: FAIL when any violation is proven; INDETERMINATE
when coverage is incomplete or a check could not be decided and the gap is not
a documented limitation; PASS otherwise. A skip caused by a documented
limitation — for example query-value withholding — is printed in the report but
does not by itself lower the verdict, because no better capture could resolve
it.

Each finding carries a stable identifier:

RQ-002-e3e590f4-0-0-737999de
└┬──┘ └────┬────┘ │ │ └────┬────┘
 rule     scope   op obs  rule inputs
          hash    seq seq  digest

scope_hash8 is derived from the declared contract structure only, so it is
stable for a given contract and changes when the contract's requirements
change. PASS confirmations are never counted as drift and never change the
CLI exit status.

INDETERMINATE is never reported as drift. Absence of a finding is never
presented as a claim of conformance.

The demo (python demo/demo.py) demonstrates all three verdicts through the
real pipeline: a PASS case (declared status observed, required parameter
present), a FAIL case (request body violates a declared enum), and an
INDETERMINATE case (a pattern-only schema the evaluator does not support).

What it detects

Request-side drift

Response-side drift

Status matching covers exact codes (200), ranges (2XX), wildcards (*),
and default.

Coverage intelligence

SpecSentry refuses to draw authoritative conclusions from incomplete
evidence. A PARTIAL, empty, or unusable contract or traffic input sets
coverage_complete: false, emits COV-001, and suppresses absence-based
rules (RQ-009, RS-001) — because an absent declaration may be a parsing
artifact rather than real drift.

Reproducible regression tests

From any parsed contract and traffic capture, SpecSentry derives a
deterministic regression suite. Each case is a pure function of its recorded
structural inputs plus the SHA-256 digests of the two source documents:

REG-<16 hex>  RQ-002  FAIL  GET /users/{id}

Cases are byte-identical across runs on the same inputs, and replay never
needs the original files. Cases derived from unusable evidence are recorded as
INDETERMINATE, never as authoritative expectations.

Projects: save, reopen, retrieve

A project is one small JSON file storing the derived, non-sensitive
results plus the SHA-256 digests of the two source documents:

stored   contract/traffic digests, coverage state, matching summary,
         drift findings with structural evidence, skip reasons,
         regression case records, rendered report text
absent   HAR entries, request/response bodies, header values, query values

Saving the same inputs twice produces byte-identical files. Opening a project
returns exactly what was saved, so a prior result can be retrieved without
re-running the analysis or keeping the original capture.

Loading is fail-closed. A missing, corrupt, truncated, oversized, or
version-incompatible project file is reported as an error and never silently
treated as a valid prior result. Project ids are restricted to a safe charset,
so a caller-supplied id cannot escape the store directory.

Privacy by construction

• Query values are withheld at parse time; only parameter names survive,
  because matching needs them.
• URL userinfo credentials and fragments are removed entirely.
• Sensitive headers and credential-shaped keys are redacted.
• Body evidence is reported structurally (<68 byte(s)>), never verbatim.
• Bodies that cannot be redacted safely are replaced with [REDACTED] and
  are never schema-evaluated — no reliable evidence, no claim.

Try it

Every command below runs offline from the project root. specsentry is the
installed console script; python -m core.cli is the same CLI from a source
checkout.

# Full demonstration: all three verdicts, live
python demo/demo.py

# Drift report with evidence (PASS / FAIL / INDETERMINATE)
specsentry drift tests/fixtures/demo/openapi.json tests/fixtures/demo/traffic.har

# Machine-readable output (same verdict, structured)
specsentry drift tests/fixtures/demo/openapi.json tests/fixtures/demo/traffic.har --json

# Matching and coverage only — not a drift verdict
specsentry analyze tests/fixtures/demo/openapi.json tests/fixtures/demo/traffic.har

# Reproducible regression suite
specsentry regress tests/fixtures/demo/openapi.json tests/fixtures/demo/traffic.har

# Save a project, list saved projects, reopen one later
specsentry save tests/fixtures/demo/openapi.json tests/fixtures/demo/traffic.har \
    --project-id demo --store .specsentry
specsentry list --store .specsentry
specsentry open --project-id demo --store .specsentry

# Offline self-check
specsentry selftest

Exit codes

Suitable for CI gating: specsentry drift ... && echo "no drift".

What a report looks like

This is real output from the bundled demo corpus (abridged):

SpecSentry Contract Drift Report
Real traffic. Real evidence. Reproducible tests.

contract: tests/fixtures/demo/openapi.json
traffic:  tests/fixtures/demo/traffic.har
coverage: complete

VERDICT: FAIL
  7 proven violation(s); 7 confirmed compatible; 2 unverified check(s).

drift: 7   confirmed: 7   unverified: 2
  drift by rule: RQ-002=1, RQ-003=2, RQ-005=1, RS-001=2, RS-003=1
  drift by severity: high=1, low=1, medium=5

DRIFT — 7 proven violation(s)

[FAIL] HIGH RQ-002  GET /users/{id} query parameter 'limit'
      rule:     Required parameter declared by the contract was absent
      expected: required query parameter present
      observed: query parameter absent
      evidence: contract: /operations/GET /users/{id}/parameters/limit required
      authority: authoritative
      id:       RQ-002-1e47da1e-0-1-f2f575eb

CONFIRMED — 7 positive confirmation(s)

[PASS] RS-003  GET /users/{id} response status
      rule:     Observed response status is declared by the contract
      expected: observed status declared by the contract
      observed: status 200 is declared
      evidence: contract: /operations/GET /users/{id}/responses 200|404
      evidence: observed: /observations/0/status 200
      authority: authoritative
      id:       RS-003-1e47da1e-0-0-5cccc2be

UNVERIFIED — 2 check(s) not decided (never a finding)
  SKIP-SENSITIVE-VALUE: 2 — query values are withheld, so value checks are skipped

Verdicts: PASS requires sufficient evidence of compatibility; FAIL requires sufficient evidence of a violation; INDETERMINATE is never reported as drift.
PASS confirms observed behaviour; it is not drift.

Finding ids are stable for a given contract and observation sequence, so a
drift finding and the regression case derived from it can be cross-referenced.

Installation

SpecSentry is a Hermes Agent plugin and a Python package. It runs on Python
3.11+ with no required third-party dependencies.

• No third-party dependencies for the analysis core. It runs on the
  standard library alone.
• PyYAML (optional) — only needed to parse YAML OpenAPI documents. JSON
  contracts work without it; YAML input without PyYAML fails with a clear
  message rather than silently.

Nothing is fetched at runtime, and the analysis core performs no network
access.

Option A — Hermes plugin directory

Download the source into the Hermes plugins directory. With Git:

export SPECSENTRY_REPO_URL="https://github.com/<your-org>/specsentry.git"
mkdir -p ~/.hermes/plugins
git -C ~/.hermes/plugins clone "$SPECSENTRY_REPO_URL" specsentry

Or download the archive for the release you want and unpack it into
~/.hermes/plugins/specsentry.

Option B — Python package

Install a pinned release from PyPI:

# Analysis core only (JSON contracts; no third-party dependencies)
python -m pip install --no-deps specsentry==0.1.0

# Core plus YAML OpenAPI support
python -m pip install --no-deps "specsentry[yaml]==0.1.0"

# Core plus test tooling
python -m pip install --no-deps "specsentry[dev]==0.1.0"

To install from a source checkout instead, build a wheel with your build
frontend, then install the pinned artifact it produces from the dist/
directory.

First-run verification

Confirm the installation from any directory:

specsentry --version
specsentry --help
specsentry selftest

specsentry --version prints the installed version. specsentry selftest runs
the offline Plugin Guard and prints a verdict, an install-allowed flag, and a
finding count. A finding count above zero does not mean the install is blocked;
the verdict and the install-allowed flag decide that.

The specsentry console script is provided by the package; python -m core.cli
works from a source checkout without installing.

As a Hermes tool

SpecSentry registers two offline tools:

• specsentry_scope_check — fail-closed authorization scope validation.
• specsentry_drift — drift analysis over two local files, returning the
  run-level verdict, proven violations (drift.findings), positive
  confirmations (drift.pass_confirmations), unverified checks, coverage, and
  the text report.

Both read only files you name. Neither performs DNS, network requests, or
traffic replay.

Privacy by construction

• Query values are withheld at parse time; only parameter names survive,
  because matching needs them.
• URL userinfo credentials and fragments are removed entirely.
• Sensitive headers and credential-shaped keys are redacted.
• Body evidence is reported structurally (<68 byte(s)>), never verbatim.
• Bodies that cannot be redacted safely are replaced with [REDACTED] and
  are never schema-evaluated — no reliable evidence, no claim.

Architecture

core/
  scope.py                     offline scope validation (fail-closed)
  contract/
    openapi.py                 OpenAPI 3.0/3.1 + Swagger 2 parsing
    har.py                     HAR 1.2 parsing, redaction, size caps
    model.py                   frozen immutable models
    redaction.py               secret detection and redaction
    paths.py                   path traversal and file safety
    analysis/
      matching.py              server/path compilation + candidate index
      engine.py                deterministic composition, serialization
      drift.py                 RQ-*/RS-* rules, schema evaluator, reports
      regression_factory.py    deterministic replayable test suites
      project_store.py         save/reopen projects (digests + derived results)
  cli.py                       command-line interface
tools.py                       Hermes tool registration
demo/demo.py                   demonstration (all six stages)

Reporting formats: JSON (serialize_report, --json) and human-readable text
(render_report). Markdown and HTML export are not implemented.

Every analysis layer is pure: no I/O, no network, no clock, no randomness.
The same inputs always produce byte-identical output.

Safety and scope

• Offline only. No DNS, no HTTP, no traffic replay, no credential testing.
• Files are size-capped (10 MiB contract, 1 MiB per body); oversized input is
  rejected, never truncated.
• Path traversal, symlinks, FIFOs, and NUL bytes in file paths are rejected.
• External $ref values are never resolved.
• The scope tool fails closed: any error or missing in_scope field is a
  deny.

Known limitations

These are the product's honest boundaries. Each is a deliberate trade in favour
of never claiming more than the evidence supports.

RQ-013 cannot produce a finding from captured traffic. Step A withholds
every query value as [REDACTED] before analysis, so the sensitive-value skip
always fires before value validation. Query names are still matched and
validated structurally. This is the security fix taking precedence over
coverage; it is not a gap in the matching layer.

Because this limitation is permanent and documented, it does not lower the
run-level verdict: a contract whose only unverified checks are withheld query
values can still report PASS, with the limitation printed in the report. A
check that the evaluator simply does not support (for example a pattern-only
schema) is different — that is genuine uncertainty and reports
INDETERMINATE.

Bodies replaced by [REDACTED] are never schema-evaluated. A body is
redacted when it exceeds the size cap, fails base64 decoding, or cannot be
redacted safely. There is then no reliable evidence, so no body-schema claim is
made in either direction.

Response-schema findings require a recorded, typed body. If a HAR entry has
no response body, or no content type, or a non-JSON media type, the check is
skipped rather than guessed.

No Markdown or HTML export. Reports are JSON and plain text only.

No confidence percentages and no aggregate risk score. Severity is a fixed
per-rule table; it is never computed from counts or confidence, and it never
changes a verdict.

No live target interaction. SpecSentry reads only the two files it is
given. It cannot scan a host, replay traffic, or verify ownership. Coverage is
therefore bounded by whatever the HAR capture contains.

Project files are records, not reproducible artifacts. A saved project
stores derived results and source digests. Re-running the analysis is what
produces reproducible output; the regression suite is the replayable part.

External $ref values are never resolved. A contract relying on external
references will be partially evaluated, and unsupported schema features
(pattern alone, patternProperties) are reported as unverified rather than
guessed.

Tests

python -m unittest discover -s tests

The suite covers parsing, redaction, matching equivalence against the
pre-index reference, drift verdicts in all three states, coverage suppression,
privacy sentinels, determinism, and malformed or unsupported input.

Decision records

Architectural and product decisions are logged append-only in
docs/DECISIONS.md (DEC-001 onward). Each record states
its status — ACCEPTED, RECONSTRUCTED, PROPOSED, SUPERSEDED, or
REJECTED — its evidence, and the condition that would reverse it. A past
record is never rewritten; a change appends a new record and supersedes the
old one.

License

MIT.

README from the reviewed commit 87edf75; it updates when the author re-pins.