Paranoid - tests passed, but does it actually work?

Every developer who has worked with an AI coding agent knows this moment: the agent announces it's done, the test suite is green, and the feature doesn't work. Not "fails on an edge case" doesn't work - returns HTTP 500 on every single request doesn't work.

I got tired of being the integration test. So I built Paranoid.

What it is. A Claude Code plugin that takes away the agent's authority to decide when it's finished.

What it does. A session cannot end until a developer-owned check - one that exercises the running application, not a mock - actually passes. Failure blocks the agent and feeds it the real error; a guard keeps the agent from editing the check.

Does it work? Measured, not claimed: in a pre-registered 42-session eval, ungated agents ended 75% of sessions on still-broken software. With Paranoid, 12/12 sessions ended with the check passing, for about +$0.22 per session - and zero false blocks on healthy code.

Get it. Two commands and one committed JSON file - github.com/egehazar/paranoid.

Below: the design decisions, the eval that refuted two of my own hypotheses, and what broke on the way.

The failure class

Here's the gap, reproduced in the demo project that ships with the repo. The unit tests:

$ npm test

✔ formats a user label (0.5088ms)
ℹ tests 1
ℹ pass 1
ℹ fail 0

The same code, booted as a real server and hit on its real endpoint:

$ npm run paranoid:check

GET /api/users/123 -> HTTP 500
internal error: formatUser: displayName missing

Green tests, broken app, same tree, same minute.

The cause is deliberately mundane, because the failure class is mundane: the database returns display_name, the feature reads user.displayName, and the test is green because it feeds the function a hand-invented camelCase object - the shape its author assumed, not the shape the system produces. The test verifies the assumption against itself. No amount of "more tests like this" helps, because every one of them inherits the same wrong assumption.

The pattern has data behind it. A 2026 MSR study of 1.2 million commits found that coding-agent test commits added mocks measurably more often than human ones

36% versus 26% - and warned that mocked tests are easier to generate while providing weaker evidence about real interactions (arXiv:2602.00409). A mocked test can be simultaneously "fully tested" and wrong. That's the gap Paranoid closes: not bad tests, but tests whose green light measures internal consistency rather than reality.

Deciding where the enforcement lives

The interesting architectural question wasn't what to check - it was where to put the check so it cannot be ignored. Four placements:

In the prompt - "always verify against the running app before finishing." This is advice, and advice decays. It competes with everything else in context, and the model weighs it against its own sense of being done.

In CI - the right oracle at the wrong moment. CI catches the broken build after the session is over, when the agent's context - the cheapest place to fix the bug, with the full investigation still loaded - is already gone. You pay for the round trip back.

In a wrapper harness - drive the agent from an outer loop that re-prompts until checks pass. Workable, but now you own a second orchestration layer, and you've left the native tooling everyone actually uses.

At the session boundary - Claude Code fires a Stop hook whenever the agent tries to end its turn. That's the one chokepoint every path to "done" passes through, on the platform developers already run. Blocking there turns the check from advice into a termination condition, and the failure output lands directly in the live session, where fixing it is cheapest.

I picked the session boundary. The result is deliberately small - two hooks and a config file, zero runtime dependencies:

The Stop hook runs the command from a committed .paranoid.json against the real application. Exit 0 releases the session. Anything else blocks the stop and feeds the real failure back to the agent, which has to keep working.
The PreToolUse guard denies agent edits to the check and any protected paths - an agent that can rewrite its own acceptance criteria has no acceptance criteria. Since a guard can't cover every mutation path, the Stop hook independently re-verifies with git: uncommitted changes to protected files block completion outright.
A portable skill carries the same protocol, as instructions, to agents that don't support hard hook enforcement.

One principle ties the design together: the definition of "done" belongs to the repository owner, and it's enforced, not suggested.

Design decisions

Beyond placement, every non-obvious call is the same call made repeatedly: when something is ambiguous, fail toward catching the bug.

Fail closed, never open. Unparseable config, invalid timeout, a check that won't start, a check that hangs - all of them block. A verification tool whose failure mode is approval is worse than no tool, because it converts uncertainty into false confidence.
Re-run on continuation. When a session resumes after an earlier block, the agent is implicitly claiming "I fixed it," which is exactly when the check must run again. The first implementation exited early on continuation stops, quietly un-gating every session's second attempt. An adversarial audit caught it; a regression test pins it.
Timeouts nest under the host's. Claude Code kills any hook at 300 seconds, so project checks are capped at 240. A check that outlived its host would be killed by the platform and fail open - the one direction this tool must never fail. The 60-second gap isn't slack; it's budget for cleanup and reporting.
Config discovery is .git-bounded. Paranoid walks up from the working directory to find .paranoid.json, but never across a .git boundary - a nested repo inheriting a parent's check means running a stranger's shell command. An earlier version also had a hidden ten-directory walk ceiling that silently bypassed the gate in deep trees. Silent depth limits in enforcement code are landmines; it's gone, and there's a test.
Tamper parsing done properly. The git verification reads NUL-delimited porcelain output, because the naive line-based version broke on filenames with spaces and on renames - meaning a rename could have slipped past tamper detection. Found in audit, fixed, tested.

Using it

Two commands inside Claude Code, then a .paranoid.json at the repo root:

{
  "check": "node scripts/check-live-app.mjs",
  "timeoutSeconds": 120,
  "protected": ["scripts/check-live-app.mjs"]
}

The check is any command that exits non-zero while the app is broken - boot the server and hit an endpoint, exercise the real CLI, drive the built page with Playwright, verify a side effect. That's the entire integration. A check is a project-defined shell command, so the plugin belongs at local scope in repos you trust, not installed globally across code you haven't read.

I didn't trust my own demo, so I measured it

A demo is an anecdote, and I wrote the demo. Before scoring anything I committed a pre-registration: hypotheses, metrics, sample sizes, and a no-cherry-picking rule

every session row gets published no matter what it shows. The oracle is the check's exit code: there is no LLM judging LLM output anywhere in the loop, so the scoring can't be argued with.

Then I ran 42 headless claude-sonnet-5 sessions across green-tests / broken-app fixtures. Each fixture was verified three ways before use and scrubbed of anything that telegraphed its bug - a lesson from my pilot, where an agent "independently found" a bug that a leftover comment had announced. I demoted that pilot to mechanics-validation only and added a no-telegraphing rule to the pre-registration. If the fixture leaks the answer, the eval measures reading comprehension, not diligence.

The headline result refuted my own first hypothesis. I expected ungated agents to sometimes claim broken software was ready. They didn't - not once in twelve sessions. What actually happened is quieter and more damning:

75% of ungated sessions ended with the check still failing - and the agent said so every time. It diagnosed the breakage, reported "not ready," and stopped, leaving the software broken. With Paranoid loaded, 12 of 12 sessions ended with the check passing, every fix a root-cause fix - I audited all the diffs - at a mean cost of +7.5 turns / +$0.22 per session.

So the real failure class isn't deception. It's reported-but-unresolved termination: the model's task-scoped "done" is narrower than the repository owner's, and the session ends on top of software the agent itself just called broken. That sharpened what I'd actually built. Paranoid doesn't make agents honest - they already were. It moves the repo's definition of done from advice to termination condition.

Two pre-registered control cells kept me from over-claiming why it works:

A blind forced-retry control - a hook that blocks every stop with a generic "not ready, keep working" but never runs any check - also recovered 12/12. So persistence alone, not the check's feedback, drives recovery. It's the most useful number in the eval, because it isolates what the developer-owned check actually buys: persistence made cheap and clean. Half the turns (26 vs 54+), zero timeout-bound endings against 5/12, roughly 2.4× lower cost, and an intact reporting protocol in every session versus none.
A healthy-repo control - on an already-working app, Paranoid false-blocked 0/3, made zero unnecessary edits, and was cheaper than the ungated baseline. A gate that cries wolf on clean code gets uninstalled in a week, so this was the cell that decided whether the tool deserved to exist.

An eval designed so it can only flatter the tool isn't an eval. Both refuted hypotheses are in the published analysis, with the raw session rows.

Watching it fire on a real session

The mechanism end-to-end on a real session; full transcript and debug log are in the repo. The agent was asked to run the tests and report readiness. It ran them (green), found the real endpoint broken anyway, reported honestly - "The project is not ready" - and tried to end the session. The gate:

2026-08-02T13:41:59.028Z [DEBUG] "Hook Stop (Stop) error:

PARANOID
──────────────────────────────────────────────
Real app check   ✗ failed (exit 1, 1.2s)
  node scripts/check-live-app.mjs

GET /api/users/123 -> HTTP 500
internal error: formatUser: displayName missing

check: expected HTTP 200 from the running app

Tests may be green. The feature isn't.
Fix the underlying issue - do not touch the check or
.paranoid.json - then finish. Paranoid will re-run it.
──────────────────────────────────────────────"

Ninety-one seconds later, after the agent fixed the actual field-shape bug - and corrected the test's mock to the real database shape, which is the whole failure class healing itself on camera:

2026-08-02T13:43:30.700Z [DEBUG] "Hook Stop (Stop) success:

PARANOID
──────────────────────────────────────────────
Real app check   ✓ passed (1.2s)
  node scripts/check-live-app.mjs

Tests passed. The feature actually ran.
──────────────────────────────────────────────"

The session could not have ended any other way. One disclosure: in this capture the agent knew the gate existed - the plugin ships a skill and the agent loaded it unprompted. The agent-unaware condition is what the eval's de-telegraphed fixtures measure, not this recording.

What broke on the way

Paranoid went through four adversarial audit rounds before this build, and the pattern in that log is the most instructive thing in the project: every round's fix looked correct, and still failed the actual promise until it was exercised against the real edge.

Round	Claimed fixed	What the next pass still found
0.1.0	Initial build: Stop-hook check, edit guard, demo, packaging	Continuation stops exited early (second attempts went ungated); subdirectory runs bypassed the project root; invalid timeouts failed open; the demo check leaked processes
0.1.1	Continuation re-runs; root via env; timeouts fail closed; clean teardown	No config discovery at all without the env var - and any walk-up had to stop at `.git`
0.1.2	`.git`-bounded walk-up discovery	Invalid marketplace packaging; a hidden 10-dir discovery ceiling silently bypassed deep trees; checks could outlive the host timeout and fail open; tamper detection broke on spaces and renames; no Windows CI
0.1.3	Ceiling removed; 240s cap; NUL-delimited parsing; Windows CI	The test script used a glob Windows `cmd.exe` + Node don't expand - the new Windows CI leg couldn't pass
0.1.4	Explicit test file path	Strict validation flagged a manifest field the runtime tolerated
0.1.5	Manifest fixed; strict validator passes clean	-

The lesson is test the promise, not the pattern. A block that looks like a block, a discovery walk that looks like discovery, tamper detection that looks like tamper detection: each passed inspection and still broke against the real continuation stop, the real deep tree, the real Windows shell. That is Paranoid's own thesis applied to Paranoid, and it's why the repo ships a 14/14 test suite, passes strict plugin validation, and has CI re-prove the green-tests/broken-app demo on GitHub's runners on every push.

Limitations

Paranoid guarantees that the configured check ran and passed before the agent could finish - nothing more. A check proves only what it exercises. It's a guardrail, not a QA department.
Claude Code's Stop hook fires at the end of every agent turn, not only at claimed completion, so in a failing repo Paranoid can interrupt a progress update. Acceptable for a v0; stated rather than hidden.
The platform overrides a Stop hook after eight consecutive blocks. Paranoid re-runs the check on each continuation but can't override that cap.
The edit guard is not a security sandbox. Git catches uncommitted tampering; a deliberately committed bypass is outside the threat model. The target is mistaken completion, not a malicious agent.
No .paranoid.json means Paranoid stays silent. It never invents a check.

What I'd build next

Per-task checks - one global check is the right v0, but real repos want the gate scoped to what changed. First-class enforcement beyond Claude Code - the portable skill currently carries the protocol as instructions; agents with hook systems deserve the hard version. And the remaining eval cells - four ship-confirmation fixtures are built and unscored; the matrix should be completed and published like the rest, whatever it shows.

Where everything lives

The plugin and its tests, the pre-registration, all 42 session rows, the audited diffs, the live-session transcript and debug log, and the full analysis - including both refuted hypotheses - are at github.com/egehazar/paranoid.