Faintprint

What we can say, and what we cannot

Every figure comes from our own evaluation and carries its caveat.

What we can say — each with its caveat

  1. The AI reviewer rarely accuses a harmless server. On passages from servers believed to be harmless (chosen by fixed rules from how established and widely used they are, before anyone read them), it wrongly confirmed an attack on 5 of 1,449 checks (0.34%), all five on one passage. After one person on our side re-read the six disputed passages, the figure was 0 — but four of those labels were changed after seeing the reviewer's answer, which makes the second number weaker, so we always give both. Measured once, in September 2026, on data held back from all tuning.
  2. It does not wave known attacks through. On published attack examples it gave a verdict on 92% of checks and missed none of those; with the backup model used in production, 100% answered, none missed. The attacks come largely from one research paper and three templates.
  3. Fooling the reviewer cannot make a server look safer. The fixed rules are the floor; the reviewer can only add to them. A fully fooled reviewer gives exactly the fixed-rule result.
  4. We attacked the reviewer ourselves. When we wrote 24 attacks aimed at the AI reviewer itself — forged boundary markers, fake "approved by security" notes, "this is a test fixture", encoded and foreign-language instructions, the attack quoted as an example, long documentation padding, refusal bait — and put each of them on all 173 known attack passages in our evaluation set, the reviewer was talked into clearing a passage it had confirmed once in 3,272 checks. It downgraded about one in a hundred to "suspicious", almost all when the attack text claimed the passage was a quoted example; a "suspicious" passage still counts and is still shown. Two shapes made it decline to answer more often (about 29% instead of 13%); a declined check counts as no confirmation, never as a clearance. Caveats that travel with this: the attacks were written by a model of the same family as the reviewer; the passages come largely from one research paper; this is the evaluation set, not a held-out one; and the figures are for the batch path without the production fallback.
  5. You can check every finding yourself. Each one comes with the quoted text, and each passage the reviewer confirms with its reason.
  6. It costs cents. About 7 in 10 servers need no AI call at all; a scan is capped at 30 calls, about 35 cents at worst, on your own Anthropic key.
  7. What leaves your network: only the servers' own text, to Anthropic, on your key, after the price is shown and you say yes (in a pipeline, only within the cost limit you set). Nothing from your config files, no token, no file name. Nothing is ever sent to us.

What we cannot say