EVALUATION
How the evaluation reads a scanner
Six methods read the same synthetic inputs in different ways: authored pairs, harmless look-alikes, changed context, altered values, one scanner against another, and a frozen holdout. Pick a question, or a method.
The six methods
Each page says how the method runs, what the run recorded for it and how to read that, then lists its exact inputs.
- TwinDoes the scanner tell a secret from its harmless twin?1,528 pairs
- BenignDoes the scanner leave harmless look-alikes alone?2,277 controls
- MetamorphicDoes a scanner find the same thing when only the surrounding text changes?4,913 source cases
- MutationDoes a scanner still find a value that was altered but is still valid?4,913 source cases
- DifferentialWhere does another scanner report something different from redact-secret?6,441 inputs
- HoldoutDid the frozen candidate run its holdout cases?12 cases
The run behind these pages
Run 6183a594, finished 2026-10-10, from revision 019a450. The method pages read this one run, so it is stated here and not on each page. Holdout reads a separate qualification run.
| Scanner | Version | How it was run | Observed | Execution |
|---|---|---|---|---|
| redact-secret | 0.1.0-beta.14 | Published npm package · default detectors | Fresh, 2026-10-10 | Complete |
| gitleaks | 8.30.1 | Directory scan · default rules | Fresh, 2026-10-10 | Complete |
| trufflehog | 3.97.4 | Filesystem scan · verification disabled | Fresh, 2026-10-10 | Complete |
| flare-redact | 1.6.1 | Published npm package · secrets-only (pii, generic_assignment disabled) · JavaScript engine | Fresh, 2026-10-10 | Complete |
How to read every page
Recorded, not judgedA figure is what a run recorded. This site does not rank scanners or call a result good or bad.
Read a number in its rowEvery scanner reads the same synthetic inputs, and a cell is a count of one kind of check. Counts are never added across rows or scanners.
Needs review is not a scoreA check whose expected outcome is unresolved is counted apart. It is neither passed nor failed.
A peer is not ground truthWhere two scanners differ, the difference is evidence to read, not a vote.