EVALUATION METHOD
Differential
The canonical input of each case is read by redact-secret and by each peer, and the ranges each one reports are compared. A difference is review evidence. A peer is not ground truth and a difference is not a failure of either tool.
1 · How it runs
Where does another scanner report something different from redact-secret?
- InputThe canonical input of each case, unchanged.
- ChangeNone.
- Checkredact-secret and the peer report the same ranges, or they differ in one of the stated ways. A difference is queued for review.
- Inputs
- 6,441
- Peers
- 3
- Suites
- 75
2 · Recorded now
Where the ranges differ
| What was compared | gitleaks8.30.1 | trufflehog3.97.4 | flare-redact1.6.1 |
|---|---|---|---|
| What each tool reported | |||
| No difference foundThe same ranges, or none from either, and no family difference where families could be compared | 4,497 of 6,441 | 3,743 of 6,441 | 3,490 of 6,441 |
| Ranges differBoth reported something, and not the same ranges | 89 of 6,441 | 67 of 6,441 | 385 of 6,441 |
| Only redact-secret reportedredact-secret reported a range the peer did not | 1,107 of 6,441 | 2,416 of 6,441 | 2,344 of 6,441 |
| Only the peer reportedThe peer reported a range redact-secret did not | 234 of 6,441 | 75 of 6,441 | 94 of 6,441 |
| Same ranges, different familyThe ranges match and the families they map to differ | 514 of 6,441 | 140 of 6,441 | 128 of 6,441 |
| Family classification | |||
| ComparableRanges match and every range maps to a familyNot listed: the same comparisons as the rows above, split by family. Open a count above. | 1,425 of 6,441 | 460 of 6,441 | 407 of 6,441 |
| Not comparableNothing to compare, an unmapped range, or ranges that differNot listed: the same comparisons as the rows above, split by family. Open a count above. | 5,016 of 6,441 | 5,981 of 6,441 | 6,034 of 6,441 |
Inputs that fell in that row, of the inputs both tools completed.
A count opens the comparisons behind it, from this run only. A zero has none to open. Which tool is right is decided in the review ledger (benchmarks/review-ledger.json), never on these pages.
Differences are review evidence
7,593 comparisons recorded a difference and are queued for review. Which tool is right is not decided here.
3 · How to read it
Reading differential
- Rows are exclusive: an input falls in one of the first five rows. The classification rows are a second view of the same inputs, not more of them.
- Family classification can be not comparable even when the ranges compare, for instance when a family is not mapped.
4 · Exact inputs
Where the cases come from
Suites
Every case comes from a published suite. Open a suite to read its fixtures.