EVALUATION METHOD
Twin
Each case is an authored pair: a value that should be redacted, and the same text with one authored change that makes it not a secret. A scanner has to treat the two sides differently.
1 · How it runs
Does the scanner tell a secret from its harmless twin?
- InputAn authored pair: a positive text and its negative twin.
- ChangeOne authored change turns the secret into something that is not one.
- CheckThe positive side is detected within its expected envelope, the twin is left alone, and the pair flips between the two.
- Pairs
- 1,528
- Texts
- 3,056
- Suites
- 66
2 · Recorded now
Do the pairs come apart?
One row for the pair and one for each side, per scanner.
| Check | redact-secret0.1.0-beta.14 | gitleaks8.30.1 | trufflehog3.97.4 | flare-redact1.6.1 |
|---|---|---|---|---|
| Pair told apartThe positive is detected and its twin is left alone | 2 of 1,472 | 559 of 1,472 | 976 of 1,472 | 1,140 of 1,472 |
| Positive sideDetected within its expected envelope | 2 of 1,525 | 479 of 1,525 | 946 of 1,525 | 1,135 of 1,525 |
| Negative twinLeft alone | 0 of 1,472 | 98 of 1,472 | 66 of 1,472 | 53 of 1,472 |
Checks that did not hold, of those scored for that scanner.
A count opens the checks behind it: the ones that did not hold, from this run only. "Needs review" opens the checks that wait for a person. A zero has none to open, and a scanner that did not run has none to list.
Needs review
56 pairs have a side whose expected outcome is unresolved (tier T0). They are counted in no row above.
3 · How to read it
Reading twin
- The first row is the pair as a whole. The next two are its sides, so a pair that was not told apart shows up in at least one of them.
- A pair with a side whose expected outcome is unresolved (tier T0) is counted apart and in none of these rows.
4 · Exact inputs
Where the cases come from
1,528 pairs read by this method. Synthetic content only.
Suites
Every case comes from a published suite. Open a suite to read its fixtures.