EVALUATION METHOD
Mutation
An operator alters the secret itself: its length, its last or first character, its alphabet, a delimiter or a segment. Whether the altered value is still a secret depends on the format contract, and the result is scored only where the contract says it is.
1 · How it runs
Does a scanner still find a value that was altered but is still valid?
- InputA source case with a single secret in a known format.
- ChangeA lexical, boundary or structural operator alters the value.
- CheckWhere the altered value still matches the format contract, the scanner is expected to detect it as before. Where it no longer matches, the expectation is deferred and nothing is scored.
- Source cases
- 4,913
- Altered values
- 12,037
- Operators
- 7
2 · Recorded now
Does detection survive an altered value?
| Operator | redact-secret0.1.0-beta.14 | gitleaks8.30.1 | trufflehog3.97.4 | flare-redact1.6.1 |
|---|---|---|---|---|
| Same detection, format still valid | ||||
| lexical.length-minus-oneDrops the last character of the secret | 0 of 414 | 184 of 414 | 288 of 414 | 352 of 414 |
| lexical.length-plus-oneAppends one character to the secret | 0 of 416 | 187 of 416 | 306 of 416 | 350 of 416 |
| lexical.replace-lastReplaces the last character of the secret | 0 of 1,505 | 502 of 1,505 | 929 of 1,505 | 1,083 of 1,505 |
| lexical.invalid-alphabetReplaces the last character with one outside the format | Needs review 2,020 unscored | Needs review 2,020 unscored | Needs review 2,020 unscored | Needs review 2,020 unscored |
| lexical.prefix-changeReplaces the first character of the secret | 0 of 43 | 11 of 43 | 14 of 43 | 41 of 43 |
| boundary.remove-delimiterRemoves one delimiter (dot, underscore or hyphen) from the secret | 0 of 112 | 76 of 112 | 86 of 112 | 93 of 112 |
| structural.remove-segmentRemoves one delimited segment (SendGrid and Slack token formats only) | Needs review 48 unscored | Needs review 48 unscored | Needs review 48 unscored | Needs review 48 unscored |
| The altered value on its own | ||||
| Value detectedA value that should be detected is, within its expected envelope | 0 of 2,490 | 955 of 2,490 | 1,604 of 2,490 | 1,899 of 2,490 |
Altered values where the check did not hold, of those scored for that scanner.
A count opens the checks behind it: the ones that did not hold, from this run only. "Needs review" opens the checks that wait for a person. A zero has none to open, and a scanner that did not run has none to list.
Needs review
9,547 altered values no longer match the format contract. Their expectation is deferred, so they are counted in no row above.
3 · How to read it
Reading mutation
- Most altered values no longer match the format. Those are deferred to review, because a scanner may reasonably report a valid substring or ignore the value.
- The authored pairs in this method are the twin method. They are not repeated here.
4 · Exact inputs
Where the cases come from
Suites
Every case comes from a published suite. Open a suite to read its fixtures.
Show the 75 suites
Operators
Generated counts exclude the unaltered source. Deferred values no longer match the format contract. Not applicable means the operator could not be applied to that source.
| Operator | What it changes | Generated | Scored | Deferred to review | Not applicable |
|---|---|---|---|---|---|
| lexical.length-minus-one | Drops the last character of the secret | 2,020 | 414 | 1,606 | 2,893 |
| lexical.length-plus-one | Appends one character to the secret | 2,020 | 416 | 1,604 | 2,893 |
| lexical.replace-last | Replaces the last character of the secret | 2,020 | 1,505 | 515 | 2,893 |
| lexical.invalid-alphabet | Replaces the last character with one outside the format | 2,020 | 0 | 2,020 | 2,893 |
| lexical.prefix-change | Replaces the first character of the secret | 2,020 | 43 | 1,977 | 2,893 |
| boundary.remove-delimiter | Removes one delimiter (dot, underscore or hyphen) from the secret | 1,889 | 112 | 1,777 | 3,024 |
| structural.remove-segment | Removes one delimited segment (SendGrid and Slack token formats only) | 48 | 0 | 48 | 4,865 |