EVALUATION METHOD
Metamorphic
A transform changes the context around a value (a prefix, indentation, line endings, a JSON, YAML, Markdown or quoted wrapper) and nothing else. The scanner should detect the same thing as it did on the original text.
1 · How it runs
Does a scanner find the same thing when only the surrounding text changes?
- InputA source case, read once as authored and once per transform.
- ChangeA context or encoding operator wraps or re-encodes the text. The value is not changed.
- CheckThe detection after the transform equals the detection before it (same-detection). The transformed text is also read on its own.
- Source cases
- 4,913
- Transformed texts
- 17,237
- Operators
- 8
2 · Recorded now
Does detection survive a change of context?
| Transform | redact-secret0.1.0-beta.14 | gitleaks8.30.1 | trufflehog3.97.4 | flare-redact1.6.1 |
|---|---|---|---|---|
| Same detection after the transform | ||||
| context.unicode-prefixPuts a line of non-ASCII text before the value | 16 of 4,885 | 993 of 4,885 | 1,708 of 4,885 | 1,978 of 4,885 |
| context.indentIndents the text by four spaces | 16 of 4,885 | 993 of 4,885 | 1,708 of 4,885 | 1,978 of 4,885 |
| encoding.crlfChanges line endings to CRLF | 10 of 4,041 | 783 of 4,041 | 1,462 of 4,041 | 1,714 of 4,041 |
| context.jsonWraps the value as a JSON string | 3 of 660 | 113 of 660 | 127 of 660 | 138 of 660 |
| context.quoteWraps the value in double quotes | 3 of 660 | 113 of 660 | 127 of 660 | 138 of 660 |
| context.single-quoteWraps the value in single quotes | 3 of 666 | 113 of 666 | 133 of 666 | 144 of 666 |
| context.yamlWraps the value as a YAML scalar | 3 of 666 | 113 of 666 | 133 of 666 | 144 of 666 |
| context.markdownWraps the value in inline code | 3 of 663 | 113 of 663 | 130 of 663 | 141 of 663 |
| The transformed text on its own | ||||
| Value detectedA value that should be detected is, within its expected envelope | 42 of 8,405 | 3,112 of 8,405 | 5,419 of 8,405 | 6,161 of 8,405 |
| Look-alike left aloneA text that should not be flagged is not | 15 of 8,721 | 222 of 8,721 | 108 of 8,721 | 214 of 8,721 |
Transformed texts where the check did not hold, of those scored for that scanner.
A count opens the checks behind it: the ones that did not hold, from this run only. "Needs review" opens the checks that wait for a person. A zero has none to open, and a scanner that did not run has none to list.
Needs review
28 source cases have a side whose expected outcome is unresolved (tier T0). They are counted in no row above.
3 · How to read it
Reading metamorphic
- A scanner that missed the value on both sides holds the relation, so the second group reads the transformed text on its own. Together they tell the two cases apart.
- A source with an unresolved expectation (tier T0) is counted apart and in none of these rows.
4 · Exact inputs
Where the cases come from
Suites
Every case comes from a published suite. Open a suite to read its fixtures.
Show the 75 suites
Operators
Generated counts exclude the unaltered source. Not applicable means the operator could not be applied to that source.
| Operator | What it changes | Generated | Not applicable |
|---|---|---|---|
| context.unicode-prefix | Puts a line of non-ASCII text before the value | 4,913 | 0 |
| context.indent | Indents the text by four spaces | 4,913 | 0 |
| encoding.crlf | Changes line endings to CRLF | 4,051 | 862 |
| context.json | Wraps the value as a JSON string | 669 | 4,244 |
| context.quote | Wraps the value in double quotes | 669 | 4,244 |
| context.single-quote | Wraps the value in single quotes | 675 | 4,238 |
| context.yaml | Wraps the value as a YAML scalar | 675 | 4,238 |
| context.markdown | Wraps the value in inline code | 672 | 4,241 |