REDACT-SECRET 0.1.0-BETA.14

What the benchmark shows

Synthetic inputs, the same for every scanner, scored span by span. Start from a provider or a family, or see what changed.

New pipelineAuthority

Built from the new pipeline

The new pipeline is the authority for credential qualification. These numbers are read from the qualification view derived from the official credential-eval run of the public-evidence-snapshot population. The regression and policy populations keep their own counts on the qualification pages and are not added in here.

Authority: new
Population: public-evidence-snapshot
Engine: credential-eval 0.1.0-alpha.16
Evidence: snapshot-2026.10.06.4
Run: sha256:4bec6e539482…
Recorded: 2026-10-07
Authorised for: @redact-secret/core@0.1.0-beta.14

Maintainer-reviewed (independent review pending)

598 fixtures of the 7,041 in evidence release snapshot-2026.10.06.4 carry this label. They are counted in the numbers on this page.

Passed the project’s own verification (source evidence, automated checks, recorded counter-arguments); not yet independently reviewed.

OpenRedaction default (all patterns): not measured in this run (optional)

The default (all-pattern) profile is slow and noisy for credential evaluation: personal data and resource identifiers dominate its findings, and its last official measurement took 7 minutes of scanner time on a population the credential profile scans in seconds (its methods run takes 40+ minutes), so it is a manual measurement and never blocks other scanners or core verification (#763). Its earlier results stay as history with their run identity; the credential profile is a separate scanner, never spliced with it.

Last measurement: policy-corpus@linux-x64 (configuration sha256:2451ce7bf380), public-evidence-snapshot+methods@linux-x64 (configuration sha256:d47d081ce5fb), public-evidence-snapshot@linux-x64 (configuration sha256:2451ce7bf380), regression-corpus@linux-x64 (configuration sha256:2451ce7bf380) · engine 0.1.0-alpha.5 · recorded 2026-10-05 · retained record, no longer listed by the run registry. It stays labelled with that run identity and is never combined with another profile or another run.

Open the qualification page for this pointer

OpenRedaction credential profile (33 types): not measured in an official run (local exploratory diagnostics only: see ADR)

The credential profile is the comparison profile chosen in docs/decisions/2026-10-06-choose-the-openredaction-credential-bearing-profile-as-the-comparison-scanner.md. It is optional in the official run class because no pinned engine configuration measures it yet: making it required would refuse every official view until the owner approves the one profile-only run. Its own history starts with that run; nothing of the default's is reused for it.

No earlier measurement of it is recorded. Nothing is shown in its place.

Official measurement: pending: requires owner approval (one profile-only run; see the decision).

Open the qualification page for this pointer

Every population and its qualification

PROVIDER-DOCUMENTED

Three answers

Run 2026-10-07
Same 7,036 inputs for 4 scanners
Accounting v1.1
Mode published · redact-secret 0.1.0-beta.14

Does it miss real secrets?

at most 2.3%

23 of 1,468 secret spans leaked

Leaked span rate. Lower is better. 95% pessimistic bound, corpus-relative.

Does it flag safe values?

at most 2.2%

0 of 167 controls flagged

False alarm rate on provider-documented controls. Lower is better. Few controls keep the bound wide.

Does it tell near-twins apart?

at least 97.9%

808 of 817 pairs discriminated

Twin discrimination: the secret is covered and its one-character fake stays quiet. Higher is better.

TOOL-CORROBORATED

Three answers

Run 2026-10-07
Same 7,036 inputs for 4 scanners
Accounting v1.1
Mode published · redact-secret 0.1.0-beta.14

Does it miss real secrets?

at most 3.1%

9 of 548 secret spans leaked

Leaked span rate. Lower is better. 95% pessimistic bound, corpus-relative.

Does it flag safe values?

at most 1.3%

14 of 1,866 controls flagged

False alarm rate on tool-corroborated controls. Lower is better. Few controls keep the bound wide.

Does it tell near-twins apart?

not published —

230 of 233 pairs discriminated Withheld

Twin discrimination: the secret is covered and its one-character fake stays quiet. Higher is better. Too few secrets have an authored near-twin to publish a rate.

PROJECT POLICY

Three answers

Run 2026-10-07
Same 7,036 inputs for 4 scanners
Accounting v1.1
Mode published · redact-secret 0.1.0-beta.14

Does it leave policy spans readable?

at most 7.1%

58 of 1,053 secret spans leaked

Leaked span rate. Lower is better. 95% pessimistic bound, corpus-relative. These spans are this project’s redaction policy: a difference here is a difference of opinion, not a defect.

Does it flag safe values?

at most 0.9%

9 of 1,880 controls flagged

False alarm rate on project policy controls. Lower is better. Few controls keep the bound wide.

Does it tell near-twins apart?

at least 96.8%

308 of 312 pairs discriminated

Twin discrimination: the secret is covered and its one-character fake stays quiet. Higher is better.

What changed

Findings this benchmark handed to the product, newest first. Ledger last reviewed 2026-10-07, last measured on 0.1.0-beta.14; not live issue status.

All 64 findings

Promoted#1236 · Square placeholders are still reported: EAAA-your-access-token, EAAA<your-production-access-token>, sandbox-sq0csb-<your-sandbox-application-secret>Flagged a safe value · 3 fixtures2026-10-06
Verified#1042 · Placeholders are still reported: vcp_/vca_/vcr_ x-run and rpa_ word fixtures, ClickHouse mykeysecret, Paddle ellipsis, Bitwarden x-runFlagged a safe value · 7 fixtures2026-09-30
Verified#1041 · generic-token redacts the placeholder phrase placeholder-not-a-key under secret_access_key (newly reached since #1026)Flagged a safe value · 1 fixture2026-09-30
Verified#1038 · Python subscript assignments os.environ["NAME"] = "<literal>" are never read as an assignment: credential values under MISTRAL_API_KEY, CO_API_KEY, API_TOKEN are missedLeft a secret readable · 1 fixture2026-09-30
Verified#1018 · A complete, unmasked Cohere key under a masked_-led name (LiteLLM proxy debug log) is not reportedLeft a secret readable · 1 fixture2026-09-30
Verified#1017 · deepgram-api-key misses the JS SDK createClient("<key>") and the WebSocket token subprotocol, and types adjacent-line Deepgram context as generic-tokenLeft a secret readable · 2 fixtures2026-09-30

Other scanners on the same inputs

We ran 3 other scanners on the same 1,440 provider-documented inputs. This shows what each one left readable. It does not show which scanner is better.

Read this before the numbers

Our inputs, our answer key. The redact-secret team wrote every input and every expected span, using redact-secret’s own definition of a secret.
redact-secret was tuned on these inputs. 56 of 64 findings from this corpus are recorded as fixed in redact-secret. The other scanners were never tuned against it.
Different jobs. Most inputs fall outside at least one scanner’s rules. A readable span there shows where its rules end, not that it failed.

Other scanners on the same inputs
Scanner	Inputs its rules target	Left readable on those inputs	Left readable everywhere else	Left readable on all inputs	Safe values flagged
flare-redact 1.6.1Runtime library · Published npm packageBuilt to redact secrets and personal data from text at runtime. Run secrets-only here: its personal-data and generic-assignment detectors are off.	468 of 1,44038 of its 81 rules target a credential family	48 of 485 spansredact-secret, same inputs: 4 of 485	870 of 983 spansNo rule of its own targets these	918 of 1,468 spansredact-secret, same inputs: 23 of 1,468	6 of 167redact-secret, same inputs: 0 of 167
gitleaks 8.30.1Repository scanner · Directory scanBuilt to find secrets in git history, files and directories before they are committed.	635 of 1,44073 of its 222 rules target a credential family	92 of 646 spansredact-secret, same inputs: 4 of 646	426 of 822 spansNo rule of its own targets these	518 of 1,468 spansredact-secret, same inputs: 23 of 1,468	10 of 167redact-secret, same inputs: 0 of 167
trufflehog 3.97.4Repository scanner · Filesystem scanBuilt to find and verify secrets in repositories and other sources. Verification is off here.	722 of 1,44083 of its 892 rules target a credential family	168 of 739 spansredact-secret, same inputs: 2 of 739	715 of 729 spansNo rule of its own targets these	883 of 1,468 spansredact-secret, same inputs: 23 of 1,468	4 of 167redact-secret, same inputs: 0 of 167

Observed once each, in the official run sha256:4bec6e539482…, recorded 2026-10-07. Listed in run order; a new scanner adds a row. Rules are matched to families from each scanner’s pinned rule file (reviewed 2026-09-30).

Quoting these numbers

Don't “redact-secret leaks far fewer secrets than flare-redact.”

Do “On 468 provider-documented inputs that match flare-redact 1.6.1’s default rules, in a corpus written and used for tuning by the redact-secret team, flare-redact left 48 of 485 secret spans readable.”

Any quote names the input slice, the version, the date, and who wrote the inputs.

Other scanners on the same inputs

We ran 3 other scanners on the same 548 tool-corroborated inputs. This shows what each one left readable. It does not show which scanner is better.

Read this before the numbers

Our inputs, our answer key. The redact-secret team wrote every input and every expected span, using redact-secret’s own definition of a secret.
redact-secret was tuned on these inputs. 56 of 64 findings from this corpus are recorded as fixed in redact-secret. The other scanners were never tuned against it.
Different jobs. Most inputs fall outside at least one scanner’s rules. A readable span there shows where its rules end, not that it failed.

Other scanners on the same inputs
Scanner	Inputs its rules target	Left readable on those inputs	Left readable everywhere else	Left readable on all inputs	Safe values flagged
flare-redact 1.6.1Runtime library · Published npm packageBuilt to redact secrets and personal data from text at runtime. Run secrets-only here: its personal-data and generic-assignment detectors are off.	195 of 54838 of its 81 rules target a credential family	44 of 195 spansredact-secret, same inputs: 9 of 195	316 of 353 spansNo rule of its own targets these	360 of 548 spansredact-secret, same inputs: 9 of 548	68 of 1,866redact-secret, same inputs: 14 of 1,866
gitleaks 8.30.1Repository scanner · Directory scanBuilt to find secrets in git history, files and directories before they are committed.	268 of 54873 of its 222 rules target a credential family	49 of 268 spansredact-secret, same inputs: 9 of 268	117 of 280 spansNo rule of its own targets these	166 of 548 spansredact-secret, same inputs: 9 of 548	121 of 1,866redact-secret, same inputs: 14 of 1,866
trufflehog 3.97.4Repository scanner · Filesystem scanBuilt to find and verify secrets in repositories and other sources. Verification is off here.	336 of 54883 of its 892 rules target a credential family	111 of 336 spansredact-secret, same inputs: 9 of 336	212 of 212 spansNo rule of its own targets these	323 of 548 spansredact-secret, same inputs: 9 of 548	62 of 1,866redact-secret, same inputs: 14 of 1,866

Quoting these numbers

Don't “redact-secret leaks far fewer secrets than flare-redact.”

Do “On 195 tool-corroborated inputs that match flare-redact 1.6.1’s default rules, in a corpus written and used for tuning by the redact-secret team, flare-redact left 44 of 195 secret spans readable.”

Any quote names the input slice, the version, the date, and who wrote the inputs.

Other scanners on the same inputs

Hidden by default. Project policy is this project’s masking policy: a peer’s rate here reflects scope, not accuracy, because a peer is not built to flag it. Show anyway.

Other scanners on the same inputs

We ran 3 other scanners on the same 1,006 project policy inputs. This shows what each one left readable. It does not show which scanner is better.

Read this before the numbers

Our inputs, our answer key. The redact-secret team wrote every input and every expected span, using redact-secret’s own definition of a secret.
redact-secret was tuned on these inputs. 56 of 64 findings from this corpus are recorded as fixed in redact-secret. The other scanners were never tuned against it.
Different jobs. Most inputs fall outside at least one scanner’s rules. A readable span there shows where its rules end, not that it failed.

Other scanners on the same inputs
Scanner	Inputs its rules target	Left readable on those inputs	Left readable everywhere else	Left readable on all inputs	Safe values flagged
flare-redact 1.6.1Runtime library · Published npm packageBuilt to redact secrets and personal data from text at runtime. Run secrets-only here: its personal-data and generic-assignment detectors are off.	365 of 1,00638 of its 81 rules target a credential family	50 of 403 spansredact-secret, same inputs: 20 of 403	564 of 650 spansNo rule of its own targets these	614 of 1,053 spansredact-secret, same inputs: 58 of 1,053	48 of 1,880redact-secret, same inputs: 9 of 1,880
gitleaks 8.30.1Repository scanner · Directory scanBuilt to find secrets in git history, files and directories before they are committed.	517 of 1,00673 of its 222 rules target a credential family	212 of 554 spansredact-secret, same inputs: 54 of 554	298 of 499 spansNo rule of its own targets these	510 of 1,053 spansredact-secret, same inputs: 58 of 1,053	52 of 1,880redact-secret, same inputs: 9 of 1,880
trufflehog 3.97.4Repository scanner · Filesystem scanBuilt to find and verify secrets in repositories and other sources. Verification is off here.	519 of 1,00683 of its 892 rules target a credential family	459 of 555 spansredact-secret, same inputs: 20 of 555	483 of 498 spansNo rule of its own targets these	942 of 1,053 spansredact-secret, same inputs: 58 of 1,053	34 of 1,880redact-secret, same inputs: 9 of 1,880

Quoting these numbers

Don't “redact-secret leaks far fewer secrets than flare-redact.”

Do “On 365 project policy inputs that match flare-redact 1.6.1’s default rules, in a corpus written and used for tuning by the redact-secret team, flare-redact left 50 of 403 secret spans readable.”

Any quote names the input slice, the version, the date, and who wrote the inputs.

Hide