Skip to content
Benchmarks

redact-secret · Report · Detector

PyPI tokens

  • 26 fixtures
  • Format evidence: T1 · Provider-documented
  • Run 2026-10-07
  • Mode published · redact-secret 0.1.0-beta.14

Format evidence

  • Provider documentation pypi- prefix, base64-serialized PyMacaroon, {85,} length floor · observed 2026-09-21 · docs.pypi.org's own detection-format page publishes this contract's exact regex, pypi-[A-Za-z0-9-_]{85,}: the pypi- prefix, a "-" separator, a base64 PyMacaroon serialization, an 85-character floor and no ceiling
  • trufflehog 3.97.4
  • gitleaks 8.30.1
  • Reference 1

Re-checked 2026-09-21 (#104/#107, docs/decisions/2026-09-21-author-pypi-macaroon-positives-synthetically.md): the prior review here read "no lexical length contract" from pypi.org/help alone; docs.pypi.org/api/secrets instead publishes the exact regex now used as this contract's pattern, and #104 verified it is also what the pinned scanners and this project's own product key on. A positive is additionally constructed as a well-formed libmacaroons v2 body (VERSION, LOCATION("pypi.org"), IDENTIFIER, one caveat, SIGNATURE) per the linked decision record, not merely an arbitrary {85,}-byte run.

What the run recorded, by group

Detector views overlap, so their groups are never summed across detectors. Other scanners are reference values on the same inputs, in run order.
Groups of PyPI tokens fixtures
GroupFixturesLeaked or false alarmsNear-twinsOutcomesOther scanners, same cell
Must not flag · Tool-corroborated13False alarmsat most 22.8%0 of 13 controls flagged13 quiet · 0 flagged
  • flare-redact at most 22.8%
  • gitleaks at most 57.6%
  • trufflehog at most 22.8%
Must not flag · Project policy4False alarmsinsufficient-evidence0 of 4 controls flagged4 quiet · 0 flagged
  • flare-redact insufficient-evidence
  • gitleaks insufficient-evidence
  • trufflehog insufficient-evidence
Must redact · Provider-documented9Secret spans left readableat most 29.9%0 of 9 spansNear-twins told apartat least 70.1%9 of 9 pairs9 redacted · 0 too much · 0 partly exposed · 0 missed
  • flare-redact at most 100.0%
  • gitleaks at most 29.9%
  • trufflehog at most 29.9%
26 of 26 rows
Fixtures
26
Left readable
0
Redacted too much
0
False alarms
0

Fixtures for this detector

26 fixtures. Choose "Every scanner" to see each scanner's outcome for the same rows.
Fixtures in PyPI tokens
FixtureKind and evidenceredact-secret
pypi-token-gitlab-ci-variablespypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-noxfile-publishpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-public-idpypi · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
pypi-token-pypircpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-shape-1-barepypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-shape-1-length-bare-twinpypi · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-length-quoted-twinpypi · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-length-unicode-crlf-twinpypi · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-quotedpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-shape-1-unicode-crlfpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-twine-envpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-twine-upload-flagpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-uv-publish-tokenpypi · documented-format-literalMust redactT1 · Provider-documentedRedacted
pypi-token-encoded-valuepypi · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
pypi-token-label-prosepypi · benign-lookalikeMust not flagT3 · Project policyQuiet
pypi-token-maskpypi · benign-lookalikeMust not flagT3 · Project policyQuiet
pypi-token-ordinary-prosepypi · benign-lookalikeMust not flagT3 · Project policyQuiet
pypi-token-prefix-onlypypi · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
pypi-token-referencepypi · benign-lookalikeMust not flagT3 · Project policyQuiet
pypi-token-shape-1-alphabet-bare-twinpypi · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-alphabet-quoted-twinpypi · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-alphabet-unicode-crlf-twinpypi · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-bare-twinpypi · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-quoted-twinpypi · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-shape-1-unicode-crlf-twinpypi · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
pypi-token-short-bodypypi · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet