Skip to content
Benchmarks

redact-secret · Report · Detector

W&B API keys

  • 35 fixtures
  • Format evidence: T1 · Provider-documented
  • Run 2026-10-07
  • Mode published · redact-secret 0.1.0-beta.14

Format evidence

Arrival evidence (#436, product redact-secret#917; #860 handoff wandb.md, READY for wandb_v1_). T1: the prefix is a W&B-authored test constant (R5), the [A-Za-z0-9_] alphabet is the SDK validator and its error text (R1), and the docs say new keys are "about 86 characters", which every provider test fixture of a new key (wandb#10688, Weave #5732) makes exactly 86. By the orchestrator decision on redact-secret#917 the product matches a bounded tolerant range around 86 until an issuance check, so positives carry totals 85, 86 and 87 and no fixture asserts silence on any length. The 27/_/49 split is scanner-only (T2) and not required: positives carry it and a body without an inner _. On-prem keys are <host>-<key>; the host label is public and is authored as an envelope around the key, so a finding with or without it passes. Excluded: the legacy 40-hex key (generic context), client JWTs (jwt), OIDC identity-token files, wandb_v1_ placeholders, and a future wandb_v2_ (unknown, so never asserted). Graduated to a registry detector at the 1127bf9 re-pin (redact-secret PR #938).

What the run recorded, by group

Detector views overlap, so their groups are never summed across detectors. Other scanners are reference values on the same inputs, in run order.
Groups of W&B API keys fixtures
GroupFixturesLeaked or false alarmsNear-twinsOutcomesOther scanners, same cell
Must not flag · Tool-corroborated11False alarmsat most 25.9%0 of 11 controls flagged11 quiet · 0 flagged
  • flare-redact at most 25.9%
  • gitleaks at most 25.9%
  • trufflehog at most 25.9%
Must not flag · Project policy8False alarmsat most 32.4%0 of 8 controls flagged8 quiet · 0 flagged
  • flare-redact at most 32.4%
  • gitleaks at most 32.4%
  • trufflehog at most 32.4%
Must redact · Provider-documented16Secret spans left readableat most 19.4%0 of 16 spansNear-twins told apartinsufficient-coverage6 of 6 pairs16 redacted · 0 too much · 0 partly exposed · 0 missed
  • flare-redact at most 98.9%
  • gitleaks at most 72.0%
  • trufflehog at most 61.4%
35 of 35 rows
Fixtures
35
Left readable
0
Redacted too much
0
False alarms
0

Fixtures for this detector

35 fixtures. Choose "Every scanner" to see each scanner's outcome for the same rows.
Fixtures in W&B API keys
FixtureKind and evidenceredact-secret
wandb-api-key-bare-prosewandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-bearer-headerwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-chat-pastewandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-dotenvwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-exportwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-json-api-keywandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-json-tokenwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-key-shape-barewandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-key-shape-quotedwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-key-shape-unicode-crlfwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-mcp-envwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-netrc-passwordwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-on-prem-host-labelwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-sdk-kwargwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-wandb-loginwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-x-api-key-headerwandb · documented-format-literalMust redactT1 · Provider-documentedRedacted
wandb-api-key-actions-secret-referencewandb · templated-referenceMust not flagT3 · Project policyQuiet
wandb-api-key-ellipsis-placeholderwandb · documentation-placeholderMust not flagT3 · Project policyQuiet
wandb-api-key-entity-and-project-public-idwandb · public-identifierMust not flagT2 · Tool-corroboratedQuiet
wandb-api-key-env-reference-referencewandb · templated-referenceMust not flagT3 · Project policyQuiet
wandb-api-key-git-sha-encoded-valuewandb · benign-encoded-valueMust not flagT2 · Tool-corroboratedQuiet
wandb-api-key-hyphen-in-body-twinwandb · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-hyphen-prefix-twinwandb · boundary-violationMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-key-guidance-prosewandb · prose-mentionMust not flagT3 · Project policyQuiet
wandb-api-key-label-prosewandb · benign-lookalikeMust not flagT3 · Project policyQuiet
wandb-api-key-leading-glue-twinwandb · boundary-violationMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-leading-underscore-twinwandb · boundary-violationMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-maskwandb · benign-lookalikeMust not flagT3 · Project policyQuiet
wandb-api-key-prefix-onlywandb · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
wandb-api-key-referencewandb · benign-lookalikeMust not flagT3 · Project policyQuiet
wandb-api-key-short-bodywandb · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
wandb-api-key-test-constant-shape-placeholderwandb · documentation-placeholderMust not flagT3 · Project policyQuiet
wandb-api-key-trailing-hyphen-twinwandb · boundary-violationMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-uppercase-prefix-twinwandb · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
wandb-api-key-version-identifiers-public-idwandb · public-identifierMust not flagT2 · Tool-corroboratedQuiet