Skip to content
Benchmarks

redact-secret · Report · Detector

LangSmith API keys

  • 42 fixtures
  • Format evidence: T2 · Tool-corroborated
  • Run 2026-10-07
  • Mode published · redact-secret 0.1.0-beta.14

Format evidence

Graduated arrival contract (#210, registry detector since redact-secret#728; research #219 observed 2026-09-24). LangSmith's own documentation establishes two API-key roles — Personal Access Tokens and workspace/org service keys — their issuance under Settings → API Keys, one-time display and the LANGSMITH_API_KEY / X-API-Key contexts, but states no prefix, length or alphabet. The docs render only masked lsv2_pt_…/lsv2_sk_… forms. The segment grammar lsv2_(pt|sk)_<32 lowercase hex>_<10 lowercase hex> is tool-corroborated: trufflehog 3.97.4's langsmith detector matches exactly that (lowercase only), while Titus/Kingfisher and Poltergeist match the same widths case-insensitively. The provider's own SDK redactor (provider code) is looser — [A-Za-z0-9]{32,} plus any number of _ tails — and is a redaction heuristic, not a format spec. Uppercase hex is therefore neither claimed nor twinned. The legacy ls__ form, LangSmith license keys, SCIM bearer tokens, OAuth access/refresh tokens, the internal X-Service-Key JWT and the uncorroborated deployment keys (sk-*/dep-srv-*) are separate credentials this contract does not claim; none is used as a benign control. gitleaks 8.30.1 has no LangSmith rule, so a gitleaks miss is the expected peer result.

What the run recorded, by group

Detector views overlap, so their groups are never summed across detectors. Other scanners are reference values on the same inputs, in run order.
Groups of LangSmith API keys fixtures
GroupFixturesLeaked or false alarmsNear-twinsOutcomesOther scanners, same cell
Must not flag · Tool-corroborated15False alarmsat most 20.4%0 of 15 controls flagged15 quiet · 0 flagged
  • flare-redact at most 20.4%
  • gitleaks at most 29.8%
  • trufflehog at most 20.4%
Must not flag · Project policy9False alarmsat most 29.9%0 of 9 controls flagged9 quiet · 0 flagged
  • flare-redact at most 29.9%
  • gitleaks at most 29.9%
  • trufflehog at most 29.9%
Must redact · Tool-corroborated18Secret spans left readableat most 17.6%0 of 18 spansNear-twins told apartinsufficient-coverage8 of 8 pairs18 redacted · 0 too much · 0 partly exposed · 0 missed
  • flare-redact at most 100.0%
  • gitleaks at most 61.4%
  • trufflehog at most 17.6%
42 of 42 rows
Fixtures
42
Left readable
0
Redacted too much
0
False alarms
0

Fixtures for this detector

42 fixtures. Choose "Every scanner" to see each scanner's outcome for the same rows.
Fixtures in LangSmith API keys
FixtureKind and evidenceredact-secret
langsmith-api-key-export-patlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-langgraph-clilangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-op-readlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-dotenvlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-nested-run-metadatalangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-otlp-header-listlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-profile-jsonlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-shape-barelangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-shape-quotedlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-pat-shape-unicode-crlflangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-runbooklangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-service-key-legacy-exportlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-service-key-python-clientlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-service-key-workflow-envlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-service-key-x-api-keylangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-short-tail-twinlangsmith · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-twin-base-cilangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-twin-base-dotenvlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-twin-base-jsonlangsmith · documented-format-literalMust redactT2 · Tool-corroboratedRedacted
langsmith-api-key-env-example-placeholderlangsmith · documentation-placeholderMust not flagT3 · Project policyQuiet
langsmith-api-key-key-roles-doc-proselangsmith · prose-mentionMust not flagT3 · Project policyQuiet
langsmith-api-key-label-proselangsmith · benign-lookalikeMust not flagT3 · Project policyQuiet
langsmith-api-key-legacy-readme-placeholderlangsmith · documentation-placeholderMust not flagT3 · Project policyQuiet
langsmith-api-key-long-tail-twinlangsmith · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-masklangsmith · benign-lookalikeMust not flagT3 · Project policyQuiet
langsmith-api-key-masked-short-key-placeholderlangsmith · documentation-placeholderMust not flagT3 · Project policyQuiet
langsmith-api-key-merged-segments-twinlangsmith · boundary-violationMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-missing-tail-near-misslangsmith · format-near-missMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-non-hex-body-twinlangsmith · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-non-hex-tail-twinlangsmith · wrong-alphabetMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-prefix-onlylangsmith · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-prefix-only-grep-near-misslangsmith · format-near-missMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-python-env-lookup-referencelangsmith · templated-referenceMust not flagT3 · Project policyQuiet
langsmith-api-key-referencelangsmith · benign-lookalikeMust not flagT3 · Project policyQuiet
langsmith-api-key-requirements-hash-encoded-valuelangsmith · benign-encoded-valueMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-role-code-twinlangsmith · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-short-bodylangsmith · benign-lookalikeMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-short-first-segment-twinlangsmith · wrong-lengthMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-trace-share-link-public-idlangsmith · public-identifierMust not flagT2 · Tool-corroboratedQuiet
langsmith-api-key-unknown-role-code-twinlangsmith · prefix-near-missMust not flagT2 · Tool-corroborated · twinQuiet
langsmith-api-key-workflow-secret-ref-referencelangsmith · templated-referenceMust not flagT3 · Project policyQuiet
langsmith-api-key-workspace-ids-env-public-idlangsmith · public-identifierMust not flagT2 · Tool-corroboratedQuiet