Skip to content

Update trufflehog to 3.100.0 - #3670

Merged
liquidsec merged 1 commit into
devfrom
update-trufflehog
Nov 20, 2026
Merged

liquidsec merged 1 commit into
devfrom
update-trufflehog

Conversation

@blsaccess

Copy link
Copy Markdown
Contributor

This PR uses https://api.github.com/repos/trufflesecurity/trufflehog/releases/latest to obtain the latest version of trufflehog and update the version in bbot/modules/trufflehog.py.

Release notes:

What's Changed

New Contributors

Full Changelog: trufflesecurity/trufflehog@v3.99.9...v3.100.0

@blsaccess
blsaccess requested a review from a team as a code owner November 20, 2026 00:20
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (1ead7cc) to head (b2a9839).
⚠️ Report is 2 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3670   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        465     465           
  Lines      52188   52188           
=====================================
+ Hits       47281   47282    +1     
+ Misses      4907    4906    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (1ead7cc26) (baseline) vs update-trufflehog (5ef3da11c) (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 2.26ms 2.24ms -1.0% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 9.67ms 9.45ms -2.2% ⚪ ✅
Large Closest Match Lookup 214.94ms 213.20ms -0.8% ⚪ ✅
Realistic Closest Match Workload 118.60ms 117.67ms -0.8% ⚪ ✅
Event Memory Medium Scan 1389 B/event 1389 B/event +0.0% ⚪ ✅
Event Memory Large Scan 1514 B/event 1514 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 20.87ms 20.73ms -0.7% ⚪ ✅
Event Validation Full Scan Startup Large Batch 108.84ms 103.13ms -5.2% ⚪ ✅
Make Event Autodetection Small 16.21ms 15.99ms -1.3% ⚪ ✅
Make Event Autodetection Large 163.84ms 161.10ms -1.7% ⚪ ✅
Make Event Explicit Types 6.88ms 6.80ms -1.1% ⚪ ✅
Excavate Single Thread Small 2.465s 2.530s +2.6% ⚪ ✅
Excavate Single Thread Large 6.530s 6.560s +0.5% ⚪ ✅
Excavate Parallel Tasks Small 2.671s 2.654s -0.6% ⚪ ✅
Excavate Parallel Tasks Large 4.308s 4.265s -1.0% ⚪ ✅
Intercept Throughput Small 352.71ms 352.90ms +0.1% ⚪ ✅
Intercept Throughput Medium 349.92ms 350.95ms +0.3% ⚪ ✅
Dns Throughput Quiet 1.202s 1.215s +1.1% ⚪ ✅
Dns Throughput Loaded 706.12ms 673.68ms -4.6% ⚪ ✅
Dns Throughput Inherited 999.65ms 1.012s +1.3% ⚪ ✅
Is Ip Performance 1.38ms 1.42ms +3.2% ⚪ ✅
Make Ip Type Performance 115.14µs 114.24µs -0.8% ⚪ ✅
Mixed Ip Operations 1.47ms 1.49ms +1.4% ⚪ ✅
Memory Use Web Crawl 1.1 MB 1.1 MB +0.9% ⚪ ✅
Memory Use Subdomain Enum 27.8 MB 27.8 MB +0.0% ⚪ ✅
Memory Use Deep Chain 5.1 MB 5.1 MB -0.2% ⚪ ✅
Memory Use Parallel Chains 16.4 MB 11.7 MB -28.7% 🟢🟢🟢 🚀
Scan Throughput 100 1.635s 1.682s +2.9% ⚪ ✅
Scan Throughput 1000 12.400s 12.598s +1.6% ⚪ ✅
Typical Queue Shuffle 3.64µs 3.66µs +0.3% ⚪ ✅
Priority Queue Shuffle 17.50µs 17.52µs +0.1% ⚪ ✅

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Memory Use Parallel Chains: 28.7% 🚀 less memory

🐍 Python Version 3.12.3

@singlerider singlerider added the enhancement New feature or request label Nov 20, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b2a9839.

The 3.100.0 release publishes all four tarballs bbot installs, their digests match the vendor checksums, every flag bbot passes still exists at the tag, and all 8 trufflehog module tests passed against the freshly downloaded binary in all five Python CI jobs with no reruns.

Reproduce this review
# save the script below as verify-3670.sh, then:
bash verify-3670.sh blacklanternsecurity/bbot b2a98396b39b48357cbb220559251521c4beb396
# expected: SUMMARY 7 PASS, 0 FAIL, 0 SKIP, 0 BROKEN
# control:  bash verify-3670.sh blacklanternsecurity/bbot b2a98396b39b48357cbb220559251521c4beb396 "$PWD/control-3670.patch"
#           -> V02 V03 V04 V06 V07 FAIL (version pointed at an unpublished release)
#!/usr/bin/env bash
# verify-3670.sh: reproducible checks for blacklanternsecurity/bbot#3670 (trufflehog 3.99.9 -> 3.100.0)
# Usage: bash verify-3670.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
# Runs no project code and no downloaded binary: the binary is exercised by CI (V05).
#
# CONTROL: patch control-3670.patch
# CONTROL-EXPECT: V02=FAIL V03=FAIL V04=FAIL V06=FAIL V07=FAIL
# AUTHOR-CODE: none
set -uo pipefail

REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass()   { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail()   { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip()   { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }

echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "curl $(curl --version | head -1 | cut -d' ' -f2)"
echo "python3 $(python3 -c 'import platform;print(platform.python_version())')"

WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
export GIT_CONFIG_COUNT=2 GIT_CONFIG_KEY_0=credential.helper GIT_CONFIG_VALUE_0= GIT_CONFIG_KEY_1=credential.helper GIT_CONFIG_VALUE_1='!gh auth git-credential'
GIT=(git)
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi

MOD=bbot/modules/trufflehog.py
# Same extraction version-updater.yml uses, read from the working tree (so a control patch is seen).
VER="$(sed -nE 's/^[[:space:]]+version: VersionStr = Field\("([^"]*)".*/\1/p' "$MOD" | head -1)"
echo "trufflehog_version_in_tree $VER"
[ -n "$VER" ] || { broken V00 "read module version" "no version line in $MOD"; summary; exit 1; }

# V01: the default passes bbot's own VersionStr validator (bbot/core/config/models.py _VERSION_RE)
RE="$(sed -nE 's/^_VERSION_RE = re\.compile\(r"(.*)"\)$/\1/p' bbot/core/config/models.py)"
if [ -z "$RE" ]; then broken V01 "version passes VersionStr validator" "_VERSION_RE not found"
elif python3 -c 'import re,sys; sys.exit(0 if re.match(sys.argv[1], sys.argv[2]) else 1)' "$RE" "$VER"; then pass V01 "version '$VER' passes VersionStr validator"
else fail V01 "version passes VersionStr validator" "'$VER' rejected by $RE"; fi

# V02: v<version> is a published, non-draft, non-prerelease trufflehog release
REL="$WORK/release.json"
if gh api --method GET "repos/trufflesecurity/trufflehog/releases/tags/v$VER" > "$REL" 2>/dev/null; then
  st="$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(r["draft"], r["prerelease"], r["published_at"])' "$REL")"
  case "$st" in "False False "*) pass V02 "v$VER is a published stable release (${st##* })";; *) fail V02 "v$VER is a published stable release" "draft/prerelease/published: $st";; esac
else fail V02 "v$VER is a published stable release" "no release with tag v$VER"; fi

# V03: the vendor's checksums file lists every tarball bbot's deps_ansible URL builds
#      (trufflehog_<ver>_<BBOT_OS_PLATFORM>_<BBOT_CPU_ARCH_GOLANG>.tar.gz for linux/darwin x amd64/arm64)
SUMS="$WORK/sums.txt"
if curl -fsSL -o "$SUMS" "https://github.com/trufflesecurity/trufflehog/releases/download/v$VER/trufflehog_${VER}_checksums.txt" 2>/dev/null; then
  missing=""
  for p in linux_amd64 linux_arm64 darwin_amd64 darwin_arm64; do
    n="$(awk -v f="trufflehog_${VER}_${p}.tar.gz" '$2==f && $1 ~ /^[0-9a-f]{64}$/' "$SUMS" | wc -l | tr -d ' ')"
    [ "$n" = "1" ] || missing="$missing $p"
  done
  if [ -z "$missing" ]; then pass V03 "checksums file lists all four platform tarballs for $VER"; else fail V03 "checksums file lists all four platform tarballs" "missing:$missing"; fi
else fail V03 "checksums file lists all four platform tarballs" "trufflehog_${VER}_checksums.txt not downloadable"; fi

# V04: each tarball URL exactly as deps_ansible builds it resolves (HEAD only, nothing executed)
TPL="$(sed -nE 's/^[[:space:]]+"src": "(https:\/\/github\.com\/trufflesecurity\/[^"]+)",$/\1/p' "$MOD" | head -1)"
if [ -z "$TPL" ]; then broken V04 "deps_ansible URL resolves" "src template not found"
else
  bad=""
  for p in linux_amd64 linux_arm64 darwin_amd64 darwin_arm64; do
    os="${p%_*}"; ar="${p#*_}"
    url="${TPL//\#\{BBOT_MODULES_TRUFFLEHOG_VERSION\}/$VER}"; url="${url//\#\{BBOT_OS_PLATFORM\}/$os}"; url="${url//\#\{BBOT_CPU_ARCH_GOLANG\}/$ar}"
    code="$(curl -sIL -o /dev/null -w '%{http_code}' "$url")"
    [ "$code" = "200" ] || bad="$bad $p=$code"
  done
  if [ -z "$bad" ]; then pass V04 "deps_ansible download URL resolves for all four platforms at $VER"; else fail V04 "deps_ansible download URL resolves for all four platforms" "$bad"; fi
fi

# V05: CI at this SHA ran the trufflehog module tests against the real binary, all PASSED, in every Python job
JQ='.check_runs[] | select((.name | startswith("test / test (")) and .head_sha == "'"$SHA"'") | "\(.id):\(.conclusion):\(.name | gsub(" "; "_"))"'
if ids="$(gh api --method GET "repos/$REPO/commits/$SHA/check-runs" -F per_page=100 --jq "$JQ" 2>&1)"; then
  if [ -z "$ids" ]; then fail V05 "trufflehog module tests passed in CI at $SHA" "no 'test / test (...)' check runs at this SHA"
  else
    bad=""; jobs=0
    for e in $ids; do
      id="${e%%:*}"; rest="${e#*:}"; concl="${rest%%:*}"; name="${rest#*:}"; jobs=$((jobs+1))
      log="$WORK/job-$id.log"
      gh api --method GET "repos/$REPO/actions/jobs/$id/logs" > "$log" 2>/dev/null || { bad="$bad $name=nolog"; continue; }
      np="$(grep -cE 'PASSED bbot/test/test_step_2/module_tests/test_module_trufflehog\.py::' "$log")"
      nr="$(grep -cE 'RERUN bbot/test/test_step_2/module_tests/test_module_trufflehog\.py::' "$log")"
      nf="$(grep -cE '(FAILED|ERROR) bbot/test/test_step_2/module_tests/test_module_trufflehog\.py::' "$log")"
      { [ "$concl" = "success" ] && [ "$np" -ge 8 ] && [ "$nf" -eq 0 ] && [ "$nr" -eq 0 ]; } || bad="$bad $name:$concl:passed=$np:failed=$nf:rerun=$nr"
    done
    if [ -z "$bad" ]; then pass V05 "8+ trufflehog module tests PASSED, none failed or rerun, in all $jobs CI Python jobs at $SHA"; else fail V05 "trufflehog module tests passed in CI at $SHA" "$bad"; fi
  fi
else broken V05 "trufflehog module tests passed in CI at $SHA" "check-runs read or jq filter failed: ${ids:0:200}"; fi

# V06: trufflehog's CLI at tag v<version> still defines every subcommand and flag bbot's execute_trufflehog passes
#      (read from the vendor's source, not by running a binary)
MAIN="$WORK/main.go"
if gh api --method GET "repos/trufflesecurity/trufflehog/contents/main.go?ref=v$VER" -H "Accept: application/vnd.github.raw+json" > "$MAIN" 2>/dev/null; then
  miss=""
  for f in json local-dev no-update only-verified config concurrency; do grep -qE "cli\.Flag\(\"$f\"" "$MAIN" || miss="$miss --$f"; done
  for c in git docker postman filesystem stdin github-experimental; do grep -qE "cli\.Command\(\"$c\"" "$MAIN" || miss="$miss cmd:$c"; done
  for f in repo object-discovery delete-cached-data token; do grep -qE "githubExperimentalScan\.Flag\(\"$f\"" "$MAIN" || miss="$miss github-experimental--$f"; done
  grep -qE 'dockerScan\.Flag\("image"' "$MAIN" || miss="$miss docker--image"
  grep -qE 'postmanScan\.Flag\("workspace-paths"' "$MAIN" || miss="$miss postman--workspace-paths"
  if [ -z "$miss" ]; then pass V06 "v$VER main.go defines every subcommand and flag bbot passes"; else fail V06 "v$VER main.go defines every subcommand and flag bbot passes" "missing:$miss"; fi
else fail V06 "v$VER main.go defines every subcommand and flag bbot passes" "main.go not readable at tag v$VER"; fi

# V07: GitHub's own per-asset sha256 digest equals the vendor checksums file for every platform tarball bbot installs
if [ -s "$REL" ] && [ -s "$SUMS" ]; then
  bad="$(python3 -I -c '
import json,sys
r=json.load(open(sys.argv[1])); ver=sys.argv[3]
d={a["name"]:(a.get("digest") or "").split(":",1)[-1] for a in r["assets"]}
s={l.split()[1]:l.split()[0] for l in open(sys.argv[2]) if len(l.split())==2}
out=[]
for p in ("linux_amd64","linux_arm64","darwin_amd64","darwin_arm64"):
    n="trufflehog_%s_%s.tar.gz"%(ver,p)
    if not d.get(n) or d.get(n)!=s.get(n): out.append(n)
print(" ".join(out))' "$REL" "$SUMS" "$VER" 2>&1)" || bad="python-error: $bad"
  if [ -z "$bad" ]; then pass V07 "GitHub asset digests match trufflehog_${VER}_checksums.txt for all four platforms"; else fail V07 "GitHub asset digests match the checksums file" "$bad"; fi
else fail V07 "GitHub asset digests match the checksums file" "release or checksums file unavailable for $VER"; fi

summary
diff --git a/bbot/modules/trufflehog.py b/bbot/modules/trufflehog.py
--- a/bbot/modules/trufflehog.py
+++ b/bbot/modules/trufflehog.py
@@ -15,7 +15,7 @@ class trufflehog(BaseModule):
     }
 
     class Config(BaseModuleConfig):
-        version: VersionStr = Field("3.100.0", description="trufflehog version")
+        version: VersionStr = Field("0.0.0-pr3670-control", description="trufflehog version")
         config: str = Field("", description="File path or URL to YAML trufflehog config")
         only_verified: bool = Field(True, description="Only report credentials that have been verified")
         concurrency: int = Field(8, description="Number of concurrent workers")

@liquidsec
liquidsec merged commit 186b188 into dev Nov 20, 2026
27 checks passed
@liquidsec
liquidsec deleted the update-trufflehog branch November 20, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants