Skip to content

excavate: skip parameters on template placeholders and escaped links - #3656

Merged
liquidsec merged 3 commits into
devfrom
excavate-template-params
Nov 21, 2026
Merged

liquidsec merged 3 commits into
devfrom
excavate-template-params

Conversation

@liquidsec

@liquidsec liquidsec commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

excavate's parameter extractor emitted WEB_PARAMETERs on URLs that aren't links:

  • <img ng-src="{{c.selected_image}}?t=small"> gave a t parameter on /{{c.selected_image}}
  • a JSON-escaped link in the page source, <a href=\&quot;https:\/\/example.com\/x?conf=1\&quot;>, gave a conf parameter on /\&quot;https:\/\/example.com\/x

The URL extractor already rejects both URLs. The parameter extractor resolves the same attribute values on its own and only checked validate_url_parsed, which accepts any path, so every module that consumes WEB_PARAMETER spent requests testing pages that don't exist.

This skips a parameter whose URL path contains { or }, or a backslash escaping a slash, quote or HTML entity (\/, \&quot;). A real link's braces get percent-encoded, and those escapes only show up when the link sits inside a JSON or JS string. A plain backslash is left alone: <a href="images\view.asp?id=1"> is a real link on IIS, which reads \ as /, so its parameter stays on images\view.asp as before. Query strings are left alone too, since they can legitimately hold braces.

TestExcavateParameterExtraction_notalink covers both cases, along with a link escaped with only \/, an escaped jQuery URL and a form action holding a placeholder, and checks that real links on the same page, IIS backslash links among them, still yield their parameters.

@liquidsec liquidsec self-assigned this Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (e0b08a4) to head (bda1e8e).
⚠️ Report is 21 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff           @@
##             dev   #3656    +/-   ##
======================================
+ Coverage     91%     91%    +1%     
======================================
  Files        465     465            
  Lines      52089   52203   +114     
======================================
+ Hits       47191   47298   +107     
- Misses      4898    4905     +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (c339f34c5) (baseline) vs excavate-template-params (75886947c) (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 3.76ms 3.79ms +0.8% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 15.10ms 15.40ms +2.0% ⚪ ✅
Large Closest Match Lookup 333.45ms 334.60ms +0.3% ⚪ ✅
Realistic Closest Match Workload 186.72ms 187.27ms +0.3% ⚪ ✅
Event Memory Medium Scan 1389 B/event 1389 B/event +0.0% ⚪ ✅
Event Memory Large Scan 1514 B/event 1514 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 33.80ms 34.89ms +3.2% ⚪ ✅
Event Validation Full Scan Startup Large Batch 162.36ms 184.43ms +13.6% 🟡🟡 ⚠️
Make Event Autodetection Small 25.99ms 25.86ms -0.5% ⚪ ✅
Make Event Autodetection Large 262.90ms 261.81ms -0.4% ⚪ ✅
Make Event Explicit Types 11.27ms 11.34ms +0.6% ⚪ ✅
Excavate Single Thread Small 3.640s 3.633s -0.2% ⚪ ✅
Excavate Single Thread Large 9.827s 9.997s +1.7% ⚪ ✅
Excavate Parallel Tasks Small 3.845s 3.862s +0.4% ⚪ ✅
Excavate Parallel Tasks Large 6.267s 6.213s -0.9% ⚪ ✅
Intercept Throughput Small 384.31ms 383.70ms -0.2% ⚪ ✅
Intercept Throughput Medium 469.64ms 383.66ms -18.3% 🟢🟢 🚀
Dns Throughput Quiet 2.059s 2.137s +3.8% ⚪ ✅
Dns Throughput Loaded 1.143s 1.125s -1.5% ⚪ ✅
Dns Throughput Inherited 1.837s 1.868s +1.7% ⚪ ✅
Is Ip Performance 2.17ms 2.16ms -0.4% ⚪ ✅
Make Ip Type Performance 240.74µs 232.19µs -3.6% ⚪ ✅
Mixed Ip Operations 2.24ms 2.21ms -1.1% ⚪ ✅
Memory Use Web Crawl 1.1 MB 1.1 MB -1.8% ⚪ ✅
Memory Use Subdomain Enum 27.8 MB 27.8 MB +0.0% ⚪ ✅
Memory Use Deep Chain 5.1 MB 5.1 MB +0.0% ⚪ ✅
Memory Use Parallel Chains 13.3 MB 11.4 MB -13.9% 🟢🟢 🚀
Scan Throughput 100 2.295s 2.458s +7.1% ⚪ ✅
Scan Throughput 1000 20.870s 20.388s -2.3% ⚪ ✅
Typical Queue Shuffle 5.30µs 5.40µs +1.8% ⚪ ✅
Priority Queue Shuffle 27.04µs 26.95µs -0.3% ⚪ ✅

🎯 Performance Summary

+ 2 improvements 🚀
! 1 regression ⚠️
  28 unchanged ✅

🔍 Significant Changes (>10%)

  • Event Validation Full Scan Startup Large Batch: 13.6% 🐌 slower
  • Intercept Throughput Medium: 18.3% 🚀 faster
  • Memory Use Parallel Chains: 13.9% 🚀 less memory

🐍 Python Version 3.12.3

@singlerider singlerider added the bug Something isn't working label Nov 18, 2026
@liquidsec
liquidsec requested a review from a team as a code owner November 19, 2026 01:12
en0f
en0f previously approved these changes Nov 19, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at dca0142.

🟢 Path-only check, with a real link on the same page as the control

The skip runs after URL validation and looks only at the path, so braces in a query string and percent-encoded braces in a path still yield parameters. The test puts a real link next to the two bad ones and asserts its parameter survives, so it can't pass by suppressing everything. At the base, the probe below emits t, conf and tp; at this head, none of them, and all 15 ParameterExtraction tests pass.

Reproduce this review
# save the script below as verify-3656.sh, then (needs docker; bbot runs in a digest-pinned Linux image):
bash verify-3656.sh blacklanternsecurity/bbot dca01429a7ab8ac0d65dd4fb4b7a5094ed414bf2
# expected: SUMMARY 8 PASS, 0 FAIL, 0 SKIP, 0 BROKEN
# control:  bash verify-3656.sh blacklanternsecurity/bbot e0b08a40a32e6a46208f490cb81cdf89aab9eaf8   -> V01 V02 V03 V08 FAIL
#!/usr/bin/env bash
# verify-3656.sh: reproducible checks for blacklanternsecurity/bbot#3656
# Usage: bash verify-3656.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# CONTROL: sha e0b08a40a32e6a46208f490cb81cdf89aab9eaf8
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V08=FAIL
# AUTHOR-CODE: container ghcr.io/astral-sh/uv:python3.13-bookworm-slim@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca
# bbot runs only inside that digest-pinned Linux image (its blasthttp dependency ships Linux wheels only),
# with the clone mounted and no host environment or credentials passed in.
# V01-V06 come from a probe test this script writes itself: one page served by bbot's test HTTP server,
# scanned with http + excavate + hunt, and the WEB_PARAMETER events it yields.
set -uo pipefail

REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass()   { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail()   { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip()   { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }

echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "docker $(docker version --format '{{.Server.Version}}' 2>/dev/null || echo none)"

WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
for i in 1 2 3; do rm -rf "$WORK/src"; "${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" && break; sleep 5; done
[ -d "$WORK/src/.git" ] || { broken V00 "clone" "clone failed (3 attempts)"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
T=bbot/test/test_step_2/module_tests
[ -f bbot/modules/internal/excavate.py ] && [ -f "$T/test_module_excavate.py" ] || { broken V00 "excavate and its test exist" "missing"; summary; exit 1; }

# ---- the probe: written by this script, identical at head and control ----------
cat > "$T/test_zz_probe_3656.py" <<'PROBE'
import json
from pathlib import Path

from .base import ModuleTestBase
from bbot.test.worker import HTTPSERVER_URL

OUT = Path(__file__).resolve().parents[4] / ".probe3656.json"


class TestExcavateProbe3656(ModuleTestBase):
    targets = [f"{HTTPSERVER_URL}/"]
    module_name = "excavate"
    modules_overrides = ["http", "excavate", "hunt"]
    html = r"""
<html>
<img ng-src="{{c.selected_image}}?t=small">
<a href=\&quot;https:\/\/www.example.com\/webcast\/register.aspx?conf=1\&quot;>webcast</a>
<a href="/{{tpl}}/x.php?tp=1">mid-path placeholder</a>
<a href="/real.php?keep=1">real</a>
<a href="/brace.php?q={x}&keep2=1">braces in the query only</a>
<a href="/%7Benc%7D.php?enc=1">percent-encoded braces</a>
</html>
"""

    async def setup_after_prep(self, module_test):
        module_test.set_expect_requests(
            respond_args={"response_data": self.html, "headers": {"Content-Type": "text/html"}}
        )

    def check(self, module_test, events):
        OUT.write_text(
            json.dumps(
                {
                    "params": {e.data["name"]: e.data["url"] for e in events if e.type == "WEB_PARAMETER"},
                }
            )
        )
PROBE

IMG='ghcr.io/astral-sh/uv:python3.13-bookworm-slim@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca'
if ! command -v docker >/dev/null || ! docker info >/dev/null 2>&1; then
  for v in V01 V02 V03 V04 V05 V06 V08 V09; do skip $v "container check" "docker not available"; done
  summary; exit $?
fi

docker run --rm --network bridge -v "$WORK/src:/src" -w /src "$IMG" bash -c '
  export UV_CACHE_DIR=/tmp/uvcache
  uv sync --frozen --group dev --quiet >/tmp/sync.log 2>&1 || { echo "SYNC_FAILED $(tail -1 /tmp/sync.log)"; exit 0; }
  uv run --no-sync pytest -q -p no:cacheprovider bbot/test/test_step_2/module_tests/test_zz_probe_3656.py > /src/.probe.log 2>&1; echo "PROBE_RC $?"
  uv run --no-sync pytest -q -p no:cacheprovider --junitxml=/src/.pe.xml bbot/test/test_step_2/module_tests/test_module_excavate.py -k ParameterExtraction > /src/.pe.log 2>&1; echo "PE_RC $?"
' > "$WORK/container.out" 2>&1

if grep -q '^SYNC_FAILED' "$WORK/container.out"; then
  for v in V01 V02 V03 V04 V05 V06 V08 V09; do broken $v "container check" "$(grep SYNC_FAILED "$WORK/container.out" | cut -c1-150)"; done
  summary; exit $?
fi
sed 's/^/# container: /' "$WORK/container.out"
tail -3 .probe.log 2>/dev/null | sed 's/^/# probe pytest: /'
tail -3 .pe.log 2>/dev/null | sed 's/^/# excavate -k ParameterExtraction: /'

# helper: q <python expression over P (params dict) and U (url list)> -> prints result
q() { python3 -c "import json,sys; d=json.load(open('.probe3656.json')); P=d['params']; print($1)"; }
if [ ! -s .probe3656.json ]; then
  for v in V01 V02 V03 V04 V05 V06; do broken $v "probe ran" "no probe output (probe rc $(grep -o 'PROBE_RC [0-9]*' "$WORK/container.out"))"; done
else
  echo "# probe params: $(q 'json.dumps(P, sort_keys=True)')"
  if [ "$(q '"t" in P')" = False ]; then pass V01 "a template placeholder src ({{...}}?t=) yields no WEB_PARAMETER"
  else fail V01 "a template placeholder src ({{...}}?t=) yields no WEB_PARAMETER" "t on $(q 'P["t"]')"; fi
  if [ "$(q '"conf" in P')" = False ]; then pass V02 "a JSON-escaped link (\\&quot;https:\\/\\/...) yields no WEB_PARAMETER"
  else fail V02 "a JSON-escaped link (\\&quot;https:\\/\\/...) yields no WEB_PARAMETER" "conf on $(q 'P["conf"]')"; fi
  if [ "$(q '"tp" in P')" = False ]; then pass V03 "a placeholder mid-path (/{{tpl}}/x.php?tp=) yields no WEB_PARAMETER"
  else fail V03 "a placeholder mid-path (/{{tpl}}/x.php?tp=) yields no WEB_PARAMETER" "tp on $(q 'P["tp"]')"; fi
  if [ "$(q 'P.get("keep","").endswith("/real.php")')" = True ]; then pass V04 "a real link on the same page still yields its parameter (keep on /real.php)"
  else fail V04 "a real link on the same page still yields its parameter" "keep -> $(q 'P.get("keep")')"; fi
  if [ "$(q 'P.get("keep2","").endswith("/brace.php")')" = True ]; then pass V05 "braces in the query string alone do not suppress the link's parameters (keep2 on /brace.php)"
  else fail V05 "braces in the query string alone do not suppress the link's parameters" "keep2 -> $(q 'P.get("keep2")')"; fi
  if [ "$(q '"enc" in P')" = True ]; then pass V06 "percent-encoded braces in the path (%7B..%7D) still yield the parameter"
  else fail V06 "percent-encoded braces in the path (%7B..%7D) still yield the parameter" "enc absent"; fi
fi

# V08: the PR's own regression test exists and passes; V09: no ParameterExtraction test regressed
pe=$(python3 - <<'PY' 2>/dev/null
import xml.etree.ElementTree as E
r = E.parse(".pe.xml").getroot(); s = r if r.tag == "testsuite" else r.find("testsuite")
cases = {"%s::%s" % (c.get("classname"), c.get("name")): (c.find("failure") is None and c.find("error") is None and c.find("skipped") is None) for c in s.iter("testcase")}
nl = [n for n in cases if "notalink" in n]
print("notalink=%s total=%d failed=%d" % ("pass" if nl and all(cases[n] for n in nl) else ("fail" if nl else "absent"), len(cases), sum(1 for v in cases.values() if not v)))
PY
)
if [ -z "$pe" ]; then broken V08 "PR test TestExcavateParameterExtraction_notalink passes" "no junit output"; broken V09 "every excavate ParameterExtraction test passes" "no junit output"
else
  case "$pe" in notalink=pass*) pass V08 "PR test TestExcavateParameterExtraction_notalink passes ($pe)";; *) fail V08 "PR test TestExcavateParameterExtraction_notalink passes" "$pe";; esac
  case "$pe" in *failed=0) pass V09 "every excavate ParameterExtraction test passes ($pe)";; *) fail V09 "every excavate ParameterExtraction test passes" "$pe";; esac
fi

summary

@liquidsec liquidsec changed the title excavate: skip parameters on template placeholders and escaped links excavate: resolve links the way a browser does Nov 19, 2026
@liquidsec
liquidsec force-pushed the excavate-template-params branch from c5c07b9 to bda1e8e Compare November 20, 2026 22:27
@liquidsec liquidsec changed the title excavate: resolve links the way a browser does excavate: skip parameters on template placeholders and escaped links Nov 20, 2026
@liquidsec
liquidsec merged commit d8cb12e into dev Nov 21, 2026
27 checks passed
@liquidsec
liquidsec deleted the excavate-template-params branch November 21, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants