Repository navigation
excavate: skip parameters on template placeholders and escaped links - #3656
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #3656 +/- ##
======================================
+ Coverage 91% 91% +1%
======================================
Files 465 465
Lines 52089 52203 +114
======================================
+ Hits 47191 47298 +107
- Misses 4898 4905 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
📊 Performance Benchmark Report
📈 Detailed Results (All Benchmarks)
🎯 Performance Summary+ 2 improvements 🚀
! 1 regression ⚠️
28 unchanged ✅🔍 Significant Changes (>10%)
🐍 Python Version 3.12.3 |
en0f
left a comment
There was a problem hiding this comment.
Reviewed at dca0142.
🟢 Path-only check, with a real link on the same page as the control
The skip runs after URL validation and looks only at the path, so braces in a query string and percent-encoded braces in a path still yield parameters. The test puts a real link next to the two bad ones and asserts its parameter survives, so it can't pass by suppressing everything. At the base, the probe below emits t, conf and tp; at this head, none of them, and all 15 ParameterExtraction tests pass.
Reproduce this review
# save the script below as verify-3656.sh, then (needs docker; bbot runs in a digest-pinned Linux image):
bash verify-3656.sh blacklanternsecurity/bbot dca01429a7ab8ac0d65dd4fb4b7a5094ed414bf2
# expected: SUMMARY 8 PASS, 0 FAIL, 0 SKIP, 0 BROKEN
# control: bash verify-3656.sh blacklanternsecurity/bbot e0b08a40a32e6a46208f490cb81cdf89aab9eaf8 -> V01 V02 V03 V08 FAIL#!/usr/bin/env bash
# verify-3656.sh: reproducible checks for blacklanternsecurity/bbot#3656
# Usage: bash verify-3656.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# CONTROL: sha e0b08a40a32e6a46208f490cb81cdf89aab9eaf8
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V08=FAIL
# AUTHOR-CODE: container ghcr.io/astral-sh/uv:python3.13-bookworm-slim@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca
# bbot runs only inside that digest-pinned Linux image (its blasthttp dependency ships Linux wheels only),
# with the clone mounted and no host environment or credentials passed in.
# V01-V06 come from a probe test this script writes itself: one page served by bbot's test HTTP server,
# scanned with http + excavate + hunt, and the WEB_PARAMETER events it yields.
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "docker $(docker version --format '{{.Server.Version}}' 2>/dev/null || echo none)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
for i in 1 2 3; do rm -rf "$WORK/src"; "${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" && break; sleep 5; done
[ -d "$WORK/src/.git" ] || { broken V00 "clone" "clone failed (3 attempts)"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
T=bbot/test/test_step_2/module_tests
[ -f bbot/modules/internal/excavate.py ] && [ -f "$T/test_module_excavate.py" ] || { broken V00 "excavate and its test exist" "missing"; summary; exit 1; }
# ---- the probe: written by this script, identical at head and control ----------
cat > "$T/test_zz_probe_3656.py" <<'PROBE'
import json
from pathlib import Path
from .base import ModuleTestBase
from bbot.test.worker import HTTPSERVER_URL
OUT = Path(__file__).resolve().parents[4] / ".probe3656.json"
class TestExcavateProbe3656(ModuleTestBase):
targets = [f"{HTTPSERVER_URL}/"]
module_name = "excavate"
modules_overrides = ["http", "excavate", "hunt"]
html = r"""
<html>
<img ng-src="{{c.selected_image}}?t=small">
<a href=\"https:\/\/www.example.com\/webcast\/register.aspx?conf=1\">webcast</a>
<a href="/{{tpl}}/x.php?tp=1">mid-path placeholder</a>
<a href="/real.php?keep=1">real</a>
<a href="/brace.php?q={x}&keep2=1">braces in the query only</a>
<a href="/%7Benc%7D.php?enc=1">percent-encoded braces</a>
</html>
"""
async def setup_after_prep(self, module_test):
module_test.set_expect_requests(
respond_args={"response_data": self.html, "headers": {"Content-Type": "text/html"}}
)
def check(self, module_test, events):
OUT.write_text(
json.dumps(
{
"params": {e.data["name"]: e.data["url"] for e in events if e.type == "WEB_PARAMETER"},
}
)
)
PROBE
IMG='ghcr.io/astral-sh/uv:python3.13-bookworm-slim@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca'
if ! command -v docker >/dev/null || ! docker info >/dev/null 2>&1; then
for v in V01 V02 V03 V04 V05 V06 V08 V09; do skip $v "container check" "docker not available"; done
summary; exit $?
fi
docker run --rm --network bridge -v "$WORK/src:/src" -w /src "$IMG" bash -c '
export UV_CACHE_DIR=/tmp/uvcache
uv sync --frozen --group dev --quiet >/tmp/sync.log 2>&1 || { echo "SYNC_FAILED $(tail -1 /tmp/sync.log)"; exit 0; }
uv run --no-sync pytest -q -p no:cacheprovider bbot/test/test_step_2/module_tests/test_zz_probe_3656.py > /src/.probe.log 2>&1; echo "PROBE_RC $?"
uv run --no-sync pytest -q -p no:cacheprovider --junitxml=/src/.pe.xml bbot/test/test_step_2/module_tests/test_module_excavate.py -k ParameterExtraction > /src/.pe.log 2>&1; echo "PE_RC $?"
' > "$WORK/container.out" 2>&1
if grep -q '^SYNC_FAILED' "$WORK/container.out"; then
for v in V01 V02 V03 V04 V05 V06 V08 V09; do broken $v "container check" "$(grep SYNC_FAILED "$WORK/container.out" | cut -c1-150)"; done
summary; exit $?
fi
sed 's/^/# container: /' "$WORK/container.out"
tail -3 .probe.log 2>/dev/null | sed 's/^/# probe pytest: /'
tail -3 .pe.log 2>/dev/null | sed 's/^/# excavate -k ParameterExtraction: /'
# helper: q <python expression over P (params dict) and U (url list)> -> prints result
q() { python3 -c "import json,sys; d=json.load(open('.probe3656.json')); P=d['params']; print($1)"; }
if [ ! -s .probe3656.json ]; then
for v in V01 V02 V03 V04 V05 V06; do broken $v "probe ran" "no probe output (probe rc $(grep -o 'PROBE_RC [0-9]*' "$WORK/container.out"))"; done
else
echo "# probe params: $(q 'json.dumps(P, sort_keys=True)')"
if [ "$(q '"t" in P')" = False ]; then pass V01 "a template placeholder src ({{...}}?t=) yields no WEB_PARAMETER"
else fail V01 "a template placeholder src ({{...}}?t=) yields no WEB_PARAMETER" "t on $(q 'P["t"]')"; fi
if [ "$(q '"conf" in P')" = False ]; then pass V02 "a JSON-escaped link (\\"https:\\/\\/...) yields no WEB_PARAMETER"
else fail V02 "a JSON-escaped link (\\"https:\\/\\/...) yields no WEB_PARAMETER" "conf on $(q 'P["conf"]')"; fi
if [ "$(q '"tp" in P')" = False ]; then pass V03 "a placeholder mid-path (/{{tpl}}/x.php?tp=) yields no WEB_PARAMETER"
else fail V03 "a placeholder mid-path (/{{tpl}}/x.php?tp=) yields no WEB_PARAMETER" "tp on $(q 'P["tp"]')"; fi
if [ "$(q 'P.get("keep","").endswith("/real.php")')" = True ]; then pass V04 "a real link on the same page still yields its parameter (keep on /real.php)"
else fail V04 "a real link on the same page still yields its parameter" "keep -> $(q 'P.get("keep")')"; fi
if [ "$(q 'P.get("keep2","").endswith("/brace.php")')" = True ]; then pass V05 "braces in the query string alone do not suppress the link's parameters (keep2 on /brace.php)"
else fail V05 "braces in the query string alone do not suppress the link's parameters" "keep2 -> $(q 'P.get("keep2")')"; fi
if [ "$(q '"enc" in P')" = True ]; then pass V06 "percent-encoded braces in the path (%7B..%7D) still yield the parameter"
else fail V06 "percent-encoded braces in the path (%7B..%7D) still yield the parameter" "enc absent"; fi
fi
# V08: the PR's own regression test exists and passes; V09: no ParameterExtraction test regressed
pe=$(python3 - <<'PY' 2>/dev/null
import xml.etree.ElementTree as E
r = E.parse(".pe.xml").getroot(); s = r if r.tag == "testsuite" else r.find("testsuite")
cases = {"%s::%s" % (c.get("classname"), c.get("name")): (c.find("failure") is None and c.find("error") is None and c.find("skipped") is None) for c in s.iter("testcase")}
nl = [n for n in cases if "notalink" in n]
print("notalink=%s total=%d failed=%d" % ("pass" if nl and all(cases[n] for n in nl) else ("fail" if nl else "absent"), len(cases), sum(1 for v in cases.values() if not v)))
PY
)
if [ -z "$pe" ]; then broken V08 "PR test TestExcavateParameterExtraction_notalink passes" "no junit output"; broken V09 "every excavate ParameterExtraction test passes" "no junit output"
else
case "$pe" in notalink=pass*) pass V08 "PR test TestExcavateParameterExtraction_notalink passes ($pe)";; *) fail V08 "PR test TestExcavateParameterExtraction_notalink passes" "$pe";; esac
case "$pe" in *failed=0) pass V09 "every excavate ParameterExtraction test passes ($pe)";; *) fail V09 "every excavate ParameterExtraction test passes" "$pe";; esac
fi
summaryc5c07b9 to
bda1e8e
Compare
excavate's parameter extractor emitted WEB_PARAMETERs on URLs that aren't links:
<img ng-src="{{c.selected_image}}?t=small">gave atparameter on/{{c.selected_image}}<a href=\"https:\/\/example.com\/x?conf=1\">, gave aconfparameter on/\"https:\/\/example.com\/xThe URL extractor already rejects both URLs. The parameter extractor resolves the same attribute values on its own and only checked
validate_url_parsed, which accepts any path, so every module that consumes WEB_PARAMETER spent requests testing pages that don't exist.This skips a parameter whose URL path contains
{or}, or a backslash escaping a slash, quote or HTML entity (\/,\"). A real link's braces get percent-encoded, and those escapes only show up when the link sits inside a JSON or JS string. A plain backslash is left alone:<a href="images\view.asp?id=1">is a real link on IIS, which reads\as/, so its parameter stays onimages\view.aspas before. Query strings are left alone too, since they can legitimately hold braces.TestExcavateParameterExtraction_notalinkcovers both cases, along with a link escaped with only\/, an escaped jQuery URL and a form action holding a placeholder, and checks that real links on the same page, IIS backslash links among them, still yield their parameters.