Repository navigation
cdn_port_filter: keep only allowed ports on hosts that list every port - #3663
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #3663 +/- ##
=====================================
+ Coverage 91% 91% +1%
=====================================
Files 465 465
Lines 52188 52272 +84
=====================================
+ Hits 47280 47370 +90
+ Misses 4908 4902 -6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
📊 Performance Benchmark Report
📈 Detailed Results (All Benchmarks)
🎯 Performance Summary+ 1 improvement 🚀
30 unchanged ✅🔍 Significant Changes (>10%)
🐍 Python Version 3.12.3 |
en0f
left a comment
There was a problem hiding this comment.
Reviewed at 5f0852a.
🟢 One scanner method behind the existing switches, and it holds on a live catch-all IP
filter_catch_all_ports() lives once on the scanner and all three port sources call it before emitting, so the cap isn't repeated per module. It reuses cdn_port_filter.enabled and allowed_ports, so --allow-cdn-ports turns it off with no new flag, and like the CDN check in ScanEgress it keeps a port you targeted on that host. On a real scan at this head, bbot -t 35.186.224.25 -m shodan_idb (a Google LB address InternetDB lists with 1,995 ports) emits only 80 and 443 for the IP and its PTR name, and with --allow-cdn-ports the IP keeps all 1,995. With the cap patched out, the default scan emits about 3,700.
🟢 censys_ip also drops the URL and PROTOCOL on a cut port, and its test guards both
Dropping only OPEN_TCP_PORT wouldn't have held: speculate turns a URL_UNVERIFIED back into an OPEN_TCP_PORT (speculate.py:145-156), and a PROTOCOL event feeds medusa and legba. Putting either emission back on a cut port, or undoing the null transport_protocol fix, makes TestCensys_IP_CatchAllPorts fail on an assertion, so the test protects the parts that matter.
Reproduce this review
# save the script below as verify-3663.sh and the patch as control-3663.patch, then (needs Docker; V08/V09 query Shodan InternetDB, passive):
bash verify-3663.sh blacklanternsecurity/bbot 5f0852a305ce3cec383b718e7f7740c5440ad793
# expected: SUMMARY 11 PASS, 1 FAIL, 0 SKIP, 0 BROKEN (V12: a negative max_ports cuts every host; an edge outside the documented 0 = off, not a merge condition)
# control: bash verify-3663.sh blacklanternsecurity/bbot 5f0852a305ce3cec383b718e7f7740c5440ad793 "$PWD/control-3663.patch" -> V01 V02 V03 V04 V08 V10 FAIL--- a/bbot/scanner/scanner.py
+++ b/bbot/scanner/scanner.py
@@ -1093,8 +1093,8 @@
def filter_catch_all_ports(self, host, ports):
"""Return a host's unique ports, cut to the allowed and targeted ones if there are more than cdn_port_filter.max_ports (it accepts connections on any port)"""
ports = list(dict.fromkeys(ports))
- if not self.cdn_port_filter or not self.cdn_max_ports or len(ports) <= self.cdn_max_ports:
- return ports
+ # CONTROL (verify-3663): the catch-all cap is never applied, as before this PR
+ return ports
host = str(host)
# like the rest of cdn_port_filter, a port explicitly targeted on this host is never cut
targeted = {#!/usr/bin/env bash
# verify-3663.sh: reproducible checks for blacklanternsecurity/bbot#3663
# Usage: bash verify-3663.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# CONTROL: patch control-3663.patch
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V04=FAIL V08=FAIL V10=FAIL
# AUTHOR-CODE: container ghcr.io/astral-sh/uv@sha256:4f52717de41541452f5318571b05da783d2ddf346a94b4d5c6512a4b51e986bd
# (tag 0.9.7-python3.12-bookworm-slim). bbot runs only inside that digest-pinned Linux image (blasthttp ships
# Linux wheels only), with the clone mounted and no host environment or credentials passed in.
# The control patch makes scanner.filter_catch_all_ports() return every port (the cap is never applied),
# which is the behaviour before this PR. V01-V04 and V10 (the PR's own tests) and V08 (a live scan) must flip to FAIL.
# V08/V09 are passive: shodan_idb asks Shodan's InternetDB about 35.186.224.25 (a Google Cloud LB address
# InternetDB listed with 1,995 open ports on 2026-11-20); no packet is sent to that address. Live data may drift.
# V09 is checked against a direct InternetDB read, so a broken --allow-cdn-ports switch prints FAIL (challenge C1).
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "docker $(docker version --format '{{.Server.Version}} {{.Server.Arch}}' 2>/dev/null || echo none)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
for i in 1 2 3; do rm -rf "$WORK/src"; "${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" && break; sleep 5; done
[ -d "$WORK/src/.git" ] || { broken V00 "clone" "clone failed (3 attempts)"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
grep -q 'def filter_catch_all_ports' bbot/scanner/scanner.py || { broken V00 "filter_catch_all_ports exists" "missing"; summary; exit 1; }
# ---- mutants for V06/V07: censys_ip emits URL / PROTOCOL on a dropped port again ----
CENSYS=bbot/modules/censys_ip.py
URL_OLD='if uri and not dropped and uri not in seen:'
PROTO_OLD='if service_name and not dropped and service_name.upper()'
grep -qF "$URL_OLD" "$CENSYS" && URL_OK=1 || URL_OK=0
grep -qF "$PROTO_OLD" "$CENSYS" && PROTO_OK=1 || PROTO_OK=0
# V11 mutant (challenge C11): undo the null transport_protocol fix (both call sites)
grep -qF '(service.get("transport_protocol") or "TCP")' "$CENSYS" && NULL_OK=1 || NULL_OK=0
# V12 probe (challenge C10): a negative max_ports must not cut a host with only 2 ports
cat > bbot/test/test_step_1/test_zz_probe_3663.py <<'PROBE'
import pytest
from ..bbot_fixtures import * # noqa: F401,F403
@pytest.mark.asyncio
async def test_probe_negative_max_ports(bbot_scanner):
scan = bbot_scanner("1.2.3.4", config={"cdn_port_filter": {"max_ports": -1}})
assert scan.filter_catch_all_ports("1.2.3.4", [22, 80]) == [22, 80]
PROBE
IMG='ghcr.io/astral-sh/uv@sha256:4f52717de41541452f5318571b05da783d2ddf346a94b4d5c6512a4b51e986bd'
T1=bbot/test/test_step_1/test_scan.py
TM=bbot/test/test_step_2/module_tests
docker run --rm --network bridge -v "$WORK/src:/src" -w /src "$IMG" bash -c '
export UV_CACHE_DIR=/tmp/uvcache
uv sync --frozen --group dev --quiet >/tmp/sync.log 2>&1 || { echo "SYNC_FAILED $(tail -1 /tmp/sync.log)"; exit 0; }
P="uv run --no-sync pytest -q -p no:cacheprovider"
$P '"$T1"' -k test_cdn_port_filter_catch_all > /src/.v01.log 2>&1; echo "V01_RC $?"
$P '"$TM"'/test_module_shodan_idb.py -k CatchAllPorts > /src/.v02.log 2>&1; echo "V02_RC $?"
$P '"$TM"'/test_module_shodan_enterprise.py -k CatchAllPorts > /src/.v03.log 2>&1; echo "V03_RC $?"
$P '"$TM"'/test_module_censys_ip.py -k "CatchAllPorts and not Targeted" > /src/.v04.log 2>&1; echo "V04_RC $?"
$P '"$TM"'/test_module_censys_ip.py -k CatchAllTargetedPort > /src/.v10.log 2>&1; echo "V10_RC $?"
$P '"$TM"'/test_module_shodan_idb.py '"$TM"'/test_module_shodan_enterprise.py '"$TM"'/test_module_censys_ip.py > /src/.v05.log 2>&1; echo "V05_RC $?"
cp '"$CENSYS"' /tmp/censys.orig
if [ '"$URL_OK"' = 1 ]; then
sed -i "s/if uri and not dropped and uri not in seen:/if uri and uri not in seen:/" '"$CENSYS"'
$P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v06.log 2>&1; echo "V06_RC $?"
cp /tmp/censys.orig '"$CENSYS"'
fi
if [ '"$PROTO_OK"' = 1 ]; then
sed -i "s/if service_name and not dropped and service_name.upper()/if service_name and service_name.upper()/" '"$CENSYS"'
$P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v07.log 2>&1; echo "V07_RC $?"
cp /tmp/censys.orig '"$CENSYS"'
fi
if [ '"$NULL_OK"' = 1 ]; then
sed -i "s/(service.get(\"transport_protocol\") or \"TCP\")/service.get(\"transport_protocol\", \"TCP\")/g" '"$CENSYS"'
$P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v11.log 2>&1; echo "V11_RC $?"
cp /tmp/censys.orig '"$CENSYS"'
fi
cmp -s /tmp/censys.orig '"$CENSYS"' && echo "CENSYS_RESTORED 1"
$P bbot/test/test_step_1/test_zz_probe_3663.py > /src/.v12.log 2>&1; echo "V12_RC $?"
live() { timeout 300 uv run --no-sync bbot -t 35.186.224.25 -m shodan_idb -y --json --silent "$@" 2>/dev/null \
| python3 -c "import sys,json
n=0
for l in sys.stdin:
try: e=json.loads(l)
except Exception: continue
if e.get(\"type\")==\"OPEN_TCP_PORT\" and e.get(\"module\")==\"shodan_idb\": n+=1; print(\"PORT\", e[\"data\"])
print(\"COUNT\", n)"; }
live > /src/.v08.out; echo "V08_DONE $?"
live --allow-cdn-ports > /src/.v09.out; echo "V09_DONE $?"
' > "$WORK/container.out" 2>&1
if grep -q '^SYNC_FAILED' "$WORK/container.out"; then
for v in V01 V02 V03 V04 V05 V06 V07 V08 V09 V10 V11 V12; do broken $v "container check" "$(grep SYNC_FAILED "$WORK/container.out" | cut -c1-150)"; done
summary; exit $?
fi
sed 's/^/# container: /' "$WORK/container.out"
for f in .v01.log .v02.log .v03.log .v04.log .v05.log .v06.log .v07.log .v10.log .v11.log .v12.log; do [ -f "$f" ] && tail -1 "$f" | sed "s/^/# $f: /"; done
rc() { grep -o "^$1 [0-9]*" "$WORK/container.out" | cut -d' ' -f2; }
grep -q '^CENSYS_RESTORED 1' "$WORK/container.out" || broken V00 "censys_ip.py restored after mutants" "file differs"
t() { # id rc-key desc
local r; r="$(rc "$2")"
if [ -z "$r" ]; then broken "$1" "$3" "no result"
elif [ "$r" = 0 ]; then pass "$1" "$3"
else fail "$1" "$3" "pytest rc $r"; fi
}
t V01 V01_RC "test_cdn_port_filter_catch_all (PR test: cap, boundary, custom allowed_ports, both off switches, targeted port kept) passes"
t V02 V02_RC "TestShodan_IDB_CatchAllPorts (2,000-port IP keeps 80/443; 4-port IP keeps all) passes"
t V03 V03_RC "TestShodan_Enterprise_CatchAllPorts (TCP cut, UDP and findings kept) passes"
t V04 V04_RC "TestCensys_IP_CatchAllPorts (TCP cut, URL/PROTOCOL on cut ports dropped) passes"
t V05 V05_RC "every test in the three changed module test files passes (no regression)"
t V10 V10_RC "TestCensys_IP_CatchAllTargetedPort (a port targeted on a catch-all host keeps its URL) passes"
# V06/V07: the PR's censys test kills a mutant that re-emits URL_UNVERIFIED / PROTOCOL on a dropped port
# killed means pytest rc 1 (tests failed) with an assertion failure in its log (challenge C4); rc 2-5 is BROKEN
mut() { # id rc-key ok desc log
local r; r="$(rc "$2")"
[ -f "$5" ] && grep -E 'AssertionError|^E +assert|[0-9]+ (passed|failed)' "$5" | tail -4 | sed "s/^/# $1 log: /"
if [ "$3" != 1 ]; then broken "$1" "$4" "mutation target not found in censys_ip.py"
elif [ -z "$r" ]; then broken "$1" "$4" "no result"
elif [ "$r" = 0 ]; then fail "$1" "$4" "mutant survived (pytest rc 0)"
elif [ "$r" = 1 ] && grep -qE 'AssertionError|^E +assert' "$5"; then pass "$1" "$4"
else broken "$1" "$4" "pytest rc $r without an assertion failure (collection/usage error?)"; fi
}
mut V06 V06_RC "$URL_OK" "PR censys test fails when URL_UNVERIFIED is emitted on a dropped port (mutant killed)" .v06.log
mut V07 V07_RC "$PROTO_OK" "PR censys test fails when PROTOCOL is emitted on a dropped port (mutant killed)" .v07.log
mut V11 V11_RC "$NULL_OK" "PR censys test fails when a null transport_protocol is no longer treated as TCP (mutant killed)" .v11.log
t V12 V12_RC "a negative cdn_port_filter.max_ports does not cut a host listing only 2 ports (probe)"
# V08/V09: user boundary, live and passive. bbot -t 35.186.224.25 -m shodan_idb
c8=$(grep -o '^COUNT [0-9]*' .v08.out 2>/dev/null | cut -d' ' -f2)
c9=$(grep -o '^COUNT [0-9]*' .v09.out 2>/dev/null | cut -d' ' -f2)
other8=$(grep '^PORT ' .v08.out 2>/dev/null | grep -v -E ':(80|443)$' | head -3 | tr '\n' ' ')
hosts8=$(grep '^PORT ' .v08.out 2>/dev/null | sed 's/^PORT //; s/:[0-9]*$//' | sort -u | tr '\n' ' ')
hosts9=$(grep '^PORT ' .v09.out 2>/dev/null | sed 's/^PORT //; s/:[0-9]*$//' | sort | uniq -c | tr -s ' ' | tr '\n' ';')
echo "# live default: ${c8:-none} shodan_idb ports; with --allow-cdn-ports: ${c9:-none}"
echo "# live default hosts: ${hosts8:-none}; --allow-cdn-ports ports per host: ${hosts9:-none}"
echo "# live default ports: $(grep '^PORT ' .v08.out 2>/dev/null | head -10 | sed 's/^PORT //' | tr '\n' ' ')"
# positive control (challenge C1): InternetDB's own listing, read directly over anonymous HTTPS
listed=$(curl -s -m 20 https://internetdb.shodan.io/35.186.224.25 | python3 -c 'import sys,json; print(len(json.load(sys.stdin).get("ports",[])))' 2>/dev/null)
echo "# InternetDB lists ${listed:-none} ports for 35.186.224.25 (direct read)"
if [ -z "$listed" ] || [ "$listed" -le 100 ]; then
skip V08 "default scan keeps only 80/443 from shodan_idb on a catch-all IP" "positive control: InternetDB lists ${listed:-none} ports for 35.186.224.25 (not >100)"
skip V09 "--allow-cdn-ports keeps the listed ports" "same: live data no longer catch-all"
else
# every emitted port is 80 or 443 (one pair per host shodan_idb queried: the IP and any name resolving to it)
if [ -n "$c8" ] && [ "$c8" -ge 1 ] && [ -z "$other8" ]; then
pass V08 "bbot -t 35.186.224.25 -m shodan_idb emits only ports 80/443 ($c8 OPEN_TCP_PORT on: $hosts8) while InternetDB lists $listed"
else fail V08 "bbot -t 35.186.224.25 -m shodan_idb emits only 80/443" "emitted ${c8:-none} ports, e.g. ${other8:-n/a}"; fi
# V09: with the switch off, the IP itself keeps more than max_ports (100) of its listed ports
ip9=$(grep '^PORT 35\.186\.224\.25:' .v09.out 2>/dev/null | wc -l | tr -d ' ')
if [ "${ip9:-0}" -gt 100 ]; then pass V09 "--allow-cdn-ports turns the cap off: 35.186.224.25 keeps $ip9 of $listed listed ports"
else fail V09 "--allow-cdn-ports turns the cap off" "35.186.224.25 kept ${ip9:-0} ports while InternetDB lists $listed"; fi
fi
summary|
📝 Optional, not a merge condition: a negative
Repro at 5f0852a: |
Some IPs accept a connection on any port, such as cloud load-balancer front ends (Google Cloud's 34.x and 35.x anycast addresses, for one) and tarpits. Passive port sources list every one of those ports as open, so shodan_idb emitted around 2,000 OPEN_TCP_PORT events per IP. These IPs aren't tagged CDN or WAF, so the CDN port filter lets the ports through, and each one costs a DNS resolution, a cloudcheck pass and an HTTP probe. In one scan, 7 such IPs produced about 14,700 open ports in its first seven minutes. Rerun with this change, each of those IPs got only 80 and 443, and the scan produced 646 open ports over the same stretch.
cdn_port_filter.max_ports, default 100. When a passive port source lists more open TCP ports than that for a host, onlycdn_port_filter.allowed_ports(80 and 443 by default) and any ports explicitly targeted on that host are kept, and the drop is noted in the verbose log.shodan_idb,shodan_enterpriseandcensys_ipapply it through one scanner method,filter_catch_all_ports(), before emitting anything, so none of the dropped ports enter the pipeline.censys_ipalso drops the URLs and PROTOCOL events it would have emitted on those ports.cdn_port_filter:--allow-cdn-ports(orenabled: false) turns it off, andmax_ports: 0turns off just this check. Activeportscanresults aren't capped; Cap open ports on catch-all hosts across every port source #3671 covers enforcing this across every port source.censys_ipno longer crashes on a service whosetransport_protocolis null.test_cdn_port_filter_catch_allcovers the cap, its boundary, custom allowed ports, targeted ports and both off switches, and each of the three modules has a catch-all test.