Skip to content

cdn_port_filter: keep only allowed ports on hosts that list every port - #3663

Merged
liquidsec merged 5 commits into
devfrom
shodan-idb-catch-all-ports
Nov 21, 2026
Merged

liquidsec merged 5 commits into
devfrom
shodan-idb-catch-all-ports

Conversation

@liquidsec

@liquidsec liquidsec commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Some IPs accept a connection on any port, such as cloud load-balancer front ends (Google Cloud's 34.x and 35.x anycast addresses, for one) and tarpits. Passive port sources list every one of those ports as open, so shodan_idb emitted around 2,000 OPEN_TCP_PORT events per IP. These IPs aren't tagged CDN or WAF, so the CDN port filter lets the ports through, and each one costs a DNS resolution, a cloudcheck pass and an HTTP probe. In one scan, 7 such IPs produced about 14,700 open ports in its first seven minutes. Rerun with this change, each of those IPs got only 80 and 443, and the scan produced 646 open ports over the same stretch.

  • New option cdn_port_filter.max_ports, default 100. When a passive port source lists more open TCP ports than that for a host, only cdn_port_filter.allowed_ports (80 and 443 by default) and any ports explicitly targeted on that host are kept, and the drop is noted in the verbose log.
  • shodan_idb, shodan_enterprise and censys_ip apply it through one scanner method, filter_catch_all_ports(), before emitting anything, so none of the dropped ports enter the pipeline. censys_ip also drops the URLs and PROTOCOL events it would have emitted on those ports.
  • It follows the rest of cdn_port_filter: --allow-cdn-ports (or enabled: false) turns it off, and max_ports: 0 turns off just this check. Active portscan results aren't capped; Cap open ports on catch-all hosts across every port source #3671 covers enforcing this across every port source.
  • censys_ip no longer crashes on a service whose transport_protocol is null.

test_cdn_port_filter_catch_all covers the cap, its boundary, custom allowed ports, targeted ports and both off switches, and each of the three modules has a catch-all test.

@liquidsec liquidsec self-assigned this Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (c339f34) to head (5f0852a).
⚠️ Report is 13 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3663   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        465     465           
  Lines      52188   52272   +84     
=====================================
+ Hits       47280   47370   +90     
+ Misses      4908    4902    -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (c339f34c5) (baseline) vs shodan-idb-catch-all-ports (d8349006d) (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 3.06ms 3.12ms +2.0% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 11.62ms 12.72ms +9.5% ⚪ ✅
Large Closest Match Lookup 244.35ms 245.71ms +0.6% ⚪ ✅
Realistic Closest Match Workload 138.32ms 135.27ms -2.2% ⚪ ✅
Event Memory Medium Scan 1389 B/event 1389 B/event +0.0% ⚪ ✅
Event Memory Large Scan 1514 B/event 1514 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 23.44ms 23.64ms +0.8% ⚪ ✅
Event Validation Full Scan Startup Large Batch 114.91ms 115.68ms +0.7% ⚪ ✅
Make Event Autodetection Small 16.20ms 16.15ms -0.3% ⚪ ✅
Make Event Autodetection Large 160.61ms 160.59ms -0.0% ⚪ ✅
Make Event Explicit Types 6.43ms 6.42ms -0.1% ⚪ ✅
Excavate Single Thread Small 2.902s 2.612s -10.0% 🟢🟢 🚀
Excavate Single Thread Large 7.589s 7.245s -4.5% ⚪ ✅
Excavate Parallel Tasks Small 3.076s 2.772s -9.9% ⚪ ✅
Excavate Parallel Tasks Large 4.879s 4.586s -6.0% ⚪ ✅
Intercept Throughput Small 383.73ms 372.33ms -3.0% ⚪ ✅
Intercept Throughput Medium 370.77ms 367.97ms -0.8% ⚪ ✅
Dns Throughput Quiet 1.395s 1.378s -1.2% ⚪ ✅
Dns Throughput Loaded 748.41ms 763.94ms +2.1% ⚪ ✅
Dns Throughput Inherited 1.188s 1.142s -3.9% ⚪ ✅
Is Ip Performance 1.70ms 1.67ms -1.5% ⚪ ✅
Make Ip Type Performance 161.59µs 157.30µs -2.7% ⚪ ✅
Mixed Ip Operations 1.79ms 1.72ms -3.6% ⚪ ✅
Memory Use Web Crawl 1.1 MB 1.1 MB -0.9% ⚪ ✅
Memory Use Subdomain Enum 27.8 MB 27.8 MB +0.0% ⚪ ✅
Memory Use Deep Chain 5.1 MB 5.1 MB +0.0% ⚪ ✅
Memory Use Parallel Chains 12.1 MB 13.2 MB +9.4% ⚪ ✅
Scan Throughput 100 1.783s 1.876s +5.2% ⚪ ✅
Scan Throughput 1000 14.355s 14.723s +2.6% ⚪ ✅
Typical Queue Shuffle 4.20µs 4.20µs +0.1% ⚪ ✅
Priority Queue Shuffle 21.21µs 21.56µs +1.7% ⚪ ✅

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Excavate Single Thread Small: 10.0% 🚀 faster

🐍 Python Version 3.12.3

@liquidsec
liquidsec requested a review from a team as a code owner November 20, 2026 22:47
@liquidsec liquidsec changed the title shodan_idb: keep only allowed ports for an IP that answers on every port cdn_port_filter: keep only allowed ports on hosts that list every port Nov 20, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 5f0852a.

🟢 One scanner method behind the existing switches, and it holds on a live catch-all IP

filter_catch_all_ports() lives once on the scanner and all three port sources call it before emitting, so the cap isn't repeated per module. It reuses cdn_port_filter.enabled and allowed_ports, so --allow-cdn-ports turns it off with no new flag, and like the CDN check in ScanEgress it keeps a port you targeted on that host. On a real scan at this head, bbot -t 35.186.224.25 -m shodan_idb (a Google LB address InternetDB lists with 1,995 ports) emits only 80 and 443 for the IP and its PTR name, and with --allow-cdn-ports the IP keeps all 1,995. With the cap patched out, the default scan emits about 3,700.

🟢 censys_ip also drops the URL and PROTOCOL on a cut port, and its test guards both

Dropping only OPEN_TCP_PORT wouldn't have held: speculate turns a URL_UNVERIFIED back into an OPEN_TCP_PORT (speculate.py:145-156), and a PROTOCOL event feeds medusa and legba. Putting either emission back on a cut port, or undoing the null transport_protocol fix, makes TestCensys_IP_CatchAllPorts fail on an assertion, so the test protects the parts that matter.

Reproduce this review
# save the script below as verify-3663.sh and the patch as control-3663.patch, then (needs Docker; V08/V09 query Shodan InternetDB, passive):
bash verify-3663.sh blacklanternsecurity/bbot 5f0852a305ce3cec383b718e7f7740c5440ad793
# expected: SUMMARY 11 PASS, 1 FAIL, 0 SKIP, 0 BROKEN (V12: a negative max_ports cuts every host; an edge outside the documented 0 = off, not a merge condition)
# control:  bash verify-3663.sh blacklanternsecurity/bbot 5f0852a305ce3cec383b718e7f7740c5440ad793 "$PWD/control-3663.patch"   -> V01 V02 V03 V04 V08 V10 FAIL
--- a/bbot/scanner/scanner.py
+++ b/bbot/scanner/scanner.py
@@ -1093,8 +1093,8 @@
     def filter_catch_all_ports(self, host, ports):
         """Return a host's unique ports, cut to the allowed and targeted ones if there are more than cdn_port_filter.max_ports (it accepts connections on any port)"""
         ports = list(dict.fromkeys(ports))
-        if not self.cdn_port_filter or not self.cdn_max_ports or len(ports) <= self.cdn_max_ports:
-            return ports
+        # CONTROL (verify-3663): the catch-all cap is never applied, as before this PR
+        return ports
         host = str(host)
         # like the rest of cdn_port_filter, a port explicitly targeted on this host is never cut
         targeted = {
#!/usr/bin/env bash
# verify-3663.sh: reproducible checks for blacklanternsecurity/bbot#3663
# Usage: bash verify-3663.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# CONTROL: patch control-3663.patch
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V04=FAIL V08=FAIL V10=FAIL
# AUTHOR-CODE: container ghcr.io/astral-sh/uv@sha256:4f52717de41541452f5318571b05da783d2ddf346a94b4d5c6512a4b51e986bd
# (tag 0.9.7-python3.12-bookworm-slim). bbot runs only inside that digest-pinned Linux image (blasthttp ships
# Linux wheels only), with the clone mounted and no host environment or credentials passed in.
# The control patch makes scanner.filter_catch_all_ports() return every port (the cap is never applied),
# which is the behaviour before this PR. V01-V04 and V10 (the PR's own tests) and V08 (a live scan) must flip to FAIL.
# V08/V09 are passive: shodan_idb asks Shodan's InternetDB about 35.186.224.25 (a Google Cloud LB address
# InternetDB listed with 1,995 open ports on 2026-11-20); no packet is sent to that address. Live data may drift.
# V09 is checked against a direct InternetDB read, so a broken --allow-cdn-ports switch prints FAIL (challenge C1).
set -uo pipefail

REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass()   { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail()   { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip()   { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }

echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "docker $(docker version --format '{{.Server.Version}} {{.Server.Arch}}' 2>/dev/null || echo none)"

WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
for i in 1 2 3; do rm -rf "$WORK/src"; "${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" && break; sleep 5; done
[ -d "$WORK/src/.git" ] || { broken V00 "clone" "clone failed (3 attempts)"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
grep -q 'def filter_catch_all_ports' bbot/scanner/scanner.py || { broken V00 "filter_catch_all_ports exists" "missing"; summary; exit 1; }

# ---- mutants for V06/V07: censys_ip emits URL / PROTOCOL on a dropped port again ----
CENSYS=bbot/modules/censys_ip.py
URL_OLD='if uri and not dropped and uri not in seen:'
PROTO_OLD='if service_name and not dropped and service_name.upper()'
grep -qF "$URL_OLD" "$CENSYS" && URL_OK=1 || URL_OK=0
grep -qF "$PROTO_OLD" "$CENSYS" && PROTO_OK=1 || PROTO_OK=0
# V11 mutant (challenge C11): undo the null transport_protocol fix (both call sites)
grep -qF '(service.get("transport_protocol") or "TCP")' "$CENSYS" && NULL_OK=1 || NULL_OK=0
# V12 probe (challenge C10): a negative max_ports must not cut a host with only 2 ports
cat > bbot/test/test_step_1/test_zz_probe_3663.py <<'PROBE'
import pytest

from ..bbot_fixtures import *  # noqa: F401,F403


@pytest.mark.asyncio
async def test_probe_negative_max_ports(bbot_scanner):
    scan = bbot_scanner("1.2.3.4", config={"cdn_port_filter": {"max_ports": -1}})
    assert scan.filter_catch_all_ports("1.2.3.4", [22, 80]) == [22, 80]
PROBE

IMG='ghcr.io/astral-sh/uv@sha256:4f52717de41541452f5318571b05da783d2ddf346a94b4d5c6512a4b51e986bd'
T1=bbot/test/test_step_1/test_scan.py
TM=bbot/test/test_step_2/module_tests
docker run --rm --network bridge -v "$WORK/src:/src" -w /src "$IMG" bash -c '
  export UV_CACHE_DIR=/tmp/uvcache
  uv sync --frozen --group dev --quiet >/tmp/sync.log 2>&1 || { echo "SYNC_FAILED $(tail -1 /tmp/sync.log)"; exit 0; }
  P="uv run --no-sync pytest -q -p no:cacheprovider"
  $P '"$T1"' -k test_cdn_port_filter_catch_all > /src/.v01.log 2>&1; echo "V01_RC $?"
  $P '"$TM"'/test_module_shodan_idb.py -k CatchAllPorts > /src/.v02.log 2>&1; echo "V02_RC $?"
  $P '"$TM"'/test_module_shodan_enterprise.py -k CatchAllPorts > /src/.v03.log 2>&1; echo "V03_RC $?"
  $P '"$TM"'/test_module_censys_ip.py -k "CatchAllPorts and not Targeted" > /src/.v04.log 2>&1; echo "V04_RC $?"
  $P '"$TM"'/test_module_censys_ip.py -k CatchAllTargetedPort > /src/.v10.log 2>&1; echo "V10_RC $?"
  $P '"$TM"'/test_module_shodan_idb.py '"$TM"'/test_module_shodan_enterprise.py '"$TM"'/test_module_censys_ip.py > /src/.v05.log 2>&1; echo "V05_RC $?"
  cp '"$CENSYS"' /tmp/censys.orig
  if [ '"$URL_OK"' = 1 ]; then
    sed -i "s/if uri and not dropped and uri not in seen:/if uri and uri not in seen:/" '"$CENSYS"'
    $P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v06.log 2>&1; echo "V06_RC $?"
    cp /tmp/censys.orig '"$CENSYS"'
  fi
  if [ '"$PROTO_OK"' = 1 ]; then
    sed -i "s/if service_name and not dropped and service_name.upper()/if service_name and service_name.upper()/" '"$CENSYS"'
    $P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v07.log 2>&1; echo "V07_RC $?"
    cp /tmp/censys.orig '"$CENSYS"'
  fi
  if [ '"$NULL_OK"' = 1 ]; then
    sed -i "s/(service.get(\"transport_protocol\") or \"TCP\")/service.get(\"transport_protocol\", \"TCP\")/g" '"$CENSYS"'
    $P '"$TM"'/test_module_censys_ip.py -k CatchAllPorts > /src/.v11.log 2>&1; echo "V11_RC $?"
    cp /tmp/censys.orig '"$CENSYS"'
  fi
  cmp -s /tmp/censys.orig '"$CENSYS"' && echo "CENSYS_RESTORED 1"
  $P bbot/test/test_step_1/test_zz_probe_3663.py > /src/.v12.log 2>&1; echo "V12_RC $?"
  live() { timeout 300 uv run --no-sync bbot -t 35.186.224.25 -m shodan_idb -y --json --silent "$@" 2>/dev/null \
    | python3 -c "import sys,json
n=0
for l in sys.stdin:
    try: e=json.loads(l)
    except Exception: continue
    if e.get(\"type\")==\"OPEN_TCP_PORT\" and e.get(\"module\")==\"shodan_idb\": n+=1; print(\"PORT\", e[\"data\"])
print(\"COUNT\", n)"; }
  live > /src/.v08.out; echo "V08_DONE $?"
  live --allow-cdn-ports > /src/.v09.out; echo "V09_DONE $?"
' > "$WORK/container.out" 2>&1

if grep -q '^SYNC_FAILED' "$WORK/container.out"; then
  for v in V01 V02 V03 V04 V05 V06 V07 V08 V09 V10 V11 V12; do broken $v "container check" "$(grep SYNC_FAILED "$WORK/container.out" | cut -c1-150)"; done
  summary; exit $?
fi
sed 's/^/# container: /' "$WORK/container.out"
for f in .v01.log .v02.log .v03.log .v04.log .v05.log .v06.log .v07.log .v10.log .v11.log .v12.log; do [ -f "$f" ] && tail -1 "$f" | sed "s/^/# $f: /"; done
rc() { grep -o "^$1 [0-9]*" "$WORK/container.out" | cut -d' ' -f2; }
grep -q '^CENSYS_RESTORED 1' "$WORK/container.out" || broken V00 "censys_ip.py restored after mutants" "file differs"

t() { # id rc-key desc
  local r; r="$(rc "$2")"
  if [ -z "$r" ]; then broken "$1" "$3" "no result"
  elif [ "$r" = 0 ]; then pass "$1" "$3"
  else fail "$1" "$3" "pytest rc $r"; fi
}
t V01 V01_RC "test_cdn_port_filter_catch_all (PR test: cap, boundary, custom allowed_ports, both off switches, targeted port kept) passes"
t V02 V02_RC "TestShodan_IDB_CatchAllPorts (2,000-port IP keeps 80/443; 4-port IP keeps all) passes"
t V03 V03_RC "TestShodan_Enterprise_CatchAllPorts (TCP cut, UDP and findings kept) passes"
t V04 V04_RC "TestCensys_IP_CatchAllPorts (TCP cut, URL/PROTOCOL on cut ports dropped) passes"
t V05 V05_RC "every test in the three changed module test files passes (no regression)"
t V10 V10_RC "TestCensys_IP_CatchAllTargetedPort (a port targeted on a catch-all host keeps its URL) passes"

# V06/V07: the PR's censys test kills a mutant that re-emits URL_UNVERIFIED / PROTOCOL on a dropped port
# killed means pytest rc 1 (tests failed) with an assertion failure in its log (challenge C4); rc 2-5 is BROKEN
mut() { # id rc-key ok desc log
  local r; r="$(rc "$2")"
  [ -f "$5" ] && grep -E 'AssertionError|^E +assert|[0-9]+ (passed|failed)' "$5" | tail -4 | sed "s/^/# $1 log: /"
  if [ "$3" != 1 ]; then broken "$1" "$4" "mutation target not found in censys_ip.py"
  elif [ -z "$r" ]; then broken "$1" "$4" "no result"
  elif [ "$r" = 0 ]; then fail "$1" "$4" "mutant survived (pytest rc 0)"
  elif [ "$r" = 1 ] && grep -qE 'AssertionError|^E +assert' "$5"; then pass "$1" "$4"
  else broken "$1" "$4" "pytest rc $r without an assertion failure (collection/usage error?)"; fi
}
mut V06 V06_RC "$URL_OK"   "PR censys test fails when URL_UNVERIFIED is emitted on a dropped port (mutant killed)" .v06.log
mut V07 V07_RC "$PROTO_OK" "PR censys test fails when PROTOCOL is emitted on a dropped port (mutant killed)" .v07.log
mut V11 V11_RC "$NULL_OK"  "PR censys test fails when a null transport_protocol is no longer treated as TCP (mutant killed)" .v11.log
t V12 V12_RC "a negative cdn_port_filter.max_ports does not cut a host listing only 2 ports (probe)"

# V08/V09: user boundary, live and passive. bbot -t 35.186.224.25 -m shodan_idb
c8=$(grep -o '^COUNT [0-9]*' .v08.out 2>/dev/null | cut -d' ' -f2)
c9=$(grep -o '^COUNT [0-9]*' .v09.out 2>/dev/null | cut -d' ' -f2)
other8=$(grep '^PORT ' .v08.out 2>/dev/null | grep -v -E ':(80|443)$' | head -3 | tr '\n' ' ')
hosts8=$(grep '^PORT ' .v08.out 2>/dev/null | sed 's/^PORT //; s/:[0-9]*$//' | sort -u | tr '\n' ' ')
hosts9=$(grep '^PORT ' .v09.out 2>/dev/null | sed 's/^PORT //; s/:[0-9]*$//' | sort | uniq -c | tr -s ' ' | tr '\n' ';')
echo "# live default: ${c8:-none} shodan_idb ports; with --allow-cdn-ports: ${c9:-none}"
echo "# live default hosts: ${hosts8:-none}; --allow-cdn-ports ports per host: ${hosts9:-none}"
echo "# live default ports: $(grep '^PORT ' .v08.out 2>/dev/null | head -10 | sed 's/^PORT //' | tr '\n' ' ')"
# positive control (challenge C1): InternetDB's own listing, read directly over anonymous HTTPS
listed=$(curl -s -m 20 https://internetdb.shodan.io/35.186.224.25 | python3 -c 'import sys,json; print(len(json.load(sys.stdin).get("ports",[])))' 2>/dev/null)
echo "# InternetDB lists ${listed:-none} ports for 35.186.224.25 (direct read)"
if [ -z "$listed" ] || [ "$listed" -le 100 ]; then
  skip V08 "default scan keeps only 80/443 from shodan_idb on a catch-all IP" "positive control: InternetDB lists ${listed:-none} ports for 35.186.224.25 (not >100)"
  skip V09 "--allow-cdn-ports keeps the listed ports" "same: live data no longer catch-all"
else
  # every emitted port is 80 or 443 (one pair per host shodan_idb queried: the IP and any name resolving to it)
  if [ -n "$c8" ] && [ "$c8" -ge 1 ] && [ -z "$other8" ]; then
    pass V08 "bbot -t 35.186.224.25 -m shodan_idb emits only ports 80/443 ($c8 OPEN_TCP_PORT on: $hosts8) while InternetDB lists $listed"
  else fail V08 "bbot -t 35.186.224.25 -m shodan_idb emits only 80/443" "emitted ${c8:-none} ports, e.g. ${other8:-n/a}"; fi
  # V09: with the switch off, the IP itself keeps more than max_ports (100) of its listed ports
  ip9=$(grep '^PORT 35\.186\.224\.25:' .v09.out 2>/dev/null | wc -l | tr -d ' ')
  if [ "${ip9:-0}" -gt 100 ]; then pass V09 "--allow-cdn-ports turns the cap off: 35.186.224.25 keeps $ip9 of $listed listed ports"
  else fail V09 "--allow-cdn-ports turns the cap off" "35.186.224.25 kept ${ip9:-0} ports while InternetDB lists $listed"; fi
fi
summary

@en0f

en0f commented Nov 21, 2026

Copy link
Copy Markdown

📝 Optional, not a merge condition: a negative cdn_port_filter.max_ports cuts every host instead of turning the cap off.

scanner.py:295 stores int(max_ports or 0), which keeps -1, and the early return at scanner.py:1096 (len(ports) <= self.cdn_max_ports) is then never taken, so a host listing only [22, 80] comes back as [80]. CdnPortFilterConfig.max_ports is Optional[int] with no lower bound, and the docs only say 0 = off, so -1 is an easy guess for "off". Either treat max_ports <= 0 as off (max(0, int(...))) or reject it with Field(ge=0).

Repro at 5f0852a: scan = bbot_scanner("1.2.3.4", config={"cdn_port_filter": {"max_ports": -1}}) then scan.filter_catch_all_ports("1.2.3.4", [22, 80]) returns [80] (V12 in the review's verify script).

@liquidsec
liquidsec merged commit 8c97ba2 into dev Nov 21, 2026
27 checks passed
@liquidsec
liquidsec deleted the shodan-idb-catch-all-ports branch November 21, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants