Repository navigation
ci: CLA caller v1.3.0 on stable - #3659
Conversation
Puts dev's cla.yml (release-workflows cla-reusable.yml v1.3.0) on stable, the default branch, which is the copy pull_request_target and issue_comment run. The old copy saves signatures on CLA main, which no longer accepts writes, so outside contributors cannot sign until this lands.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## stable #3659 +/- ##
======================================
+ Coverage 91% 91% +1%
======================================
Files 465 465
Lines 51927 51927
======================================
+ Hits 47031 47048 +17
+ Misses 4896 4879 -17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
en0f
left a comment
There was a problem hiding this comment.
Reviewed at adda64f.
🟢 Signature saves move off CLA's write-refused main, in the copy the triggers actually load
issue_comment and pull_request_target load stable's cla.yml, so only this copy decides where a signature is saved. Today it calls CLA/cla-reusable.yml@4f85d3c, which saves on CLA's main, and run 44293705592 this morning failed with Could not update the JSON file: Repository rule violations found. v1.3.0 saves on the signatures branch instead (V12). Targeting stable directly is the right exception to "target dev": the file is byte-identical to dev's (blob 31bc4d38, V07), so the next release merges nothing in it.
🟢 Least privilege for a pull_request_target job
secrets: inherit is gone. The caller passes exactly APP_ID and APP_PRIVATE_KEY, the two secrets v1.3.0 declares, and both reach bbot as org secrets. contents: read is dropped because the called job asks only for pull-requests: write and statuses: write, which is exactly what's granted here. The called job checks out no code, and the pin is tag v1.3.0's commit (ci.md § Permissions, § Pin everything to a SHA; V01 to V06).
📝 Test artifact pending: the first signature saved on CLA's signatures branch
No run of this caller can exist before merge, and nowhere in the org has v1.3.0 saved a signature yet: signatures holds only its seed commit f8010baf (V11), and the one v1.3.0 run that minted the App token (MANSPIDER 43966084823, V10) took the all-exempt path. The save needs the App to be a bypass actor on CLA's signatures-bot-only ruleset, and my token can't see bypass actors. Nothing is needed in this PR. Confirm the App is a bypass actor there, or link the first post-merge run that saves an outside contributor's signature. I'll check that run.
Reproduce this review
# save the script below as verify-3659.sh, then:
bash verify-3659.sh blacklanternsecurity/bbot adda64f3ab18b4a059e77d82ad118d55c3c40f75
# expected: SUMMARY 12 PASS, 2 FAIL, 0 SKIP, 0 BROKEN (V11 until a signature is saved; V14 is release-workflows' 100-commit cap)
# control: bash verify-3659.sh blacklanternsecurity/bbot beca7d2fadeae998acb23b6c5c375f42acceb316 -> V01 V02 V03 V04 V07 V08 V12 FAIL#!/usr/bin/env bash
# verify-3659.sh: reproducible checks for blacklanternsecurity/bbot#3659
# Usage: bash verify-3659.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
# Reads the called reusable workflow, org secrets names, rules and CI runs via gh api --method GET.
#
# CONTROL: sha beca7d2fadeae998acb23b6c5c375f42acceb316
# A fix to stable's cla.yml: the base carries the old caller.
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V04=FAIL V07=FAIL V08=FAIL V12=FAIL
# AUTHOR-CODE: none
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
BASE=beca7d2fadeae998acb23b6c5c375f42acceb316
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
echo "jq $(jq --version)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
CALLER=.github/workflows/cla.yml
[ -f "$CALLER" ] || { broken V00 "caller exists" "$CALLER missing"; summary; exit 1; }
RW=blacklanternsecurity/release-workflows
USES=$(awk '/^[[:space:]]*uses:[[:space:]]/{sub(/^[[:space:]]*uses:[[:space:]]*/,""); sub(/[[:space:]]*#.*/,""); print; exit}' "$CALLER")
PIN=${USES##*@}
echo "info caller uses: $USES"
# The called workflow, at whatever ref the caller pins
CALLED="$WORK/called.yml"
CREPO=$(cut -d/ -f1-2 <<<"${USES%@*}"); CPATH=$(cut -d/ -f3- <<<"${USES%@*}")
gh api --method GET "repos/$CREPO/contents/$CPATH" -f ref="$PIN" -H 'Accept: application/vnd.github.raw+json' > "$CALLED" 2>/dev/null || : > "$CALLED"
[ -s "$CALLED" ] || { broken V00 "read called workflow" "could not read $USES"; summary; exit 1; }
echo "info called_sha256 $(shasum -a 256 "$CALLED" | cut -d' ' -f1)"
# V01: the caller pins release-workflows' cla-reusable.yml to the full SHA that tag v1.3.0 names
TAG_SHA=$(gh api --method GET "repos/$RW/git/ref/tags/v1.3.0" --jq '.object.sha' 2>/dev/null || true)
if [ -z "$TAG_SHA" ]; then skip V01 "cla.yml pins release-workflows cla-reusable.yml to the v1.3.0 commit" "could not read tag v1.3.0"
elif [ "${USES%@*}" = "$RW/.github/workflows/cla-reusable.yml" ] && [ "$PIN" = "$TAG_SHA" ]; then
pass V01 "cla.yml pins release-workflows cla-reusable.yml to the v1.3.0 commit ($TAG_SHA)"
else fail V01 "cla.yml pins release-workflows cla-reusable.yml to the v1.3.0 commit" "uses=$USES tag=$TAG_SHA"; fi
# V02: no secrets: inherit into a pull_request_target job
if grep -Eq '^[[:space:]]*secrets:[[:space:]]*inherit' "$CALLER"; then fail V02 "cla.yml does not pass secrets: inherit" "secrets: inherit present"
else pass V02 "cla.yml does not pass secrets: inherit"; fi
# V03: the caller passes exactly the secrets the called workflow declares under workflow_call
DECL=$(awk '/^ workflow_call:/{w=1;next} w&&/^ secrets:/{s=1;next} s&&/^ [A-Za-z_]+:/{gsub(/[ :]/,"");print;next} s&&/^ [a-z]/{exit} /^[a-z]/&&w{exit}' "$CALLED" | sort | tr '\n' ' ')
PASSED=$(awk '/^ secrets:[[:space:]]*$/{s=1;next} s&&/^ [A-Za-z_]+:/{sub(/^ /,"");sub(/:.*/,"");print;next} s{exit}' "$CALLER" | sort | tr '\n' ' ')
if [ -z "$DECL" ]; then fail V03 "cla.yml passes exactly the secrets the called workflow declares" "called workflow declares none; passed=[$PASSED]"
elif [ "$DECL" = "$PASSED" ]; then pass V03 "cla.yml passes exactly the secrets the called workflow declares ($DECL)"
else fail V03 "cla.yml passes exactly the secrets the called workflow declares" "declared=[$DECL] passed=[$PASSED]"; fi
# V04: the called job declares its permissions, and the caller grants exactly those (least privilege, ci.md)
CALLER_PERMS=$(awk '/^permissions:/{p=1;next} p&&/^ [a-z-]+:/{gsub(/ /,"");print;next} p&&!/^ /{exit}' "$CALLER" | sort | tr '\n' ' ')
JOB_PERMS=$(awk '/^jobs:/{j=1} j&&/^ permissions:/{p=1;next} p&&/^ [a-z-]+:/{gsub(/ /,"");print;next} p&&!/^ /{exit}' "$CALLED" | sort | tr '\n' ' ')
if [ -z "$JOB_PERMS" ]; then fail V04 "caller grants exactly the called job's declared permissions" "called job declares none; caller=[$CALLER_PERMS]"
elif [ "$JOB_PERMS" = "$CALLER_PERMS" ]; then pass V04 "caller grants exactly the called job's declared permissions ($CALLER_PERMS)"
else fail V04 "caller grants exactly the called job's declared permissions" "caller=[$CALLER_PERMS] job=[$JOB_PERMS]"; fi
# V05: the called workflow checks out no code (pull_request_target safety, ci.md)
if grep -Eq 'actions/checkout|git (clone|checkout|fetch)' "$CALLED"; then fail V05 "called workflow checks out no code" "$(grep -Ein 'actions/checkout|git (clone|checkout|fetch)' "$CALLED" | head -2 | tr '\n' ' ')"
else pass V05 "called workflow checks out no code"; fi
# V06: APP_ID and APP_PRIVATE_KEY reach this repository as org secrets
NAMES=$(gh api --method GET "repos/$REPO/actions/organization-secrets" --jq '[.secrets[].name]|sort|join(" ")' 2>/dev/null || true)
if [ -z "$NAMES" ]; then skip V06 "APP_ID and APP_PRIVATE_KEY reach this repository" "org secrets list unreadable with this token"
elif grep -qw APP_ID <<<"$NAMES" && grep -qw APP_PRIVATE_KEY <<<"$NAMES"; then pass V06 "APP_ID and APP_PRIVATE_KEY reach this repository as org secrets"
else fail V06 "APP_ID and APP_PRIVATE_KEY reach this repository" "org secrets: $NAMES"; fi
# V07: this cla.yml is byte-identical to dev's (so the next dev->stable release has nothing to merge in it)
"${GIT[@]}" fetch --quiet origin "+refs/heads/dev:refs/remotes/origin/dev" || true
DEV_BLOB=$(git rev-parse --verify --quiet "origin/dev:$CALLER" || true); HEAD_BLOB=$(git hash-object "$CALLER")
if [ -z "$DEV_BLOB" ]; then skip V07 "cla.yml equals dev's cla.yml" "could not read dev"
elif [ "$DEV_BLOB" = "$HEAD_BLOB" ]; then pass V07 "cla.yml equals dev's cla.yml (blob ${HEAD_BLOB:0:8})"
else fail V07 "cla.yml equals dev's cla.yml" "here ${HEAD_BLOB:0:8}, dev ${DEV_BLOB:0:8}"; fi
# V08: against the base, the change touches only .github/workflows/cla.yml
"${GIT[@]}" cat-file -e "$BASE^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "$BASE" || true
CHANGED=$(git diff --name-only "$BASE" HEAD 2>/dev/null | tr '\n' ' ')
if [ "$CHANGED" = "$CALLER " ]; then pass V08 "the change touches only $CALLER"
else fail V08 "the change touches only $CALLER" "changed=[$CHANGED]"; fi
# V09: no required status check on stable or dev names the old job check, so the job rename blocks nothing
REQ=""
for BR in stable dev; do
R=$(gh api --method GET "repos/$REPO/rules/branches/$BR" --jq '[.[] | select(.type=="required_status_checks") | .parameters.required_status_checks[].context] | join("|")' 2>/dev/null) || { REQ="ERR"; break; }
REQ="$REQ $BR:[$R]"
done
if [ "$REQ" = "ERR" ]; then skip V09 "no required check names 'check the CLA is signed / CLAAssistant'" "rules unreadable"
elif grep -q 'check the CLA is signed' <<<"$REQ"; then fail V09 "no required check names 'check the CLA is signed / CLAAssistant'" "$REQ"
else pass V09 "no required check names 'check the CLA is signed / CLAAssistant' ($REQ )"; fi
# V10: a run of release-workflows cla-reusable.yml@v1.3.0 in this org minted the App token with these secrets
# (MANSPIDER run 43966084823, pull_request_target, the only v1.3.0 run found that reached the token step)
MRUN=43966084823
MREF=$(gh api --method GET "repos/blacklanternsecurity/MANSPIDER/actions/runs/$MRUN" --jq '[.referenced_workflows[]?.path]|join(",")' 2>/dev/null || true)
MTOK=$(gh api --method GET "repos/blacklanternsecurity/MANSPIDER/actions/runs/$MRUN/jobs" --jq '[.jobs[].steps[] | select(.name=="Generate token from GitHub App") | .conclusion] | join(",")' 2>/dev/null || true)
if [ -z "$MREF" ] || [ -z "$MTOK" ]; then skip V10 "an org run of cla-reusable.yml@v1.3.0 minted the App token" "could not read MANSPIDER run $MRUN"
elif grep -q "cla-reusable.yml@$TAG_SHA" <<<"$MREF" && [ "$MTOK" = "success" ]; then pass V10 "an org run of cla-reusable.yml@v1.3.0 minted the App token (MANSPIDER run $MRUN)"
else fail V10 "an org run of cla-reusable.yml@v1.3.0 minted the App token" "ref=$MREF token_step=$MTOK"; fi
# V11: the App has saved a signature to CLA's signatures branch (a commit after the seed f8010baf)
NSIG=$(gh api --method GET "repos/blacklanternsecurity/CLA/commits?sha=signatures&per_page=100" --jq '[.[] | select(.sha | startswith("f8010baf") | not)] | length' 2>/dev/null || true)
if [ -z "$NSIG" ]; then skip V11 "the App has saved a signature to CLA's signatures branch" "could not read CLA signatures"
elif [ "$NSIG" -gt 0 ]; then pass V11 "the App has saved a signature to CLA's signatures branch ($NSIG commits after the seed)"
else fail V11 "the App has saved a signature to CLA's signatures branch" "only the seed commit f8010baf; no run has saved one yet"; fi
# V12: the called workflow saves signatures on CLA's signatures branch, not on main (main refuses writes)
BRANCH=$(awk '/^[[:space:]]*branch:[[:space:]]/{gsub(/["[:space:]]/,""); sub(/^branch:/,""); print; exit}' "$CALLED")
if [ "$BRANCH" = "signatures" ] && grep -q 'remote-repository-name: "CLA"' "$CALLED"; then pass V12 "called workflow saves signatures on CLA's signatures branch"
else fail V12 "called workflow saves signatures on CLA's signatures branch" "branch=${BRANCH:-unset}"; fi
# V13 (challenge C11): classic branch protection on stable and dev does not require the old job check either
CREQ=""
for BR in stable dev; do
R=$(gh api --method GET "repos/$REPO/branches/$BR" --jq '[.protection.required_status_checks.contexts[]?] | join("|")' 2>/dev/null) || { CREQ="ERR"; break; }
CREQ="$CREQ $BR:[$R]"
done
if [ "$CREQ" = "ERR" ]; then skip V13 "classic protection requires no 'check the CLA is signed' check" "branches unreadable"
elif grep -q 'check the CLA is signed' <<<"$CREQ"; then fail V13 "classic protection requires no 'check the CLA is signed' check" "$CREQ"
else pass V13 "classic protection requires no 'check the CLA is signed' check"; fi
# V14 (challenge C12): the called workflow can pass a pull request with more than 100 commits, as bbot's
# dev->stable release merges have (346 at beca7d2f, 121 at eabac72a, 1407 at 5355bd1f)
"${GIT[@]}" fetch --quiet origin eabac72a4 5355bd1f9 2>/dev/null || true
RC=$(for c in "$BASE" eabac72a4 5355bd1f9; do git rev-list --count "$c^1..$c^2" 2>/dev/null; done | tr '\n' ' ')
if grep -Eq '"\$N_COMMITS" -gt 100' "$CALLED"; then fail V14 "called workflow passes pull requests with more than 100 commits" "exit 1 when commits > 100, before exemption; recent release merges: $RC"
else pass V14 "called workflow passes pull requests with more than 100 commits"; fi
summary
liquidsec
left a comment
There was a problem hiding this comment.
Approving: this restores CLA signing on bbot. Outside contributors can't sign today: #3658 (into dev) still ran stable's old CLA/cla-reusable.yml@4f85d3c (run 44284637473) and failed. GitHub runs this workflow from the default branch, so dev's v1.3.0 copy has never run in bbot.
Please hold off on making cla / CLAAssistant required on stable or dev until release-workflows fixes the first two points below. The next release can't merge without a bypass until then.
- More than 100 commits fails the check. v1.3.0 exits on
N_COMMITS > 100, and that runs before the org-member exemption, so a dev→stable release opened by a maintainer fails too. The last two releases had 346 commits (#3584) and 121 (#3271). Paginating the commits (--paginate) instead of capping them fixes it. - Signing or
recheckcan't clear a failed check. After saving the signature, the action re-runs the PR's failed run, which needsactions: write. Neither this caller nor the reusable grants it, so the call returns 403 and is only logged. The contributor stays red until they push again. This was already the case with the old workflow, but it only matters once the check is required. - The check name changes from
check the CLA is signed / CLAAssistanttocla / CLAAssistant. Open PRs only report the old name, so each one needs a new push before a required check can pass. - A commit whose author email isn't linked to a GitHub account fails the whole check. One such commit in a release's hundreds blocks it.
Also: when the opener isn't an org member, v1.3.0 requires every commit author to have signed, members included. That's deliberate, since an author email can be spoofed. But #3638 includes a maintainer's merge commit from #3584, and no maintainer is in cla.json, so that PR will ask a maintainer to sign. Maintainers should sign the ICLA once.
Puts the v1.3.0 CLA caller that dev already carries (identical file) on stable, the default branch. CLA runs from the default branch's copy, and the old copy saves signatures where they can no longer be written, so outside contributors cannot sign until the next release (#3658 today). Only this file changes; the next release has no conflict on it. Org CLA rollout; same change as MANSPIDER#138 and badsecrets#472.
After merge:
cla / CLAAssistantbecomes required on stable and dev. Contributors whose signature failed to save commentrecheckor sign again.