Skip to content

Commit be487d2

Browse files
yaythomasyaythomas
committed
test: run the JS SDK examples on the local runner
Add scripts/js-examples/run.sh. It runs the JS SDK's example tests against the testing package's local runner. It creates a venv, fetches and builds the JS SDK at a pinned commit, runs the tests, and exits non-zero on any failure. The workflow runs it on changes to the testing package, and weekly against JS main. The Lambda shim runs each function on reusable worker threads. When an invocation passes the function's Timeout, the shim terminates the worker, as Lambda does. step/interrupted-no-retry depends on this. All 121 test files with cloud tests pass in about 2.5 minutes on 4 CPUs.
1 parent 631b78d commit be487d2

12 files changed

Lines changed: 1951 additions & 0 deletions

File tree

‎.github/workflows/js-examples.yml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
name: JS Examples
2+
3+
# Runs the JS SDK's example tests against the Python local runner in
4+
# packages/aws-durable-execution-sdk-python-testing. Everything runs on the
5+
# job's loopback: no AWS account or credentials are involved.
6+
#
7+
# The harness is scripts/js-examples/run.sh, the same script developers run
8+
# locally. See scripts/js-examples/README.md.
9+
#
10+
# Pull requests and pushes test against the JS SDK commit pinned in
11+
# scripts/js-examples/js-sdk.ref, so a failure points at the Python change.
12+
# The weekly run tests against the JS SDK's main branch, to find JS changes
13+
# the runner does not handle yet before the pin is moved.
14+
15+
on:
16+
push:
17+
branches: ["main"]
18+
paths:
19+
- "packages/aws-durable-execution-sdk-python-testing/**"
20+
- "scripts/js-examples/**"
21+
- ".github/workflows/js-examples.yml"
22+
pull_request:
23+
branches: ["main"]
24+
paths:
25+
- "packages/aws-durable-execution-sdk-python-testing/**"
26+
- "scripts/js-examples/**"
27+
- ".github/workflows/js-examples.yml"
28+
schedule:
29+
- cron: "17 6 * * 1"
30+
workflow_dispatch:
31+
inputs:
32+
js_ref:
33+
description: "JS SDK branch, tag or commit. Empty: the commit in scripts/js-examples/js-sdk.ref"
34+
required: false
35+
default: ""
36+
type: string
37+
38+
permissions:
39+
contents: read
40+
41+
concurrency:
42+
group: js-examples-${{ github.ref }}
43+
cancel-in-progress: true
44+
45+
jobs:
46+
js-examples:
47+
runs-on: ubuntu-latest
48+
timeout-minutes: 30
49+
steps:
50+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
51+
52+
- name: Choose JS SDK ref
53+
id: ref
54+
env:
55+
INPUT_REF: ${{ github.event.inputs.js_ref }}
56+
EVENT: ${{ github.event_name }}
57+
run: |
58+
if [ -n "$INPUT_REF" ]; then
59+
ref="$INPUT_REF"
60+
elif [ "$EVENT" = "schedule" ]; then
61+
ref=main
62+
else
63+
ref="$(cut -d' ' -f1 scripts/js-examples/js-sdk.ref)"
64+
fi
65+
echo "JS SDK ref: $ref"
66+
echo "ref=$ref" >> "$GITHUB_OUTPUT"
67+
68+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
69+
with:
70+
repository: aws/aws-durable-execution-sdk-js
71+
ref: ${{ steps.ref.outputs.ref }}
72+
path: js-sdk
73+
persist-credentials: false
74+
75+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
76+
with:
77+
python-version: "3.14"
78+
79+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
80+
with:
81+
# The JS SDK's CI runs on Node.js 22.
82+
node-version: "22.x"
83+
cache: npm
84+
cache-dependency-path: js-sdk/package-lock.json
85+
86+
- name: Unit tests for the harness
87+
run: |
88+
scripts/js-examples/run.sh --help > /dev/null
89+
scripts/js-examples/.venv/bin/python -m pytest scripts/js-examples
90+
91+
- name: Run the JS examples
92+
run: scripts/js-examples/run.sh --js-dir js-sdk --out "$RUNNER_TEMP/js-examples"
93+
94+
- name: Upload logs and report
95+
if: always()
96+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
97+
with:
98+
name: js-examples-logs
99+
path: ${{ runner.temp }}/js-examples
100+
if-no-files-found: ignore

‎CONTRIBUTING.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,19 @@ Mimic the package structure in the src/aws_durable_execution_sdk_python director
383383
Name your module so that src/mypackage/mymodule.py has a dedicated unit test file
384384
tests/mypackage/mymodule_test.py
385385

386+
### JS SDK examples against the local runner
387+
388+
If you change the testing package, also run the JS SDK's example tests
389+
against its local runner. The script needs Node.js 22 or newer and sets up
390+
everything else:
391+
392+
```
393+
scripts/js-examples/run.sh
394+
```
395+
396+
CI runs the same script on pull requests that change the testing package.
397+
See [scripts/js-examples/README.md](scripts/js-examples/README.md).
398+
386399
### Conformance tests
387400

388401
Conformance tests verify cross-SDK behavioral parity. The runner and requirement specifications are maintained in a separate repository: [aws-durable-execution-conformance-tests](https://github.com/aws/aws-durable-execution-conformance-tests). The Python handlers live in-repo under `packages/aws-durable-execution-sdk-python-conformance-tests/`, but new requirement IDs must first be registered upstream. Contributors do not need to add conformance tests. If you believe a change warrants one, mention it in your PR description or [open an issue](https://github.com/aws/aws-durable-execution-conformance-tests/issues/new?template=new_requirement.yml) in that repository.

‎scripts/js-examples/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.venv/
2+
.out/

‎scripts/js-examples/README.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# JS examples against the local runner
2+
3+
The JS SDK repository ([aws/aws-durable-execution-sdk-js]) has about 125
4+
example tests. Each test runs a durable function and checks its result and
5+
history. The JS repository runs them against real Lambda. This directory runs
6+
the same tests against the Python local runner in
7+
`packages/aws-durable-execution-sdk-python-testing`. Everything runs on your
8+
machine. No AWS account or credentials are needed.
9+
10+
Run it after you change the testing package. The same script runs in CI
11+
(`.github/workflows/js-examples.yml`) on pull requests that touch the testing
12+
package.
13+
14+
[aws/aws-durable-execution-sdk-js]: https://github.com/aws/aws-durable-execution-sdk-js
15+
16+
## Quick start
17+
18+
You need `git`, Python 3.11 or newer, and Node.js 22 or newer with `npm`.
19+
20+
```sh
21+
scripts/js-examples/run.sh # every example, about 2.5 minutes
22+
scripts/js-examples/run.sh invoke # only tests whose path matches "invoke"
23+
```
24+
25+
The first run takes a few more minutes. It does two setup steps:
26+
27+
1. It creates `scripts/js-examples/.venv` and installs the SDK and the testing
28+
package from your checkout. The install is editable, so later runs use
29+
your current source without reinstalling.
30+
2. It clones the JS SDK into `~/.cache/dex-js-examples/js-sdk` at the commit
31+
in `js-sdk.ref`, then runs `npm ci` and builds the example bundles. Later
32+
runs reuse the build until the commit changes.
33+
34+
The last lines of the output give the result:
35+
36+
```text
37+
[js-examples] selected 125 test files: 121 passed (0 on retry), 0 failed, 4 have no cloud tests, 0 not run
38+
```
39+
40+
The script exits 0 when every selected test passes and 1 otherwise. It exits
41+
2 when setup fails.
42+
43+
## Common tasks
44+
45+
| Task | Command |
46+
| --- | --- |
47+
| Run tests whose path matches a regex | `run.sh step/ 'wait-for-callback/.*heartbeat'` |
48+
| Test against another JS SDK branch, tag or commit | `run.sh --js-ref main` |
49+
| Use your own JS SDK checkout | `run.sh --js-dir ~/src/aws-durable-execution-sdk-js` |
50+
| Skip the JS build for that checkout | `run.sh --js-dir ~/src/aws-durable-execution-sdk-js --no-build` |
51+
| Force a JS rebuild | `run.sh --rebuild` |
52+
| See runner debug logs | `run.sh --log-level DEBUG hello-world` |
53+
| Pass options to jest | `run.sh invoke -- --verbose` |
54+
| All options | `run.sh --help` |
55+
56+
With `--js-dir`, the harness builds your checkout in place and changes nothing
57+
else in it. Its lock and build stamp live in `~/.cache/dex-js-examples`.
58+
59+
## When a test fails
60+
61+
Each run writes its files to `scripts/js-examples/.out/`, or to the directory
62+
given with `--out`:
63+
64+
| File | Contents |
65+
| --- | --- |
66+
| `report.json` | Passed, failed, flaky and skipped test files |
67+
| `runner.log` | The Python local runner, which is the code under test |
68+
| `shim.log` | One line per handler invocation: request ID, function, duration, outcome |
69+
| `proxy.log` | The invoke proxy |
70+
| `jest-*.json` | Raw jest results |
71+
| `maps/` | The function maps generated from the examples' `template.yml` |
72+
73+
To investigate one example, run it alone with debug logs:
74+
75+
```sh
76+
scripts/js-examples/run.sh --log-level DEBUG --retries 0 step/interrupted-no-retry -- --verbose
77+
```
78+
79+
Set `PROXY_DUMP_DIR=/tmp/dump` to also save every history, state and
80+
checkpoint response the tests read.
81+
82+
A test file that fails is run once more on fresh servers. If it then passes,
83+
the run succeeds and the file is reported as `FLAKY`. A few examples race real
84+
timers, for example parallel branches that must checkpoint within
85+
milliseconds of each other, and they can fail on a loaded machine. Use
86+
`--retries 0` to see every failure.
87+
88+
## How it works
89+
90+
Four processes run on the loopback, on free ports chosen for each run:
91+
92+
```text
93+
jest (CloudDurableTestRunner)
94+
-> invoke_proxy.py turns Lambda Invoke into POST /start-durable-execution;
95+
forwards every other request unchanged
96+
-> local runner from this checkout: the code under test
97+
-> lambda-shim.cjs answers the runner's Invoke calls by running the
98+
built example bundles
99+
```
100+
101+
1. jest runs the examples in integration mode (`NODE_ENV=integration`), the
102+
mode the JS repository uses against real Lambda. The JS test runner starts
103+
each execution with Lambda `Invoke` and reads the execution ARN from the
104+
response. `invoke_proxy.py` gives the local runner that endpoint.
105+
2. The local runner invokes handlers at its `--lambda-endpoint`, which is the
106+
shim. It resolves chained-invoke targets with `--function-configs`.
107+
3. `lambda-shim.cjs` implements Lambda `Invoke` for the example bundles. It
108+
runs each function on worker threads. A worker handles one invocation at a
109+
time and is reused afterwards, like a warm Lambda sandbox. When an
110+
invocation runs past the function's `Timeout`, the shim returns Lambda's
111+
`Sandbox.Timedout` error and terminates the worker. Some examples depend
112+
on this, for example `step/interrupted-no-retry`.
113+
4. `run.py` reads the examples' `template.yml` and writes the three maps the
114+
other processes need: test file to function name (for jest), function
115+
name to bundle, timeout and environment (for the shim), and function name
116+
to `DurableConfig` (for the runner).
117+
118+
The runner emulates region `us-west-2` and account `123456789012`. The proxy
119+
and the shim use the same values. If they differ, the runner rejects chained
120+
invokes whose target ARN names another region or account.
121+
122+
## Which tests run
123+
124+
Every `*.test.ts` under the examples' `src/examples` is selected, except for
125+
three groups:
126+
127+
- `otel/` examples. They export spans to an OpenTelemetry collector, which the
128+
harness does not run.
129+
- Examples marked `localOnly` in the JS catalog. `template.yml` omits them,
130+
and the JS repository's integration run skips them too. The output reports
131+
how many were skipped.
132+
- Tests without "cloud" in their name. The integration jest config runs only
133+
those. A file with no such test is reported as "no cloud tests".
134+
135+
## Known failures
136+
137+
`known-failures.txt` lists test files that are expected to fail, one per line,
138+
with the reason after `#`. It is empty today. A listed file that fails does
139+
not fail the run. A listed file that passes does fail the run, so that the
140+
list stays accurate.
141+
142+
## Updating the pinned JS SDK
143+
144+
Pull requests test against the commit in `js-sdk.ref`. So a failure on a pull
145+
request points at the Python change, not at a new JS commit. The weekly CI
146+
run tests against JS `main` instead. When it is green, move the pin:
147+
148+
```sh
149+
git ls-remote https://github.com/aws/aws-durable-execution-sdk-js.git refs/heads/main
150+
# write that commit into scripts/js-examples/js-sdk.ref, then:
151+
scripts/js-examples/run.sh
152+
```
153+
154+
## Harness tests
155+
156+
`test_run.py` tests the template parsing, test selection and result handling
157+
in `run.py`. It needs no JS SDK:
158+
159+
```sh
160+
scripts/js-examples/.venv/bin/python -m pytest scripts/js-examples
161+
```

‎scripts/js-examples/conftest.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""Make run.py importable from test_run.py.
2+
3+
The repository's pytest configuration uses --import-mode=importlib, which
4+
does not add a test file's directory to sys.path. So this adds it.
5+
"""
6+
7+
import sys
8+
from pathlib import Path
9+
10+
sys.path.insert(0, str(Path(__file__).resolve().parent))

0 commit comments

Comments
 (0)