Skip to content

Commit ca059e5

Browse files
yaythomasyaythomas
committed
test: run the JS SDK examples on the local runner
Add .github/scripts/js_examples/run.py. It runs the JS SDK's example tests against the testing package's local runner, and exits non-zero on any failure. It uses a JS SDK checkout from --js-dir or JS_SDK_DIR, or clones JS main. Run it with hatch run dev-testing:js-examples. run.py serves WebRunner in a child process, so each server session starts with no runner state. A small proxy on a thread turns the tests' Lambda Invoke into POST /start-durable-execution. The Lambda shim runs each function on reusable worker threads. When an invocation passes the function's Timeout, the shim terminates the worker, as Lambda does. step/interrupted-no-retry depends on this. The js-examples workflow runs the harness on changes to the testing package. All 121 test files with cloud tests pass in about 2.5 minutes on 4 CPUs.
1 parent 631b78d commit ca059e5

10 files changed

Lines changed: 2910 additions & 1 deletion

File tree

‎.github/scripts/js_examples/README.md‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
# JS examples against the local runner
2+
3+
The JS SDK repository ([aws/aws-durable-execution-sdk-js]) has about 125
4+
example tests. Each test runs a durable function and checks its result and
5+
history. The JS repository runs them against real Lambda. This directory runs
6+
the same tests against the Python local runner in
7+
`packages/aws-durable-execution-sdk-python-testing`. Everything runs on your
8+
machine. No AWS account or credentials are needed.
9+
10+
Run it after you change the testing package. The same script runs in CI
11+
(`.github/workflows/js-examples.yml`) on pull requests that touch the testing
12+
package.
13+
14+
[aws/aws-durable-execution-sdk-js]: https://github.com/aws/aws-durable-execution-sdk-js
15+
16+
## Quick start
17+
18+
You need `git`, [hatch](https://hatch.pypa.io/), and Node.js 22 or newer with
19+
`npm`. Run from the repository root:
20+
21+
```sh
22+
hatch run dev-testing:js-examples # every example, about 2.5 minutes
23+
hatch run dev-testing:js-examples invoke # only tests whose path matches "invoke"
24+
```
25+
26+
`dev-testing` is the hatch environment for the testing package. It has the SDK
27+
and the testing package from your checkout, installed in editable mode, so a
28+
run always tests your current source.
29+
30+
To use a JS SDK checkout you already have, pass it with `--js-dir`, or set
31+
`JS_SDK_DIR` once in your shell:
32+
33+
```sh
34+
export JS_SDK_DIR=~/github/aws-durable-execution-sdk-js
35+
hatch run dev-testing:js-examples invoke
36+
```
37+
38+
The harness uses that checkout as it is. It does not fetch, switch branches or
39+
clone. It runs `npm ci` and builds the example bundles when the checkout's
40+
source differs from the last build: a new commit, or an uncommitted change
41+
to a tracked or untracked file.
42+
43+
Without `--js-dir`, the harness clones the JS SDK's `main` branch into
44+
`~/.cache/dex-js-examples/js-sdk` on the first run, and builds it. Each later
45+
run fetches `main` again, and rebuilds only when `main` has moved.
46+
47+
The last lines of the output give the result:
48+
49+
```text
50+
[js-examples] selected 125 test files: 121 passed (0 on retry), 0 failed, 4 have no cloud tests, 0 not run
51+
```
52+
53+
The script exits 0 when every selected test passes and 1 otherwise. It exits
54+
2 when setup fails, for example when the interpreter running it does not have
55+
the testing package from this checkout.
56+
57+
## Common tasks
58+
59+
Each command below follows `hatch run dev-testing:js-examples`.
60+
61+
| Task | Options |
62+
| --- | --- |
63+
| Run tests whose path matches a regex | `step/ 'wait-for-callback/.*heartbeat'` |
64+
| Test against another JS SDK branch, tag or commit | `--js-ref v1.2.0` |
65+
| Use your own JS SDK checkout | `--js-dir ~/github/aws-durable-execution-sdk-js` |
66+
| Skip the JS build for that checkout | `--js-dir ~/github/aws-durable-execution-sdk-js --no-build` |
67+
| Force a JS rebuild | `--rebuild` |
68+
| See runner debug logs | `--log-level DEBUG hello-world` |
69+
| Pass options to jest | `invoke --jest-args='--verbose'` |
70+
| All options | `--help` |
71+
72+
`--js-ref` works only with the cloned JS SDK, so it cannot be combined with
73+
`--js-dir` or `JS_SDK_DIR`. With `--js-dir`, the harness writes only build
74+
output into your checkout. Its lock and build stamp live in
75+
`~/.cache/dex-js-examples`.
76+
77+
A run holds its JS checkout's lock until its tests finish. So a second run on
78+
the same checkout waits for the first. Runs on different checkouts, for
79+
example the cloned SDK and your own, run at the same time.
80+
81+
## When a test fails
82+
83+
Each run writes its files to a new directory under
84+
`~/.cache/dex-js-examples/runs/`, so runs from two worktrees never overwrite
85+
each other. `runs/latest` points at the newest run, and only the last 10 runs
86+
are kept. With `--out`, a run writes to that directory instead. It empties the
87+
directory first, so it refuses one that has files in it and was not created by
88+
an earlier run. It also refuses a directory that another run is still using.
89+
90+
| File | Contents |
91+
| --- | --- |
92+
| `report.json` | Passed, failed, flaky and skipped test files |
93+
| `runner.log` | The Python local runner, which is the code under test |
94+
| `shim.log` | One line per handler invocation: request ID, function, duration, outcome |
95+
| `jest.json`, `retry1.json` | Raw jest results, for the first run and the retry |
96+
| `maps/` | The function maps generated from the examples' `template.yml` |
97+
98+
To investigate one example, run it alone with debug logs:
99+
100+
```sh
101+
hatch run dev-testing:js-examples --log-level DEBUG --retries 0 --jest-args='--verbose' step/interrupted-no-retry
102+
```
103+
104+
Add `--dump-dir /tmp/dump` to also save every history, state and checkpoint
105+
response the tests read.
106+
107+
A test file that fails is run once more on fresh servers. If it then passes,
108+
the run succeeds and the file is reported as `FLAKY`. A few examples race real
109+
timers, for example parallel branches that must checkpoint within
110+
milliseconds of each other, and they can fail on a loaded machine. Use
111+
`--retries 0` to see every failure.
112+
113+
## How it works
114+
115+
The Python local runner is the backend, in place of the Lambda service. Three
116+
helpers run around it, on the loopback, on free ports chosen for each run:
117+
118+
```text
119+
jest ----> invoke_proxy.py ----> local runner ----> lambda-shim.cjs
120+
(test (starts executions) (the code under (runs the JS handlers,
121+
driver) test) like the Lambda runtime)
122+
```
123+
124+
1. **jest** runs the JS example tests unchanged, in the mode the JS repository
125+
uses against real Lambda (`NODE_ENV=integration`). Each test is a client:
126+
it starts an execution, then reads history and state, and some tests send
127+
callbacks. `LAMBDA_ENDPOINT` points the tests' Lambda client at the proxy.
128+
2. **invoke_proxy.py** exists because the tests start an execution with Lambda
129+
`Invoke`, and the local runner has no `Invoke` route. The proxy turns that
130+
call into the runner's `POST /start-durable-execution`, and returns the
131+
execution ARN in the header the JS SDK reads. It forwards every other
132+
request to the runner unchanged.
133+
3. **The local runner** runs the executions. To run a handler, it calls
134+
Lambda `Invoke` on its Lambda endpoint, which is the shim. It resolves
135+
chained-invoke targets from the function configs `run.py` gives it.
136+
4. **lambda-shim.cjs** hosts the built example bundles, as the Lambda runtime
137+
does in AWS. It runs each function on worker threads. A worker handles one
138+
invocation at a time and is reused afterwards, like a warm Lambda sandbox.
139+
When an invocation runs past the function's `Timeout`, the shim returns
140+
Lambda's `Sandbox.Timedout` error and terminates the worker.
141+
`step/interrupted-no-retry` depends on this. Inside the handler, the durable
142+
SDK checkpoints straight to the runner, because the shim sets
143+
`AWS_ENDPOINT_URL_LAMBDA` to the runner's address.
144+
145+
`run.py` starts the runner with `WebRunner` from the testing package, in a
146+
child process, so each server session gets a runner with no state left from
147+
an earlier one. A run has one session, plus one per retry. The proxy runs on a
148+
thread in `run.py`, and the shim and jest run as separate Node processes.
149+
Before the run, `run.py` reads the examples' `template.yml` and writes three maps: test file to
150+
function name (for jest), function name to bundle, timeout and environment
151+
(for the shim), and function name to `DurableConfig` (for the runner).
152+
153+
The runner emulates region `us-west-2` and account `123456789012`. The proxy
154+
and the shim use the same values. If they differ, the runner rejects chained
155+
invokes whose target ARN names another region or account.
156+
157+
## Which tests run
158+
159+
Every `*.test.ts` under the examples' `src/examples` is selected, except for
160+
three groups:
161+
162+
- `otel/` examples. They export spans to an OpenTelemetry collector, which the
163+
harness does not run.
164+
- Examples marked `localOnly` in the JS catalog. `template.yml` omits them,
165+
and the JS repository's integration run skips them too. The output reports
166+
how many were skipped.
167+
- Tests without "cloud" in their name. The integration jest config runs only
168+
those. A file with no such test is reported as "no cloud tests".
169+
170+
## Known failures
171+
172+
`known-failures.txt` lists test files that are expected to fail, one per line,
173+
with the reason after `#`. It is empty today. A listed file that fails does
174+
not fail the run. A listed file that passes does fail the run, so that the
175+
list stays accurate.
176+
177+
## A failure caused by a JS change
178+
179+
Every run tests against the JS SDK's `main` branch. So a run can fail because
180+
of a new JS commit, with no Python change. To check, run the failing test
181+
against an older JS commit:
182+
183+
```sh
184+
hatch run dev-testing:js-examples --js-ref <older JS commit> <failing test>
185+
```
186+
187+
If it passes there, the new JS commit exposed something the runner does not
188+
handle yet.
189+
190+
## Harness tests
191+
192+
`.github/scripts/tests/test_js_examples.py` tests the template parsing, test
193+
selection and result handling in `run.py`. It needs no JS SDK:
194+
195+
```sh
196+
hatch run dev-testing:python -m pytest .github/scripts/tests/test_js_examples.py
197+
```

0 commit comments

Comments
 (0)