Skip to content

Commit 16d7d74

Browse files
behinddwallsbehinddwalls
committed
refactor(test): consolidate testing infrastructure with Docker Compose
Migrate from manual server setup to Docker Compose-based hermetic testing. Consolidate tests under test/ directory with consistent patterns. Key changes: - New test structure: test/e2e/, test/integration/{gateway,orchestrator,extensions}/ - Container naming: sq-test-{context}-{unique-id} for parallel execution - Shared utilities in test/testutil/ (compose, docker, mysql, schema helpers) - Removed old {service}/integration_test/ directories and speculator service - Documentation: TESTING.md → doc/howto/, STRUCTURE.md → PROJECT_STRUCTURE.md - Makefile: alphabetical targets, new integration test targets, local-start/stop naming
1 parent c80effa commit 16d7d74

78 files changed

Lines changed: 3160 additions & 2563 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build_and_test.yml‎

Lines changed: 19 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Build and Test
1+
name: Build and Unit Tests
22

33
on:
44
push:
@@ -11,40 +11,30 @@ on:
1111
- synchronize
1212

1313
jobs:
14-
build-and-test:
14+
build:
1515
runs-on: ubuntu-latest
1616

1717
steps:
1818
- name: Checkout code
1919
uses: actions/checkout@v4
2020

21-
- name: Build all services
21+
- name: Check BUILD files are up to date
22+
run: |
23+
echo "Running Gazelle to check BUILD files..."
24+
make gazelle
25+
if ! git diff --quiet; then
26+
echo "❌ BUILD files are out of date!"
27+
echo ""
28+
echo "The following files were modified by Gazelle:"
29+
git diff --name-only
30+
echo ""
31+
echo "Please run 'make gazelle' locally and commit the changes."
32+
exit 1
33+
fi
34+
echo "✅ BUILD files are up to date"
35+
36+
- name: Build project
2237
run: make build
2338

2439
- name: Run unit tests
25-
run: make test || echo "No tests found yet"
26-
27-
- name: Start all servers
28-
run: make start-servers
29-
30-
- name: Run service integration tests
31-
run: make integration-test
32-
33-
- name: Run end-to-end tests
34-
run: make e2e-test
35-
36-
- name: Stop servers
37-
if: always()
38-
run: make stop-servers
39-
40-
- name: Display server logs on failure
41-
if: failure()
42-
run: |
43-
echo "=== Gateway logs ==="
44-
cat /tmp/gateway.log || echo "No gateway logs found"
45-
echo ""
46-
echo "=== Orchestrator logs ==="
47-
cat /tmp/orchestrator.log || echo "No orchestrator logs found"
48-
echo ""
49-
echo "=== Speculator logs ==="
50-
cat /tmp/speculator.log || echo "No speculator logs found"
40+
run: make test || echo "No unit tests found"

‎.github/workflows/e2e_test.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: E2E Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
types:
9+
- opened
10+
- reopened
11+
- synchronize
12+
13+
jobs:
14+
e2e:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Run E2E tests
21+
run: make e2e-test
22+
23+
- name: Display container logs on failure
24+
if: failure()
25+
run: |
26+
echo "=== Listing all Docker containers ==="
27+
docker ps -a --filter "name=sq-test-" || true
28+
echo ""
29+
echo "=== Dumping container logs ==="
30+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
31+
echo ">>> Logs for $container <<<"
32+
docker logs "$container" 2>&1 || true
33+
echo ""
34+
done

‎.github/workflows/integration_test_extension.yml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: Integration Tests - Extensions
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
types:
9+
- opened
10+
- reopened
11+
- synchronize
12+
13+
permissions:
14+
contents: read
15+
pull-requests: read
16+
17+
jobs:
18+
# Detect which extensions changed
19+
detect-changes:
20+
runs-on: ubuntu-latest
21+
outputs:
22+
counter: ${{ steps.filter.outputs.counter }}
23+
queue: ${{ steps.filter.outputs.queue }}
24+
storage: ${{ steps.filter.outputs.storage }}
25+
steps:
26+
- uses: actions/checkout@v4
27+
- uses: dorny/paths-filter@v3
28+
id: filter
29+
with:
30+
filters: |
31+
counter:
32+
- 'extension/counter/**'
33+
- 'test/integration/extension/counter/**'
34+
- 'entity/**'
35+
- '.github/workflows/integration_test_extension.yml'
36+
queue:
37+
- 'extension/queue/**'
38+
- 'test/integration/extension/queue/**'
39+
- 'entity/**'
40+
- '.github/workflows/integration_test_extension.yml'
41+
storage:
42+
- 'extension/storage/**'
43+
- 'test/integration/extension/storage/**'
44+
- 'entity/**'
45+
- '.github/workflows/integration_test_extension.yml'
46+
47+
# Counter extension tests
48+
test-counter:
49+
needs: detect-changes
50+
if: needs.detect-changes.outputs.counter == 'true'
51+
runs-on: ubuntu-latest
52+
steps:
53+
- uses: actions/checkout@v4
54+
55+
- name: Run counter extension tests
56+
run: ./tool/bazel test //test/integration/extension/counter/... --test_output=errors
57+
58+
- name: Display container logs on failure
59+
if: failure()
60+
run: |
61+
echo "=== Listing all Docker containers ==="
62+
docker ps -a --filter "name=sq-test-" || true
63+
echo ""
64+
echo "=== Dumping container logs ==="
65+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
66+
echo ">>> Logs for $container <<<"
67+
docker logs "$container" 2>&1 || true
68+
echo ""
69+
done
70+
71+
# Queue extension tests
72+
test-queue:
73+
needs: detect-changes
74+
if: needs.detect-changes.outputs.queue == 'true'
75+
runs-on: ubuntu-latest
76+
steps:
77+
- uses: actions/checkout@v4
78+
79+
- name: Run queue extension tests
80+
run: ./tool/bazel test //test/integration/extension/queue/... --test_output=errors
81+
82+
- name: Display container logs on failure
83+
if: failure()
84+
run: |
85+
echo "=== Listing all Docker containers ==="
86+
docker ps -a --filter "name=sq-test-" || true
87+
echo ""
88+
echo "=== Dumping container logs ==="
89+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
90+
echo ">>> Logs for $container <<<"
91+
docker logs "$container" 2>&1 || true
92+
echo ""
93+
done
94+
95+
# Storage extension tests
96+
test-storage:
97+
needs: detect-changes
98+
if: needs.detect-changes.outputs.storage == 'true'
99+
runs-on: ubuntu-latest
100+
steps:
101+
- uses: actions/checkout@v4
102+
103+
- name: Run storage extension tests
104+
run: ./tool/bazel test //test/integration/extension/storage/... --test_output=errors
105+
106+
- name: Display container logs on failure
107+
if: failure()
108+
run: |
109+
echo "=== Listing all Docker containers ==="
110+
docker ps -a --filter "name=sq-test-" || true
111+
echo ""
112+
echo "=== Dumping container logs ==="
113+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
114+
echo ">>> Logs for $container <<<"
115+
docker logs "$container" 2>&1 || true
116+
echo ""
117+
done

‎.github/workflows/integration_test_gateway.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: Integration Tests - Gateway Service
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
types:
9+
- opened
10+
- reopened
11+
- synchronize
12+
13+
jobs:
14+
test-gateway:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Run Gateway integration tests
21+
run: make integration-test-gateway
22+
23+
- name: Display container logs on failure
24+
if: failure()
25+
run: |
26+
echo "=== Listing all Docker containers ==="
27+
docker ps -a --filter "name=sq-test-" || true
28+
echo ""
29+
echo "=== Dumping container logs ==="
30+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
31+
echo ">>> Logs for $container <<<"
32+
docker logs "$container" 2>&1 || true
33+
echo ""
34+
done

‎.github/workflows/integration_test_orchestrator.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: Integration Tests - Orchestrator Service
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
types:
9+
- opened
10+
- reopened
11+
- synchronize
12+
13+
jobs:
14+
test-orchestrator:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Run Orchestrator integration tests
21+
run: make integration-test-orchestrator
22+
23+
- name: Display container logs on failure
24+
if: failure()
25+
run: |
26+
echo "=== Listing all Docker containers ==="
27+
docker ps -a --filter "name=sq-test-" || true
28+
echo ""
29+
echo "=== Dumping container logs ==="
30+
for container in $(docker ps -a --filter "name=sq-test-" --format "{{.Names}}"); do
31+
echo ">>> Logs for $container <<<"
32+
docker logs "$container" 2>&1 || true
33+
echo ""
34+
done

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ MODULE.bazel.lock
1010

1111
# Built binaries
1212
bin/
13+
.docker-bin/

‎BUILD.bazel‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,13 @@ load("@gazelle//:def.bzl", "gazelle")
77
# gazelle:resolve go github.com/uber/submitqueue/orchestrator/protopb //orchestrator/protopb
88
# gazelle:resolve go github.com/uber/submitqueue/speculator/protopb //speculator/protopb
99

10+
# Export marker files for test data dependencies (used by FindRepoRoot in tests)
11+
exports_files(
12+
[
13+
"MODULE.bazel",
14+
"go.mod",
15+
],
16+
visibility = ["//visibility:public"],
17+
)
18+
1019
gazelle(name = "gazelle")

0 commit comments

Comments
 (0)