Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 0 additions & 18 deletions .buildkite/db_sync_full_sync.yml

This file was deleted.

18 changes: 0 additions & 18 deletions .buildkite/node-db-sync.yml

This file was deleted.

16 changes: 0 additions & 16 deletions .buildkite/node_sync_tests.yml

This file was deleted.

1 change: 0 additions & 1 deletion .github/workflows/node_mainnet_tx_count_per_epoch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ jobs:

- name: Get the automated tests results
env:
BUILDKITE_API_ACCESS_TOKEN: ${{ secrets.BUILDKITE_API_ACCESS_TOKEN }}
AWS_DB_USERNAME: ${{ secrets.AWS_DB_USERNAME }}
AWS_DB_PASS: ${{ secrets.AWS_DB_PASS }}
AWS_DB_NAME: ${{ secrets.AWS_DB_NAME }}
Expand Down
112 changes: 0 additions & 112 deletions .github/workflows/node_sync_test_genesis.yaml

This file was deleted.

46 changes: 0 additions & 46 deletions .github/workflows/weekly_node_sync_test.yaml

This file was deleted.

31 changes: 4 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,6 @@

Sync tests for `cardano-node` and `cardano-db-sync`.

## Documentation

| Doc | Purpose |
|-----|---------|
| [`doc/LEARNING.md`](doc/LEARNING.md) | Index of learning and troubleshooting docs |
| [`doc/SYNC_TEST_TROUBLESHOOTING.md`](doc/SYNC_TEST_TROUBLESHOOTING.md) | Buildkite, Postgres, Nix, stuck runs |
| [`history.md`](history.md) | Branch timeline, incidents, owner conventions |
| [`ISSUES_ENCOUNTERED.md`](ISSUES_ENCOUNTERED.md) | Known issues and fixes |
| [`CURSOR_GUIDE.md`](CURSOR_GUIDE.md) | Style and where to save chat takeaways |

## Setup (local, without Nix)

From the repository root:
Expand Down Expand Up @@ -105,15 +95,16 @@ pytest sync_tests/tests/ \
When db-sync tests are included, `test_dbsync_artifacts.py` generates a comprehensive
test results JSON and checks db-sync logs for errors and rollbacks.

Current CI artifact behavior (Buildkite `artifact_paths`):
GitHub Actions uploads these bundles when CI finishes:

- `sync_logs.zip`
- `sync_results.zip`
- `monitor.zip`

Bundle generation is CI-agnostic and keyed off standard CI variables
(`CI`, `GITHUB_ACTIONS`, `BUILDKITE`, `GITLAB_CI`, `CIRCLECI`). The runner uploads
the zips; local runs keep raw files under `test_workdir/` for debugging.
(`CI`, `GITHUB_ACTIONS`, `GITLAB_CI`, `CIRCLECI`). The workflow uploads the zips
with `actions/upload-artifact`; local runs keep raw files under `test_workdir/`
for debugging.

Results JSON includes enriched performance samples under `system_metrics`.

Expand Down Expand Up @@ -220,17 +211,3 @@ When using `test_workdir/cardano-db-sync/db_sync_performance_stats.json` (list o
- **Available graphs**: CPU, RSS, Combined Resources
- **Empty graphs**: Epoch Duration, Blocks per Epoch, Block Throughput
- **Recommendation**: Use full `test_workdir/db_sync_results.json` for complete graph set

## Document Index

| Document | Purpose |
|----------|---------|
| [`doc/LEARNING.md`](doc/LEARNING.md) | Short index: doc map, high-signal facts, debugging entry point |
| [`history.md`](history.md) | Branch timeline, local notes, *What Martin wants*, chat archive pointers |
| [`doc/SYNC_TEST_TROUBLESHOOTING.md`](doc/SYNC_TEST_TROUBLESHOOTING.md) | Stuck runs, Buildkite, disk / Postgres, Nix `cardano-cli`, agent variance, pipeline YAML (§11) |
| [`ISSUES_ENCOUNTERED.md`](ISSUES_ENCOUNTERED.md) | Short issue → fix index |
| [`CURSOR_GUIDE.md`](CURSOR_GUIDE.md) | Style, pytest, Buildkite, where to save chat takeaways |
| [`SYNC_TESTS_ARCHITECTURE.md`](SYNC_TESTS_ARCHITECTURE.md) | Module layout and pytest model |
| [`SYNC_TESTS_COMPREHENSIVE_GUIDE.md`](SYNC_TESTS_COMPREHENSIVE_GUIDE.md) | Setup, CLI, markers, Buildkite env mapping |
| [`NODE_SYNC_TEST_EXECUTION_FLOW.md`](NODE_SYNC_TEST_EXECUTION_FLOW.md) | Node-only pytest flow vs legacy script context |
| [`REFRACTOR_PLAN.md`](REFRACTOR_PLAN.md) | Planned refactors |
2 changes: 1 addition & 1 deletion sync_tests/scripts/postgres-start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ POSTGRES_DIR="${1:?"Need path to postgres dir"}"
POSTGRES_DIR="$(readlink -m "$POSTGRES_DIR")"

# Postgres refuses to start if the full Unix socket path exceeds ~107 bytes
# (directory + "/.s.PGSQL.<port>"). Long CI workdirs (e.g. Buildkite) need a
# (directory + "/.s.PGSQL.<port>"). Long CI workdirs need a
# short -k directory; PGDATA stays under POSTGRES_DIR/data.
_MAX_SOCK_PARENT_LEN=$((107 - 22))
SOCKET_DIR="$POSTGRES_DIR"
Expand Down
12 changes: 11 additions & 1 deletion sync_tests/tests/test_local_snapshot_restoration.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from __future__ import annotations

import datetime
import json
import logging
import os
import pathlib as pl
Expand All @@ -14,6 +15,7 @@
from _pytest.fixtures import FixtureRequest

from sync_tests.tests.conftest import SyncContext
from sync_tests.tests.test_snapshot_creation import snapshot_created # noqa: F401
from sync_tests.utils import db_sync
from sync_tests.utils import helpers
from sync_tests.utils import node
Expand All @@ -30,12 +32,14 @@
def local_restoration_result(
request: FixtureRequest,
sync_context: SyncContext,
snapshot_created: dict[str, tp.Any], # noqa: ARG001,F811
) -> tp.Generator[dict[str, tp.Any], None, None]:
"""Restore db-sync from a local snapshot, sync, and yield result data.

Args:
request: Pytest fixture request for CLI options.
sync_context: Shared session context.
snapshot_created: Fixture dependency that creates snapshot metadata first.

Yields:
Dict with restoration timing, tip data, and sync results.
Expand Down Expand Up @@ -75,7 +79,13 @@ def local_restoration_result(
db_sync.create_database()

# restore snapshot
snapshot_file = db_sync.get_buildkite_meta_data("snapshot_file")
snapshot_state_file = sync_context.workdir / "sync_session_state.json"
if not snapshot_state_file.exists():
msg = f"Snapshot metadata file not found: {snapshot_state_file}"
raise FileNotFoundError(msg)
with open(snapshot_state_file, encoding="utf-8") as state_fh:
snapshot_data = json.load(state_fh)
snapshot_file = snapshot_data["snapshot_file"]
LOGGER.info("Restoring from snapshot: %s", snapshot_file)
restoration_time = db_sync.restore_db_sync_from_snapshot(
config,
Expand Down
5 changes: 3 additions & 2 deletions sync_tests/tests/test_snapshot_creation.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,20 @@ def snapshot_created(
end_time = datetime.datetime.now(tz=datetime.timezone.utc)

snapshot_file = stage_2_result
db_sync.set_buildkite_meta_data("snapshot_file", snapshot_file)

creation_secs = int((end_time - start_time).total_seconds())
LOGGER.info("Snapshot creation time: %d seconds", creation_secs)

return {
snapshot_data = {
"snapshot_file": snapshot_file,
"stage_2_cmd": stage_2_cmd,
"stage_2_result": stage_2_result,
"creation_time_secs": creation_secs,
"start_time": start_time.strftime("%d/%m/%Y %H:%M:%S"),
"end_time": end_time.strftime("%d/%m/%Y %H:%M:%S"),
}
helpers.write_json_to_file(sync_context.workdir / "sync_session_state.json", snapshot_data)
return snapshot_data


class TestSnapshotCreation:
Expand Down
Loading