Skip to content

Commit 723046f

Browse files
Mohanad49Mohanad49
committed
docs: rewrite the README around what the suite actually had to solve
The old version was a setup guide. It listed what was tested and how to install it and said nothing about why any of it was interesting, which for a suite whose target is a live rate-limited API is most of the value. Adds the rate-limit story with the real failure output, the reasoning behind not retrying every 403, a live Allure and TestPulse link, the CI badge, and the token scopes it actually needs - delete_repo was undocumented and its absence silently leaves repositories behind.
1 parent 9d01788 commit 723046f

1 file changed

Lines changed: 122 additions & 62 deletions

File tree

‎README.md‎

Lines changed: 122 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,90 +1,150 @@
11
# GitHub REST API — Automated Test Suite
22

3-
Automated API test suite for the [GitHub REST API](https://docs.github.com/en/rest) using **Python + Requests + PyTest**.
3+
[![GitHub API Tests](https://github.com/Mohanad49/github-api-tests/actions/workflows/api-tests.yml/badge.svg)](https://github.com/Mohanad49/github-api-tests/actions/workflows/api-tests.yml)
4+
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
5+
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
46

5-
## Tech Stack
7+
API test suite for the [GitHub REST API](https://docs.github.com/en/rest), built with
8+
**Python, requests and pytest**. 56 tests over repositories, issues, users and
9+
authentication, running nightly against the live API.
610

7-
| Layer | Tool |
8-
|----------------|---------------------------|
9-
| HTTP Client | `requests` |
10-
| Test Framework | `pytest` |
11-
| Schema Valid. | `jsonschema` |
12-
| Reporting | `allure-pytest`, `pytest-html` |
13-
| CI/CD | GitHub Actions |
11+
**[Allure report →](https://mohanad49.github.io/github-api-tests/)** ·
12+
**[Flake history on TestPulse →](https://testpulse-eight.vercel.app/suites/github-api)**
1413

15-
## What's Tested
14+
The target is a real, public, rate-limited API rather than a practice sandbox, which is
15+
the point: the interesting problems in API testing are the ones a sandbox does not have.
1616

17-
| Module | Coverage |
18-
|----------------------|--------------------------------------------------------------------------|
19-
| `test_repositories` | CRUD, schema validation, pagination, rate-limit headers, negative cases |
20-
| `test_issues` | Create, update, close, labels, comments, filtering, negative cases |
21-
| `test_users` | Authenticated & public user lookup, schema validation, bio update |
22-
| `test_auth` | Invalid/missing/malformed tokens, 401 body checks, public endpoints |
17+
---
2318

24-
## Setup
19+
## What it covers
2520

26-
```bash
27-
# 1. Clone & enter
28-
git clone <repo-url> && cd github-api-tests
21+
| Module | Coverage |
22+
|---|---|
23+
| `test_repositories` | CRUD, JSON Schema validation, pagination and `Link` headers, rate-limit headers, negative cases |
24+
| `test_issues` | Create, update, close, labels, comments, state filtering, negative cases |
25+
| `test_users` | Authenticated and public lookup, schema validation, an idempotent write |
26+
| `test_auth` | Invalid, missing and malformed tokens; 401 bodies; public endpoints |
27+
| `test_rate_limiting` | The suite's own throttling client — offline, no token needed |
2928

30-
# 2. Create & activate virtual env
31-
python -m venv venv
32-
source venv/bin/activate # Windows: venv\Scripts\activate
29+
Every response body that has a schema is validated against one in `schemas/`, so a test
30+
asserting `200` is also asserting the shape of what came back.
3331

34-
# 3. Install dependencies
35-
pip install -r requirements.txt
32+
## Three things worth reading the code for
33+
34+
### 1. The suite used to rate-limit itself, and blamed the API
3635

37-
# 4. Configure your token
38-
cp .env.example .env
39-
# Edit .env and paste your GitHub PAT (needs repo + user scopes)
36+
Every scheduled run failed for ten consecutive nights. The failures looked like this:
37+
38+
```
39+
E assert 403 == 201
40+
ERROR tests/test_issues.py::TestIssues::test_add_label_to_issue
41+
AssertionError: Repo creation failed — status 403: {"message":"You have exceeded a
42+
secondary rate limit and have been temporarily blocked from content creation..."}
4043
```
4144

42-
## Running Tests
45+
GitHub enforces two different limits. The **primary** one — 5,000 requests an hour — is
46+
the famous one, and this suite never came close to it. The **secondary** limits govern
47+
*rates*, and one of them caps how fast an account may create content. A repository is
48+
content. So is an issue, a comment, a label, and the initial commit `auto_init` makes.
49+
50+
The suite created a fresh repository for every test that needed one: nine repositories,
51+
nine initial commits and six issues, as fast as the network allowed. Then a nightly job
52+
was added that ran three more copies of the suite in parallel, all sharing one token.
53+
Four concurrent bursts of content creation is exactly what the limit exists to stop.
54+
55+
The fix has three parts, and only one of them is "send fewer requests":
56+
57+
- **`utils/api_client.py`** — `GitHubSession` paces writes a second apart (GitHub's own
58+
documented guidance), honours `Retry-After`, waits out an exhausted primary limit and
59+
backs off on secondary ones. The rate-limit helpers were already in this file. Nothing
60+
imported them.
61+
- **`conftest.py`** — one repository shared across the tests that only read from it. The
62+
cost is isolation, and the reason it is affordable is written down in the fixture
63+
rather than glossed over.
64+
- **`.github/workflows/api-tests.yml`** — the nightly repeats run one at a time. Three
65+
runs of one commit disagreeing only means something if nothing else about the runs
66+
differed, and "how many siblings were competing for the same token" is a difference.
67+
68+
### 2. The retry logic deliberately refuses to retry some 403s
69+
70+
GitHub returns `403` both for *you are going too fast* and for *you may not do that*,
71+
and this suite asserts on the second kind. A retry layer that cannot tell them apart
72+
turns a fast, correct negative test into a five-minute sleep ending in the same answer.
73+
74+
So `is_rate_limited()` requires positive evidence of throttling — a `Retry-After`
75+
header, an exhausted `x-ratelimit-remaining`, or the secondary-limit message in the body
76+
— and `tests/test_rate_limiting.py` pins that behaviour with a test named
77+
`test_plain_403_is_not_retried`. Those tests are offline: they construct responses
78+
rather than provoking real limits, because a test that abuses the API to prove it
79+
handles abuse is not one anyone can run on a fork.
80+
81+
### 3. The write test writes the same value back
82+
83+
`test_patch_authenticated_user_is_accepted` exercises `PATCH /user` by setting the bio to
84+
whatever it already is.
85+
86+
An earlier version set a test string and restored it on the next line — safe right up
87+
until the process dies between those two calls. A cancelled workflow or a runner timeout
88+
would leave a real, public profile advertising that a test suite writes to it, and on a
89+
nightly schedule that is several chances a week. Writing the current value back exercises
90+
the same endpoint, auth, status code and response shape with no window to clean up.
91+
92+
## Running it
4393

4494
```bash
45-
# All tests, verbose
95+
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
96+
pip install -r requirements.txt
97+
98+
cp .env.example .env # then paste a PAT with repo + user + delete_repo scopes
4699
pytest tests/ -v
100+
```
47101

48-
# Smoke tests only
49-
pytest tests/ -v -m smoke
102+
The suite creates and deletes private repositories under the authenticated account, so
103+
point it at a token you are happy to have do that. It needs three scopes: `repo` to
104+
create, `user` to exercise `PATCH /user`, and `delete_repo` — without the last one,
105+
cleanup silently 403s and leaves `test-repo-*` repositories behind.
50106

51-
# Negative tests only
52-
pytest tests/ -v -m negative
107+
Selected runs:
53108

54-
# Generate HTML report
55-
pytest tests/ -v --html=report.html --self-contained-html
109+
```bash
110+
pytest tests/ -v -m smoke # smoke only
111+
pytest tests/ -v -m negative # negative cases only
112+
pytest tests/test_rate_limiting.py # offline; no token, no network
56113

57-
# Generate Allure results
58-
pytest tests/ -v --alluredir=allure-results
59-
allure serve allure-results
114+
pytest tests/ --alluredir=allure-results && allure serve allure-results
60115
```
61116

62-
## Project Structure
117+
## Layout
63118

64119
```
65-
├── .github/workflows/api-tests.yml # CI pipeline
66-
├── tests/
67-
│ ├── test_repositories.py # Repo CRUD, schema, pagination
68-
│ ├── test_issues.py # Issue CRUD, labels, comments
69-
│ ├── test_users.py # User info, schema, bio update
70-
│ └── test_auth.py # Auth negative testing
71-
├── schemas/
72-
│ ├── repository.json # JSON Schema for repos
73-
│ ├── issue.json # JSON Schema for issues
74-
│ └── user.json # JSON Schema for users
75-
├── utils/
76-
│ ├── api_client.py # Reusable API client wrapper
77-
│ └── schema_validator.py # Schema loading & validation
78-
├── conftest.py # Shared fixtures (session, test_repo, test_issue)
79-
├── .env.example # Token template
80-
├── pytest.ini # Pytest config & markers
81-
└── requirements.txt # Pinned dependencies
120+
.github/workflows/api-tests.yml CI: nightly, plus repeat runs for flake detection
121+
tests/
122+
test_repositories.py repo CRUD, schema, pagination
123+
test_issues.py issue CRUD, labels, comments
124+
test_users.py user info, schema, idempotent write
125+
test_auth.py auth negative testing
126+
test_rate_limiting.py the throttling client itself (offline)
127+
schemas/ JSON Schemas: repository, issue, user
128+
utils/
129+
api_client.py GitHubSession — pacing, backoff, retry policy
130+
schema_validator.py schema loading and validation
131+
conftest.py shared fixtures
82132
```
83133

84-
## CI/CD
134+
## CI
135+
136+
Runs on push, on pull request, and nightly at 02:00 UTC. Each run:
137+
138+
1. Executes the suite and emits both Allure results and JUnit XML
139+
2. Publishes the Allure report to GitHub Pages
140+
3. Ingests the JUnit file into [TestPulse](https://github.com/Mohanad49/testpulse) —
141+
including on failure, because a red run is the data point that matters most
142+
143+
The nightly schedule additionally runs the same commit three more times, one after
144+
another. pytest cannot report retries the way Playwright can, so the only way to produce
145+
same-commit evidence of flakiness is to run one unchanged SHA several times and see
146+
whether the outcomes agree.
85147

86-
The GitHub Actions workflow (`.github/workflows/api-tests.yml`) runs on every push/PR to `main`:
148+
## License
87149

88-
1. Installs Python 3.11 + dependencies
89-
2. Runs the full test suite with Allure output
90-
3. Publishes an Allure report to `gh-pages`
150+
[MIT](LICENSE)

0 commit comments

Comments
 (0)