Skip to content

Commit bad4e28

Browse files
authored
test(google_genai): migrate golden coverage to vcr tests (#196)
Move the remaining Google GenAI scenarios that were only exercised by the manual golden suite into the integration test module with recorded VCR cassettes. The new coverage focuses on image and document inputs, prefill, short max-token truncation, tool-result follow-ups, and reasoning metrics while avoiding duplicate tests for flows already covered in the integration suite. Remove the now-redundant genai golden suite and trim the golden README so it only describes the remaining suites.
1 parent ed258f6 commit bad4e28

10 files changed

Lines changed: 1340 additions & 670 deletions

File tree

internal/golden/README.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,32 +17,32 @@ Each test suite validates:
1717

1818
## Running Tests
1919

20-
### Python Tests
20+
Run a specific golden suite from its directory, for example:
2121

2222
```bash
23-
cd genai-py-v1
24-
python google_genai.py
23+
cd langchain-py-v1
24+
python langchain.py
2525
```
2626

27-
## Requirements
27+
```bash
28+
cd pydantic-ai-v1
29+
python pydantic_ai_test.py
30+
```
2831

29-
Before running the tests, ensure you have the appropriate API keys set as environment variables:
3032

31-
- `BRAINTRUST_API_KEY` to log to braintrust
32-
- `GOOGLE_API_KEY` for Google Generative AI tests
33+
## Requirements
3334

34-
The tests will automatically log traces to Braintrust projects named `golden-python-genai`.
35+
Before running a suite, ensure you have the appropriate API keys set as environment variables for that provider, along with `BRAINTRUST_API_KEY` if you want to log traces to Braintrust.
3536

3637
## Contributing
3738

3839
### Adding a New Provider
3940

4041
To add tests for a new AI provider:
4142

42-
1. Use `google_genai.py` as a reference implementation
43-
2. Provide it as context to an LLM and ask it to create an equivalent file for the new provider
44-
3. Ensure all test cases are covered with provider-specific adaptations
45-
4. Follow the naming convention: `<provider>.py`
43+
1. Use an existing golden suite as a reference implementation
44+
2. Ensure all test cases are covered with provider-specific adaptations
45+
3. Follow the naming convention already used by the surrounding suites
4646

4747
### Adding New Feature Coverage
4848

@@ -52,4 +52,4 @@ When adding a new feature (like reasoning, extended context, or new modalities):
5252
2. Ensure consistency in test structure and naming across providers
5353
3. Update this README to document the new feature coverage
5454

55-
This ensures comprehensive validation across all supported providers and maintains test parity.
55+
This helps maintain broad feature coverage across the remaining golden suites.

0 commit comments

Comments
 (0)