GSoC2026 : Docs Refactoring (Issue #86)

databusclient/cli.py

@@ -333,7 +333,7 @@ def download(
 ):
     """
     Download datasets from databus, optionally using vault access if vault options are provided.
-    Supports on-the-fly compression format conversion using --convert-to and --convert-from options.
+    Supports on-the-fly compression format conversion using the --compression option.
     """
     # Determine auth method for manifest (never store the token itself)
     auth_method = None

doc/README.md

@@ -0,0 +1,7 @@
+# Documentation
+
+- [CLI Usage](cli-usage.md#cli-download) - Complete command-line documentation for download, deploy, delete, manifests, and workflows. Use the section links in the same file for download, deploy, delete, manifest, and workflow details.
+- [Module Usage](module-usage.md) - Python API examples for creating distributions, datasets, and deployments.
+- [Reproducible Download](examples/reproducible-download.md) - Record and replay a download with a JSON-LD manifest.
+- [Workflow Examples](examples/workflows/README.md) - Example download, deploy, delete, and manifest workflows.
+- [GSoC 2026](gsoc-2026/README.md) - Project overview and proposal.

examples/workflows/README.md → doc/examples/workflows/README.md

@@ -25,4 +25,4 @@ databusclient workflow run download-deploy.yml
 
 Workflows can write a unified manifest covering every step in two ways: pass `--manifest path.jsonld` on the command line, or set a top-level `manifest:` key inside the YAML file itself (the command-line flag takes priority if both are given). Several of the examples above use the YAML key. Every manifest file entry that came from a workflow step is tagged with `dbus:stepName`, so a multi-step run stays traceable to which step produced or failed on which file.
 
-See the main [README's Workflow section](../../README.md#cli-workflow) for the full YAML format, step chaining, error handling, and WebDAV deploy mode documentation.
+See the [CLI usage documentation](../../cli-usage.md#cli-workflow) for the full YAML format, step chaining, error handling, and WebDAV deploy mode documentation.

doc/gsoc-2026/README.md

@@ -0,0 +1,15 @@
+# GSoC 2026
+
+Hi, I'm Dhanashree Petare ([GitHub](https://github.com/DhanashreePetare)), and I contributed to this project as part of Google Summer of Code 2026, under the DBpedia organization.
+
+My project extended the Databus Python Client with reproducible, workflow-aware data operations, delivered across five milestones:
+
+1. **Format and Mapping Conversion Layer** - RDF triple, RDF quad, and tabular format conversion during download, bringing the Python client to feature parity with the Java client. [Download docs](../cli-usage.md#cli-download).
+2. **Structured Run Manifest System** - JSON-LD manifests recording operation parameters, file metadata, checksums, and execution results for every `download`, `deploy`, and `delete` run. [Manifest docs](../cli-usage.md#cli-manifest).
+3. **Manifest Replay and Summary** - re-executing a past operation from its saved manifest, and printing a readable console summary of any manifest. [Replay docs](../cli-usage.md#cli-manifest-replay) and [summary docs](../cli-usage.md#cli-manifest-summary).
+4. **Declarative Workflow Engine** - YAML-defined pipelines chaining `download`/`deploy`/`delete` steps, with step-to-step output chaining and per-step error handling (`fail`/`continue`/`retry`). [Workflow docs](../cli-usage.md#cli-workflow).
+5. **Workflow-Manifest Integration and Example Workflows** - a unified manifest covering an entire workflow run, an automatic console summary, and example workflows. [Workflow examples](../examples/workflows/README.md).
+
+My project proposal is available [here](proposal_DhanashreePetare.pdf).
+
+Thank you.

doc/module-usage.md

@@ -0,0 +1,82 @@
+# Module Usage
+
+The client exposes Python functions for creating distributions and datasets and deploying them programmatically.
+
+<a id="module-deploy"></a>
+### Deploy
+
+#### Step 1: Create lists of distributions for the dataset
+
+```python
+from databusclient import create_distribution
+
+# create a list
+distributions = []
+
+# minimal requirements
+# compression and filetype will be inferred from the path
+# this will trigger the download of the file to evaluate the shasum and content length
+distributions.append(
+    create_distribution(url="https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml", cvs={"type": "swagger"})
+)
+
+# full parameters
+# will just place parameters correctly, nothing will be downloaded or inferred
+distributions.append(
+    create_distribution(
+        url="https://example.org/some/random/file.csv.bz2",
+        cvs={"type": "example", "realfile": "false"},
+        file_format="csv",
+        compression="bz2",
+        sha256_length_tuple=("7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653", 367116)
+    )
+)
+```
+
+A few notes:
+
+* The dict for content variants can be empty ONLY IF there is just one distribution
+* There can be no compression if there is no file format
+
+#### Step 2: Create dataset
+
+```python
+from databusclient import create_dataset
+
+# minimal way
+dataset = create_dataset(
+  version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18",
+  title="Client Testing",
+  abstract="Testing the client....",
+  description="Testing the client....",
+  license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10",
+  distributions=distributions,
+)
+
+# with group metadata
+dataset = create_dataset(
+  version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18",
+  title="Client Testing",
+  abstract="Testing the client....",
+  description="Testing the client....",
+  license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10",
+  distributions=distributions,
+  group_title="Title of group1",
+  group_abstract="Abstract of group1",
+  group_description="Description of group1"
+)
+```
+
+NOTE: Group metadata is applied only if all group parameters are set.
+
+#### Step 3: Deploy to Databus
+
+```python
+from databusclient import deploy
+
+# to deploy something you just need the dataset from the previous step and an API key
+# API key can be found (or generated) at https://$$DATABUS_BASE$$/$$USER$$#settings
+deploy(dataset, "mysterious API key")
+```
+
+The API key can be found or generated in the Databus account settings.

run_all_conversion_tests.py → tests/manual/run_all_conversion_tests.py

@@ -1,6 +1,7 @@
 """
 Layer 2 Conversion Testing Script
 Tests every conversion combination systematically.
+Base fixture files live under tests/resources/ (base.ttl, base.nq, base.csv).
 Outputs go to test_outputs/ folder.
 Test file for testing with real datasets from databus.
 """
@@ -70,7 +71,7 @@ def run_test(test_id, description, func, input_file, output_file, *args):
 
 print("\n=== GROUP 1: RDF TRIPLE FORMAT CONVERSIONS ===\n")
 
-BASE_TTL = "test_outputs/base/base.ttl"
+BASE_TTL = "tests/resources/base.ttl"
 
 # T1: turtle -> ntriples (from base turtle file)
 t1_out = "test_outputs/triples/T1_turtle_to_ntriples/output.nt"
@@ -142,7 +143,7 @@ def run_test(test_id, description, func, input_file, output_file, *args):
 
 print("\n=== GROUP 2: RDF QUAD FORMAT CONVERSIONS ===\n")
 
-BASE_NQ = "test_outputs/base/base.nq"
+BASE_NQ = "tests/resources/base.nq"
 
 # Q1: nquads -> trig
 q1_out = "test_outputs/quads/Q1_nquads_to_trig/output.trig"
@@ -275,8 +276,8 @@ def run_test(test_id, description, func, input_file, output_file, *args):
 
 print("\n=== GROUP 3: TABULAR FORMAT CONVERSIONS ===\n")
 
-BASE_CSV = "test_outputs/base/base.csv"
-BASE_TSV = "test_outputs/base/base.tsv"
+BASE_CSV = "tests/resources/base.csv"
+BASE_TSV = "tests/resources/base.tsv"
 
 # TAB1: csv -> tsv
 tab1_out = "test_outputs/tabular/TAB1_csv_to_tsv/output.tsv"
@@ -306,19 +307,19 @@ def run_test(test_id, description, func, input_file, output_file, *args):
 print("\n=== GROUP 4: CLI END-TO-END (run these manually) ===\n")
 cli_tests = [
     "CLI1: turtle->ntriples from compressed Databus file",
-    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --convert-format ntriples --localdir ./test_outputs/cli/CLI1",
+    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --format ntriples --localdir ./test_outputs/cli/CLI1",
     "",
     "CLI2: turtle->rdf-xml from compressed Databus file",
-    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --convert-format rdf-xml --localdir ./test_outputs/cli/CLI2",
+    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --format rdf-xml --localdir ./test_outputs/cli/CLI2",
     "",
     "CLI3: turtle->ntriples + compression bz2->gz",
-    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --convert-format ntriples --convert-to gz --localdir ./test_outputs/cli/CLI3",
+    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --format ntriples --compression gz --localdir ./test_outputs/cli/CLI3",
     "",
     "CLI4: turtle->ntriples + compression bz2->xz",
-    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --convert-format ntriples --convert-to xz --localdir ./test_outputs/cli/CLI4",
+    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --format ntriples --compression xz --localdir ./test_outputs/cli/CLI4",
     "",
     "CLI5: unsupported cross-class error (expect ValueError)",
-    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --convert-format nquads --localdir ./test_outputs/cli/CLI5",
+    "  poetry run databusclient download \"https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=cy.ttl.bz2\" --format nquads --localdir ./test_outputs/cli/CLI5",
 ]
 for line in cli_tests:
     print(line)

tests/resources/base.csv

@@ -0,0 +1,6 @@
+@'
+id,name,country
+1,Leipzig,Germany
+2,Berlin,Germany
+3,Paris,France
+'@ | Out-File -FilePath tests\resources\base.csv -Encoding ascii

tests/resources/base.nq

@@ -0,0 +1,3 @@
+<http://dbpedia.org/resource/Leipzig> <http://www.w3.org/2000/01/rdf-schema#label> "Leipzig"@en <http://example.org/graph1> .
+<http://dbpedia.org/resource/Leipzig> <http://dbpedia.org/ontology/country> <http://dbpedia.org/resource/Germany> <http://example.org/graph1> .
+<http://dbpedia.org/resource/Germany> <http://www.w3.org/2000/01/rdf-schema#label> "Germany"@en <http://example.org/graph2> .

tests/resources/base.ttl

@@ -0,0 +1,8 @@
+@prefix dbo: <http://dbpedia.org/ontology/> .
+@prefix dbr: <http://dbpedia.org/resource/> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+
+dbr:Leipzig rdfs:label "Leipzig"@en .
+dbr:Leipzig dbo:country dbr:Germany .
+dbr:Leipzig dbo:populationTotal "593145"^^<http://www.w3.org/2001/XMLSchema#integer> .
+dbr:Germany rdfs:label "Germany"@en .