GSoC2026 : Docs Refactoring (Issue #86) - #87
Conversation
…oc/ folder for CLI/module/examples usage
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The proposal link attached is correct but not showing the document, I will update that with a drive link of proposal. |
|
|
||
| <a id="cli-manifest"></a> | ||
| ### Manifest | ||
|
|
||
| The `download`, `deploy`, and `delete` commands accept `--manifest PATH` to write a structured JSON-LD record of the operation. | ||
|
|
||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --manifest ./manifests/download-run.jsonld | ||
| ``` | ||
|
|
||
| The manifest records input parameters, file URLs, checksums, byte sizes, timestamps, and success or failure status. API keys and vault tokens are not stored. If the operation fails, the manifest includes a `dbus:operationError` record with the error type and message. Existing target paths are preserved by writing an auto-suffixed manifest path. | ||
|
|
||
| See [Reproducible Download](examples/reproducible-download.md) for a complete download recording and replay example. | ||
|
|
||
| <a id="cli-manifest-replay"></a> | ||
| #### Replay | ||
|
|
||
| Replay a recorded download, deploy, or delete operation with: | ||
|
|
||
| ```bash | ||
| databusclient manifest replay [OPTIONS] MANIFEST_PATH | ||
| ``` | ||
|
|
||
| Credentials are never stored in manifests and must be supplied again when needed. Use `--localdir` for download output, `--databus` for the endpoint, `--vault-token`, `--databus-key`, or `--apikey` for authentication, and `--force` or `--dry-run` for delete replay. Deploy replay supports classic and metadata-file deployments, but not WebDAV deployments because their original local files may no longer exist. | ||
|
|
||
| ```bash | ||
| databusclient manifest replay ./manifests/download-run.jsonld --localdir ./replayed-data | ||
| ``` | ||
|
|
||
| <a id="cli-manifest-summary"></a> | ||
| #### Summary | ||
|
|
||
| Print the stored results from a manifest without replaying the operation or accessing the network: | ||
|
|
||
| ```bash | ||
| databusclient manifest summary ./manifests/download-run.jsonld | ||
| ``` | ||
|
|
||
| The summary displays the command, execution time, file counts, byte total when available, overall status, operation errors, and individual failed files. | ||
|
|
||
| <a id="cli-workflow"></a> | ||
| ### Workflow | ||
|
|
||
| Run a multi-step `download`, `deploy`, and `delete` pipeline from YAML: | ||
|
|
||
| ```bash | ||
| databusclient workflow run [OPTIONS] WORKFLOW_PATH | ||
| ``` | ||
|
|
||
| Each workflow has a top-level `steps` list. Steps run in order, can reference earlier outputs using `${steps.step_name.output_files}` or `${steps.step_name.output_urls}`, and support `on_error: fail`, `continue`, or `retry`. Retry settings use `max_attempts` and `delay_seconds`. | ||
|
|
||
| ```yaml | ||
| steps: | ||
| - name: fetch_dataset | ||
| command: download | ||
| uri: https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 | ||
| localdir: ./data | ||
| - name: publish_dataset | ||
| command: deploy | ||
| files: ${steps.fetch_dataset.output_urls} | ||
| on_error: fail | ||
| ``` | ||
|
|
||
| Workflow runs always produce a console summary. Pass `--manifest PATH`, or set the top-level YAML `manifest` value, to also write one unified JSON-LD manifest covering the complete workflow and each step. | ||
|
|
||
| See [Workflow Examples](examples/workflows/README.md) for ready-to-use download, deploy, delete, and failure-handling workflows. |
There was a problem hiding this comment.
Compared to before, quite a bit of documentation has been removed. Was that intentional? For example, the section on workflows used to be very informative. It explained things like deployment variants within a workflow and the different types of error handling available.
Let's say a new user wants to use workflows. Do you think the current information is sufficient for them to understand and use this feature?
| #### Examples of using the download command | ||
|
|
||
| **Download File**: download of a single file | ||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 | ||
| ``` | ||
|
|
||
| **Download Version**: download of all files of a specific version | ||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 | ||
| ``` | ||
|
|
||
| **Download Artifact**: download of all files with the latest version of an artifact | ||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals | ||
| ``` | ||
|
|
||
| **Download Group**: download of all files with the latest version of all artifacts of a group | ||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings | ||
| ``` | ||
|
|
||
| **Download Collection**: download of all files within a collection | ||
| ```bash | ||
| databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 | ||
| ``` | ||
|
|
||
| **Download Query**: download of all files returned by a query (SPARQL endpoint must be provided with `--databus`) | ||
| ```bash | ||
| databusclient download 'PREFIX dcat: <http://www.w3.org/ns/dcat#> SELECT ?x WHERE { ?sub dcat:downloadURL ?x . } LIMIT 10' --databus https://databus.dbpedia.org/sparql | ||
| ``` | ||
|
|
||
| **Download with Compression Conversion**: download files and convert them to a different compression format on-the-fly | ||
| ```bash | ||
| # Convert all compressed files to gzip format | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --compression gz | ||
|
|
||
| # Decompress files without recompressing | ||
| databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals --compression none | ||
|
|
||
| # Download a collection and unify all files to bz2 format | ||
| databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --compression bz2 |
There was a problem hiding this comment.
It should not just list a few examples. A lot of existing documentation is missing. There is no information about localdir, vault-token, all-version, and so on
Yes, those sample commands are stale and should be updated to the current CLI syntax. They are not part of the public docs, but they are still examples that can mislead users, used it for testing though. |
|
Based on the changes, are you relying solely on agents/LLMs to refactor the documentation? |
| ```bash | ||
| databusclient download --help | ||
|
|
||
| # Output: | ||
| Usage: python -m databusclient.cli download [OPTIONS] DATABUSURIS... | ||
|
|
||
| Download datasets from databus, optionally using vault access if vault | ||
| options are provided. Supports on-the-fly compression format conversion | ||
| using the --compression option. | ||
|
|
||
| Options: | ||
| --localdir TEXT Local databus folder (if not given, databus | ||
| folder structure is created in current | ||
| working directory) | ||
| --databus TEXT Databus URL (if not given, inferred from | ||
| databusuri, e.g. | ||
| https://databus.dbpedia.org/sparql) | ||
| --vault-token TEXT Path to Vault refresh token file | ||
| --databus-key TEXT Databus API key to download from protected | ||
| databus | ||
| --all-versions When downloading artifacts, download all | ||
| versions instead of only the latest | ||
| --authurl TEXT Keycloak token endpoint URL [default: https | ||
| ://auth.dbpedia.org/realms/dbpedia/protocol/ | ||
| openid-connect/token] | ||
| --clientid TEXT Client ID for token exchange [default: | ||
| vault-token-exchange] | ||
| --compression [bz2|gz|xz|none] Target compression format for on-the-fly | ||
| conversion during download. Source | ||
| compression is detected automatically from | ||
| the file extension. Use 'none' to decompress | ||
| files without recompressing. | ||
| --format [ntriples|nt|turtle|ttl|rdf-xml|rdf|xml|nquads|nq|trig|trix|json-ld|jsonld|csv|tsv] | ||
| Target format for on-the-fly format | ||
| conversion during download (Layer 2 and | ||
| Layer 3). Accepts full names or short aliases. | ||
| --graph-name TEXT Named graph URI for Triple -> Quad | ||
| conversion. Required when converting RDF | ||
| triple formats to quad formats. | ||
| --base-uri TEXT Base URI for CSV -> RDF Triple conversion. | ||
| Required when converting CSV/TSV to RDF | ||
| triple formats. | ||
| --manifest TEXT Write a JSON-LD manifest of this operation | ||
| to PATH. | ||
| --validate-checksum Validate checksums of downloaded files | ||
| --help Show this message and exit. | ||
| ``` |
There was a problem hiding this comment.
You removed the help output earlier, as we discussed, but have now reintroduced it. Is there a specific reason for this?
| # Output: | ||
| Usage: databusclient deploy [OPTIONS] [DISTRIBUTIONS]... | ||
|
|
||
| Flexible deploy to Databus command supporting three modes: | ||
|
|
||
| - Classic deploy (distributions as arguments) | ||
|
|
||
| - Metadata-based deploy (--metadata <file>) | ||
|
|
||
| - Upload & deploy via Nextcloud (--webdav-url, --remote, --path) | ||
|
|
||
| Options: | ||
| --version-id TEXT Target databus version/dataset identifier of the form <h | ||
| ttps://databus.dbpedia.org/$ACCOUNT/$GROUP/$ARTIFACT/$VE | ||
| RSION> [required] | ||
| --title TEXT Artifact & Version Title: used for BOTH artifact and | ||
| version. Keep stable across releases; identifies the | ||
| data series. [required] | ||
| --abstract TEXT Artifact & Version Abstract: used for BOTH artifact and | ||
| version (max 200 chars). Updating it changes both | ||
| artifact and version metadata. [required] | ||
| --description TEXT Artifact & Version Description: used for BOTH artifact | ||
| and version. Supports Markdown. Updating it changes both | ||
| artifact and version metadata. [required] | ||
| --license TEXT License (see dalicc.net) [required] | ||
| --apikey TEXT API key [required] | ||
| --metadata PATH Path to metadata JSON file (for metadata mode) | ||
| --webdav-url TEXT WebDAV URL (e.g., | ||
| https://cloud.example.com/remote.php/webdav) | ||
| --remote TEXT rclone remote name (e.g., 'nextcloud') | ||
| --path TEXT Remote path on Nextcloud (e.g., 'datasets/mydataset') | ||
| --help Show this message and exit. | ||
| ``` |
There was a problem hiding this comment.
You removed the help output earlier, as we discussed, but have now reintroduced it. Is there a specific reason for this?
The reason why I thought we remove the help output is, that it gets deprecated/wrong easily. This help output for example is not up to date. For example, it does not contain --manifest
| # Output: | ||
| Usage: databusclient delete [OPTIONS] DATABUSURIS... | ||
|
|
||
| Delete a dataset from the databus. | ||
|
|
||
| Delete a group, artifact, or version identified by the given databus URI. | ||
| Will recursively delete all data associated with the dataset. | ||
|
|
||
| Options: | ||
| --databus-key TEXT Databus API key to access protected databus [required] | ||
| --dry-run Perform a dry run without actual deletion | ||
| --force Force deletion without confirmation prompt | ||
| --help Show this message and exit. | ||
| ``` |
There was a problem hiding this comment.
You removed the help output earlier, as we discussed, but have now reintroduced it. Is there a specific reason for this?
However, also not up to date, --manifest missing
So removing or keeping? |
No, actually I faced some commit issues, my commits somehow got mixed up, the codes I mean..and so it is happening likewise, still trying to fix all work. I have decided that, if this doesn't get resolved in time, I will fetch the merged work and redo every doc change. |
Keeping it, I have updated it with newer work, and have replaced it is test/manual folder just so that it stays clean. |
The current commit is done by restoring the actual previous documentation from merged commit of milestone 5, with doc fixes applied. I hope it tries to clean what was messed up, will recheck everything myself once again. |
2 participants
Description
Restructures project documentation :
doc/cli-usage.md.doc/module-usage.md.doc/examples/, including all workflow example files.doc/README.mdas a navigation index for the documentation folder.Related Issues
Closes #86
Type of change