Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions test-cnm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,46 @@ same project.
If no environment is specified the `default` environment will be used. If a
config value is missing for the currently specified environment, the value will
be pulled from the entry in the `[default]` section of the config instead.

## Adding New Data to the Bucket

- Get the Granule S3 Path
- For example, with **`granule_name`**=[OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z](https://cumulus-dashboard.asf.earthdatacloud.nasa.gov/granules/granule/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z), scroll down to the columns.
- Combine the `link` with the `bucket`, and its s3 path is: `s3://<bucket>/<link>`, but you'll remove the last part of the link.
- i.e: **`input_granule`**=`s3://asf-cumulus-prod-opera-products/OPERA_L3_DISP-S1_V1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/`
- Figure out the tcnm bucket/prefix to store it
- In the same cumulus-dashboard link, copy the `Collection` value. That'll be the `collection/version` prefix. For example, `OPERA_L3_DISP-S1_V1/1` for this link (but remove any spaces).
- **`tcnm_bucket`**=`s3://asf-cumulus-dev-e2e-tests/OPERA_L3_DISP-S1_V1/1/`
- Upload the data with:
- `aws s3 cp --recursive <input_granule> <tcnm_bucket>/<granule_name>/`
- i.e:

```bash
aws s3 cp --recursive \
s3://asf-cumulus-prod-opera-products/OPERA_L3_DISP-S1_V1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/ \
s3://asf-cumulus-dev-e2e-tests/OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/
```

- Now do the same for the browse bucket. In the columns in the dashboard, one is the browse bucket. (The only thing that changes in the above command is the input bucket. In this case, to `asf-cumulus-prod-opera-browse`):

```bash
aws s3 cp --recursive \
s3://asf-cumulus-prod-opera-browse/OPERA_L3_DISP-S1_V1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/ \
s3://asf-cumulus-dev-e2e-tests/OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/
```

- Check the bucket you synced everything to.
- If you synced over any `*.zarr.json.gz` files, **delete** them. Sometimes there's browse and other files not sent by the provider, that we have to remove too.
- You'll notice the `*.cmr.json` file won't copy over from the original bucket, **that's expected / desired**.

- If you uploaded through the console directly, or deleted any files from the upload: run `tcnm tidy`. (It doesn't hurt to just run it either).
Comment thread
Cameronsplaze marked this conversation as resolved.
- Finally, `tcnm update-metadata <collection_name>`
- ESPECIALLY if you do this with larger volume collections, run this in AWS CloudShell. It needs to download each file to md5sum it, so locally can take forever.
- If you already know the md5sum, you can add `--interactive` to the command so it'll ask you instead of downloading the file.
- **Note**: `tcnm update-metadata OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F21517_VV_20170516T055331Z_20170528T055332Z_v1.0_20260225T005629Z/` works to *just* update the above.
Comment thread
Cameronsplaze marked this conversation as resolved.


## Removing Data from the Bucket

- Delete whatever from the testing bucket that's defined in `testcnm.cfg`.
- Run `tcnm tidy`