Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This repository contains Jupyter notebooks & Github actions pipelines to be used

### To get started with local development

1. Install Deno and Deno Jupyter kernel
1. Install [Deno and Deno Jupyter kernel](https://docs.deno.com/runtime/getting_started/installation/)
2. Install [Jupyter extension for VSCode](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter)
3. Open up a notebook, select Deno as kernel and run

41 changes: 40 additions & 1 deletion v1-to-v2-data-migration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
- Run `./migrate.ipynb`
- If the migration is successful, it will print out a list of successful migrations

**NOTE: to re-run a notebook after making changes, you need to Click Restart and Run All**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**NOTE: to re-run a notebook after making changes, you need to Click Restart and Run All**
**NOTE: to re-run a notebook after making changes to other files, you need to Click Restart and Run All**



## How to set up migrations for countries

Expand All @@ -35,4 +37,41 @@ In the case where your production does not use a VPN, just ignore those steps
- Login as an admin user
- Go to `Configuration -> Integrations -> Create client`
- Create client with name 'Migrations' and type 'Import/Export'
- Copy the Client ID and Client secret and save them in the github secrets for each environment
- Copy the Client ID and Client secret and save them in the github secrets for each environment

## Example process
- Approach:
- Configure 1.9 completely for the birth & death events you are trying to migrate
- Fork the OpenCRVS Notebook and install your Node runtime e.g. Deno to run these Jupyter notebooks in VSCode
- Update your address / name mappings and resolvers to those that suit your country here: https://github.com/opencrvs/notebooks/blob/main/v1-to-v2-data-migration/countryData/addressResolver.ts. The administrativeArea prop should be set to the field id for the leaf level select for admin levels that you have configured in your 1.9 `application-config.ts`
- Update your country mappings and resolvers that are appropriate to your form - usually for custom fields


- Madagascar example:
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/addressMappings.ts
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/addressResolver.ts
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/countryMappings.ts
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/countryResolvers.ts
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/nameMappings.ts
https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/countryData/nameResolver.ts


- Locally run the tasks in the https://github.com/opencrvs/notebooks/blob/main/v1-to-v2-data-migration/get-field-diff.ipynb notebook to make sure that your 1.9 forms have been configured so that every field in 1.8 has a new field in 1.9 to migrate data too.  This will obviously fail first time as you need to make your country specific mappers and resolvers


Once that script passes, ...


In an incognito window, load the 1.8 version of your app and create some test records to migrate:


http://localhost:3000/?V2_EVENTS=false


- Run the migrate notebook with test system client API details and a test Registrar locally to see if the records migrated OK:  https://github.com/opencrvs/notebooks/blob/mdg-migrations/v1-to-v2-data-migration/migrate.ipynb
- Deploy 1.9 to the target (QA) environment
- Obtain QA server env var/secrets for the notebook and set them in Github Secrets
- Create test records to migrate on your QA environment (There probably already are a lot of records there)
- Run the https://github.com/opencrvs/notebooks/blob/mdg-migrations/.github/workflows/migrate.yml Github Action to migrate the environment.  This action will need edited with VPN access commands as you have already for other actions in your forked countryconfig repo
- Quality assure your 1.9 config thoroughly
- Repeat in staging and production
Loading