Transform/reshape data first instead of wrangling the metadata directly

[joelostblom]

During our update meeting last week we talked about considering transforming the data first and then using Sprout's capabilities to autogenerate the metadata rather than transforming the metadata directly as we are doing now. Since we would still need to transform the data regardless, doing it the other way around might save some effort/reduce the code.

[martonvago]

Would the rough flow be:

1. Download data dict and event metadata.
2. Download all data as one table.
3. Split data into instruments/resources using instrument-field mapping in data dict. Preserve ID cols in each new data frame.
4. Post-process data: add center col, drop descriptive cols, do our own resource-specific transformations, etc. Also, remove empty rows coming from events where the instrument is not filled in (using event metadata).
5. Extract metadata (field properties) using Sprout.
6. Post-process metadata: add resource name, description, list PKs, extract field type from data dict, etc.

[lwjohnst86]

Yea, I think that's a decent flow for now @martonvago. I think 3. will need to be iterated on a bit depending on the data and how the REDCap gets updated with the clearer keys (e.g. visit id).