Skip to content

rootcoder007/rmorie

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

287 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

R-MORIE rmorie hex logo

Status Lifecycle r-universe CI Coverage AGPL-3.0

R-MORIE (Multi-domain Open Research and Inferential Estimation) is an R package for causal inference, sampling, psychometrics, point-process modeling, and criminological accountability analysis, with no Python dependencies.

Statement of need

Applied observational research on Canadian carceral, policing, and oversight data usually means stitching together a dozen single-purpose packages — one for difference-in-differences, one for propensity-score matching, one for spatial scan statistics, one for self-exciting point processes — each with its own data contract, and none aware of the survey-weighting, provenance, and privacy constraints these data carry. rmorie is for criminologists, quantitative social scientists, and accountability researchers who need those estimators in one consistent, provenance-preserving toolkit, with the MRM (Multilevel Reconciliation Methodology) framework as its motivating application. Every public function is prefixed morie_* so it composes safely alongside the specialist packages it wraps, and results carry the labeling that keeps synthetic development runs from being mistaken for inferential findings. Primary methodological references for each estimator family are listed in the package-level help (?rmorie).

Documentation

With over 1,900 exported functions, the full reference is large — use the manual or the package site above rather than scrolling the function index. This README covers install + the most common workflows only.

What's in v1.1.3

  • Over 1,900 exported morie_* R functions (1,963 at the time of writing) — every public callable is now prefixed to avoid name collisions with other CRAN packages (morie_chi_square_test, morie_kmeans_clustering, morie_decision_tree_split, etc.). The companion morie.fn Python library mirrors these for cross-language parity. Two deliberate exceptions keep their unprefixed names to match the MRM papers and the Python implementation exactly: mrm_otis_mandela_spectrum() and mrm_classify_mandela().
  • SIU subsystem — a full pipeline for the Ontario Special Investigations Unit director's-report corpus (English + French, 2005-present). See SIU pipeline below.
  • Free-first AI helpers — local Ollama by default (gemma3:4b, translategemma:latest), with optional Gemini, Claude, or Vertex AI fallback. No paid API key is required for the default workflow.
  • Polite-by-default HTTP fetcher — token-bucket throttling at 4 req/s, exponential backoff on 429/5xx, on-disk page cache.
  • Built-in datasets — 41 datasets accessible through the shared SQLite store (morie_datasets.db), plus the SIU manifest (4,743 drids, 2,218 unique cases, language-classified).
  • CPADS contract helpers and IPW / eBAC workflow functions.
  • Outputs-manifest tooling — read, validate, audit, and build outputs_manifest.csv tables for reproducible research projects.
  • Synthetic data generators for development and CI.
  • C/C++ computational backend — Hawkes self-exciting point process likelihood (Markovian + non-Markovian), HTML-to-text state machine, SIU parser. See src/.
  • Causal-taphonomy suite — Bayesian hierarchical preservation model (cmdstanr / brms / rstanarm HMC backends), absorbing-DTMC decay chains, forensic likelihood ratios, pXRF compositional transforms, and USGS NGDB / MorphoSource open-data ingest.
  • agent() — call the rmorie CLI agent from R (with agent_available() to probe for the binary).

Scientific guardrail

  • Synthetic data is for development, testing, demos, and CI only.
  • Final inferential or policy-facing results must be produced from approved real data with full provenance.
  • Synthetic runs must be explicitly labeled as synthetic in outputs and reporting text.

Install

From local source:

install.packages("r-package/morie", repos = NULL, type = "source")

From r-universe (development snapshot):

install.packages(
  "rmorie",
  repos = c(rootcoder007 = "https://rootcoder007.r-universe.dev",
            CRAN         = "https://cloud.r-project.org")
)

# want every optional package too, in one shot? add dependencies = TRUE
# (large: compiles many specialist packages — see "Optional packages" below)
install.packages(
  "rmorie", dependencies = TRUE,
  repos = c(rootcoder007 = "https://rootcoder007.r-universe.dev",
            CRAN         = "https://cloud.r-project.org")
)

The assistant bridge supports a local fallback through the Python package when no live OpenAI / Anthropic credentials are configured.

Optional packages (the R equivalent of pip install pkg[extra])

The base install stays deliberately light: rmorie wraps many specialist CRAN packages and declares them in Suggests, so nothing heavy compiles until you need it. Every function that uses one tells you exactly what to install when it's missing, and the test suite skips (never fails) without them. To provision up front instead:

# install every optional package rmorie can use (one-time, ~15 min)
morie_install_extras(which = "all", ask = FALSE)

# or just what's missing, interactively
morie_install_extras()

# or a specific family, e.g. machine learning
morie_install_extras(which = c("randomForest", "glmnet", "xgboost",
                               "ranger", "caret", "pROC"))

Common families: ML (randomForest, glmnet, xgboost/gbm, ranger, caret, pROC, Rtsne, e1071, dbscan), DSP (signal, pracma, wavelets), causal (DoubleML, mlr3, mlr3learners, ivreg, fixest), storage (RSQLite, duckdb).

Outputs-manifest example

library(rmorie)

manifest <- morie_read_outputs_manifest(project_root = "/path/to/project")
audit    <- morie_audit_public_outputs(project_root = "/path/to/project",
                                       manifest     = manifest)
morie_summarize_output_audit(audit)

Synthetic data example

library(rmorie)

synthetic_path <- morie_write_synthetic_data(
  path      = "data/private/synthetic_study_data.csv",
  n         = 8000,
  seed      = 2026,
  overwrite = TRUE
)

Cross-project adaptation

library(rmorie)

name_map <- morie_default_synthetic_name_map("generic")
name_map["cannabis_use"] <- "exposure_any"
name_map["bac"]          <- "outcome_continuous"

dat <- morie_generate_synthetic_data(
  n        = 5000,
  seed     = 1,
  name_map = name_map
)

SIU pipeline

A first-class subsystem for the Ontario Special Investigations Unit director's-report corpus. The fetcher handles both English and French templates from 2005 onward; the parser is hand-rolled C++ for correctness under SIU's heterogeneous markup.

Fetch and parse the full corpus

library(rmorie)

# Use the shipped language-aware DRID manifest; English-only,
# cache pages so re-runs are fast.
df <- morie_fetch_siu(
  lang       = "en",         # skip French drids automatically
  cache_html = TRUE,         # persist every fetched page locally
  rate_limit = 4             # requests per second (polite default)
)

# 2,218 unique cases x 64 columns; 100% format-clean on the
# shipping corpus per morie_siu_sanity_check().
nrow(df)

Audit a single case

# Inspect parser row + raw HTML + cleaned text side-by-side.
morie_siu_audit_case("16-OFI-019")

# Per-field "does the HTML actually support this value?" check.
morie_siu_anomaly_check("16-OFI-019")

# Diff parser output against an external table.
morie_siu_compare(
  case_number = "16-OFI-019",
  external    = my_other_table,
  field_map   = c(officer_count = "n_officers")
)

AI extraction (free local model by default)

# Default: local Ollama with gemma3:4b. No API key required.
morie_siu_llm_extract("16-OFI-019")

# Failover chain: try local first, fall back to Gemini only on error.
morie_siu_llm_extract("16-OFI-019", model = c("ollama", "gemini"))

# French to English translation via translategemma.
morie_siu_translate(text = "L'enquete a ete close...", target_lang = "en")

Supported providers: ollama (default), gemini, claude, vertex. Environment knobs: OLLAMA_HOST (defaults to http://localhost:11434), OLLAMA_MODEL (defaults to gemma3:4b), OLLAMA_KEEP_ALIVE (30m).

Format-validity sweep

sane <- morie_siu_sanity_check(df)
sum(!sane$ok)  # rows with format issues (regex / ISO date / Yes-No / chrome leak)

Aggregate accuracy

# How accurate is each column across a sample of cases?
morie_siu_audit_columns(case_numbers = sample(df$case_number, 50))

Canonical override system

The parser learns. Ship-time corrections live in inst/extdata/siu_canonical_overrides.csv.gz (47 hand-verified corrections covering 10 spot-checked cases). Users can add their own:

morie_siu_record_correction(
  case_number = "20-OFD-082",
  field       = "officer_count",
  value       = 3L
)

Overrides are applied automatically at the end of morie_fetch_siu(), per cell, by case number.

Inspect the manifest

manifest <- morie_siu_index()
table(manifest$`_language`)  # en=2531, fr=2212, unknown=0

Continuous integration

The R CMD check matrix covers six cells, all green on main:

Platform R version
macos-latest release
windows-2025 release
ubuntu-latest release
ubuntu-latest release + postgres-15
ubuntu-latest oldrel-1
ubuntu-latest devel

Plus: pkgcheck, covr + Codecov upload, lintr, goodpractice, and CodeQL.

Citation

If you use rmorie in your research, please cite the software:

Ruhela, V. S. (2026). rmorie: Multi-domain Open Research and Inferential Estimation in R. https://github.com/rootcoder007/rmorie

BibTeX (or run citation("rmorie") after installation for the entry stamped with the exact installed version, sourced from inst/CITATION):

@Manual{ruhela_rmorie_2026,
  title   = {rmorie: Multi-domain Open Research and Inferential Estimation in R},
  author  = {Ruhela, Vansh Singh},
  year    = {2026},
  url     = {https://github.com/rootcoder007/rmorie}
}

See CITATION.cff for the machine-readable metadata GitHub's "Cite this repository" button uses.

License

R-MORIE is licensed under AGPL-3.0-or-later. See LICENSE for the full text and LICENSING.md for the per-component breakdown.

Bayesian priors

rmorie's Bayesian regression (morie_bayes_lm) places zero-mean Normal priors on the regression coefficients; the prior_sd argument is the prior standard deviation (the scale of plausible coefficient values). Larger prior_sd is weakly informative; smaller values pull estimates toward zero (regularisation). Example:

d <- data.frame(x = rnorm(100)); d$y <- 1 + 2 * d$x + rnorm(100)
# weakly-informative prior (sd = 10) vs a tight regularising prior (sd = 0.5)
fit_weak  <- morie_bayes_lm(y ~ x, d, prior_sd = 10)
fit_tight <- morie_bayes_lm(y ~ x, d, prior_sd = 0.5)

See the bayesian-priors vignette for applied guidance.

Packages

 
 
 

Contributors

Languages