Skip to content

Commit fd4e9b5

Browse files
agusnieto77agusnieto77
committed
Deploying to gh-pages from @ 6203a81 🚀
1 parent 4c04b8b commit fd4e9b5

71 files changed

Lines changed: 6094 additions & 1451 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.en.html‎

Lines changed: 151 additions & 48 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.en.md‎

Lines changed: 178 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,117 +1,211 @@
11
# entropiaR
22

3-
A **read-only** R interface to the SQLite databases generated by the
4-
[EntropIA](https://github.com/HumaLab/EntropIA-Pro-Lite) desktop
5-
application. Built in a tidyverse style: explore your full document
6-
corpus — items, transcriptions, entities, LLM analysis, searches —
7-
without writing a single line of SQL.
8-
9-
Every access is lazy (`tbl_sql` via `dbplyr`), typed on materialisation
10-
(timestamps to `POSIXct`, JSON to list-columns) and backed by schema
11-
compatibility checks against the live database. And by design: the
12-
database is opened **read-only**, never modified.
13-
14-
## Features
15-
16-
- **Lazy access to 18 tables**:
17-
[`entropia_items()`](https://humalab.github.io/EntropIA-R/reference/entropia_items.md),
18-
[`entropia_entities()`](https://humalab.github.io/EntropIA-R/reference/entropia_entities.md),
19-
[`entropia_transcriptions()`](https://humalab.github.io/EntropIA-R/reference/entropia_transcriptions.md),
20-
[`entropia_llm_results()`](https://humalab.github.io/EntropIA-R/reference/entropia_llm_results.md),
21-
and more. Compose with `dplyr` verbs and materialise with
22-
[`entropia_collect()`](https://humalab.github.io/EntropIA-R/reference/entropia_collect.md).
23-
- **Typed materialisation**: timestamp columns to `POSIXct` (automatic
24-
milliseconds/seconds handling) and JSON columns to list-columns.
25-
- **Full-text search** (FTS5), parameter-safe:
26-
[`entropia_search()`](https://humalab.github.io/EntropIA-R/reference/entropia_search.md).
27-
- **Domain layer**: unified corpus with the best available text, parsed
28-
metadata and text-extraction helpers.
29-
- **Corpus diagnostics**: OCR/metadata coverage, orphaned-reference
30-
detection and connection validation.
31-
- **Ready-made analysis**: temporal profiles, document lengths, entity
32-
and topic frequencies, collection comparison, and reproducible
33-
datasets with v2 provenance (schema, snapshot and data hashes).
34-
- **Shared EDA**:
35-
[`entropia_overview()`](https://humalab.github.io/EntropIA-R/reference/entropia_overview.md)
36-
SQL-aggregates the study universe (counts, collections, temporality,
37-
eligibility-aware quality, entity and topic occurrence plus per-item
38-
prevalence) and
39-
[`entropia_profile()`](https://humalab.github.io/EntropIA-R/reference/entropia_profile.md)
40-
profiles collected tibbles locally (missingness, distributions,
41-
duplicates, reproducible sampling).
42-
- **Local dashboard and frozen reports**:
43-
[`entropia_dashboard()`](https://humalab.github.io/EntropIA-R/reference/entropia_dashboard.md)
44-
builds an optional Shiny app over a snapshot (isolated sessions, one
45-
selection across panels, on-demand text, bounded downloads) and
46-
[`entropia_report()`](https://humalab.github.io/EntropIA-R/reference/entropia_report.md)
47-
renders a Quarto dashboard with optional redaction.
48-
- **Visualisation** with `ggplot2` (`entropia_plot_*`) and
49-
**reproducible export** to CSV, TSV, JSON, RDS, Parquet or Arrow, with
50-
provenance sidecars.
51-
52-
## Installation
3+
A **read-only** interface between R and the SQLite databases from
4+
[EntropIA](https://github.com/HumaLab/EntropIA-Pro-Lite). Tidyverse
5+
style: explore the corpus — items, texts, entities, topics, LLM results
6+
— without writing SQL.
7+
8+
The database is **always opened read-only**. Nothing in this package
9+
writes to it.
10+
11+
Spanish is the primary documentation language. This file is the English
12+
secondary version. Articles:
13+
[`vignette("connect")`](https://humalab.github.io/EntropIA-R/articles/connect.md)
14+
(Spanish) /
15+
[`vignette("connect.en")`](https://humalab.github.io/EntropIA-R/articles/connect.en.md)
16+
(English).
17+
18+
## Quick path
5319

5420
``` r
5521

56-
remotes::install_github("HumaLab/EntropIA-R")
22+
library(entropiaR)
23+
library(dplyr)
24+
25+
con <- entropia_connect(system.file(
26+
"extdata", "entropia-example.sqlite",
27+
package = "entropiaR"
28+
))
29+
30+
# 1. What is in the corpus? SQL aggregates, no text download.
31+
eda <- entropia_overview(con)
32+
eda$counts
33+
eda$collections
34+
35+
# 2. A reproducible dataset (one universe, one recipe).
36+
ds <- entropia_analysis_dataset(
37+
con,
38+
asset_type == "pdf",
39+
name = "pdfs",
40+
text = FALSE
41+
)
42+
entropia_provenance(ds)$dataset_sha256
43+
44+
# 3. Plots from the same tables.
45+
entropia_plot_collections(eda$collections)
46+
entropia_plot_entities(eda$entities)
47+
entropia_plot_coverage(eda$quality, metric = "ocr_coverage")
48+
49+
entropia_disconnect(con)
5750
```
5851

59-
Requires **R \>= 4.1**.
52+
Expected: `eda$counts` reports items/assets/collections; the dataset is
53+
a tibble with an `entropia_prov` stamp; `entropia_plot_*` return
54+
extensible `ggplot` objects.
6055

61-
## Quick start
56+
## Your own database
6257

6358
``` r
6459

65-
library(entropiaR)
60+
# If EntropIA may be running, snapshot first (VACUUM INTO, WAL-aware).
61+
live <- entropia_connect("path/to/entropia.sqlite")
62+
snap <- tempfile(fileext = ".sqlite")
63+
entropia_copy(live, snap)
64+
entropia_disconnect(live)
6665

67-
# The package ships a small example database so you can try it right away.
68-
con <- entropia_connect(system.file("extdata", "entropia-example.sqlite",
69-
package = "entropiaR"))
66+
con <- entropia_connect(snap)
67+
```
7068

71-
entropia_items(con) # lazy tbl_sql: nothing loaded yet
72-
entropia_collect(entropia_items(con)) # typed tibble: POSIXct + JSON list-columns
73-
entropia_search(con, "huelga") # parameter-safe full-text search (FTS5)
74-
entropia_corpus(con) |> entropia_collect() |> entropia_document_lengths()
69+
Compatibility on open (`warn` by default):
7570

76-
entropia_disconnect(con)
71+
``` r
72+
73+
options(entropiaR.schema_policy = "warn") # or "error" | "allow"
74+
entropia_schema_compat(con)$compatible
7775
```
7876

79-
## Connect your own database
77+
Without the core tables (`collections`, `items`, `assets`), `warn` and
78+
`error` refuse the connection. `allow` opens for diagnostics.
8079

81-
If you use the EntropIA application, connect the database it generates
82-
the same way:
80+
## Shared EDA
81+
82+
[`entropia_overview()`](https://humalab.github.io/EntropIA-R/reference/entropia_overview.md)
83+
aggregates **the selected universe** in SQL. It does not materialise
84+
text or embeddings.
8385

8486
``` r
8587

86-
con <- entropia_connect("path/to/your/entropia.sqlite")
88+
eda <- entropia_overview(
89+
con,
90+
asset_types = c("pdf", "image"),
91+
page_assets = FALSE
92+
)
93+
eda$quality # n, total, pct, status, group_id
94+
eda$entities # n = occurrences; pct = prevalence per item
95+
eda$topics
96+
attr(eda$temporal, "exclusions")
8797
```
8898

89-
The connection is read-only and runs a schema-compatibility check
90-
against the live database on open (controlled with
91-
`options(entropiaR.schema_policy = "warn" | "error" | "allow")`).
99+
[`entropia_profile()`](https://humalab.github.io/EntropIA-R/reference/entropia_profile.md)
100+
profiles an already collected tibble:
92101

93-
## Documentation
102+
``` r
94103

95-
- [Package website](https://humalab.github.io/EntropIA-R/) with
96-
reference and vignettes: connect, corpus, text, dplyr, datasets,
97-
analysis and administration.
98-
- Help in R:
99-
[`?entropia_connect`](https://humalab.github.io/EntropIA-R/reference/entropia_connect.md),
100-
[`?entropia_search`](https://humalab.github.io/EntropIA-R/reference/entropia_search.md),
101-
etc.
104+
lengths <- entropia_text(con) |>
105+
entropia_collect() |>
106+
entropia_document_lengths()
102107

103-
## Project status
108+
entropia_profile(lengths, columns = c("n_chars", "n_words"))
109+
```
104110

105-
Early development. The current release (v1) is **read-only**: it opens
106-
the database for querying and never writes to it. The write API is
107-
already designed and available as stubs with clear errors; its full
108-
implementation arrives in v2.
111+
Caveats the package does not hide:
112+
113+
| Fact | Consequence |
114+
|----|----|
115+
| `created_at` timestamps | Operational (created/imported), not necessarily the document date |
116+
| entity `n` | Occurrences; `pct` is prevalence over items in the universe |
117+
| Same collection names | Distinguished by `collection_id`, not by the label |
118+
| PDF pages | Each page is an asset; an item is not “one document × N pages” |
119+
| AI entities | Extractions, not verified facts |
120+
121+
## Dashboard and report
122+
123+
``` r
124+
125+
# Does not open a browser: returns a shiny.appobj.
126+
app <- entropia_dashboard(snap)
127+
# shiny::runApp(app)
128+
129+
# Frozen HTML report (requires Quarto on PATH).
130+
entropia_report(eda, "study.html") # redacts paths and labels
131+
entropia_report(eda, "study-internal.html", redact = FALSE)
132+
```
133+
134+
Shiny, bslib, ggplot2 and Quarto are **optional**. The core (connect,
135+
corpus, overview, export) works without them.
136+
137+
## What the package covers
138+
139+
| Layer | Typical entry |
140+
|----|----|
141+
| Connection / schema | [`entropia_connect()`](https://humalab.github.io/EntropIA-R/reference/entropia_connect.md), [`entropia_copy()`](https://humalab.github.io/EntropIA-R/reference/entropia_copy.md), `entropia_schema_*()`, [`entropia_validate()`](https://humalab.github.io/EntropIA-R/reference/entropia_validate.md) |
142+
| Lazy tables | [`entropia_items()`](https://humalab.github.io/EntropIA-R/reference/entropia_items.md), [`entropia_entities()`](https://humalab.github.io/EntropIA-R/reference/entropia_entities.md), … + dplyr |
143+
| Corpus and text | [`entropia_corpus()`](https://humalab.github.io/EntropIA-R/reference/entropia_corpus.md), [`entropia_text()`](https://humalab.github.io/EntropIA-R/reference/entropia_text.md), [`entropia_metadata()`](https://humalab.github.io/EntropIA-R/reference/entropia_metadata.md), [`entropia_search()`](https://humalab.github.io/EntropIA-R/reference/entropia_search.md) |
144+
| EDA | [`entropia_overview()`](https://humalab.github.io/EntropIA-R/reference/entropia_overview.md), [`entropia_profile()`](https://humalab.github.io/EntropIA-R/reference/entropia_profile.md) |
145+
| Analysis | [`entropia_temporal_profile()`](https://humalab.github.io/EntropIA-R/reference/entropia_temporal_profile.md), `entropia_*_frequency()`, [`entropia_compare_collections()`](https://humalab.github.io/EntropIA-R/reference/entropia_compare_collections.md) |
146+
| Datasets | [`entropia_analysis_dataset()`](https://humalab.github.io/EntropIA-R/reference/entropia_analysis_dataset.md), [`entropia_provenance()`](https://humalab.github.io/EntropIA-R/reference/entropia_provenance.md), [`entropia_export()`](https://humalab.github.io/EntropIA-R/reference/entropia_export.md) |
147+
| Plots | `entropia_plot_*()` (Suggests: ggplot2) |
148+
| Apps | [`entropia_dashboard()`](https://humalab.github.io/EntropIA-R/reference/entropia_dashboard.md), [`entropia_report()`](https://humalab.github.io/EntropIA-R/reference/entropia_report.md) |
149+
150+
## Documentation
151+
152+
Articles (vignettes), Spanish first, English with the `.en` suffix:
153+
154+
1. [`vignette("connect")`](https://humalab.github.io/EntropIA-R/articles/connect.md)
155+
/
156+
[`vignette("connect.en")`](https://humalab.github.io/EntropIA-R/articles/connect.en.md)
157+
— open, validate, snapshot
158+
2. [`vignette("corpus")`](https://humalab.github.io/EntropIA-R/articles/corpus.md)
159+
/
160+
[`vignette("corpus.en")`](https://humalab.github.io/EntropIA-R/articles/corpus.en.md)
161+
— collections, items, assets
162+
3. [`vignette("text")`](https://humalab.github.io/EntropIA-R/articles/text.md)
163+
/
164+
[`vignette("text.en")`](https://humalab.github.io/EntropIA-R/articles/text.en.md)
165+
— OCR, transcriptions, metadata
166+
4. [`vignette("dplyr")`](https://humalab.github.io/EntropIA-R/articles/dplyr.md)
167+
/
168+
[`vignette("dplyr.en")`](https://humalab.github.io/EntropIA-R/articles/dplyr.en.md)
169+
— lazy filters and typing
170+
5. [`vignette("eda")`](https://humalab.github.io/EntropIA-R/articles/eda.md)
171+
/
172+
[`vignette("eda.en")`](https://humalab.github.io/EntropIA-R/articles/eda.en.md)
173+
— overview and profile
174+
6. [`vignette("visualize")`](https://humalab.github.io/EntropIA-R/articles/visualize.md)
175+
/
176+
[`vignette("visualize.en")`](https://humalab.github.io/EntropIA-R/articles/visualize.en.md)
177+
— individual plots
178+
7. [`vignette("datasets")`](https://humalab.github.io/EntropIA-R/articles/datasets.md)
179+
/
180+
[`vignette("datasets.en")`](https://humalab.github.io/EntropIA-R/articles/datasets.en.md)
181+
— provenance v2 and export
182+
8. [`vignette("analysis")`](https://humalab.github.io/EntropIA-R/articles/analysis.md)
183+
/
184+
[`vignette("analysis.en")`](https://humalab.github.io/EntropIA-R/articles/analysis.en.md)
185+
— a complete analysis
186+
9. [`vignette("dashboard")`](https://humalab.github.io/EntropIA-R/articles/dashboard.md)
187+
/
188+
[`vignette("dashboard.en")`](https://humalab.github.io/EntropIA-R/articles/dashboard.en.md)
189+
— Shiny and Quarto
190+
10. [`vignette("administration")`](https://humalab.github.io/EntropIA-R/articles/administration.md)
191+
/
192+
[`vignette("administration.en")`](https://humalab.github.io/EntropIA-R/articles/administration.en.md)
193+
— read-only, WAL, write stubs
194+
195+
Site: <https://humalab.github.io/EntropIA-R/>
196+
197+
## Status
198+
199+
Early development, **v1 read-only**. Stubs
200+
`entropia_insert/update/upsert/delete` fail with
201+
`entropia_error_write_disabled`. Real writes: v2.
202+
203+
Requires **R \>= 4.1**.
109204

110205
## License
111206

112207
MIT.
113208

114209
------------------------------------------------------------------------
115210

116-
[Spanish
117-
version](https://github.com/HumaLab/EntropIA-R/blob/main/README.md)
211+
[Versión en español](https://humalab.github.io/EntropIA-R/README.md)

0 commit comments

Comments
 (0)