This repository implements the page stream segmentation stage of TRACE, which turns raw California police record disclosures into a structured, searchable database and links every component repository and the technical report.
This is code for splitting files (which can consist of multiple documents concatenated together) into distinct documents and extracting key metadata.
dc is the main processing code:
dc
├── classify.py
├── doctext.py
├── llm.py
├── prompts.py
└── toc.py
classify takes a file and classifies each page as a new document or a
continuation of the previous document.
toc takes the page classifications and bundles them into distinct documents
with extracted document-level metadata, producing a table-of-contents for a
file.
The tasks directory contains code for exploring and experimenting with the
current codebase. case-eda has code to process a sample of 25 cases, and also
hand-labeled page-classification data for those cases.
tasks/case-eda/src/summarize-case-from-metadata.py is a demonstration for how
to extract things like "investigated officer" from just the extracted metadata
from case files, although a more robust approach might be to use the metadata
to filter for key documents to highlight or to pass to an extraction model.
tasks/case-eda/src/case-document-summary.py formats the table-of-contents
data into a chronological summary of case documents, and spins up the webserver
found here