Repository navigation
Imputation-gap capstone: get_reported_*(), get_nibrs_estimate(), imputation_gap() - #4
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Summary rape is the revised definition (NIBRS 11A, 11B and 11C). The estimates' "Rape" indicator (118) is 11A alone, so imputation_gap() compared an 11A estimate with all three codes: Texas 2022 came out 47% below its own NIBRS counts. "Revised Rape" (120) covers all three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
The API answers Arizona (both years) with {"data": null} and
Pennsylvania 2021 with a fragment that has no offense total.
imputation_gap() now says the FBI publishes no total there, and
get_nibrs_estimate() documents the zero-row result. The new fixture is
Arizona's recorded 2022 response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Six states' 2021 reported counts include summary-format reports (New Jersey: 5,353 of 12,972 violent crimes), so the estimate models old-format reporters in 2021 as well as 2022. Measured where every report arrived in NIBRS (Texas 2022), the counting difference is about -1% for violent crime, not the +1% the national 2021 figures showed. gap compares two NIBRS figures, so the counting difference does not enter it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
The vignette follows the plan with the changes agreed in review: the counting difference is measured on Texas 2022, where every report arrived in NIBRS, rather than on the nation in 2021, which includes summary-format reports from six states; a New Jersey example shows those 2021 reports; and the gap is kept apart from the counting difference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
|
Thanks for this — and one thing worth knowing before it happens. Development happens in a private Gitea repository. Your pull request Because that merge preserves your commits at their original SHAs, GitHub will
If it is not going to be merged, you will get an actual reply saying so. Full details: CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
specs/plans/2026-10-15-imputation-gap.md(Gitea issue 42). Not for merging on GitHub: the operator lands it locally.What changed
Task 1,
21a110a: rename.get_estimated_crime()/get_estimated_arson()are nowget_reported_crime()/get_reported_arson(). The old names remain as exported aliases that warn through.Deprecated()and are documented on an internalfbiCDE-deprecatedpage. Tests, README, thefbivignette, pkgdown and NEWS are updated.Task 2,
3eeeece: vocabularies..parse_estimation_lookup()and the.ESTIMATE_OFFENSESmap are new.data-raw/api_vocabularies.Rnow also buildsnibrs_estimation_states(57 rows),nibrs_estimation_indicators(71) andnibrs_estimation_years(2021, 2022) from the live lookup. Rebuilding left every existing.rdabyte-identical.Task 3,
e4c2b27:get_nibrs_estimate(). One request per call. It returns one row per category with 95% bounds; null bounds becomeNAand negative bounds are kept as published.Task 4,
b0a0d0f:imputation_gap(). One row per year. Columns:reported,reported_nibrs,reported_other,coverage,coverage_nibrs,publishedwith its bounds,gapwith bounds,share_estimatedwith bounds.Review fixes, found while knitting the vignette and agreed with the operator in the session:
2f42682fix:RPEuses the "Revised Rape" indicator (120). The plan mapped it to "Rape" (118), which is NIBRS 11A alone, soimputation_gap(offense = "RPE")compared an 11A estimate with 11A+11B+11C. Texas 2022 came out at −47%; it is −7.3% now. An offline test pins 120, and a live guard checks every offense's Texas 2022share_estimatedstays within 10%.c93a28afix: clearer message where no estimate total exists. Arizona answers{"data": null}in both years, and Pennsylvania 2021 is a fragment with no total. The error now reads "The FBI publishes no V total for AZ in 2022." Arizona's response is recorded verbatim as a new fixture,nibrs-estimation-state-5-133-2022.json; no existing fixture was touched.a9a122edocs: reporting-format and counting-difference wording, explained under "Changes from the plan".Task 5,
278ab62: terms and vignette. Newvignette("estimates"), a README "Reported, estimated and imputed" section,@seealsolinks on five reference pages, a sentence inimpute_reporting_gaps()saying its imputation is not the FBI's estimate, one wording fix in thegeographyvignette, and the CLAUDE.md roadmap entry.Verification
99eeef3FBI_CDE_LIVE=true), this branchR CMD check --no-manual: Status OK, with 0 errors, 0 warnings and 0 notes (R 4.3.3, UTF-8 session). Every Suggests package was installed, plusqpdf.pkgdown::check_pkgdown(): "No problems found".test-ascii-docs.Rcannot translate the em-dashes already inman/and reports 34 warnings; they count as results, so the base shows 1,333 and the check only passes on paper.Knitted vignette against spec §3.3
share_estimated(vignette)The rest of the California rows match too:
reported193,824,reported_nibrs71,625,reported_other122,199, coverage 98.8% against 46.6% NIBRS-only.reported_other−504.Changes from the plan
summarizedcounts include summary-format reports. New Jersey: 75% coverage either format against 45% NIBRS-only, and 5,353 of 12,972 violent crimes arrived in the old format. Paterson PD (NIBRS start 2021-06-01) has 607 violent crimes for January to May 2021 and no NIBRS counts behind them. Without those six states, the 2021 counting difference for violent crime is −1.2% (NIBRS counts more), not the +1.1% the national table showed. Agreed changes:imputation_gap()page, NEWS).imputation_gap()page quotes the Texas figures in place of "about 1% / 5% / 7%".gapis kept apart from the counting difference: it compares two NIBRS figures. In Texas 2022 it runs from −7.3% to 2.9% of the estimate.claude/inspiring-wright-usthec, so the handoff'sclaude/imputation-gapwas not used.coveragevalue, 0.263218, fails its owntolerance = 1e-6, because testthat's tolerance is relative (off by 1.7e-6). It is written as 0.2632175; the fixture gives 0.2632175442.fbivignette drops its "'Estimated' is a legacy name" aside, which the rename made moot. The@seealsolines putvignette("estimates")in backticks.Not in this PR
fbi_api_agencies).summarizedviolent crime (7 to 34 a month) is below its NIBRS aggravated assaults alone (55 to 113).🤖 Generated with Claude Code
https://claude.ai/code/session_01UY9bUXWGai1iHZLE63yAG8
Generated by Claude Code