Skip to content

Package review: live-verified fixes, geography, Gitea #33 #46 #52 #58 #61 - #1

Merged
jknowles merged 30 commits into
mainfrom
claude/peaceful-curie-578gvv
Oct 9, 2026
Merged

jknowles merged 30 commits into
mainfrom
claude/peaceful-curie-578gvv

Conversation

@jknowles

@jknowles jknowles commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Summary

This is the full package review, from v0.1.0 (the tag r-universe builds) to now: 30 commits, each with its NEWS entry. Every claim about the API was checked against the live CDE. Issue numbers below are Gitea issues; there are no matching GitHub issues.

Correctness: wrong numbers or silently empty results

  • County resolution
    • Multi-county agencies (Columbus PD) are attributed to the queried county, which fixes split county totals.
    • county_to_fips() checks the crosswalk first, which fixes the Virginia independent cities, and counties_with_fips() now returns rows.
    • The county detail carries county_fips, so join_census_pop() works.
  • Missing agencies: the NYPD, DC's Metropolitan Police and the Baltimore City Sheriff had no county, so the New York and Washington metros lacked their largest department. They are now attributed by ORI; the NYPD goes to all five boroughs, as documented.
  • Agency directory and fan-out: get_agencies() uses the right path and parses the live response. The crime fan-out records failed agencies instead of dropping them silently.
  • Imputation: impute_reporting_gaps() now fills real gaps. Before, it filled 0 of 232 in Alameda County, 2021.
  • NIBRS (#33): it was never an outage. The endpoint takes short codes (ROB, 13B), and the package documented long names that return all-nulls. The vocabularies are rebuilt from the live API.
  • Arrests by offense (#58):
    • The API takes numeric offense codes. The package had read the HTTP 400 for a name as "not available".
    • get_arrest_count(offense = ) now returns a monthly series, and get_arrest_demographics(offense = ) works: Ohio 2023 had 1,666 under-18 larceny arrests.
    • Found along the way: the API's all-offense demographics omit arrests under the five "(Unspecified)" codes (883 of 188,836 in Ohio). This is documented, with a live test.
  • Police employment (#58): the state path is pe/{ST} and the nation is plain pe; the package used forms that return nulls. The output gains coverage columns because state totals move with how many agencies report.
  • LEOKA: counts are officers feloniously killed, not assaults.

Interface changes (breaking; I'd tag 0.2.0)

  • Comparison rows (state/national rates) are stripped by default; comparison = TRUE keeps them.
  • include_statewide replaces default_only / default_member.
  • fbi_api_agencies has real column types.
  • get_arrest_count(offense = ) returns a monthly series instead of one total row.
  • get_police_employment(region = ) is now an error: no region path in the API returns data.

New features

  • Series outputs carry population and participated_population, so reporting coverage is visible.
  • Place codes (#46): place_fips, or cousub_fips for townships and New England/NY towns. 98.7% of city agencies resolve, matched by name within the agency's county.
  • Connecticut metros (#52): the CDE now reports CT agencies by planning region, so all seven CT metros resolve. An agency is reachable by traditional county and by region.
  • cde_request() retries transient failures with backoff.

Docs, process and distribution

  • All three vignettes are rewritten and re-rendered live, including a new geography vignette. Inline numbers come from the results.
  • Help pages render their markdown. Before, text after any % was silently dropped.
  • Design specs moved to specs/ (#61), with a CLAUDE.md rule, a hook in .claude/settings.json and a Gitea CI guard.
  • DESCRIPTION, README and CONTRIBUTING are updated for r-universe and GitHub, with the maintainer email set.
  • The bulk-downloads decision is recorded (#58): no reader for now, and a tested download recipe is included.
  • Live tests are off on GitHub; .gitea/live-api.yaml.example is the opt-in workflow.

Where to look first

  • NEWS.md, top section: the user-facing story, grouped by theme.
  • specs/2026-10-06-*.md: the place-codes design, the Connecticut design, and the bulk-downloads decision.
  • R/geography.R: the two attribution layers in agencies_table().
  • R/ucr_arrest.R: offense code resolution and per-code summing.

Testing

  • Offline: 1,211 expectations pass, 0 fail. Fixtures are verbatim recorded responses, and new ones were recorded from the corrected paths.
  • Live (FBI_CDE_LIVE=true, run from the session container, not GitHub): every touched test file passes.
  • R CMD check --no-manual gives 1 NOTE: the suggested packages covr and censusapi aren't installed in the check environment.

Notes for landing

  • The commits are unsigned (relevant to Gitea #60).
  • After merging, r-universe needs a new tag and a branch update in the universe's packages.json.
  • Yale and UConn Health are missing from their CT metros until the 2019 agency snapshot is refreshed. Six more agencies now have a county in the live directory that the snapshot lacks.

🤖 Generated with Claude Code

https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6


Generated by Claude Code

claude added 30 commits October 5, 2026 19:15
…rows

county_to_fips() applied its patch table before looking the name up in the
bundled crosswalk. For the Virginia independent cities that share a name with
a county, the patch strips " CITY" and lands on the county: RICHMOND CITY
resolved to 51159 (Richmond County) instead of 51760, and likewise Fairfax,
Franklin and Roanoke. The crosswalk already had the right codes, so it is now
consulted first; a new test checks agreement for all 3,733 CDE county names.

counties_with_fips() returned crosswalk[NULL, ], a zero-row frame. It now
returns one row per resolvable county.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
…eal shape

get_agencies() requested agency/{state} and coerced each response with
as.data.frame(). On the county-keyed shape the live seam returns
(simplifyVector = FALSE) that produced one junk row per state. It now uses the
agency/byStateAbbr/{state} endpoint the participation functions already use,
flattens county -> agency records into one row per agency, and skips a failing
state with a warning instead of aborting the whole sweep.

Its fixture, agency-byStateAbbr-CA.json, was a synthetic flat array that did
not match the endpoint, which is why the test never caught this. The test now
uses the recorded Rhode Island response, parses it both ways, and asserts the
request path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Since #56 an agency that polices several counties matches each of them, but
its raw county_name ("DELAWARE; FAIRFIELD; FRANKLIN") was still used as the
row's county downstream:

- get_county_crime() grouped by it, so Franklin County, OH aggregated to 7
  rows per period instead of 1.
- county_agencies() took county_fips from the first matching row, so Licking
  County, OH got Fairfield's FIPS and join_census_pop() the wrong population.
- place_agencies(county = "Franklin") compared exactly and found nothing.

county_agencies() now sets county_name and county_fips from the queried
county and keeps the CDE's list in a new agency_county_names column (also on
metro_agencies()). get_county_crime_detail() now carries county_fips, without
which join_census_pop() errored on real detail output; the tests had been
adding that column by hand.

The fan-out also keeps each dropped agency's error in
attr(x, "dropped_reasons"), and get_county_crime() warns about and carries
attr(x, "dropped"): those agencies are missing from the denominator too, so
coverage_fraction cannot see them.

New end-to-end tests run the bundled agency table through the real resolver
and fan-out, mocking only cde_request().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
- get_arrest_demographics(offense = "murder") always errors; only "all" is
  supported. The NIBRS offender example now uses a call the juvenile-arrests
  vignette shows returning data.
- ?cde_base_url still named the pre-rename option fbi.cde.base_url.
- NEWS entry for the review fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
read_fixture() simplified fixtures to data.frames (simplifyVector = TRUE)
while cde_request() returns nested lists (simplifyVector = FALSE), so the
offline suite exercised a shape production never sees. Every parser already
handles the live shape -- the whole suite passes unchanged after the switch --
so the is.data.frame() branches that existed only for simplified fixtures are
removed from get_offense_codes(), get_states(), get_agencies() and the
participation parser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Live tests were gated on FBI_API_KEY (the CDE needs no key) plus
skip_on_ci(), so nothing automated could ever run them, and upstream schema
drift was only noticed by hand. skip_if_no_fbi_api() now runs a test when
FBI_CDE_LIVE=true (still never on CRAN), and a new live-api workflow runs the
suite that way every Monday and on demand.

set_api_key() no longer mentions FBI_API_KEY. The fixtures README now says
fixtures must be verbatim recordings, never synthetic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
A single dropped connection or 503 failed the call outright, and in a county
or metro fan-out silently removed that agency from the totals. cde_request()
now retries network errors and HTTP 408/429/500/502/503/504 with exponential
backoff (1, 2, 4 s, capped at 30), honours Retry-After, and sets a timeout per
attempt. Other statuses still fail immediately. Tunable with
options(fbiCDE.max_retries, fbiCDE.timeout); the sleep is isolated so tests
retry without waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
…kage

tests/testthat/setup.R loaded inst/testdata/state_level_estimated_crime.rda
and defined column vectors and a fix_arrest_test() helper for the retired
api.usa.gov API; only one vector was still used, and it now lives in the one
test file that uses it. inst/testdata (86 CSVs plus an .rda) shipped in every
build and was read only by that helper, so it and read.csv_system_file() are
gone. With that, utils is used only via a top-level globalVariables() call,
so the import is now declared to keep R CMD check clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Live tests should run on the Gitea runner, not GitHub-hosted infrastructure.
The live-api GitHub workflow is removed (it never ran: schedules only fire on
the default branch). .gitea/live-api.yaml.example holds the same weekly job
for Gitea Actions; it is inert until moved into .gitea/workflows/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
For an agency or state the CDE also sends its state's and the nation's series
(a rate, no count). get_agency_crime(), get_estimated_crime(),
get_estimated_arson() and get_arrest_count() returned them mixed in with the
geography's own rows -- 18 rows for Oakland, 6 of them Oakland's -- and put
series labels ("California Clearances") in the offense column, so the
requested code appeared nowhere and mean(rate) mixed geographies.

A shared parser (R/series.R) now keeps only the queried geography's own
series. Comparison series appear only under rates, never with counts, so the
labels in the counts map identify the own series without matching names.
comparison = TRUE returns the others, labelled by new series and series_name
columns. offense is now the requested code, a new measure column says
offenses/clearances/arrests, and periods sort chronologically rather than as
strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Every column was character, including placeholder strings from the source:
nibrs held "TRUE"/"FALSE", latitude/longitude held 545 "NULL"s, and
nibrs_start_date held 9,914 "NULL"s plus MM/DD/YYYY text. Code had to
coerce around them (and empty result frames had to mimic character
coordinates). nibrs is now logical, coordinates numeric, nibrs_start_date a
Date, with NA for the placeholders; one -9/-9 placeholder coordinate is NA
too. Every other value is unchanged. The conversion is recorded in
data-raw/fbi_api_agencies_types.R; xz compression keeps the file smaller
than before (255 KB vs 285 KB).

The docs now say the table is a snapshot whose NIBRS start dates run to
September 2019, and point to get_agency_participation() for current status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The geography functions now query, by default, agencies whose jurisdiction is
a specific area below the state: sheriffs, city police and campus police.
include_statewide = TRUE adds state agencies (state police, highway patrol,
other state agencies). Special-purpose agencies (transit, school, airport,
port, park and railroad police, task forces) and tribal agencies are queried
only when named in agency_class; both are deferred for a later design.
Membership is decided by agency type alone, never by parsing names.

The default_member column and default_only argument are removed from county,
place and metro functions. Behaviour changes: county queries no longer
include special/state/tribal agencies unless asked; place and metro now
include campus police by default. agency_class values are validated.

Class corrections: the 11 "Census Area" agencies are Alaska city police
departments (Nome, Bethel, ...), now municipal and resolvable as places
instead of special; "Other State Agency" is now state instead of special.

Also passes comparison through get_county_agency_crime(), and updates the
juvenile-arrests vignette code (same selection, explicit classes) and
CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
NIBRS never had an outage. The nibrs/ endpoint takes short offense codes
(V, P, ROB, BUR, 13B, 35A, 120, ...) and answers anything else with an
all-null payload. list_nibrs_offenses() returned long names ("robbery",
"burglary-breaking-and-entering") and the functions defaulted to "robbery"
and "all", so every documented call returned nothing. The variable lists
offered count, bias and five other variables the response does not have;
count was the offender and offense default.

- list_nibrs_offenses() is now a code/label table (V, P + the CDE's
  crime-trend lookup); defaults are offense = "V" and variables race/race/
  weapons; offense names and "all" are rejected before any request with a
  suggested code; unknown variables are an error.
- get_arrest_count(offense =) validated against the 34 offense names only,
  rejecting the 29 categories and 49 breakdowns -- including
  "Drug/Narcotic Offenses", the only total of drug arrests. It now checks
  the response itself, which always carries all 80 names.
- The vocabularies are rebuilt from the live API by
  data-raw/api_vocabularies.R. R/data.R also defined stale copies of them as
  package objects, which unqualified references picked up; removed.
- The NIBRS fixtures were hand-written with variables the API never returns;
  replaced by recorded 2023 responses, including an all-null one (a
  geography with no NIBRS data).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
impute_reporting_gaps() scaled the interpolated rate by
participated_population, which the CDE reports as missing for every month an
agency did not report -- i.e. every gap -- so it filled nothing on real data
(0 of 232 gaps in Alameda County, 2021), including the 2021 California
NIBRS hole it was written for. It now falls back to the agency's population
in those months (39 of 184 interior gaps filled for Alameda, 2020-2021).
Agencies with no population (campus, transit) are left alone.

get_leoka() docs: the totals are officers feloniously killed (they match the
FBI's published 46 for 2020 and 73 for 2021), not assaults; the unlabelled
_dod/_doi splits are documented as such.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
get_agency_crime(), get_estimated_crime(), get_estimated_arson() and
get_arrest_count() now return population and participated_population for
each row, joined from the response's populations map by series name and
period. Their ratio is the reporting coverage, which until now was only
available from the geography detail functions. A single-offense arrest total
carries no populations, so those columns are NA there.

list_ucr_arrest_offenses(level =) lists one level of the CDE's arrest names
(name 34, category 29, breakdown 49). Counts within a level do not overlap,
so it is what ranking offenses needs. The ucr_arrest_offenses dataset becomes
a data frame of offense and level, rebuilt by data-raw/api_vocabularies.R.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
- Getting Started: current interface (offense codes, comparison, coverage,
  arrest offense levels, NIBRS codes, SHR gaps, PE scope, LEOKA wording).
- Juvenile arrests: real reporting coverage from participated_population,
  the full 34-name offense ranking, a Columbus month-completeness check,
  campus police in the metro fan-out with failures recorded, inline numbers
  and data-driven figure alt text.
- New "Counties, cities and metro areas" vignette covering membership,
  classes, unsummed detail and the reported flag, coverage, imputation,
  multi-county agencies, places and metros.
- Precompute script checks for pkgload; .Rmd.orig sources are build-ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
…ties

The CDE gives these three agencies no county ("N/A"; "NOT SPECIFIED" in
its live directory), although each polices a whole county-equivalent.
county_agencies("District of Columbia", "DC") returned nothing, Manhattan
returned only a SUNY campus and the State Police, and the New York and
Washington metros lacked their largest department.

- .AGENCY_COUNTY_ATTRIBUTIONS, keyed by ORI, fills county_name inside
  agencies_table() only where the CDE's value is "N/A". The NYPD goes to
  all five boroughs and counts in full in each, so a borough returns
  citywide figures; documented in county_agencies(),
  get_county_crime_detail(), the geography vignette and NEWS.
- The FIPS crosswalk gains Queens, Richmond and the District of Columbia
  (data-raw/crosswalk_attributed_counties.R, FIPS from tigris::fips_codes;
  both sysdata objects re-saved).
- fbi_api_agencies keeps the CDE's own values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
make_state(), make_year(), clean_column_names() and combine_url_section()
were unexported and called nowhere in the package; only their own tests
kept them alive. They served the retired api.data.gov API (agencies/
states/ paths, its arrest column names). make_state() also carried a dead
branch comparing a lower-cased value to "CZ".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The roxygen comments are written in markdown (backticks, [fn()] links,
**bold**, bullet lists) but the package never enabled it, so the help
pages showed them literally. Worse, an unescaped % in plain Rd starts a
comment: help text such as "covers 98.6% of unique (state, county) pairs"
was cut off at the percent sign. Rd regenerated with markdown on; every
[fn()] reference now links to its topic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The CDE answers state, region and national police employment requests
with every value null -- counts, rates and populations alike (the
recorded fixtures show the same) -- so those calls returned zero rows
with no explanation. The docs and README offered state and national
examples, and the Getting Started vignette said the API returned "rates
without counts".

- An empty result now comes with a message; for a non-agency request it
  says staffing is published for individual agencies only.
- Docs, README and the vignette describe the function as agency-level,
  and from/to are documented as four-digit years (they inherited the
  MM-YYYY text and defaults of the monthly functions).
- Test comments no longer claim rates are populated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
fbiCDE is distributed through r-universe (civilytics.r-universe.dev,
built from the v0.1.0 tag of this repository), not CRAN.

- DESCRIPTION: version 0.1.0.9000 (development past v0.1.0); the
  maintainer address replaces the placeholder; URL and BugReports point at
  the public GitHub repository and the r-universe page instead of the
  private Gitea; the redundant Author/Maintainer fields are gone
  (Authors@R generates them).
- NEWS: header names the package; a "fbiCDE 0.1.0" heading marks what the
  release contained.
- README: installs from r-universe; badges point at civilytics/fbi_cde.
- CONTRIBUTING and the PR welcome comment: issues go to GitHub; the
  CONTRIBUTING link pointed at a repository that does not exist.
- CLAUDE.md: package name and distribution.
- pkgdown: reference index covers every exported topic; removed a
  placeholder analytics ID and a site URL that was never deployed.
- Removed cran-comments.md, stale boilerplate from the original package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The superpowers skills write specs to docs/superpowers/specs/ and plans
to docs/superpowers/plans/ by default, inside pkgdown's output folder.

- Specs now live in specs/, plans in specs/plans/; every reference (CLAUDE.md,
  R/place.R, the specs' and plans' cross-links) points at the new paths.
- CLAUDE.md states the location as a project rule, which the superpowers
  skills honour over their default.
- .claude/settings.json adds a PreToolUse hook that blocks Write, Edit,
  MultiEdit and NotebookEdit under docs/superpowers/ and says where the
  file belongs.
- The Gitea R-CMD-check workflow fails fast if docs/superpowers/ exists.
- specs/ is build-ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
place_agencies() and get_place_crime_detail() carry the Census code of the
unit each municipal agency polices, as a promised join key:

- place_fips (7 digits) for a city, town or village; cousub_fips (10 digits)
  for a township or a New England/New York town, which are governments but
  not Census places; place_type says which ("incorporated",
  "county_subdivision", or "cdp").
- From place_crosswalk in R/sysdata.rda, built by
  data-raw/place_fips_crosswalk.R from the Census 2020 reference code files
  (PLACE_VINTAGE). Matched by name only among Census units in the agency's
  own county; agency coordinates are not used (correct matches sit up to
  1,000 km from them). 98.7% of 11,646 municipal agencies resolve; the rest
  are NA, not guessed.
- derive_place_name() strips the ", <Name> County" disambiguator 146
  PA/OH/NJ/MI agencies carry, in either position; those places were
  unreachable by name.
- add_place_spatial_members() keeps the place agency's codes instead of
  resetting them to NA.

Design: specs/2026-10-06-place-fips-design.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The pe endpoint answers pe/{ST} (state), plain pe (nation) and
pe/{ST}/{ORI} (agency). The package requested pe/state/{ST} and
pe/national, which come back with every value null, so state and
national calls returned no rows. The recorded fixtures captured those
null answers and the tests called it "suppressed upstream"; the previous
commit then documented it as an API limit. It was the package's path.

- Correct paths; region (no form returns data) is now an error.
- Fixtures re-recorded verbatim from the correct paths.
- New participated_population and employees_per_1000 columns: a state's
  counts are sums over reporting agencies and move with coverage (Texas
  +39% in 2018-2020 at a flat ~3.4 per 1,000).
- Docs, README and the Getting Started vignette corrected.

Found by the #58 investigation into the FBI's bulk downloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The arrest endpoint addresses one offense by numeric code
(arrest/<level>/<code>); a name is an HTTP 400, which the package read as
"a specific offense is no longer addressable". So get_arrest_count()
returned a single total picked out of the all-offense response,
get_arrest_demographics() refused any offense, and the juvenile vignette
said the API had no offense-by-age table. It has one.

- New ucr_arrest_offense_codes dataset (47 codes), built from the live API
  by data-raw/api_vocabularies.R: each code placed by the name, category
  and breakdown its own response reports. Suspicion (320) is served but
  missing from the lookup; Rape and Runaway have no code and no arrests.
- get_arrest_count(offense = ) returns the offense's monthly series (and
  comparison series); a multi-code name is the sum of its codes.
- get_arrest_demographics(offense = ) accepts any offense name.
- Documented an API quirk: all-offense demographics omit arrests under the
  five "(Unspecified)" codes (OH 2023: 883 of 188,836), verified exactly
  in OH, TX and CA; a live test watches it.
- Juvenile vignette ranks offenses by juvenile arrests; Getting Started
  and README show offense series and demographics.

Found by the #58 investigation into the FBI's bulk downloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The FBI's Downloads page fetches files through
/LATEST/s3/signedurl?key=..., which works through cde_request(). Two of
the four documented API gaps turned out to be package bugs, fixed in the
previous commits. The two real ones, state/national estimates and LEOKA
assaults, have bulk files: an XLSX served under a .csv key, and a
fixed-width master file of ~200 MB per year. Decision: no bulk-file
reader for now; the spec records the mechanism, a tested download
recipe, and what a future reader would need (its own mockable network
seam, key discovery, readxl in Suggests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
The 2023 delineation builds Connecticut's seven CBSAs from planning
regions, its county equivalents since 2022, while the bundled agency table
has traditional counties, so every Connecticut metro was empty. The CDE's
live directory now reports each Connecticut agency's planning region; the
package records that, so the attribution is the CDE's, not an
approximation.

- ct_planning_regions (101 of 107 bundled CT agencies), built by
  data-raw/ct_planning_regions.R; agencies_table() appends the region to
  the agency's county list, so it is reachable by traditional county and
  by region.
- The nine regions join the county FIPS crosswalk (09110-09190, from
  tigris::fips_codes); bare region names now resolve to the region, not
  to one traditional county.
- All seven CT CBSAs resolve without warning, each attributed agency in
  exactly one; the CT-specific warning is gone (Puerto Rico still warns).
- Yale and UConn Health are not in the live directory under their bundled
  ORIs, so their metros lack them until a snapshot refresh.

Design: specs/2026-10-06-connecticut-planning-regions-design.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UFR5KRqrcaCZEB2xgCCG6
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Thanks for this — and one thing worth knowing before it happens.

This repository is a mirror. Development happens on Gitea at
gitea.civilytics.org/Civilytics/fbiCDE. Your pull request will be fetched
from here, landed there, and synced back.

Because that merge preserves your commits at their original SHAs, GitHub will
mark this pull request "Merged" on its own
— without anyone visibly clicking
the Merge button, and possibly without a review comment on this page first.

If your pull request closes as "Merged" and nobody appears to have merged it,
that is the normal, successful outcome — not a rejection.

If it is not going to be merged, you will get an actual reply saying so.

Full details: CONTRIBUTING.md.

@jknowles
jknowles merged commit 5594bad into main Oct 9, 2026
5 checks passed
@jknowles
jknowles deleted the claude/peaceful-curie-578gvv branch October 9, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants