fix(us_dol_oflc): upload the table root so staging keeps its year= partitions

[rdahis]

Every us_dol_oflc dev run fails at dbt run on the first model:

Database Error in model us_dol_oflc__lca (models/us_dol_oflc/us_dol_oflc__lca.sql)
  Error while reading table: basedosdados-dev.us_dol_oflc_staging.lca,
  error message: Incompatible partition schemas.
  Expected schema ([year:TYPE_STRING]) has 1 columns. Observed schema ([]) has 0 columns.

Latest occurrence: flow run 27bf41b1-19cd-4303-a121-94be3ae20673, 2026-09-08 06:07 UTC.

Stacked on #1995. That PR fixes the crosswalk lookup that this run hit first; its fix worked and exposed this one. Base is fix/us_dol_oflc-pipeline-crosswalk-lookup, so this diff is the one commit on top. Retarget to main if #1995 merges first.

Cause

Not a stale object in the bucket. The staging prefix was clean when checked — 19 objects per table, every one inside a year=YYYY/ directory. The pipeline creates the offending object on each run.

basedosdados.Storage.upload derives an object's hive partition from the file's path relative to the path argument it is given:

parts = [f.as_posix().replace(path.as_posix() + "/", "").replace(f.name, "") for f in paths]

The flow passed the partition directory (…/output/lca/year=2025), so that relative path came out empty, _resolve_partitions("") returned "", and the object landed at staging/us_dol_oflc/lca/data.parquet — the prefix root, carrying no partition key, each fiscal year overwriting the last. BigQuery fails the entire external table on one such object, not just that object.

The tell in the run logs is Upload concluído: gs://…/lca printed once per fiscal year rather than once per table.

All four program tables (lca, perm, h2a, h2b) go through the same loop, so all four were hit — each got a root object at 06:04 in that run. dictionary is unpartitioned and unaffected; the flow never uploads it.

Fix

Pass the table root. This does not widen what a run touches: build writes only the refreshed fiscal years into the run's own temp directory, so the root holds exactly those partitions, and dump_mode="append" leaves the closed years already in the staging prefix alone — "overwrite" would delete the whole prefix, and the prod table with it.

partition_paths loses its only caller, and its docstring asserts the wrong invariant ("Uploading the partition directories rather than the table root keeps a refresh from touching fiscal years the run did not rebuild"), so it is removed rather than left for someone to pick up again.

Verification

Both branches exercised against the real basedosdados.Storage.upload with the blob upload stubbed:

data_path	blob names produced
…/lca/year=2025 (before)	staging/us_dol_oflc/lca/data.parquet
…/lca (after)	staging/us_dol_oflc/lca/year=2025/data.parquet staging/us_dol_oflc/lca/year=2026/data.parquet

Also checked locally: module import, deploy_flows.load_flows_from_file discovers us_dol_oflc_flow, ruff, and pyrefly (0 diagnostics — run directly, since the pre-commit pyrefly hook matches 0 files inside a git worktree and always exits 1).

Bucket state: all four dev prefixes are clean of root-level objects, so a run starts from a good state. gs://basedosdados/ and gs://basedosdados-staging/ carry none either — no run has ever reached the prod upload.

Not yet done: the dev run. This PR changes flows.py, so the deploy-flow label does deploy it; once cd-prefect3 (staging) reports registrado, trigger with {"materialize_to_prod": false, "update_metadata": false, "force_run": true} and confirm the clone path in the logs reads /app/pipelines-fix-us_dol_oflc-staging-partition-path/. Done means dbt run OK and dbt test OK for every table.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 22424f14-29df-4fcc-b72d-ed25cf73ae9e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request