-
Notifications
You must be signed in to change notification settings - Fork 22
feat(br_senatran_estatisticas): adiciona os seis recortes restantes da frota #1939
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rdahis
wants to merge
112
commits into
main
Choose a base branch
from
data/br_senatran_estatisticas
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
112 commits
Select commit
Hold shift + click to select a range
2ea14ca
feat(br_senatran_estatisticas): add the six remaining frota breakdowns
rdahis 50035d8
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b4b97be
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b54c42a
fix(br_senatran_estatisticas): ano_modelo e ano_fabricacao sao STRING
rdahis a7adab7
fix(br_senatran_estatisticas): le todas as abas de dados, nao so a pr…
rdahis f7fe0ed
fix(br_senatran_estatisticas): soma as chaves que o strip torna iguais
rdahis ff99354
test(br_senatran_estatisticas): fixa a deduplicacao do clean_breakdown
rdahis 4db01c6
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 8df1de4
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 6eeed65
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a33f4f4
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 4e869fa
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c44fde2
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e293b17
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a2fb7a9
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 6f9b7fe
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 81261cf
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 9ef1d0a
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b756f51
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 98b5270
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 0244ff9
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 98d7d35
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b6a8662
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 5ad261f
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 32133c5
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e80e36e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 7e5e388
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] dce5a38
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 07c09d5
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] d246887
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 1ad2d78
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 53ee495
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a0a733d
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 69da02a
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c651519
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 0c13045
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 6669b9a
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 495e8be
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 865c3de
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 930bc17
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 00a1125
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] ee6ce8e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 4510beb
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 2759f07
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] d0eed7f
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 9998426
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 0cf7111
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 58d5b80
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e0683ce
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 3c45e6c
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 8cc6666
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] f8be1d8
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 9da3250
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] de7fd32
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e8710d9
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b9d2810
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] fc89ea4
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 6314cf7
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c40b02e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 29f84d3
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c8d42c0
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 822b2a7
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a42967f
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 3b75e50
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 92a0c3c
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 325c262
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] febdc1e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c3eeec8
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c7bf7f1
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 39ab3b2
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a95c4d4
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 385d1f2
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 1b0bb6e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 84850fc
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 8c72ef8
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 23f611f
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 581f9c1
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] fcf809d
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] d08b5de
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 2cd624c
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a5c96b8
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 35f5dee
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] a2c79f2
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] fd2f97d
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 3465079
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e9cf6a1
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 9c864aa
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] b4d6b23
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] e570525
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 723070f
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 0f199ec
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] dc33b28
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 06744a4
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] ec20ba1
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 81e52fd
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 8d3bb2e
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 06ce6bb
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 02f0fe9
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 44c35f2
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 719c9ce
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] ad4d475
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 8ad14ca
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 60041e0
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 1137788
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 9c893ad
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] fef5273
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 45a4d82
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] bc14d05
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 60bd20a
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] c0066ef
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] dc5e341
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] 3190f88
Merge branch 'main' into data/br_senatran_estatisticas
mergify[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
34 changes: 34 additions & 0 deletions
34
...ls/br_senatran_estatisticas/br_senatran_estatisticas__municipio_ano_fabricacao_modelo.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_ano_fabricacao_modelo", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2015, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(ano_modelo) as string) ano_modelo, | ||
| safe_cast(lower(ano_fabricacao) as string) ano_fabricacao, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from | ||
| {{ | ||
| set_datalake_project( | ||
| "br_senatran_estatisticas_staging.municipio_ano_fabricacao_modelo" | ||
| ) | ||
| }} as t |
28 changes: 28 additions & 0 deletions
28
models/br_senatran_estatisticas/br_senatran_estatisticas__municipio_cep.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_cep", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2014, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(cep) as string) cep, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from {{ set_datalake_project("br_senatran_estatisticas_staging.municipio_cep") }} as t |
28 changes: 28 additions & 0 deletions
28
models/br_senatran_estatisticas/br_senatran_estatisticas__municipio_cor.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_cor", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2014, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(cor) as string) cor, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from {{ set_datalake_project("br_senatran_estatisticas_staging.municipio_cor") }} as t | ||
30 changes: 30 additions & 0 deletions
30
models/br_senatran_estatisticas/br_senatran_estatisticas__municipio_potencia.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_potencia", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2013, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(potencia) as string) potencia, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from | ||
| {{ set_datalake_project("br_senatran_estatisticas_staging.municipio_potencia") }} | ||
| as t |
30 changes: 30 additions & 0 deletions
30
models/br_senatran_estatisticas/br_senatran_estatisticas__municipio_restricao.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_restricao", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2013, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(restricao) as string) restricao, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from | ||
| {{ set_datalake_project("br_senatran_estatisticas_staging.municipio_restricao") }} | ||
| as t |
35 changes: 35 additions & 0 deletions
35
models/br_senatran_estatisticas/br_senatran_estatisticas__municipio_tipo_especie_eixos.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| {{ | ||
| config( | ||
| alias="municipio_tipo_especie_eixos", | ||
| schema="br_senatran_estatisticas", | ||
| materialized="table", | ||
| partition_by={ | ||
| "field": "ano", | ||
| "data_type": "int64", | ||
| "range": { | ||
| "start": 2013, | ||
| "end": 2031, | ||
| "interval": 1, | ||
| }, | ||
| }, | ||
| cluster_by=["mes"], | ||
| pre_hook="{% if adapter.get_relation(this.database, this.schema, this.identifier) %}DROP ALL ROW ACCESS POLICIES ON {{ this }}{% else %}SELECT 1{% endif %}", | ||
| ) | ||
| }} | ||
|
|
||
|
|
||
| select | ||
| safe_cast(ano as int64) ano, | ||
| safe_cast(mes as int64) mes, | ||
| safe_cast(sigla_uf as string) sigla_uf, | ||
| safe_cast(id_municipio as string) id_municipio, | ||
| safe_cast(lower(tipo_veiculo) as string) tipo_veiculo, | ||
| safe_cast(lower(especie) as string) especie, | ||
| safe_cast(lower(eixos) as string) eixos, | ||
| safe_cast(quantidade as int64) quantidade | ||
| from | ||
| {{ | ||
| set_datalake_project( | ||
| "br_senatran_estatisticas_staging.municipio_tipo_especie_eixos" | ||
| ) | ||
| }} as t |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🛡️ Analyzed with Security Review | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: basedosdados/pipelines
Length of output: 50381
🏁 Script executed:
Repository: basedosdados/pipelines
Length of output: 50378
🏁 Script executed:
Repository: basedosdados/pipelines
Length of output: 10254
Security Misconfiguration
Reachability: Internal
Exploitability: Difficult
CWE: CWE-284
Garanta a restauração das políticas após cada rebuild de produção.
Os seis modelos removem as políticas no
pre_hook. O fluxo recriabdpro_filtereallusers_filterdepois dorun_dbt, mas somente quandoupdate_metadata=Truee quandoregister_table_materialization_tasktermina com sucesso. Se essa etapa for ignorada ou falhar, a tabela pode permanecer acessível sem filtros.Torne a restauração independente de
update_metadatae execute-a como etapa obrigatória após a materialização. Inclua tambémmunicipio_ano_fabricacao_modelo, que usa o mesmopre_hooke o mesmo fluxo.🤖 Prompt for AI Agents